Vehicle steering wheel holding detection device

Abstract

To reduce computational complexity required for processing, such as object recognition in an image, and improve the accuracy of detecting whether a driver is holding a steering wheel or not.SOLUTION: A in-vehicle camera captures an image including hands of a driver and a steering wheel. A predetermined easy-to-process steering wheel model is prepared, to transform the captured image into a coordinate system of the steering wheel model (S52), using a transformation matrix prepared in advance. A determination processing unit identifies whether the position of the steering wheel in the captured image overlaps the steering wheel model in the coordinate system of the steering wheel model (S53). A rotation angle θ on the steering wheel model corresponding to the change in position of the hands and the steering wheel in the captured image, is compared with the change in steering angle Φ acquired by a steering angle acquisition unit (S55). When the hands are on a circumference of the steering wheel model and when the rotation angle θ matches the change in steering angle Φ, holding of the steering wheel is detected.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to a steering wheel holding detection device for a vehicle. [Background technology]

[0002] In a vehicle, the function of detecting whether the driver is holding the steering wheel, i.e., the handle for steering, is a necessary function to support safe driving. In particular, in recent years, with the possibility of vehicles being equipped with autonomous driving functions, it is necessary to be able to correctly determine whether the driver is holding the steering wheel.

[0003] For example, in vehicles equipped with Level 2 autonomous driving functions, the driver must keep their hands on the steering wheel during autonomous driving. In addition, in vehicles equipped with Level 3 autonomous driving functions, it is necessary to detect whether the driver is actually holding the steering wheel and is ready to drive in order to safely hand over driving operations from the autonomous driving system to the driver when the autonomous driving system reaches its limit.

[0004] For example, Patent Document 1 discloses a driver monitoring device that accurately detects whether the driver sitting in the driver's seat is gripping the steering wheel when switching from an autonomous driving mode to a manual driving mode. Specifically, the device reads an image captured by a driver imaging camera, processes the driver image, detects whether the steering wheel is being gripped, and if so, executes a process to detect the position where the steering wheel is being gripped.

[0005] In processing the driver image, the edges (contours) of the steering wheel are first extracted using image processing such as edge detection. Next, edges that intersect with the extracted edges of the steering wheel are extracted, and if such edges are detected, it is determined whether they correspond to fingers based on the length and spacing of the edges. If it is determined that an edge corresponds to a finger, the position of the edge corresponding to the finger is detected as the steering wheel grip position.

[0006] The posture detection unit processes the driver image to detect the posture of the driver's shoulders and arms. Following the processing by the posture detection unit, the grip determination unit determines whether the imaged driver is gripping the steering wheel based on the grip position of the steering wheel detected by the grip position detection unit and the posture of the driver's shoulders and arms detected by the posture detection unit.

[0007] For example, a judgment table showing the relationship between the grip position of the steering wheel and the posture (direction and angle) of the driver's shoulders and arms is read out, and the detected grip position of the steering wheel and the posture of the driver's shoulders and arms are applied to the judgment table to determine whether the condition of gripping the steering wheel is met.

[0008] Furthermore, Patent Document 1 discloses that a contact detection sensor provided on the steering wheel is used to detect a hand touching the steering wheel. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Application Publication No. 2018-190217 Summary of the Invention [Problem to be solved by the invention]

[0010] However, even if the technology of Patent Document 1 is adopted and the position of the driver's hands on the steering wheel can be correctly detected based on an image of the driver, it is not certain whether the driver is actually holding the steering wheel. For example, it is possible that the driver is simply bringing their hands close to the steering wheel but is not actually holding the steering wheel, making it impossible for the automated driving system to safely hand over driving operations to the driver.

[0011] Therefore, as disclosed in Patent Document 1, it is necessary to perform special processing such as detecting hand contact using a contact detection sensor or detecting the posture of the driver's shoulders and arms. However, when realizing a function such as that disclosed in Patent Document 1 based on an image of the driver, it is necessary to repeatedly recognize the shapes and objects of the steering wheel and each part of the driver's body in almost real time, which usually requires very complex image processing to be performed at high speed. Therefore, a very expensive image processing device must be installed. Furthermore, if a relatively low-performance image processing device is used, there is a concern that the detection time delay will increase.

[0012] The present invention has been made in consideration of the above-mentioned circumstances, and its purpose is to provide a steering wheel holding detection device for a vehicle that can reduce the amount of calculation required for processing such as object recognition in an image and can detect with high accuracy whether or not the driver is holding the steering wheel. [Means for solving the problem]

[0013] The above object of the present invention can be achieved by the following configuration.

[0014] an imaging unit capable of capturing an image of a steering wheel of a vehicle and an image of the hands of a driver of the vehicle; a position conversion unit that associates the position of the steering wheel in the image acquired by the imaging unit with a position in a steering wheel model having a coordinate system based on an axial direction different from the imaging direction of the imaging unit; a steering angle acquisition unit that acquires a steering angle of the steering wheel; an identification unit that identifies whether the handle is being held by hand; Equipped with the identification unit recognizes that the steering wheel is being held by a hand when the position of the steering wheel in the image acquired by the imaging unit overlaps with the steering wheel model in the coordinate system of the steering wheel model, and when a rotation angle on the steering wheel model corresponding to a change in the position of the steering wheel in the image acquired by the imaging unit matches a change in steering angle acquired by the steering angle acquisition unit. A steering wheel holding detection device for a vehicle. [Effects of the Invention]

[0015] The vehicle steering wheel holding detection device of the present invention can significantly reduce the amount of calculation required for processing such as object recognition in images, thereby eliminating the need for an expensive image processing device and suppressing detection time delays.In addition, since it identifies whether or not the steering wheel is being held by combining position detection and rotation detection, it can detect with high accuracy whether or not the driver is holding the steering wheel.

[0016] The present invention has been briefly described above. The details of the present invention will become clearer by reading the following detailed description of the invention (hereinafter referred to as "embodiments") with reference to the accompanying drawings. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an in-vehicle control system. [Figure 2] FIG. 2 is a front view showing an example of the installation position of the vehicle-mounted camera in the vehicle interior. [Figure 3] FIG. 3 is a side view showing an example of the installation position of the vehicle-mounted camera in the vehicle interior. [Figure 4] FIG. 4 is a front view showing a plurality of examples of captured images. [Figure 5] FIG. 5 is a flowchart showing an outline of the steering wheel holding detection process. [Figure 6] FIG. 6 is a flowchart showing the procedure for detecting whether the steering wheel is held down according to the first method. [Figure 7] FIG. 7 is a schematic diagram showing the flow of data processing in the steering wheel holding detection of Method 1. [Figure 8] FIG. 8 is a schematic diagram showing the flow of data processing in the steering wheel holding detection of Method 2. [Figure 9] FIG. 9 is a flowchart showing a part of the processing procedure for detecting whether the steering wheel is held down, which is a combination of the methods 1 and 2. [Figure 10]FIG. 10 is a flowchart showing a part of the processing procedure for detecting whether the steering wheel is held down, which is a combination of the methods 1 and 2. DETAILED DESCRIPTION OF THE INVENTION

[0018] Specific embodiments of the present invention will be described below with reference to the accompanying drawings.

[0019] <System configuration> FIG. 1 shows an example of the configuration of an in-vehicle control system 100 including a steering wheel holding detection device for a vehicle according to an embodiment of the present invention.

[0020] The vehicle control system 100 in FIG. 1 includes an in-vehicle camera 11, a steering angle sensor 12, a judgment processing unit 13, an alarm output unit 14, an information display unit 15, a driving control unit 16, a data storage unit 17, and a steering wheel 18.

[0021] The steering handle 18, i.e., the steering wheel, has a circular operating portion that is easy for the driver to operate, and its center is supported by a freely rotatable rotation shaft. The rotation shaft of the steering handle 18 is connected to the steering mechanism of the front wheels via, for example, a mechanical connection mechanism. Therefore, the driver can adjust the steering angle of the front wheels by rotating the steering handle 18.

[0022] In the case of a vehicle equipped with an automatic driving function, an electrically controllable actuator is connected to the steering mechanism, which means that the steering angle of the wheels can be automatically adjusted to change the direction of travel of the vehicle without the driver having to operate the steering wheel 18.

[0023] The steering angle sensor 12 is attached to, for example, the rotation axis of a steering wheel 18, and is capable of detecting the actual steering angle of the automobile and changes therein.

[0024] The vehicle-mounted camera 11 is installed inside the vehicle cabin as described below, and its imaging direction and imaging range are fixed so that it can simultaneously capture the steering wheel 18 located in front of the driver's seat and the left and right hands of the driver seated in the driver's seat.

[0025] The determination processing unit 13 is configured as an electronic control unit including components such as a microcomputer, and has a function for automatically determining whether or not the driver is holding the steering wheel 18 .

[0026] As will be described later, the judgment processing unit 13 of this embodiment detects the position of the steering wheel 18 and the position of the hands from the image captured by the in-vehicle camera 11, and identifies whether the driver is holding the steering wheel 18 by taking into account the change in the steering angle detected by the steering angle sensor 12.

[0027] The warning output unit 14 can issue a warning to the driver when necessary using a buzzer or voice output. For example, during autonomous driving, which requires the driver to keep their hands on the steering wheel, the warning output unit 14 outputs a warning if the driver's hands are off the steering wheel. Also, when the driver takes over driving the vehicle from the autonomous driving system, the warning output unit 14 can be used to output appropriate guidance or warning depending on whether the driver is holding the steering wheel or not.

[0028] The information display unit 15 has a two-dimensional display capable of displaying images, figures, characters, etc. The information display unit 15 can be used to display information necessary to assist the driver in driving operations, to display the status of automatic driving, and to display information necessary for smooth switching between automatic driving and manual driving.

[0029] The driving control unit 16 includes an actuator for controlling the steering of the vehicle, an actuator for accelerator control, an actuator for brake control, sensors that can grasp road conditions using radar, stereo cameras, etc., and an automatic driving unit that controls each actuator and automatic transmission according to the situation.

[0030] The data storage unit 17 has the function of storing, as history, image data showing the status inside the vehicle cabin captured by the vehicle-mounted camera 11, data showing the road status, data showing the status of automatic driving, and the like.

[0031] <Installation location of in-vehicle camera> Examples of the installation positions of the vehicle-mounted camera 11 in the cabin of the automobile 20 are shown in FIGS. For example, by installing the in-vehicle camera 11 at the position of the overhead section 21 in Figure 2 and photographing the shooting range A01, it is possible to capture an image that shows the steering wheel 18 and the hands of the driver sitting in the driver's seat 26.

[0032] Similarly, by installing the in-vehicle camera 11 at the position of the pillar portion 22 in Figure 2 and photographing the shooting range A02, it is possible to capture an image that shows the steering wheel 18 and the hands of the driver seated in the driver's seat 26.

[0033] In addition, by installing the in-vehicle camera 11 at the center portion 23 in Figure 2 and photographing the shooting range A03, it is possible to capture an image that shows the steering wheel 18 and the hands of the driver seated in the driver's seat 26.

[0034] In addition, by installing the vehicle-mounted camera 11 at the position of the dome lamp section 24 in Figure 3 and photographing the shooting range A04, it is possible to capture an image that shows the steering wheel 18 and the hands of the driver seated in the driver's seat 26.

[0035] In other words, the on-board camera 11 of the on-board control system 100 shown in Figure 1 only needs to be capable of simultaneously capturing an image of the steering wheel and the driver's hands holding the steering wheel, and the on-board camera 11 may be installed in any of the overhead section 21, pillar section 22, center section 23, and dome lamp section 24.

[0036] <Example of a captured image> An example of multiple captured images is shown in FIG. For example, when the vehicle-mounted camera 11 is installed at the position of the center section 23 in Fig. 2, an image like the captured image 30A in Fig. 4 can be captured. When the vehicle-mounted camera 11 is installed at the position of the overhead section 21 in Fig. 2, an image like the captured image 30B in Fig. 4 can be captured.

[0037] 4 includes an image of a steering wheel 31 and the right hand 32a and left hand 32b of a driver 32. Therefore, based on the captured image 30A or 30B, it is possible to grasp the state of the steering wheel 31 and the state of the right hand 32a and left hand 32b.

[0038] <Outline of handle hold detection process> An outline of the steering wheel holding detection process in the embodiment of the present invention is shown in Fig. 5. The steering wheel holding detection process in Fig. 5 will be described below.

[0039] The determination processing unit 13 shown in FIG. 1 receives as input the data of the image captured by the vehicle-mounted camera 11 (S11). The determination processing unit 13 acquires data of a transformation matrix ([A] described later) required to grasp the handle position shape from the image data input in S11 (S12).

[0040] In this embodiment, a handle model HM (described later) suitable for grasping the position and shape of the handle 31 is created in advance. This handle model HM is represented, for example, in a two-dimensional coordinate system parallel to a plane perpendicular to the rotation axis of the handle 31. On the other hand, for example, the handle 31 in the photographed image 30A in FIG. 4 was photographed from an oblique direction, so the two-dimensional coordinates of the photographed image 30A are significantly different from those of the handle model HM. Specifically, since the actual shape of the handle 31 is circular, it is expected that the handle shape in the handle model HM will also be circular, but the handle 31 in the photographed image 30A is elliptical.

[0041] Therefore, it is necessary to convert the coordinates of the handle 31 in the photographed image 30A into a two-dimensional coordinate system on the handle model HM. By acquiring and using the constant data of the conversion matrix created in advance in S12, the determination processing unit 13 can easily perform the coordinate conversion.

[0042] The determination processing unit 13 acquires the actual angle (steering angle) of the steering wheel 18 from the steering angle sensor 12 (S13).

[0043] The determination processing unit 13 detects the positions of the left and right hands by performing image processing on the image data input in S11 (S14). For example, the positions of the hands in the image can be detected by comparing reference pattern data representing the characteristics of the hands or the characteristics of the steering wheel with the feature amounts of each area in the image. Furthermore, by tracking the detected hand positions, movement of the hand positions in the image can be detected. Existing technology can be used for the basic processing required to detect various elements in such an image.

[0044] The determination processing unit 13 uses the data acquired in S11 to S14 to perform processing to detect whether the driver is holding the steering wheel (hands-on) (S15). In this processing, in order to improve the determination accuracy, the determination is made by combining two types of methods, "Method 1" and "Method 2" described later.

[0045] <Details of "Method 1" handle holding detection> The processing procedure and data processing flow for steering wheel holding detection in "Method 1" are shown in Figure 6 and Figure 7, respectively. Steering wheel holding detection in "Method 1" will be explained below.

[0046] The determination processing unit 13 acquires data such as the captured image 30A shown in Fig. 7 from the in-vehicle camera 11 (S21). The captured image 30A includes images of the steering wheel 31 and the right hand 32a and left hand 32b of the driver 32.

[0047] In S22, the determination processing unit 13 performs image processing on the captured image 30A to obtain position information of the handle region 31A representing the handle 31. Here, the shape of the handle region 31A is elliptical, and the position and shape are expressed by a two-dimensional coordinate system (x / y coordinate system) specified by the position in the x-axis direction and the position in the y-axis direction in the captured image 30A.

[0048] In S23, the determination processing unit 13 obtains information on the position R1 (xr1, yr1) of the right hand 32a and the position L1 (xl1, yl1) of the left hand 32b in the x / y coordinate system by image processing of the captured image 30A.

[0049] In S24, the determination processing unit 13 determines whether or not the hand position has been detected in S23, and if the hand position has been detected, the process proceeds to S25.

[0050] In order to utilize the handle model HM created in advance, the judgment processing unit 13 performs coordinate conversion of the data in S25, and maps the position of the handle area 31A obtained in S22 and the hand positions R1 and L1 obtained in S23 into the coordinate system of the handle model HM.

[0051] In this embodiment, the position and shape of the handle model HM are represented by a two-dimensional coordinate system (x' / y' coordinate system) specified by the position in the x'-axis direction and the position in the y'-axis direction, as shown in Fig. 7. The origin coordinates (0,0) of the x' / y' coordinate system are set as the center P0, and the handle model HM is a circle on the circumference at a distance of radius q (constant) from this center. Because the handle model HM has a simple circular shape, it is easy to grasp each position on the handle and the rotation angle on the handle model HM.

[0052] Therefore, in S25, the determination processing unit 13 converts the position and shape of each data from the x / y coordinate system to the x' / y' coordinate system. Here, by multiplying the input data in the x / y coordinate system by a transformation matrix A prepared in advance, output data in the x' / y' coordinate system can be easily generated.

[0053] The judgment processing unit 13 compares the circular (circumference) area of ​​the handle model HM (corresponding to the handle area 31A) with the position R'1 (xr'1, yr'1) of the right hand and the position L'1 (xl'1, yl'1) of the left hand in the x' / y' coordinate system, and determines in S36 whether the hands (32a, 32b) are resting on the handle 31 (are overlapping).

[0054] Specifically, in S26, the determination processing unit 13 determines whether or not the coordinates (xr'1, yr'1) of the right hand position R'1 satisfy the condition of the following [Equation 1].

number

[0055] Also, the coordinates (xl'1, yl'1) of the left hand position L'1 are compared under the same conditions as in [Equation 1]. Then, when it is recognized that the hands (32a, 32b) are on the handlebars 31, the determination processing unit 13 detects the hands-on state in "method 1" (S27).

[0056] The contents of the transformation matrix A can be appropriately determined by experiments or analysis of past data. For example, in the case of a vehicle that automatically adjusts the steering angle of the steering wheel 18 using an actuator, data correlating changes in the position of the hands on the steering wheel 18 with the captured image 30A taken by the in-vehicle camera 11 can be recorded and accumulated as actual results. By analyzing the accumulated actual data, the relationship between each position in the x / y coordinate system of the image taken by the in-vehicle camera 11 and each position in the x' / y' coordinate system of the steering wheel model HM can be identified, and the contents of the transformation matrix A appropriate for the relationship can be determined, which is composed of multiple constants representing the relationship. The data of the transformation matrix A determined in this manner is stored in a non-volatile memory on the in-vehicle control system 100.

[0057] <Details of "Method 2" for detecting handle holding> The data processing flow for steering wheel holding detection in "Method 2" is shown in Figure 8. Steering wheel holding detection in "Method 2" is explained below.

[0058] The determination processing unit 13 acquires an image including an image of the steering wheel and hands from the vehicle-mounted camera 11, such as the captured images 30A1 and 30A2 shown in FIG. 8, and acquires the position of the steering wheel area 31A and the position of the hands from this image.

[0059] Moreover, as the hand position changes over time as seen in photographed images 30A1 and 30A2, the determination processing unit 13 tracks the hand position. In the example shown in Fig. 8, the left hand position L1 (xl1, yl1) is detected from photographed image 30A1, and the left hand position L2 (xl2, yl2) is detected from photographed image 30A2.

[0060] The judgment processing unit 13 performs coordinate transformation in S31 using transformation matrix A on the left hand position L1 (xl1, yl1) detected from the captured image 30A1 and the left hand position L2 (xl2, yl2) detected from the captured image 30A2, and detects the hand positions L'1 and L'2 on the same x' / y' coordinates as the handle model HM.

[0061] In addition, the judgment processing unit 13 acquires the steering angle Φ1 of the steering wheel 18 detected by the steering angle sensor 12 when the captured image 30A1 was taken, and the steering angle Φ2 of the steering wheel 18 detected by the steering angle sensor 12 when the captured image 30A2 was taken, and calculates the change in the actual steering angle Φ (Φ1-Φ2) in S32.

[0062] The determination processing unit 13 calculates the rotation angle θ corresponding to the amount of change in the hand positions L'1 and L'2 on the handle model HM in S33. Specifically, the determination processing unit 13 calculates the rotation angle θ from the coordinates (x1, y1) of the hand position L'1 and the coordinates (x2, y2) of the hand position L'2 on the handle model HM based on the following [Equation 2].

[0063]

number

[0064] In S34, the determination processing unit 13 compares the change in the actual steering angle Φ (Φ1-Φ2) with the hand rotation angle θ detected in S33, and if they are nearly identical, it identifies the state as the hands-on state of "Method 2." Specifically, a match is detected if the condition of the following [Equation 3] is met.

number

[0065] <Handle detection using a combination of "Method 1" and "Method 2"> The processing procedure for handlebar holding detection that combines the above-mentioned "method 1" and "method 2" is shown in Figures 9 and 10. That is, in the processing of step S15 shown in Figure 5, the judgment processing unit 13 executes the processing of S41 to S60 shown in Figures 9 and 10 as a processing that combines both handlebar holding detection of "method 1" shown in Figures 6 and 7 and handlebar holding detection of "method 2" shown in Figure 8. The processing of Figures 9 and 10 will be explained below.

[0066] In step S41, the determination processing unit 13 detects the position L1 of the driver's left hand based on an image such as the captured image 30A1 shown in Fig. 8. If possible, the determination processing unit 13 also detects the position R1 of the right hand at the same time.

[0067] If the hand position is detected in S41, the determination processing unit 13 proceeds to the process from S42 to S43, and acquires the steering angle Φ of the steering angle sensor 12 at this time.

[0068] The determination processing unit 13 registers both the hand position detected in S41 and the steering angle Φ detected in S43 as a set of data D1 in the internal memory in S44.

[0069] In step S45, the determination processing unit 13 detects the position L2 of the driver's left hand based on an image such as the captured image 30A2 shown in Fig. 8. If possible, the determination processing unit 13 also simultaneously detects the position R2 of the driver's right hand.

[0070] If the hand position is detected in S45, the determination processing unit 13 proceeds to the process from S46 to S47, and acquires the steering angle Φ of the steering angle sensor 12 at this time.

[0071] The determination processing unit 13 registers both the hand position detected in S45 and the steering angle Φ detected in S47 as a set of data D2 in the internal memory in S48.

[0072] In S49, the determination processing unit 13 determines whether the change in steering angle output by the steering angle sensor 12 is large or small. If the change in steering angle is large, the determination processing unit 13 proceeds from S49 to S50 to detect whether or not there is a change in the hand position in the two sets of data D1 and D2. If there is movement in the hand position, the process proceeds from S50 to S52.

[0073] Also, for example, if the driver is simply placing his / her hands on the steering wheel 18, there is no change in the position of the hands, so in this case the process proceeds from S50 to S58 and the determination processing unit 13 determines that the hands are off.

[0074] If the steering angle change is small, the determination processing unit 13 proceeds from S49 to S51, and detects whether or not there is a change in the hand position in the two sets of data D1 and D2. If there is movement in the hand position, the process proceeds from S51 to S58. For example, if the driver changes his or her grip on the steering wheel 18 and changes the position of his or her hands, the hand position may move significantly even if the steering angle change is small, so the process proceeds to S58.

[0075] If the change in steering angle is large and the movement of the hand position is also large, the determination processing unit 13 performs coordinate transformation of the data D1 and D2 using the transformation matrix A described above (S52).

[0076] As in the example of Figure 7, the determination processing unit 13 compares the coordinates (xl'1, yl'1) of the position of the left hand L'1 after coordinate transformation and the coordinates (xr'1, yr'1) of the position of the right hand R'1 with the circumference of the circle of the steering wheel model HM, and determines in S53 whether the hands are in a position that almost overlaps with the steering wheel model HM. If the hands are in a position that almost overlaps with the steering wheel model HM, the process proceeds from S53 to S54, and if they are not in an overlapping position, the process proceeds from S53 to S58. If the hands are not in a position that overlaps with the steering wheel model HM, it can be assumed that the hands have left the steering wheel 18.

[0077] 8, the determination processing unit 13 calculates the rotation angle θ corresponding to the amount of change in the hand positions L'1 and L'2 on the handle model HM in S54. That is, the determination processing unit 13 calculates the rotation angle θ based on the coordinates (x1, y1) of the hand position L'1 and the coordinates (x2, y2) of the hand position L'2 on the handle model HM, based on the above-mentioned [Equation 2].

[0078] In S55, the determination processing unit 13 compares the difference (Φ1-Φ2) between the steering angles Φ in the two sets of data D1 and D2 with the rotation angle θ detected in S54 to determine whether they substantially match.

[0079] If the difference in the steering angle Φ and the rotation angle θ match, the determination processing unit 13 proceeds to the process from S55 to S56. In this case, the determination processing unit 13 determines in S56 that the vehicle is in a hands-on state, and registers the contents of data D2 in data D1 (S57).

[0080] If the difference in the steering angle Φ and the rotation angle θ do not match, the determination processing unit 13 proceeds to the process from S55 to S58, determines that the hands are off, and erases the contents of the data D1 and D2 (S59).

[0081] If the change in the steering angle is small and there is no movement in the hand position, the determination processing unit 13 proceeds from S51 to S60. In this case, the hands-on / hands-off determination state is maintained in the same state as the previous determination result.

[0082] As described above, the determination processing unit 13 of the in-vehicle control system 100 shown in Fig. 1 can detect whether the steering wheel is moving or not based on the steering angle detected by the steering angle sensor 12. The determination processing unit 13 can also detect whether the hands are moving or not based on the image captured by the in-vehicle camera 11.

[0083] Furthermore, by executing the processing shown in Figures 9 and 10, the judgment processing unit 13 can grasp the situation where the hands are on the circumference of the steering wheel model and the change in the hand rotation angle θ is equal to the change in the steering angle Φ, and can make a hands-on judgment in S56.

[0084] Also, in a situation where the driver has changed his grip on the steering wheel with his hands, the determination processing unit 13 can grasp the situation where the hand rotation angle θ and the change in the steering angle Φ are not equal, and can make a hands-off determination in S58. Also, in a situation where the hands are not on the circumference of the steering wheel model, the determination processing unit 13 can make a hands-off determination in S58.

[0085] Furthermore, in a situation where the driver only has their hands on the steering wheel, such as in a forward steering operation, there is no hand movement and steering wheel movement is detected, so the judgment processing unit 13 can make a hands-off judgment in S58. Also, in a situation where there is no steering wheel movement and only the hands move, it is assumed that the steering wheel is being adjusted, so the judgment processing unit 13 can make a hands-off judgment in S58. Also, in a situation where there is no steering wheel movement and no hand movement, the judgment processing unit 13 can maintain the previous judgment result as it is in S60.

[0086] <Advantages of vehicle steering wheel holding detection device> The determination processing unit 13 of the in-vehicle control system 100 shown in FIG. 1 can obtain data that matches the coordinate system of the steering wheel model HM by coordinate transformation using the transformation matrix A, even when processing an image captured from an oblique angle, such as the captured image 30A shown in FIG. 7. Furthermore, by examining the overlap of hand positions L'1, R'1 on the simple circular steering wheel model HM, it is possible to determine with a simple calculation whether the hands are on the steering wheel. Furthermore, by comparing the rotation angle θ corresponding to changes in hand positions L'1, L'2 on the steering wheel model HM with the change in steering wheel angle Φ in S34, as shown in FIG. 8, it is easy to accurately distinguish between a state in which the driver is gripping the steering wheel and a state in which the hands are simply resting on the steering wheel. Furthermore, by using the transformation matrix A, which includes predetermined constant data, the coordinate system (x / y) of the captured image 30A captured by the in-vehicle camera 11 can be transformed into the coordinate system (x' / y') of the steering wheel model HM with a simple calculation.

[0087] The present invention is not limited to the above-described embodiments, and can be appropriately modified, improved, etc. Furthermore, the material, shape, size, number, location, etc. of each component in the above-described embodiments are arbitrary and not limited as long as they can achieve the present invention.

[0088] For example, in the above-described embodiment, the state of steering wheel holding is grasped using the image captured by the in-vehicle camera 11 and the steering angle detected by the steering angle sensor 12, but control may also be performed taking other information into consideration. For example, a capacitance sensor may be placed on the steering wheel 18 to detect whether or not the hands are in contact with the steering wheel 18. It is also possible to detect whether or not the hands are in contact with the steering wheel 18 based on fluctuations in the steering angle detected by the steering angle sensor 12. Furthermore, in situations where the hands-off state continues for a long period of time, it is possible to use the images captured by the in-vehicle camera 11 and the processing functions of the determination processing unit 13 to detect various situations other than holding the steering wheel. For example, it is expected that a function will be added to detect a state in which the person sitting in the driver's seat is holding an object such as a smartphone in their hand, or to detect gesture input operations from the hand movements of the person sitting in the driver's seat.

[0089] The characteristic features of the above-mentioned vehicle handle holding detection device are briefly summarized and listed in the following [1] to [5]. [1] An imaging unit (vehicle-mounted camera 11) capable of capturing an image of a steering wheel (steering wheel 18) for steering a vehicle and the hands of a driver of the vehicle; a position conversion unit (S25, S31, S52) that associates the position of the handle in the image acquired by the imaging unit with a position in a handle model (HM) having a coordinate system (x' / y') based on an axial direction different from the imaging direction of the imaging unit; a steering angle acquisition unit (S32, S43, S47) that acquires the steering angle of the steering wheel; an identification unit (determination processing unit 13) that identifies whether the handle is being held by hand; Equipped with The identification unit recognizes that the steering wheel is being held by a hand when the position of the steering wheel in the image acquired by the imaging unit overlaps with the steering wheel model in the coordinate system of the steering wheel model, and the rotation angle (θ) on the steering wheel model corresponding to the change in position of the steering wheel in the image acquired by the imaging unit matches the change in steering angle (Φ) acquired by the steering angle acquisition unit (S56). A steering wheel holding detection device for a vehicle.

[0090] According to the vehicle steering wheel holding detection device configured as described in [1] above, when the driver's hands are on the steering wheel and the change in the steering angle Φ of the steering wheel corresponds to the rotation angle θ of the hand movement, it is possible to detect the steering wheel being held with high detection accuracy. Moreover, since the steering wheel model is in a coordinate system different from that of the imaging unit, the amount of calculation can be significantly reduced. Therefore, steering wheel holding can be detected in real time without the need for a high-performance control device.

[0091] [2] The position conversion unit pre-stores data of a transformation matrix (A) used for coordinate conversion between the position of the steering wheel in the image acquired by the imaging unit and the position on the steering wheel model. The vehicle steering wheel holding detection device according to [1] above.

[0092] According to the vehicle steering wheel holding detection device configured as described above in [2], coordinate transformation is performed using predetermined data, so that the steering wheel model can be used without performing complex calculations. Furthermore, because coordinate transformation is performed, there is no need to adjust the shooting direction or shooting range of the imaging unit to match the coordinate system of the steering wheel model, which increases the flexibility of where the imaging unit is installed. In other words, even an image of the steering wheel photographed from an oblique angle can be used to detect steering wheel holding.

[0093] [3] The handle model has position information specified in a two-dimensional coordinate system (x' / y') based on the position (P0) of the rotation axis of the handle in a plane perpendicular to the rotation axis of the handle, and has at least constant data (radius q) representing the position of an operating part that can hold the handle on the two-dimensional coordinate system. The vehicle steering wheel holding detection device according to [1] or [2] above.

[0094] The vehicle steering wheel holding detection device configured as described in [3] above can represent a steering wheel model as a simple circular area on a two-dimensional coordinate system, making it possible to determine whether the hand position overlaps with a position on the steering wheel model through simple calculations.

[0095] [4] The operation part of the handle model includes data of a circumferential area whose center is the position of the rotation axis and whose distance from the center is represented by a constant radius (q). The vehicle steering wheel holding detection device according to [3] above.

[0096] According to the vehicle steering wheel holding detection device having the configuration [4] above, the operating part of the steering wheel model is represented as a circular circumferential area centered on the position of the rotation axis, thereby reducing the load of the calculation process for identifying the position of the operating part.

[0097] [5] The imaging unit captures an image of a photographing range (A01 to A04) that simultaneously includes images of both the steering wheel and the driver's hands, from a direction shifted from the direction of the rotation axis of the steering wheel. The vehicle handle holding detection device according to any one of [1] to [4] above.

[0098] According to the vehicle steering wheel holding detection device configured as described above in [5], the steering wheel and the driver's hands are both included in a common image, so the steering wheel and the driver's hands in the image can be treated as a common coordinate system, thereby reducing the computational load associated with coordinate transformation. [Explanation of symbols]

[0099] 11. In-car camera 12 Steering angle sensor 13. Judgment processing unit 14 Alarm output section 15 Information display section 16 Operation control unit 17 Data storage unit 18 Steering wheel 20. Automobiles 21 Overhead section 22 Pillar section 23 Center Section 24 Dome lamp section 25 Windshield 26 Driver's seat 30A, 30A1, 30A2, 30B Photographed images 31 Handle 31A Handle area 32 Driver 32a right hand 32b left hand 100 In-vehicle control system A: Transformation matrix A01, A02, A03, A04 Shooting range D1, D2 data HM Handle Model L1, L2 left hand position R1 Right Hand Position Φ Rudder angle θ rotation angle

Claims

1. an imaging unit capable of capturing an image of a steering wheel of a vehicle and an image of the hands of a driver of the vehicle; a position conversion unit that associates the position of the steering wheel in the image acquired by the imaging unit with a position in a steering wheel model having a coordinate system based on an axial direction different from the imaging direction of the imaging unit; a steering angle acquisition unit that acquires a steering angle of the steering wheel; an identification unit that identifies whether the handle is being held by hand; Equipped with the identification unit recognizes that the steering wheel is being held by a hand when the position of the steering wheel in the image acquired by the imaging unit overlaps with the steering wheel model in the coordinate system of the steering wheel model, and when a rotation angle on the steering wheel model corresponding to a change in the position of the steering wheel in the image acquired by the imaging unit matches a change in steering angle acquired by the steering angle acquisition unit. A steering wheel holding detection device for a vehicle.

2. the position conversion unit pre-stores data of a transformation matrix used for coordinate conversion between the position of the handle in the image acquired by the imaging unit and the position on the handle model; 2. The steering wheel holding detection device for a vehicle according to claim 1.

3. the handle model has position information specified in a two-dimensional coordinate system based on the position of the rotation axis of the handle in a plane perpendicular to the rotation axis of the handle, and has at least constant data representing the position of an operating part capable of holding the handle on the two-dimensional coordinate system; 2. The steering wheel holding detection device for a vehicle according to claim 1.

4. The operation portion of the handle model includes data of a circumferential area having a center at the position of the rotation axis and a distance from the center represented by a constant radius.

4. The steering wheel holding detection device for a vehicle according to claim 3.

5. the imaging unit captures an image of a photographing range including images of both the steering wheel and the driver's hands from a direction shifted from the direction of the rotation axis of the steering wheel, The steering wheel holding detection device for a vehicle according to any one of claims 1 to 4.

Images