Steering wheel position information detection device

Abstract

To reduce computational complexity required for processing, such as object recognition, when identifying positions of hands of a driver on a steering wheel from a captured image, and easily improve position detection accuracy.SOLUTION: An in-vehicle camera captures an image including right and left hands of a driver and a steering wheel, detects a steering angle of the steering wheel through a steering angle sensor, and records result data on the positions of the hands and the steering angle. A determination processing unit automatically calculates constant data of a transformation matrix A required for coordinate transformation from a first coordinate system (X / Y) for the steering handle captured from an oblique direction to a second coordinate system (X' / Y') of a steering wheel model of the steering wheel viewed from the front, on the basis of the recorded result data. The determination processing unit acquires two or more sets of detection data obtained by associating the detected steering angle with the positions of the right and left hands in the captured image, to calculate the constant data based on the acquired data.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to a steering wheel position information detection device that can be used to detect the position of a driver's hands on a steering wheel for steering a vehicle. [Background technology]

[0002] In a vehicle, the function of detecting whether the driver is holding the steering wheel, i.e., the handle for steering, is a necessary function to support safe driving. In particular, in recent years, with the possibility of vehicles being equipped with autonomous driving functions, it is necessary to be able to correctly determine whether the driver is holding the steering wheel.

[0003] For example, in vehicles equipped with Level 2 autonomous driving functions, the driver must keep their hands on the steering wheel during autonomous driving. In addition, in vehicles equipped with Level 3 autonomous driving functions, it is necessary to detect whether the driver is actually holding the steering wheel and is ready to drive in order to safely hand over driving operations from the autonomous driving system to the driver when the autonomous driving system reaches its limit.

[0004] For example, Patent Document 1 discloses a driver monitoring device that accurately detects whether the driver sitting in the driver's seat is gripping the steering wheel when switching from an autonomous driving mode to a manual driving mode. Specifically, the device reads an image captured by a driver imaging camera, processes the driver image, detects whether the steering wheel is being gripped, and if so, executes a process to detect the position where the steering wheel is being gripped.

[0005] In processing the driver image, the edges (contours) of the steering wheel are first extracted using image processing such as edge detection. Next, edges that intersect with the extracted edges of the steering wheel are extracted, and if such edges are detected, it is determined whether they correspond to fingers based on the length and spacing of the edges. If it is determined that an edge corresponds to a finger, the position of the edge corresponding to the finger is detected as the steering wheel grip position.

[0006] The posture detection unit processes the driver image to detect the posture of the driver's shoulders and arms. Following the processing by the posture detection unit, the grip determination unit determines whether the imaged driver is gripping the steering wheel based on the grip position of the steering wheel detected by the grip position detection unit and the posture of the driver's shoulders and arms detected by the posture detection unit.

[0007] For example, a judgment table showing the relationship between the grip position of the steering wheel and the posture (direction and angle) of the driver's shoulders and arms is read out, and the detected grip position of the steering wheel and the posture of the driver's shoulders and arms are applied to the judgment table to determine whether the condition of gripping the steering wheel is met. Furthermore, as described in Patent Document 1, it is disclosed that a contact detection sensor provided on the steering wheel is used to detect a hand touching the steering wheel. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Publication No. 2018-190217 Summary of the Invention [Problem to be solved by the invention]

[0009] However, even if the technology of Patent Document 1 is adopted and the position of the driver's hands on the steering wheel can be correctly detected based on an image of the driver, it is not certain whether the driver is actually holding the steering wheel. For example, it is possible that the driver is simply bringing their hands close to the steering wheel but is not actually holding the steering wheel, making it impossible for the automated driving system to safely hand over driving operations to the driver.

[0010] Therefore, as disclosed in Patent Document 1, it is necessary to perform special processing such as detecting hand contact using a contact detection sensor or detecting the posture of the driver's shoulders and arms. However, when realizing a function such as that disclosed in Patent Document 1 based on an image of the driver, it is necessary to repeatedly recognize the shapes and objects of the steering wheel and each part of the driver's body in almost real time, which usually requires very complex image processing to be performed at high speed. Therefore, a very expensive image processing device must be installed. Furthermore, if a relatively low-performance image processing device is used, there is a concern that the detection time delay will increase.

[0011] The present invention has been made in consideration of the above-mentioned circumstances, and its purpose is to provide a steering wheel position information detection device that can reduce the amount of calculation required for processing such as object recognition when determining the position of a driver's hands on the steering wheel from a captured image, and that can improve the accuracy of position detection. [Means for solving the problem]

[0012] The above object of the present invention can be achieved by the following configuration.

[0013] an imaging unit capable of capturing an image of a steering wheel of a vehicle and an image of the hands of a driver of the vehicle; a steering angle acquisition unit that acquires a steering angle of the steering wheel; a constant calculation unit that calculates constant data that represents the correlation between a first coordinate system that represents a position in the image acquired by the imaging unit and a second coordinate system that represents a position in a predetermined handle model that is associated with the handle; Equipped with The constant calculation unit acquires at least two sets of detection data that associates the positions of the left and right hands in the first coordinate system in the image acquired by the imaging unit with the steering angle acquired by the steering angle acquisition unit, and calculates the constant data based on the acquired detection data. death , When the handle has a shape other than a circle, the constant calculation unit acquires at least three sets of the detection data and calculates the constant data based on the acquired detection data. Steering wheel position information detection device. [Effects of the Invention]

[0014] According to the steering wheel position information detection device of the present invention, constant data representing the relationship between the first coordinate system and the second coordinate system can be easily calculated. By using this constant data, when detecting the position of the driver's hands on the steering wheel in an image, coordinate conversion from the first coordinate system to the second coordinate system can be performed by simple calculation. Furthermore, since an image of the steering wheel is usually taken from an oblique direction, the steering wheel appears as an oval or other shape in the first coordinate system, but the steering wheel model in the second coordinate system can be represented as a very simple shape such as a circle, making it easy to calculate the hand position.

[0015] The present invention has been briefly described above. The details of the present invention will become clearer by reading the following detailed description of the invention (hereinafter referred to as "embodiments") with reference to the accompanying drawings. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an in-vehicle control system. [Figure 2] FIG. 2 is a front view showing an example of the installation position of the vehicle-mounted camera in the vehicle interior. [Figure 3] FIG. 3 is a side view showing an example of the installation position of the vehicle-mounted camera in the vehicle interior. [Figure 4] FIG. 4 is a front view showing a plurality of examples of captured images. [Figure 5] FIG. 5 is a flowchart showing an outline of the steering wheel holding detection process. [Figure 6] FIG. 6 is a flowchart showing the procedure for detecting whether the steering wheel is held down according to the first method. [Figure 7] FIG. 7 is a schematic diagram showing the flow of data processing in the steering wheel holding detection of Method 1. [Figure 8] FIG. 8 is a schematic diagram showing the flow of data processing in the steering wheel holding detection of Method 2. [Figure 9]FIG. 9 is a flowchart showing a part of the processing procedure for detecting whether the steering wheel is held down, which is a combination of the methods 1 and 2. [Figure 10] FIG. 10 is a flowchart showing a part of the processing procedure for detecting whether the steering wheel is held down, which is a combination of the methods 1 and 2. [Figure 11] FIG. 11 is a flowchart showing a processing procedure for obtaining constant data necessary for coordinate transformation. [Figure 12] FIG. 12 is a schematic diagram showing the flow of main data processing for obtaining constant data necessary for coordinate transformation. [Figure 13] FIG. 13 is a front view showing the states of a specially shaped handle before and after rotation. [Figure 14] FIG. 14 is a flowchart showing a modification of the process shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0017] Specific embodiments of the present invention will be described below with reference to the accompanying drawings.

[0018] <System configuration> FIG. 1 shows an example of the configuration of an in-vehicle control system 100 including a steering wheel position information detection device according to an embodiment of the present invention.

[0019] The vehicle control system 100 in FIG. 1 includes an in-vehicle camera 11, a steering angle sensor 12, a judgment processing unit 13, an alarm output unit 14, an information display unit 15, a driving control unit 16, a data storage unit 17, and a steering wheel 18.

[0020] The steering handle 18, i.e., the steering wheel, has a circular operating portion that is easy for the driver to operate, and its center is supported by a freely rotatable rotation shaft. The rotation shaft of the steering handle 18 is connected to the steering mechanism of the front wheels via, for example, a mechanical connection mechanism. Therefore, the driver can adjust the steering angle of the front wheels by rotating the steering handle 18.

[0021] In the case of a vehicle equipped with an automatic driving function, an electrically controllable actuator is connected to the steering mechanism, which means that the steering angle of the wheels can be automatically adjusted to change the direction of travel of the vehicle without the driver having to operate the steering wheel 18.

[0022] The steering angle sensor 12 is attached to, for example, the rotation axis of a steering wheel 18, and is capable of detecting the actual steering angle of the automobile and changes therein.

[0023] The vehicle-mounted camera 11 is installed inside the vehicle cabin as described below, and its imaging direction and imaging range are fixed so that it can simultaneously capture the steering wheel 18 located in front of the driver's seat and the left and right hands of the driver seated in the driver's seat.

[0024] The determination processing unit 13 is configured as an electronic control unit including components such as a microcomputer, and has a function for automatically identifying whether or not the driver is holding the steering wheel 18. The determination processing unit 13 also has a function for determining in advance constant data of a transformation matrix used in coordinate transformation.

[0025] As will be described later, the judgment processing unit 13 of this embodiment detects the position of the steering wheel 18 and the position of the hands from the image captured by the in-vehicle camera 11, and identifies whether the driver is holding the steering wheel 18 by taking into account the change in the steering angle detected by the steering angle sensor 12.

[0026] The warning output unit 14 can issue a warning to the driver when necessary using a buzzer or voice output. For example, during autonomous driving, which requires the driver to keep their hands on the steering wheel, the warning output unit 14 outputs a warning if the driver's hands are off the steering wheel. Also, when the driver takes over driving the vehicle from the autonomous driving system, the warning output unit 14 can be used to output appropriate guidance or warning depending on whether the driver is holding the steering wheel or not.

[0027] The information display unit 15 has a two-dimensional display capable of displaying images, figures, characters, etc. The information display unit 15 can be used to display information necessary to assist the driver in driving operations, to display the status of automatic driving, and to display information necessary for smooth switching between automatic driving and manual driving.

[0028] The driving control unit 16 includes an actuator for controlling the steering of the vehicle, an actuator for accelerator control, an actuator for brake control, sensors that can grasp road conditions using radar, stereo cameras, etc., and an automatic driving unit that controls each actuator and automatic transmission according to the situation.

[0029] The data storage unit 17 has the function of storing, as history, image data showing the status inside the vehicle cabin captured by the vehicle-mounted camera 11, data showing the road status, data showing the status of automatic driving, and the like.

[0030] <Installation location of in-vehicle camera> Examples of the installation positions of the vehicle-mounted camera 11 in the cabin of the automobile 20 are shown in FIGS. For example, by installing the in-vehicle camera 11 at the position of the overhead section 21 in Figure 2 and photographing the shooting range A01, it is possible to capture an image that shows the steering wheel 18 and the hands of the driver sitting in the driver's seat 26.

[0031] Similarly, by installing the in-vehicle camera 11 at the position of the pillar portion 22 in Figure 2 and photographing the shooting range A02, it is possible to capture an image that shows the steering wheel 18 and the hands of the driver seated in the driver's seat 26.

[0032] In addition, by installing the in-vehicle camera 11 at the center portion 23 in Figure 2 and photographing the shooting range A03, it is possible to capture an image that shows the steering wheel 18 and the hands of the driver seated in the driver's seat 26.

[0033] In addition, by installing the vehicle-mounted camera 11 at the position of the dome lamp section 24 in Figure 3 and photographing the shooting range A04, it is possible to capture an image that shows the steering wheel 18 and the hands of the driver seated in the driver's seat 26.

[0034] In other words, the on-board camera 11 of the on-board control system 100 shown in Figure 1 only needs to be capable of simultaneously capturing an image of the steering wheel and the driver's hands holding the steering wheel, and the on-board camera 11 may be installed in any of the overhead section 21, pillar section 22, center section 23, and dome lamp section 24.

[0035] <Example of captured image> An example of multiple captured images is shown in FIG. For example, when the vehicle-mounted camera 11 is installed at the position of the center section 23 in Fig. 2, an image like the captured image 30A in Fig. 4 can be captured. When the vehicle-mounted camera 11 is installed at the position of the overhead section 21 in Fig. 2, an image like the captured image 30B in Fig. 4 can be captured.

[0036] 4 includes an image of a steering wheel 31 and the right hand 32a and left hand 32b of a driver 32. Therefore, based on the captured image 30A or 30B, it is possible to grasp the state of the steering wheel 31 and the state of the right hand 32a and left hand 32b.

[0037] <Outline of handle holding detection process> An overview of the steering wheel holding detection process is shown in Figure 5. The process in Figure 5 is explained below. The determination processing unit 13 shown in FIG. 1 receives as input the data of the image captured by the vehicle-mounted camera 11 (S11).

[0038] The determination processing unit 13 acquires data of a transformation matrix ([A] described later) required to grasp the handle position shape from the image data input in S11 (S12).

[0039] In this embodiment, a handle model HM (described later) suitable for grasping the position and shape of the handle 31 is created in advance. This handle model HM is represented, for example, in a two-dimensional coordinate system parallel to a plane perpendicular to the rotation axis of the handle 31. On the other hand, for example, the handle 31 in the photographed image 30A in FIG. 4 was photographed from an oblique direction, so the two-dimensional coordinates of the photographed image 30A are significantly different from those of the handle model HM. Specifically, since the actual shape of the handle 31 is circular, it is expected that the handle shape in the handle model HM will also be circular, but the handle 31 in the photographed image 30A is elliptical.

[0040] Therefore, it is necessary to convert the coordinates of the handle 31 in the photographed image 30A into a two-dimensional coordinate system on the handle model HM. By acquiring and using the constant data of the conversion matrix created in advance in S12, the determination processing unit 13 can easily perform the coordinate conversion.

[0041] The determination processing unit 13 acquires the actual angle (steering angle) of the steering wheel 18 from the steering angle sensor 12 (S13).

[0042] The determination processing unit 13 detects the positions of the left and right hands by performing image processing on the image data input in S11 (S14). For example, the positions of the hands in the image can be detected by comparing reference pattern data representing the characteristics of the hands or the characteristics of the steering wheel with the feature amounts of each area in the image. Furthermore, by tracking the detected hand positions, movement of the hand positions in the image can be detected. Existing technology can be used for the basic processing required to detect various elements in such an image.

[0043] The determination processing unit 13 uses the data acquired in S11 to S14 to perform processing to detect whether the driver is holding the steering wheel (hands-on) (S15). In this processing, in order to improve the determination accuracy, the determination is made by combining two types of methods, "Method 1" and "Method 2" described later.

[0044] <Details of "Method 1" handle holding detection> The processing procedure and data processing flow for steering wheel holding detection in "Method 1" are shown in Figure 6 and Figure 7, respectively. Steering wheel holding detection in "Method 1" will be explained below.

[0045] The determination processing unit 13 acquires data such as the captured image 30A shown in Fig. 7 from the in-vehicle camera 11 (S21). The captured image 30A includes images of the steering wheel 31 and the right hand 32a and left hand 32b of the driver 32.

[0046] In S22, the determination processing unit 13 performs image processing on the captured image 30A to obtain position information of the handle region 31A representing the handle 31. Here, the shape of the handle region 31A is elliptical, and the position and shape are expressed by a two-dimensional coordinate system (x / y coordinate system) specified by the position in the x-axis direction and the position in the y-axis direction in the captured image 30A.

[0047] In S23, the determination processing unit 13 obtains information on the position R1 (xr1, yr1) of the right hand 32a and the position L1 (xl1, yl1) of the left hand 32b in the x / y coordinate system by image processing of the captured image 30A.

[0048] In S24, the determination processing unit 13 determines whether or not the hand position has been detected in S23, and if the hand position has been detected, the process proceeds to S25.

[0049] In order to utilize the handle model HM created in advance, the judgment processing unit 13 performs coordinate conversion of the data in S25, and maps the position of the handle area 31A obtained in S22 and the hand positions R1 and L1 obtained in S23 into the coordinate system of the handle model HM.

[0050] In this embodiment, the position and shape of the handle model HM are represented by a two-dimensional coordinate system (x' / y' coordinate system) specified by the position in the x'-axis direction and the position in the y'-axis direction, as shown in Fig. 7. The origin coordinates (0,0) of the x' / y' coordinate system are set as the center P0, and the handle model HM is a circle on the circumference at a distance of radius q (constant) from this center. Because the handle model HM has a simple circular shape, it is easy to grasp each position on the handle and the rotation angle on the handle model HM.

[0051] Therefore, in S25, the determination processing unit 13 converts the position and shape of each data from the x / y coordinate system to the x' / y' coordinate system. Here, by multiplying the input data in the x / y coordinate system by a transformation matrix A prepared in advance, output data in the x' / y' coordinate system can be easily generated.

[0052] The judgment processing unit 13 compares the circular (circumference) area of ​​the handle model HM (corresponding to the handle area 31A) with the position R'1 (xr'1, yr'1) of the right hand and the position L'1 (xl'1, yl'1) of the left hand in the x' / y' coordinate system, and determines in S36 whether the hands (32a, 32b) are resting on the handle 31 (are overlapping).

[0053] Specifically, in S26, the determination processing unit 13 determines whether or not the coordinates (xr'1, yr'1) of the right hand position R'1 satisfy the condition of the following [Equation 1].

number

[0054] The determination processing unit 13 also compares the coordinates (xl'1, yl'1) of the left hand position L'1 under the same conditions as in [Equation 1]. Then, when it is recognized that the hands (32a, 32b) are on the handlebars 31, the determination processing unit 13 detects the hands-on state in "method 1" (S27).

[0055] The contents of the transformation matrix A can be appropriately determined by experiments or analysis of past data. For example, in the case of a vehicle that automatically adjusts the steering angle of the steering wheel 18 using an actuator, data correlating changes in the position of the hands on the steering wheel 18 with the captured image 30A taken by the in-vehicle camera 11 can be recorded and accumulated as actual results. By analyzing the accumulated actual data, the relationship between each position in the x / y coordinate system of the image taken by the in-vehicle camera 11 and each position in the x' / y' coordinate system of the steering wheel model HM can be identified, and the contents of the transformation matrix A appropriate for the relationship can be determined, which is composed of multiple constants representing the relationship. The data of the transformation matrix A determined in this manner is stored in a non-volatile memory on the in-vehicle control system 100.

[0056] <Details of "Method 2" for detecting handle holding> The data processing flow for steering wheel holding detection in "Method 2" is shown in Figure 8. Steering wheel holding detection in "Method 2" is explained below.

[0057] The determination processing unit 13 acquires an image including an image of the steering wheel and hands from the vehicle-mounted camera 11, such as the captured images 30A1 and 30A2 shown in FIG. 8, and acquires the position of the steering wheel area 31A and the position of the hands from this image.

[0058] Moreover, as the hand position changes over time as seen in photographed images 30A1 and 30A2, the determination processing unit 13 tracks the hand position. In the example shown in Fig. 8, the left hand position L1 (xl1, yl1) is detected from photographed image 30A1, and the left hand position L2 (xl2, yl2) is detected from photographed image 30A2.

[0059] The judgment processing unit 13 performs coordinate transformation in S31 using transformation matrix A on the left hand position L1 (xl1, yl1) detected from the captured image 30A1 and the left hand position L2 (xl2, yl2) detected from the captured image 30A2, and detects the hand positions L''1 and L'2 on the same x' / y' coordinates as the handle model HM.

[0060] In addition, the judgment processing unit 13 acquires the steering angle Φ1 of the steering wheel 18 detected by the steering angle sensor 12 when the captured image 30A1 was taken, and the steering angle Φ2 of the steering wheel 18 detected by the steering angle sensor 12 when the captured image 30A2 was taken, and calculates the change in the actual steering angle Φ (Φ1-Φ2) in S32.

[0061] The determination processing unit 13 calculates the rotation angle θ corresponding to the amount of change in the hand positions L'1 and L'2 on the handle model HM in S33. Specifically, the determination processing unit 13 calculates the rotation angle θ from the coordinates (x1, y1) of the hand position L'1 and the coordinates (x2, y2) of the hand position L'2 on the handle model HM based on the following [Equation 2].

[0062]

number

number

[0063] <Handle detection using a combination of "Method 1" and "Method 2"> The processing procedure for handlebar holding detection that combines the above-mentioned "method 1" and "method 2" is shown in Figures 9 and 10. That is, in the processing of step S15 shown in Figure 5, the judgment processing unit 13 executes the processing of S41 to S60 shown in Figures 9 and 10 as a processing that combines both handlebar holding detection of "method 1" shown in Figures 6 and 7 and handlebar holding detection of "method 2" shown in Figure 8. The processing of Figures 9 and 10 will be explained below.

[0064] In step S41, the determination processing unit 13 detects the position L1 of the driver's left hand based on an image such as the captured image 30A1 shown in Fig. 8. If possible, the determination processing unit 13 also detects the position R1 of the right hand at the same time.

[0065] If the hand position is detected in S41, the determination processing unit 13 proceeds to the process from S42 to S43, and acquires the steering angle Φ of the steering angle sensor 12 at this time.

[0066] The determination processing unit 13 registers both the hand position detected in S41 and the steering angle Φ detected in S43 as a set of data D1 in the internal memory in S44.

[0067] In step S45, the determination processing unit 13 detects the position L2 of the driver's left hand based on an image such as the captured image 30A2 shown in Fig. 8. If possible, the determination processing unit 13 also simultaneously detects the position R2 of the driver's right hand.

[0068] If the hand position is detected in S45, the determination processing unit 13 proceeds to the process from S46 to S47, and acquires the steering angle Φ of the steering angle sensor 12 at this time.

[0069] The determination processing unit 13 registers both the hand position detected in S45 and the steering angle Φ detected in S47 as a set of data D2 in the internal memory in S48.

[0070] In S49, the determination processing unit 13 determines whether the change in steering angle output by the steering angle sensor 12 is large or small. If the change in steering angle is large, the determination processing unit 13 proceeds from S49 to S50 to detect whether or not there is a change in the hand position in the two sets of data D1 and D2. If there is movement in the hand position, the process proceeds from S50 to S52.

[0071] Also, for example, if the driver is simply placing his / her hands on the steering wheel 18, there is no change in the position of the hands, so in this case the process proceeds from S50 to S58 and the determination processing unit 13 determines that the hands are off.

[0072] If the steering angle change is small, the determination processing unit 13 proceeds from S49 to S51, and detects whether or not there is a change in the hand position in the two sets of data D1 and D2. If there is movement in the hand position, the process proceeds from S51 to S58. For example, if the driver changes his or her grip on the steering wheel 18 and changes the position of his or her hands, the hand position may move significantly even if the steering angle change is small, so the process proceeds to S58.

[0073] If the change in steering angle is large and the movement of the hand position is also large, the determination processing unit 13 performs coordinate transformation of the data D1 and D2 using the transformation matrix A described above (S52).

[0074] As in the example of Figure 7, the determination processing unit 13 compares the coordinates (xl'1, yl'1) of the position of the left hand L'1 after coordinate transformation and the coordinates (xr'1, yr'1) of the position of the right hand R'1 with the circumference of the circle of the steering wheel model HM, and determines in S53 whether the hands are in a position that almost overlaps with the steering wheel model HM. If the hands are in a position that almost overlaps with the steering wheel model HM, the process proceeds from S53 to S54, and if they are not in an overlapping position, the process proceeds from S53 to S58. If the hands are not in a position that overlaps with the steering wheel model HM, it can be assumed that the hands have left the steering wheel 18.

[0075] 8, the determination processing unit 13 calculates the rotation angle θ corresponding to the amount of change in the hand positions L'1 and L'2 on the steering wheel model HM in S54. That is, the determination processing unit 13 calculates the rotation angle θ based on the coordinates (x1, y1) of the hand position L'1 and the coordinates (x2, y2) of the hand position L'2 on the steering wheel model HM, based on the above-mentioned [Equation 2].

[0076] In S55, the determination processing unit 13 compares the difference (Φ1-Φ2) between the steering angles Φ in the two sets of data D1 and D2 with the rotation angle θ detected in S54 to determine whether they substantially match.

[0077] If the difference in the steering angle Φ and the rotation angle θ match, the determination processing unit 13 proceeds to the process from S55 to S56. In this case, the determination processing unit 13 determines in S56 that the vehicle is in a hands-on state, and registers the contents of data D2 in data D1 (S57).

[0078] If the difference in the steering angle Φ and the rotation angle θ do not match, the determination processing unit 13 proceeds to the process from S55 to S58, determines that the hands are off, and erases the contents of the data D1 and D2 (S59).

[0079] If the change in the steering angle is small and there is no movement in the hand position, the determination processing unit 13 proceeds from S51 to S60. In this case, the hands-on / hands-off determination state is maintained in the same state as the previous determination result.

[0080] <Explanation of the steering wheel position information detection device> The steering wheel position information detection device of this embodiment can be used to automate the task of determining constant data representing the contents of the "transformation matrix A" used in the process of converting the coordinate system (X / Y) of the captured image 30A shown in FIG. 7 into the coordinate system (X' / Y') of the steering wheel model HM.

[0081] The function of this steering wheel position information detection device is included in the in-vehicle control system 100 shown in Fig. 1. Furthermore, before using the steering wheel holding detection function of the in-vehicle control system 100, constant data representing the contents of "transformation matrix A" is determined using the steering wheel position information detection device and saved in a predetermined storage device. The processing procedure and main data flow in this steering wheel position information detection device are shown in Fig. 11 and Fig. 12, respectively.

[0082] When the process shown in FIG. 11 is executed by the determination processing unit 13 in FIG. 1, the determination processing unit 13 collects the performance data required for calculating the constant data in steps S71 to S78.

[0083] The judgment processing unit 13 receives image data such as the captured images 50A1 and 50A2 shown in Fig. 12 from the in-vehicle camera 11 (S71). Here, the captured images 50A1 and 50A2 are images captured at different times, with the captured image 50A1 being an image captured when the steering wheel steering angle Φ [°] is Φ1, and the captured image 50A2 being an image captured when the steering wheel steering angle Φ is Φ2. The steering wheel steering angle Φ at each time point can be determined from the detection value of the steering angle sensor 12.

[0084] As with the captured images 50A1 and 50A2 shown in FIG. 12, valid images that can be used to calculate constant data allow the positions of the steering wheel and the driver's left and right hands to be known.

[0085] In addition, each position in each of the captured images 50A1 and 50A2 captured by the vehicle-mounted camera 11 is represented by position coordinates in an X / Y coordinate system in which the horizontal direction is the X axis and the vertical direction is the Y axis on a two-dimensional plane perpendicular to the axis of the capturing direction of the vehicle-mounted camera 11.

[0086] The judgment processing unit 13 detects the positions R1(xr1, yr1), R2(xr2, yr2) of the right hand and the positions L1(xl1, yl1), L2(xl2, yl2) of the left hand in the steering wheel area 51A in each of the captured images 50A1, 50A2 input from the in-vehicle camera 11 (S72).

[0087] When the number of detected hand positions reaches two or more, that is, when the positions of both the left and right hands can be detected, the determination processing unit 13 proceeds from S73 to S74 and obtains the steering angle Φ detected by the steering angle sensor 12.

[0088] The determination processing unit 13 records the data on the positions of the left and right hands and the steering angle Φ acquired in S74 for each of the captured images 50A1 and 50A2 in the data holding unit 40 as one set of recorded data (S75).

[0089] If the number of recorded data sets stored in the data storage unit 40 is one set or less, the determination processing unit 13 returns from S76 to S71 and repeats the above process. If the number of recorded data sets is two or more, the determination processing unit 13 proceeds from S76 to S77 and compares the torque value (rotation angle corresponding to position change) between the multiple data sets with a predetermined threshold value (for example, approximately 10°). This threshold value can be determined appropriately according to the hand position detection performance.

[0090] For example, if the hand positions are too close between the first and second data sets, it becomes difficult to accurately grasp the movement of the hand position. Therefore, if the hand positions are too close, at least one of the multiple data sets is deemed invalid and deleted from the data storage unit 40 (S78), and the process returns to S71. If the hand positions are sufficiently far apart between the multiple data sets, the determination processing unit 13 considers them to be valid data sets and proceeds to the processes of S77 to S79.

[0091] In S79, the determination processing unit 13 calculates a homography matrix based on the multiple sets of recorded data stored in the data storage unit 40. That is, in S79, the determination processing unit 13 calculates constant data representing the contents of a "transformation matrix A" that can be used for coordinate transformation between the X / Y coordinate system of the captured images 50A1 and 50A2 and the X' / Y' coordinate system of the handle model HM, as shown in FIG.

[0092] 12, the steering wheel model HM is a circular model on a two-dimensional plane perpendicular to the rotation axis of the steering wheel 18, with the origin being the position of the rotation axis, and is on a circumference at a position of a certain radius from the origin position. Each position on this steering wheel model HM is represented by an X' / Y' coordinate system with the horizontal direction as the X' axis and the vertical direction as the Y' axis.

[0093] The positions of the right hand R1 (xr1, yr1) and R2 (xr2, yr2) in the X / Y coordinate system are expressed as R'1 (xr'1, yr'1) and R'2 (xr'2, yr'2) in the X' / Y' coordinate system. The positions of the left hand L1 (xl1, yl1) and L2 (xl2, yl2) are expressed as L'1 (xl'1, yl'1) and L'2 (xl'2, yl'2).

[0094] Furthermore, the rotation angle θ and steering angle change (Φ1-Φ2) corresponding to the change in the right hand positions R1(xr1, yr1) and R2(xr2, yr2) are equal to the rotation angle θ and steering angle change (Φ1-Φ2) corresponding to the change in the left hand positions L1(xl1, yl1) and L2(xl2, yl2).

[0095] Therefore, the determination processing unit 13 calculates the constants of the "transformation matrix A" based on the following relational expressions [Equation 4] and [Equation 5] in S79 and stores them in the data holding unit 41.

number

number

[0096] <Modification of the steering wheel position information detection device> The states of the specially shaped handle 42 before and after rotation are shown in Fig. 13. Also, a modified example of the process shown in Fig. 11 is shown in Fig. 14.

[0097] 13 has an H-shape on a plane perpendicular to the direction of the rotation axis 42a at its center. Therefore, the part of the steering wheel 42 that the driver holds in his / her hands is linear, and the distance from the rotation axis 42a is not constant.

[0098] In the example shown in Figure 13, a situation is assumed in which the driver holds the steering wheel 42 at left hand position PL1 and right hand position PR1, and then rotates the steering wheel 42 to move to left hand position PL2 and right hand position PR2.

[0099] 13, the left hand position PL1 and the right hand position PR1 are not symmetrical with respect to the position of the rotation axis 42a. Therefore, the radius of the circumference 43L representing the movement trajectory of the left hand positions PL1 and PL2 is different from the radius of the circumference 43R representing the movement trajectory of the right hand positions PR1 and PR2. In other words, the steering wheel 42 shown in FIG. 13 cannot be represented by a circular model with a constant radius like the above-mentioned steering wheel model HM.

[0100] Therefore, when the positions of the left and right hands are asymmetric, as in the case of steering wheel 42 in Fig. 13, it is necessary to specify steering wheel models as separate circles for the left and right hands. Therefore, it is also necessary to prepare separate transformation matrices A for the right hand and AL for the left hand as the transformation matrix A used in coordinate transformation.

[0101] Furthermore, when calculating constant data for the transformation matrices AL and AR separately for the left and right hand positions in the case of the steering wheel 42 shown in FIG. 13, it is necessary to prepare actual data for three or more hand positions on each side.

[0102] Therefore, in this case, the determination processing unit 13 calculates the constant data of the transformation matrices AL and AR by the process shown in Fig. 14. In the process of Fig. 14, the determination processing unit 13 proceeds to the process from S76 to A77 after three or more data sets have been recorded in the data holding unit 40.

[0103] Furthermore, the determination processing unit 13 calculates constant data of the transformation matrices AL and AR corresponding to the positions of the left and right hands, respectively, based on the three or more data sets recorded in the data storage unit 40 in S79A.

[0104] <Uses of the steering wheel position information detection device> The above-described steering wheel position information detecting device can be used to automatically generate constant data of the transformation matrix A used in coordinate transformation between different coordinate systems (X / Y, X' / Y'). Therefore, the constant data of the transformation matrix A generated by the steering wheel position information detecting device can be used for various purposes requiring coordinate transformation.

[0105] For example, the steering wheel position information detection device can be used to detect whether or not a driver is holding the steering wheel 18 in their hands, as in the in-vehicle control system 100 shown in FIG. 1. It can also be used to detect hand movements near the steering wheel. Specifically, it is expected to be used when a function is implemented to detect whether or not a driver is holding a smartphone in their hands from images from the in-vehicle camera 11. It can also be used when a function is implemented to detect the driver's hand movements as gestures, and the detected gestures are used for various input operations.

[0106] The present invention is not limited to the above-described embodiments, and can be appropriately modified, improved, etc. Furthermore, the material, shape, size, number, location, etc. of each component in the above-described embodiments are arbitrary and not limited as long as they can achieve the present invention.

[0107] Here, the features of the steering wheel position information detecting device according to the embodiment of the present invention described above will be briefly summarized and listed below in [1] to [5]. [1] An imaging unit (vehicle-mounted camera 11) capable of capturing an image of a steering wheel (steering wheel 18) for steering a vehicle and the hands of a driver of the vehicle; a steering angle acquisition unit (steering angle sensor 12) that acquires the steering angle of the steering wheel; a constant calculation unit (determination processing unit 13) that calculates constant data (contents of transformation matrix A) that represent the correlation between a first coordinate system (X / Y) that represents a position in the image acquired by the imaging unit and a second coordinate system (X' / Y') that represents a position in a predetermined handle model (HM) associated with the handle; Equipped with the constant calculation unit acquires at least two sets of detection data associating the positions of the left and right hands in the first coordinate system in the image acquired by the imaging unit with the steering angle acquired by the steering angle acquisition unit (S75, S76), and calculates the constant data based on the acquired detection data (S79); Steering wheel position information detection device.

[0108] The steering wheel position information detection device configured as described in [1] above can automatically acquire constant data that represents the relationship between the first coordinate system and the second coordinate system. By using this constant data, when detecting the position of the driver's hands on the steering wheel in an image, coordinate conversion from the first coordinate system to the second coordinate system can be performed by simple calculation. Furthermore, since an image of the steering wheel is usually taken from an oblique angle, the steering wheel appears as an oval or other shape in the first coordinate system. However, the steering wheel model in the second coordinate system can be represented as a very simple shape, such as a circle, making it easier to calculate the hand position.

[0109] [2] When the handle has a shape other than a circle, the constant calculation unit acquires at least three sets of the detection data and calculates the constant data based on the acquired detection data (see FIG. 14). The steering wheel position information detection device according to [1] above.

[0110] According to the steering wheel position information detection device configured as above in [2], even if the steering wheel is not circular, it is possible to obtain constant data for coordinate conversion required to identify the positions of the left and right hands.

[0111] [3] When there is a difference of a predetermined angle or more between two or more sets of the acquired detection data, the constant calculation unit regards the detected data as valid detection data and calculates the constant data (S77, S79). The steering wheel position information detection device according to [1] or [2] above.

[0112] According to the steering wheel position information detecting device having the configuration [3] above, it is possible to avoid an increase in errors occurring in the constant data due to the influence of the position detection accuracy and the like.

[0113] [4] When the difference between the two or more sets of acquired detection data is small, the constant calculation unit determines that at least one of the sets of acquired detection data is invalid and deletes it (S78), and attempts to acquire new detection data. The steering wheel position information detection device according to any one of [1] to [3] above.

[0114] According to the steering wheel position information detecting device configured as described above in [4], invalid detection data that may cause errors can be eliminated, and constant data can be calculated based on two or more sets of valid detection data. Therefore, it is possible to prevent an increase in errors occurring in the constant data.

[0115] [5] If the handle is circular, the constant calculation unit calculates the constant data by assuming that the positions of the left and right hands corresponding to the detection data are on the same circumference of the handle model in the second coordinate system (S79). The steering wheel position information detection device according to the above [1], [3], or [4].

[0116] According to the steering wheel position information detection device configured as described above in [5], the shape of the steering wheel model is a simple circle, so the amount of calculation required to identify the position of the hands on the steering wheel can be reduced. [Explanation of symbols]

[0117] 11. In-car camera 12 Steering angle sensor 13. Judgment processing unit 14 Alarm output section 15 Information display section 16 Operation control unit 17 Data storage unit 18 Steering wheel 20. Automobiles 21 Overhead section 22 Pillar section 23 Center Section 24 Dome lamp section 25 Windshield 26 Driver's seat 30A, 30A1, 30A2, 30B Photographed images 31 Handle 31A Handle area 32 Driver 32a right hand 32b left hand 40,41 Data storage unit 42 Handle 43L, 43R circumference 50A1, 50A2 photographed images 51A Handle area 100 In-vehicle control system A, AR, AL: transformation matrices A01, A02, A03, A04 Shooting range D1, D2 data HM Handle Model L1, L2, PL1, PL2 Left hand position R1, R2, PR1, PR2 Right hand position Φ Rudder angle θ rotation angle

Claims

1. an imaging unit capable of capturing an image of a steering wheel of a vehicle and an image of the hands of a driver of the vehicle; a steering angle acquisition unit that acquires a steering angle of the steering wheel; a constant calculation unit that calculates constant data that represents the correlation between a first coordinate system that represents a position in the image acquired by the imaging unit and a second coordinate system that represents a position in a predetermined handle model that is associated with the handle; Equipped with the constant calculation unit acquires at least two sets of detection data that associates the positions of the left and right hands in the first coordinate system in the image acquired by the imaging unit with the steering angle acquired by the steering angle acquisition unit, and calculates the constant data based on the acquired detection data; When the handle has a shape other than a circle, the constant calculation unit acquires at least three sets of the detection data and calculates the constant data based on the acquired detection data. Steering wheel position information detection device.

2. An imaging unit capable of capturing an image of a steering wheel of a vehicle and an image of a driver's hands of the vehicle; a steering angle acquisition unit that acquires a steering angle of the steering wheel; a constant calculation unit that calculates constant data that represents the correlation between a first coordinate system that represents a position in the image acquired by the imaging unit and a second coordinate system that represents a position in a predetermined handle model that is associated with the handle; Equipped with the constant calculation unit acquires at least two sets of detection data that associates the positions of the left and right hands in the first coordinate system in the image acquired by the imaging unit with the steering angle acquired by the steering angle acquisition unit, and calculates the constant data based on the acquired detection data; the constant calculation unit, when there is a difference of a predetermined angle or more between the two or more sets of acquired detection data, regards the detected data as valid detection data and calculates the constant data; Steering wheel position information detection device.

3. An imaging unit capable of capturing an image of a steering wheel in a vehicle and an image of a driver's hands of the vehicle; a steering angle acquisition unit that acquires a steering angle of the steering wheel; a constant calculation unit that calculates constant data that represents the correlation between a first coordinate system that represents a position in the image acquired by the imaging unit and a second coordinate system that represents a position in a predetermined handle model that is associated with the handle; Equipped with the constant calculation unit acquires at least two sets of detection data that associates the positions of the left and right hands in the first coordinate system in the image acquired by the imaging unit with the steering angle acquired by the steering angle acquisition unit, and calculates the constant data based on the acquired detection data; When a difference between two or more sets of acquired detection data is small, the constant calculation unit determines at least one of the sets of acquired detection data to be invalid and deletes it, and attempts to acquire new detection data. Steering wheel position information detection device.

4. When the handle has a circular shape, the constant calculation unit calculates the constant data by assuming that the positions of the left and right hands corresponding to the detection data are on the same circumference of the handle model in the second coordinate system. The steering wheel position information detecting device according to claim 2 or 3.

Images