Vehicle control device

Abstract

Provided is a vehicle control device that is capable of increasing functions that can be implemented by a camera in relation to vehicle control while maintaining the cost of the camera. An imaging device (camera) is attached at a position and an inclination such that the wheels of a two-wheeled vehicle and the steering wheel thereof are intentionally captured, and a travel state of the vehicle is estimated by using information on the status, in the image, of the wheels and the steering wheel. By estimating the travel state of the vehicle in combination with a conventional object detection function, further driving assistance becomes possible. A vehicle control device 100 includes: an image acquisition unit 101 that acquires an image captured by an imaging device 110; and a vehicle travel state estimation unit 103 that processes the image acquired by the image acquisition unit 101 and estimates the travel state of the two-wheeled vehicle by means of at least one of the wheels and the steering wheel in the processed image.

Description

Vehicle control device

[0001] The present invention relates to a vehicle control device using a camera for a motorcycle (hereinafter also referred to as an imaging device).

[0002] The use of cameras for advanced driver assistance systems (hereinafter referred to as ADAS) for motorcycles is becoming increasingly widespread, especially for large motorcycles.

[0003] Conventional ADAS cameras are mounted on the front of a vehicle to detect people, three-dimensional objects, and other objects, and are intended to be used solely for object detection. However, because the camera cost is a high proportion of the vehicle cost for motorcycle ADAS cameras, using them in the same way as ADAS cameras for four-wheel vehicles is perceived as being less cost-effective.

[0004] It is expected that in the future, ADAS cameras for motorcycles will spread from large motorcycles to small and medium-sized motorcycles as well. Before that happens, it is necessary to come up with ideas that allow cameras to be used in conjunction with other functions, rather than just being equipped for object detection. Currently, ADAS cameras for small and medium-sized motorcycles tend to have a higher camera cost relative to the vehicle cost than ADAS cameras for large motorcycles, so it is important to increase the number of functions that can be realized with the camera and increase added value.

[0005] In the known technology described in Patent Document 1, a camera is attached in a position where the steering state of the motorcycle driver can be seen, and the driver's steering state is grasped by combining image data captured by the camera with data from an acceleration detection means.

[0006] Japanese Patent Application Laid-Open No. 2008-225977

[0007] However, the known technology described in Patent Document 1 merely grasps the steering status of a motorcycle driver using image data captured by a camera in order to investigate the cause of an accident, and does not estimate the vehicle's running state using image data captured of specific parts such as the wheels or handlebars. Therefore, it is difficult to increase the number of vehicle control functions that can be realized by a camera.

[0008] In view of the above-mentioned problems, an object of the present invention is to provide a vehicle control device that can increase the number of functions that can be realized by a camera in relation to vehicle control while maintaining the cost of the camera.

[0009] In order to solve the above problem, the present invention provides a vehicle control device that uses an imaging device that can be attached to a two-wheeled vehicle, wherein the imaging device can be attached in a position where it can image the wheels and handlebars of the two-wheeled vehicle, and the vehicle control device has an image acquisition unit that acquires images captured by the imaging device, and a vehicle driving state estimation unit that processes the images acquired by the image acquisition unit and estimates the driving state of the two-wheeled vehicle based on at least one of the wheels and the handlebars in the processed image.

[0010] According to the present invention, it is possible to increase the functions that can be realized by the camera in terms of vehicle control while maintaining the cost of the camera.

[0011] FIG. 1 is a functional block diagram of a vehicle control device according to an embodiment of the present invention. FIG. 2 is a side view showing an example of a mounting configuration of an imaging device on a motorcycle according to an embodiment of the present invention (the optical axis of the imaging device is directed substantially horizontally). FIG. 3 is a side view showing another example of a mounting configuration of an imaging device on a motorcycle according to an embodiment of the present invention (the optical axis of the imaging device is directed downward relative to the horizontal). FIG. 4 is an explanatory diagram of a vehicle speed estimation method according to an embodiment of the present invention. FIG. 5 is an explanatory diagram of a turning angle estimation method according to an embodiment of the present invention. FIG. 6 is an explanatory diagram of an example of a bank angle estimation method (parallax information) according to an embodiment of the present invention. FIG. 7 is an explanatory diagram of another example of a bank angle estimation method (trigonometry) according to an embodiment of the present invention. FIG. 8 is a processing flowchart of a vehicle control device according to an embodiment of the present invention.

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0013] In this embodiment, an imaging device (camera) is installed at a position and angle that deliberately captures the wheels and handlebars of a motorcycle, and the vehicle's driving state is estimated using information on the state of the wheels and handlebars in the image. By estimating the vehicle's driving state by overlaying this on a conventional object detection function, further driving assistance becomes possible.

[0014] 1 is a functional block diagram showing an example of the overall configuration of a vehicle control device 100 according to this embodiment. As shown in FIG. 1, the vehicle control device 100 is connected to an imaging device 110 and an IMU (Inertial Measurement Unit) sensor 120.

[0015] The imaging device 110 may be, for example, a stereo camera including multiple cameras, a monocular camera, or the like. The imaging device 110 can be attached to a motorcycle and captures images of the motorcycle's surroundings by recognizing the surroundings. In this embodiment, the imaging device 110 can be attached to a position that allows it to capture images of the front of the motorcycle and the wheels (front wheels) and handlebars of the motorcycle. The imaging device 110 can capture images of the wheels and handlebars by lowering it rearward to a position that allows it to capture images of the wheels and handlebars. As shown in FIG. 2A , the imaging device 110 may be attached to the motorcycle V with an inclination such that the optical axis 111 of the imaging device 110 is substantially horizontal (facing forward), or as shown in FIG. 2B , the imaging device 110 may be attached to the motorcycle V with an inclination such that the optical axis 111 of the imaging device 110 is downward relative to the horizontal (facing diagonally downward forward). This configuration allows the imaging device 110 to add information about the wheels and handlebars to the image data in addition to conventional road information, enabling the estimation of the vehicle's driving state, as described below. In particular, by tilting the optical axis 111 of the imaging device 110 downward relative to the horizontal direction (FIG. 2B), it is possible to capture an image of part of the wheels (front wheels) while ensuring road information for the forward range.

[0016] The IMU sensor 120 can be attached to any position on the two-wheeled vehicle, and can measure acceleration and angular velocity as sensor information for estimating the vehicle's running state (posture).

[0017] Image data captured by the imaging device 110 and sensor information data measured by the IMU sensor 120 are transmitted to an image processing unit 104 of the vehicle driving state estimation unit 103 (to be described later) and used for image processing.

[0018] The vehicle control device 100 includes an image acquisition unit 101, a sensor information acquisition unit 102, a vehicle driving state estimation unit 103, and a driving control unit 106. The vehicle driving state estimation unit 103 includes an image processing unit 104 and a vehicle driving state integration unit 105. The image processing unit 104 includes a vehicle speed estimation unit 201, a turning angle estimation unit 202, a bank angle estimation unit 203, and a driving situation estimation unit 204. Specifically, the vehicle control device 100 is a computer including hardware such as a calculation unit such as a CPU, a storage device such as a semiconductor memory, and a communication device. The calculation unit executes a predetermined program to realize each functional unit such as the image acquisition unit 101. Below, each unit will be described in detail, omitting such well-known techniques as appropriate.

[0019] The image acquisition unit 101 acquires image data captured by the imaging device 110 and transmits the image data to the image processing unit 104 of the vehicle traveling state estimation unit 103 .

[0020] The sensor information acquisition unit 102 acquires sensor information data acquired by the IMU sensor 120 and transmits it to the image processing unit 104 of the vehicle traveling state estimation unit 103 .

[0021] The vehicle driving state estimation unit 103 processes the image data acquired by the image acquisition unit 101, and estimates the driving state of the two-wheeled vehicle based on at least one of the wheels and the handlebars in the processed image data (processed image).

[0022] The vehicle speed estimation unit 201, turning angle estimation unit 202, and bank angle estimation unit 203 that constitute the image processing unit 104 of the vehicle driving state estimation unit 103 each process the image data transmitted from the image acquisition unit 101, and calculate the vehicle speed, turning angle, and bank angle of the vehicle based on at least one of the wheels (front wheels) and the steering wheel in the processed image.

[0023] In detail, the vehicle speed estimation unit 201 estimates the vehicle speed of the two-wheeled vehicle V from the rotation speed of the wheel (front wheel) in the processed image.

[0024] As shown in FIG. 3 , the vehicle speed estimation unit 201 uses arbitrary reference points (e.g., painted dots) and reference lines (e.g., slits) on the wheel, captures images of them with an imaging device, and calculates the distance they move per unit time to derive the vehicle speed. When estimating using reference points, since only the upper part of the wheel can be imaged, multiple reference points are provided on the wheel to prevent the wheel's position from being lost when the image is captured. When estimating using reference lines, a single diagonal line is drawn around the outer circumference of the wheel, and the position of the reference line is used to determine the vehicle speed. Furthermore, since the trajectory of the reference point cannot be tracked unless the wheel is imaged at least once per full rotation, the imaging timing of the imaging device (camera frame rate) must be considered in addition to the vehicle speed and the outer shape of the wheel. Note that if the wheel is worn and the painted dots or slits cannot be properly recognized, estimation is not performed and the rider is notified of this.

[0025] In addition, the turning angle estimation unit 202 estimates the turning angle of the two-wheeled vehicle V from at least one of the movement distance of the reference point of the wheel (front wheel) in the processed image, the angle of the wheel (front wheel) in the processed image, and the steering angle of the handlebars in the processed image.

[0026] As shown in FIG. 4 , the turning angle estimation unit 202 can estimate the turning angle of the vehicle by capturing images of the steering wheel angle and wheel inclination when the vehicle turns right or left using an imaging device. Generally, when turning at high speeds, the steering wheel angle tends to be barely turned (banking is more dominant), but when turning at low speeds, the steering wheel angle tends to be turned. Therefore, this estimation is mainly applied to low-speed driving. Note that this estimation assumes that the imaging device attached to the vehicle does not move in conjunction with the steering wheel or wheels. The turning angle can also be estimated from the distance traveled by an arbitrary reference point on the wheel. In other words, as shown in the lower diagram of FIG. 4 , the reference point set on the wheel moves within the image when turning right or left, making it possible to estimate the degree to which the steering wheel was tilted.

[0027] In addition, the bank angle estimation unit 203 estimates the bank angle of the wheels (front wheels) from the angle of incidence of the wheels (front wheels) with respect to the road plane in the processed image, or from parallax information between the wheels (front wheels) and the surrounding roads in the processed image.

[0028] As shown in Figure 5, the bank angle estimation unit 203 calculates the distance relationship between the surrounding road and the wheels based on parallax information when the vehicle banks, and estimates the bank angle of the wheels. When the vehicle banks, parallax occurs between the road surface on the tilted side and the road surface on the opposite side, so the angle of the image capture device relative to the road surface is calculated from the distance relationship on both sides (when the vehicle banks, the image capture device also banks). The relative angle between the image capture device and the wheels is calculated from the parallax information between the image capture device and the wheels. If this information can be calculated, the angle of the wheels relative to the road surface can be estimated.

[0029] Furthermore, as shown in FIG. 6, the bank angle estimation unit 203 can also estimate the bank angle of the wheel based on the angle of incidence of the wheel with respect to the road plane using the cosine law, which uses the lengths of the three sides (vertical and horizontal lines) centered on the outline of the wheel.

[0030] Furthermore, when the inertial information of the IMU sensor 120 is also used in combination with the image data, the vehicle speed estimation unit 201, the turning angle estimation unit 202, and the bank angle estimation unit 203 that constitute the image processing unit 104 of the vehicle traveling state estimation unit 103 each supplement information to the IMU sensor 120. In other words, at least one of the vehicle speed estimated by the vehicle speed estimation unit 201, the turning angle estimated by the turning angle estimation unit 202, and the bank angle estimated by the bank angle estimation unit 203 that constitute this image processing unit 104 is used as supplement information that supplements the information of the IMU sensor 120 provided on the motorcycle V (sensor information transmitted from the sensor information acquisition unit 102).

[0031] Generally, motorcycles are often equipped with an IMU sensor for vehicle attitude control. Using image data from an imaging device in conjunction with an IMU sensor allows for more accurate estimation. However, since IMU sensors also have errors (typically sensitivity errors and offset errors) and the parameters they can detect are angular velocity and acceleration, separate calculations are required to estimate vehicle attitude, so naturally errors will occur even when using the IMU sensor alone. If information from the imaging device can also be added, it will be possible to complement each other's missing information.

[0032] The driving situation estimation unit 204, which constitutes the image processing unit 104 of the vehicle driving state estimation unit 103, processes the image data transmitted from the image acquisition unit 101 and estimates the driving situation of the driver (rider) of the two-wheeled vehicle V based on at least one of the wheels (front wheels) and the handlebars in the processed image.

[0033] For example, when capturing an image of the handlebars, or capturing an image of the handlebars reflected by a mirror, etc., it is possible to grasp the rider's steering condition (hand grip condition). With this configuration, the rider's hand grip condition can be confirmed before executing the driving control described below.

[0034] In this embodiment, the rider's driving condition is grasped (estimated) from image data, but if the vehicle is equipped with a handlebar touch sensor, the rider's driving condition may also be grasped (estimated) from the handlebar touch sensor, for example.

[0035] The vehicle driving state integration unit 105 estimates the driving state of the vehicle (vehicle speed, turning angle, bank angle, etc.) based on the information calculated by the image processing unit 104, specifically by integrating the estimated information transmitted from the vehicle speed estimation unit 201, turning angle estimation unit 202, bank angle estimation unit 203, and driving situation estimation unit 204 that constitute the image processing unit 104 (vehicle driving state estimation unit 103). The estimated information from the vehicle driving state estimation unit 103 is transmitted to a driving control unit 106, which will be described later, and is used for driving control.

[0036] After the vehicle driving state estimation unit 103 estimates the vehicle driving state, the driving control unit 106 performs driving assistance (brake control, steering control, etc.) based on the estimated information transmitted from the vehicle driving state estimation unit 103.

[0037] FIG. 7 is a flowchart showing an example of the processing flow of the vehicle control device 100 according to this embodiment.

[0038] First, in step S301 , the image acquisition unit 101 acquires image data from the imaging device 110 .

[0039] In step S302 , the sensor information acquisition unit 102 acquires sensor information data (acceleration and angular velocity) from the IMU sensor 120 .

[0040] In step S303, the image processing unit 104 processes the image data acquired in step S301. More specifically, in step S303, the vehicle speed estimation unit 201, the turning angle estimation unit 202, and the bank angle estimation unit 203 constituting the image processing unit 104 each process the image data acquired in step S301 and calculate the vehicle speed, turning angle, and bank angle of the vehicle based on at least one of the wheels (front wheels) and the steering wheel in the processed image.

[0041] In addition, when the inertial information of the IMU sensor 120 is also used in conjunction with the image data, the vehicle speed estimation unit 201, the turning angle estimation unit 202, and the bank angle estimation unit 203 constituting the image processing unit 104 each use at least one of the vehicle speed estimated by the vehicle speed estimation unit 201, the turning angle estimated by the turning angle estimation unit 202, and the bank angle estimated by the bank angle estimation unit 203 constituting this image processing unit 104 as complementary information to complement the information of the IMU sensor 120 provided on the two-wheeled vehicle V (sensor information acquired in step S302).

[0042] Also, in step S303, the driving situation estimation unit 204 that constitutes the image processing unit 104 processes the image data acquired in step S301 and estimates the driving situation of the driver (rider) of the two-wheeled vehicle V based on at least one of the wheels (front wheels) and the handlebars in the processed image.

[0043] In step S304, the vehicle traveling state integration unit 105 integrates the information (estimated information) calculated in step S303. As a result, in step S305, the vehicle traveling state estimation unit 103 estimates the traveling state of the vehicle (vehicle speed, turning angle, bank angle, etc.).

[0044] In step S306, the driving control unit 106 executes driving assistance (brake control, steering control, etc.) based on the information on the vehicle's running state estimated in step S305.

[0045] As described above, the vehicle control device 100 of this embodiment uses an imaging device 110 that can be attached to a two-wheeled vehicle, and the imaging device 110 can be attached to a position where it can image the wheels and handlebars of the two-wheeled vehicle. The vehicle control device 100 has an image acquisition unit 101 that acquires images captured by the imaging device 110, and a vehicle driving state estimation unit 103 that processes the images acquired by the image acquisition unit 101 and estimates the driving state of the two-wheeled vehicle based on at least one of the wheels and the handlebars in the processed image.

[0046] This allows us to capture information about the wheels and handlebars of motorcycles that cannot be obtained with conventional installation methods, and this image data can be used to estimate the vehicle's driving conditions.

[0047] In the vehicle control device 100 of this embodiment, the imaging device 110 can be attached at an angle such that the optical axis 111 of the imaging device 110 faces downward relative to the horizontal direction.

[0048] This makes it possible to reliably capture an image of a portion of the wheel (front wheel) of the motorcycle (FIG. 2B).

[0049] In the vehicle control device 100 of this embodiment, the vehicle traveling state estimating unit 103 includes a vehicle speed estimating unit 201 that estimates the vehicle speed of the two-wheeled vehicle from the number of rotations of the wheel in the processed image.

[0050] This allows us to use the painted dots or slits on the wheels to calculate the vehicle speed in terms of the distance traveled / time when the wheels rotate (Figure 3).

[0051] Furthermore, in the vehicle control device 100 of this embodiment, the vehicle traveling state estimation unit 103 has a turning angle estimation unit 202 that estimates the turning angle of the two-wheeled vehicle from at least one of the movement distance of the reference point of the wheel in the processed image, the angle of the wheel in the processed image, and the steering angle of the handlebars in the processed image.

[0052] This allows the vehicle turning angle to be estimated (Figure 4).

[0053] Furthermore, in the vehicle control device 100 of this embodiment, the vehicle traveling state estimation unit 103 has a bank angle estimation unit 203 that estimates the bank angle of the wheel from the incident angle of the wheel with respect to the road plane in the processed image or from disparity information between the wheel and the surrounding road in the processed image.

[0054] This allows the bank angle of the wheels to be estimated (FIGS. 5 and 6).

[0055] The vehicle control device 100 of this embodiment also has a driving situation estimation unit 204 that estimates the driving situation (hand grip situation) of the driver riding the two-wheeled vehicle based on at least one of the wheels and the handlebars in the processed image.

[0056] This makes it possible to determine whether to control the driving after understanding the rider's driving situation (hand grip situation).

[0057] In addition, the vehicle control device 100 of this embodiment uses the running state of the two-wheeled vehicle estimated from at least one of the wheels and the handlebars in the processed image (at least one of the vehicle speed estimated by the vehicle speed estimation unit 201, the turning angle estimated by the turning angle estimation unit 202, and the bank angle estimated by the bank angle estimation unit 203) as complementary information that complements the information of the IMU sensor 120 provided on the two-wheeled vehicle.

[0058] This allows the estimation accuracy to be further improved compared to estimation using only the imaging device or the IMU sensor.

[0059] The vehicle control device 100 of this embodiment also includes a driving control unit 106 that controls the driving of the two-wheeled vehicle based on the information from the IMU sensor 120 and the complementary information.

[0060] In other words, by capturing images of the handlebars and wheels of a two-wheeled vehicle, it is possible to estimate the vehicle speed, turning angle, and bank angle, thereby estimating the vehicle's running state and performing driving control.

[0061] This allows the system to not only provide a general object detection function, but also estimate the vehicle's driving state from image data from the imaging device, and then perform driving control.

[0062] That is, in this embodiment, a camera is intentionally attached in a position where it can capture images of the wheels and handlebars of the motorcycle, and the traveling state of the vehicle is estimated by making full use of image data of the wheels and handlebars of the motorcycle.

[0063] By changing the camera's mounting position or tilting its optical axis, it is possible to capture images of the motorcycle's wheels and handlebars. From the image data captured by the camera, it is possible to estimate the vehicle's speed, turning angle, and bank angle based on the rotation and tilt of the wheels. Furthermore, by using it in conjunction with an IMU sensor, it is possible to supplement the original amount of information.

[0064] This allows the vehicle's driving state to be estimated using camera image data, making it possible to utilize this technology in conjunction with existing functions. For example, the vehicle's driving state can be estimated and driving assistance (brake control, steering control, etc.) becomes possible. In addition, by using this information as a supplement to the IMU (inertial measurement unit) sensor, the accuracy of estimating the vehicle's driving state can be further improved. In addition, sensors with functions that can be replaced by cameras (vehicle speed sensor, steering angle sensor, etc.) are no longer necessary, which contributes to cost reduction.

[0065] According to the vehicle control device 100 of this embodiment described above, it is possible to increase the number of functions that can be realized by the camera in relation to vehicle control while maintaining the cost of the camera.

[0066] It should be noted that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0067] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0068] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0069] 100 Vehicle control device, 101 Image acquisition unit, 102 Sensor information acquisition unit, 103 Vehicle driving state estimation unit, 104 Image processing unit, 105 Vehicle driving state integration unit, 106 Driving control unit, 110 Imaging device, 111 Optical axis, 120 IMU sensor, 201 Vehicle speed estimation unit, 202 Turning angle estimation unit, 203 Bank angle estimation unit, 204 Driving situation estimation unit, V Two-wheeled vehicle (vehicle)

Claims

1. In a vehicle control device using an imaging device attachable to a two-wheeled vehicle, the imaging device is attachable at a position capable of imaging a wheel and a handle in the two-wheeled vehicle, and the vehicle control device includes an image acquisition unit that acquires an image captured by the imaging device, and a vehicle running state estimation unit that processes the image acquired by the image acquisition unit and estimates the running state of the two-wheeled vehicle based on at least one of the wheel and the handle in the processed image.

2. The vehicle control device according to claim 1, wherein the imaging device is attachable at an inclination such that the optical axis of the imaging device faces downward with respect to the horizontal direction.

3. The vehicle control device according to claim 1, wherein the vehicle running state estimation unit includes a vehicle speed estimation unit that estimates the vehicle speed of the two-wheeled vehicle from the rotational speed of the wheel in the processed image.

4. The vehicle control device according to claim 1, wherein the vehicle running state estimation unit includes a turning angle estimation unit that estimates the turning angle of the two-wheeled vehicle from at least one of the moving distance of a reference point of the wheel in the processed image, the angle of the wheel in the processed image, and the steering angle of the handle in the processed image.

5. The vehicle control device according to claim 1, wherein the vehicle running state estimation unit includes a bank angle estimation unit that estimates the bank angle of the wheel from the incident angle of the wheel with respect to the road plane in the processed image or the parallax information between the surrounding road and the wheel in the processed image.

6. The vehicle control device according to claim 1, wherein the vehicle control device includes a driving situation estimation unit that estimates the driving situation of a driver riding on the two-wheeled vehicle based on at least one of the wheel and the handle in the processed image.

7. The vehicle control device according to claim 1, wherein the vehicle control device uses the running state of the two-wheeled vehicle estimated based on at least one of the wheel and the handle in the processed image as complementary information for complementing information of an IMU sensor provided in the two-wheeled vehicle.

8. The vehicle control device according to claim 7, wherein the vehicle control device includes a driving control unit that performs driving control of the two-wheeled vehicle based on the information of the IMU sensor and the complementary information.

Images