Image processing device for leaning vehicle

Abstract

According to the present invention, in a case in which five automobiles (61 to 65) are traveling side by side in one row in a right-left direction ahead of a leaning vehicle (1) on a five-lane (51 to 55) road, and a distance in a front-rear direction between the five automobiles (61 to 65) and the leaning vehicle (1) is a distance necessary for capturing an image so as to complete image processing at a time point of TTC of three seconds when assuming that the time from image-capturing to completion of image processing is 0.5 seconds and an object is approaching the leaning vehicle 1 at a relative speed of 15 km / h, an image-capturing device (11) includes a monocular camera the lateral-direction angle of view of which is configured such that, when the leaning vehicle (1) is traveling in a right side area (50R) of a center lane (53), a left end (61BL) of a rear face of the automobile (61) is not included in the image, but a rear face (63B) and a right side face (63R) of the automobile (63) are included in the image, and an arrival time estimation device (12) estimates arrival time until the automobile (63) reaches the leaning vehicle (1) on the basis of the image captured by the image-capturing device (11).

Description

Image processing device for lean vehicles

[0001] The present invention relates to an image processing device for a lean vehicle that is mounted on a lean vehicle.

[0002] Conventionally, vehicle periphery monitoring devices are known that include an arrival time estimation device mounted on a vehicle and that estimates the time it takes for the vehicle to reach an object, such as another vehicle or a pedestrian, located ahead of the vehicle. An example of a vehicle periphery monitoring device is disclosed in Patent Document 1. The vehicle periphery monitoring device of Patent Document 1 includes an arrival time estimation device that calculates a time change in the width of an object in an image, i.e., a change rate, from multiple images captured by an imaging device at predetermined time intervals and estimates the arrival time using the calculated change rate. Patent Document 2 proposes an arrival time estimation device that extracts feature points from images of an object captured in a time series by an imaging device mounted on the vehicle and estimates the time it takes for the vehicle to reach the object based on the extracted feature points, and a driving assistance control device that uses the estimated arrival time to control the vehicle to assist the driver in operating the vehicle. Patent Document 2 also proposes a lean vehicle equipped with an imaging device, an arrival time estimation device, and a driving assistance control device. The lean vehicle is, for example, a motorcycle. The imaging device generally includes an optical device, such as a lens, and an imaging element, such as a complementary metal oxide semiconductor (CMOS) sensor. The arrival time estimation device generally includes a processor, such as a field-programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), or a system on chip (SoC) for processing images, and a memory, such as a random access memory (RAM). Note that an SoC has a configuration in which the processor and memory are integrated. Patent Document 2 describes that the imaging device may be a monocular camera or a stereo camera.

[0003] Furthermore, the time it takes for the vehicle to reach an object described in Patent Documents 1 and 2 corresponds to the so-called Time to Collision (TTC). Non-Patent Document 1 describes that a TTC threshold value of 3 seconds is appropriate for evaluating a potential collision.

[0004] The width of lanes and the width of vehicles are regulated by national or regional administrative bodies. In Japan, lane widths are regulated to 2.75 to 4 meters depending on the road classification under Article 5, Paragraphs 4 and 5 of the Road Structure Ordinance (see Non-Patent Document 2). Generally, lane widths are 3.5 to 4 meters. Furthermore, in Japan, Article 59 of the Road Transport Vehicle Safety Standards stipulates that the width of motorcycles must be 1.3 meters or less, and Article 2 of the Road Transport Vehicle Safety Standards stipulates that the width of automobiles must be 2.5 meters or less (see Non-Patent Document 3). Generally, the width of lean-to vehicles is less than 1 meter, and the width of automobiles is 2 to 2.5 meters.

[0005] Japanese Patent Application Laid-Open No. 2007-213561 Japanese Patent Application Laid-Open No. 2021-111246

[0006] Oumaima Barhoumi, Mohamed H. Zaki, SOFIENE TAHAR, "A Formal Approach to Road Safety Assessment Using Traffic Conflict Techniques," "VI. SENSITIVITY STUDY," [online], last updated April 26, 2024, IEEE Open Journal of Vehicular Technology, [Retrieved June 4, 2024], Internet <URL: https: / / ieeexplore.ieee.org / stamp / stamp.jsp?arnumber=10496854> Road Structure Ordinance, [Retrieved June 4, 2024], Internet <URL: https: / / elaws.e-gov.go.jp / document?lawid=345CO0000000320> Road Transport Vehicle Safety Standards, [Retrieved June 4, 2024], Internet <URL: https: / / elaws.e-gov.go.jp / document?lawid=326M50000800067>

[0007] The calculation of the rate of change in Patent Document 1 requires the determination of the change over time in the width of an object in a captured image, which requires an imaging device that captures high-resolution images. An imaging device that captures high-resolution images tends to be large because it employs a larger imaging element. Additionally, the calculation of the rate of change requires the determination of the change over time in the width of an object in a captured image, which requires an arrival time estimation device that processes high-resolution images to estimate the arrival time. An arrival time estimation device that processes high-resolution images to estimate the arrival time, for example, requires a heat dissipation structure for the processor due to the high load on the processor that processes the images, which tends to result in an increase in size.

[0008] However, when an image processing device mounted on an automobile as in Patent Document 2 is mounted on a lean vehicle, the following problems arise.

[0009] Since lean vehicles have slimmer (narrower) bodies than automobiles, an image processing device for lean vehicles that can be easily installed in lean vehicles is desired.

[0010] Furthermore, the width of a lean-to vehicle is much smaller than that of an automobile. In other words, the body of a lean-to vehicle is slimmer (narrower) than that of an automobile. Furthermore, the width of a lean-to vehicle is also much smaller than the width of a lane. Therefore, when a lean-to vehicle travels in a single lane, it may travel in either the right or left side area of ​​the lane. Therefore, compared to an automobile, a lean-to vehicle has a much greater degree of freedom in its lateral travel position within a single lane. Therefore, there is a demand for an image processing device for a lean-to vehicle that includes an imaging device and an arrival time estimation device that can handle usage scenarios unique to lean-to vehicles, such as traveling in the right or left side area of ​​a lane while traveling in a single lane. Note that in this specification, when a single lane is divided into thirds horizontally, the area including the right edge of the lane is defined as the right side area. When a single lane is divided into thirds horizontally, the area including the left edge of the lane is defined as the left side area.

[0011] The object of the present invention is to provide an image processing device for a lean vehicle that can be adapted to situations in which a lean vehicle, which has a slimmer body than an automobile, travels in the right side area and left side area of ​​a lane while traveling in the same lane as an automobile, and that is highly mountable on a lean vehicle, which has a slimmer body than an automobile.

[0012] (1) An image processing device for a lean vehicle according to one embodiment of the present invention has the following configuration: The image processing device for a lean vehicle is mounted on the lean vehicle and includes an imaging device that captures images, and an arrival time estimation device that processes the images captured by the imaging device to estimate an arrival time for an object to reach the lean vehicle, and when a single lane is divided into thirds in the left-right direction, an area including the right edge of the single lane is defined as a right side area, an area including the left edge of the single lane is defined as a left side area, and an area between the right side area and the left side area is defined as a center area, and each lane of a five-lane road is defined as a left side area. When the lanes are defined as the leftmost lane, center left lane, center lane, center right lane, and rightmost lane, respectively from top to bottom to the right, the lean vehicle travels in the right side area or left side area of ​​the center lane of the five-lane road, and five automobiles travel in a line in the left-right direction ahead of the lean vehicle, and the distance between the five automobiles and the lean vehicle in the longitudinal direction is TTC (Time to Target). When the distance (To Collision) is the distance required to capture an image so that the image processing is completed at 3 seconds, the imaging device and the arrival time estimation device are configured as follows, so that the imaging device and the arrival time estimation device are configured to estimate the arrival time of the vehicle traveling in the center lane to reach the lean vehicle based on the image including the left side or right side of the vehicle traveling in the center lane, both when the lean vehicle is traveling in the right side area of ​​the center lane of the five-lane road and when the lean vehicle is traveling in the left side area of ​​the center lane of the five-lane road.The imaging device includes a monocular camera having a horizontal angle of view configured so that, while the leaned vehicle is traveling in the right side area of ​​the center lane of the five-lane road, the left edge of the rear of the vehicle traveling in the leftmost lane is not included in the image, but at least the rear of the vehicle traveling in the center lane and the right side of the vehicle traveling in the center lane are included in the image, and, while the leaned vehicle is traveling in the left side area of ​​the center lane of the five-lane road, the right edge of the rear of the vehicle traveling in the rightmost lane is not included in the image, but at least the rear of the vehicle traveling in the center lane and the left side of the vehicle traveling in the center lane are included in the image. The arrival time estimation device is configured to estimate the arrival time of the vehicle traveling in the center lane to reach the lean vehicle based on the image that includes at least the rear of the vehicle traveling in the center lane and the right side of the vehicle traveling in the center lane while the lean vehicle is traveling in the right side area of ​​the center lane of the five-lane road, and to estimate the arrival time of the vehicle traveling in the center lane to reach the lean vehicle based on the image that includes at least the rear of the vehicle traveling in the center lane and the left side of the vehicle traveling in the center lane while the lean vehicle is traveling in the left side area of ​​the center lane of the five-lane road.

[0013] The width of a lean-to vehicle is very small compared to the width of an automobile. In other words, the body of a lean-to vehicle is slimmer than that of an automobile. The width of a lean-to vehicle is very small compared to the width of a lane. Therefore, when a lean-to vehicle and an automobile are traveling in the same lane, the lean-to vehicle may travel in the right side area or the left side area of ​​the lane. When a lean-to vehicle is traveling in the right side area of ​​the center lane of a five-lane road, the rear and right side of the automobile traveling in the center lane may be located in front of the lean-to vehicle. When a lean-to vehicle is traveling in the left side area of ​​the center lane of a five-lane road, the rear and left side of the automobile traveling in the center lane may be located in front of the lean-to vehicle. The imaging device includes a monocular camera. The monocular camera of the imaging device has a horizontal angle of view configured so that when the lean-to vehicle is traveling in the right side area of ​​the center lane of a five-lane road, at least the rear and right side of the automobile traveling in the center lane are included in the image. The monocular camera of the imaging device is configured with a horizontal angle of view such that, when a leaned vehicle is traveling in the left side area of ​​the center lane of a five-lane road, at least the rear and left side of the vehicle traveling in the center lane are included in the image. The monocular camera of the imaging device is configured with a horizontal angle of view such that, when a leaned vehicle is traveling in the right side area of ​​the center lane of a five-lane road, the left edge of the rear of the vehicle traveling in the leftmost lane is not included in the image. The monocular camera of the imaging device is configured with a horizontal angle of view such that, when a leaned vehicle is traveling in the left side area of ​​the center lane of a five-lane road, the right edge of the rear of the vehicle traveling in the rightmost lane is not included in the image. When a leaned vehicle is traveling in the right side area of ​​the center lane of a five-lane road, the vehicle traveling in the leftmost lane is located more than two car lengths to the left of the leaned vehicle. When a leaned vehicle is traveling in the left side area of ​​the center lane of a five-lane road, the vehicle traveling in the rightmost lane is located more than two car lengths to the right of the leaned vehicle. The width of a lean vehicle is very small compared to the width of an automobile, in other words, the body of a lean vehicle is slimmer than that of an automobile.Therefore, compared with the two vehicles positioned in front of the lean vehicle, vehicles positioned more than two vehicle lengths away from the lean vehicle in the left-right direction have less impact on the driving of the lean vehicle, which has a slimmer body than the vehicles. With the above configuration, whether the lean vehicle is traveling in the right side area or the left side area, imaging of areas positioned more than two vehicle lengths away from the lean vehicle in the left-right direction can be limited. As a result, the resolution of the images of the two vehicles positioned in front of the lean vehicle can be increased while reducing the number of pixels in the entire image. In other words, the resolution of the images of the two vehicles positioned in front of the lean vehicle can be increased without increasing the size of the imaging device. Therefore, the imaging device can be made smaller by miniaturizing the imaging element while ensuring high resolution of the images of the two vehicles positioned in front of the lean vehicle. The monocular camera of the imaging device has a horizontal angle of view configured so that, when the lean vehicle is traveling in the right side area of ​​the center lane of a five-lane road, at least the rear and right side of the vehicle traveling in the center lane are included in the image. The monocular camera of the imaging device has a horizontal angle of view configured so that, while a lean-to vehicle is traveling in the left side area of ​​a center lane of a five-lane road, at least the rear of the vehicle traveling in the center lane and the left side of the vehicle traveling in the center lane are included in the image.The arrival time estimation device estimates the arrival time for the vehicle traveling in the center lane to reach the lean-to vehicle based on an image that includes at least the rear of the vehicle traveling in the center lane and the right side of the vehicle traveling in the center lane while the lean-to vehicle is traveling in the right side area of ​​the center lane.The arrival time estimation device estimates the arrival time for the vehicle traveling in the center lane to reach the lean-to vehicle based on an image that includes at least the rear of the vehicle traveling in the center lane and the left side of the vehicle traveling in the center lane while the lean-to vehicle is traveling in the left side area of ​​the center lane.The arrival time is estimated based on an image that includes at least the rear of a vehicle traveling in the center lane as well as the left or right side of the vehicle traveling in the center lane. This allows for estimation of arrival time in scenarios specific to lean-to-vehicle use, such as when the vehicle is traveling in the right side area of ​​the lane and when the vehicle is traveling in the left side area. Since the image includes at least the rear of a vehicle traveling in the center lane as well as the left or right side of the vehicle traveling in the center lane, the amount of information about the vehicle traveling in the center lane is large. This increases the design flexibility of the arrival time estimation algorithm, thereby enabling the hardware of the arrival time estimation device to be miniaturized. This improves the mountability of lean-to-vehicle image processing devices on lean-to-vehicles, which have slimmer bodies than cars. For example, in an algorithm that estimates arrival time based on the size (horizontal, vertical, and diagonal width) of the rear of a vehicle in an image, using an image that includes the rear and side of the vehicle makes it easy to recognize the size of the rear of the vehicle. For example, in an algorithm that estimates arrival time based on the distance between multiple feature points of a vehicle in an image, using an image that includes the rear and side of the vehicle increases the freedom of feature point selection. For example, in an algorithm that estimates arrival time using a deep learning algorithm to estimate the rate of change in the size of a vehicle in an image from image information, using an image that includes the rear and side of the vehicle increases the amount of information, thereby improving estimation accuracy. In this way, the arrival time is estimated based on an image that includes at least the rear of a vehicle traveling in the center lane, as well as the left or right side of the vehicle traveling in the center lane, thereby increasing the design flexibility of the arrival time estimation algorithm. As a result, it is possible to provide an image processing device for a lean vehicle that is adaptable to scenes in which a lean vehicle, which has a slimmer body than an automobile, travels in the right or left side area of ​​a lane while traveling in the same lane as the automobile, and that is highly mountable on a lean vehicle, which has a slimmer body than an automobile.

[0014] (2) An image processing device for a lean vehicle according to one embodiment of the present invention may have the following configuration in addition to the configuration described in (1) above: The image processing device for a lean vehicle further includes an alarm device that outputs an alarm based on the arrival time of an object to reach the lean vehicle, estimated by the arrival time estimation device, and the alarm device is configured not to output the alarm when two automobiles are traveling straight ahead of the lean vehicle side by side with a gap between them in the lateral direction, and the size of the gap is too small for the width of the automobiles but large enough for the width of the lean vehicle, even if the lean vehicle travels straight towards and approaches the center between the two automobiles; and is configured to output the alarm when two automobiles are traveling straight ahead of the lean vehicle side by side with a gap between them in the lateral direction, and the size of the gap is too small for the width of the lean vehicle, and the lean vehicle travels straight towards and approaches the center between the two automobiles, and the alarm output corresponds to the width of the lean vehicle.

[0015] According to this configuration, the image processing device for the lean-mounted vehicle further includes an alarm device that outputs an alarm based on the arrival time of the object to the lean-mounted vehicle estimated by the arrival time estimation device. The alarm device is configured not to output an alarm when two vehicles are traveling straight ahead of the lean-mounted vehicle, spaced apart in the left-right direction, and the distance between them is too small for the width of the vehicles but large enough for the width of the lean-mounted vehicle. The alarm device is configured to output an alarm when the lean-mounted vehicle travels straight ahead toward the center between the two vehicles and approaches them, when two vehicles are traveling straight ahead of the lean-mounted vehicle, spaced apart in the left-right direction, and the distance between them is too small for the width of the lean-mounted vehicle. In this way, the alarm device outputs an alarm according to the width of the lean-mounted vehicle. For example, when a lean-mounted vehicle is traveling in the right side area of ​​the center lane of a five-lane road, two vehicles traveling in the center lane and the center-right lane will be located in front of the lean-mounted vehicle. For example, when a lean-to vehicle is traveling in the right side area of ​​the center lane, a portion of a vehicle traveling in the center lane is located on the left side area of ​​the center lane and a portion of a vehicle traveling in the center-right lane is located on the right side area of ​​the center-right lane, and a portion of a vehicle traveling in the center lane is located on the right side area of ​​the center-right lane. Alternatively, a warning may be issued depending on whether or not the vehicle traveling in the center lane is located on the right side area of ​​the center lane and a portion of a vehicle traveling in the center-right lane is located on the left side area of ​​the center-right lane. For example, when a lean-to vehicle is traveling in the left side area of ​​the center lane of a five-lane road, two vehicles traveling in the center lane and the center-left lane will be located in front of the lean-to vehicle.For example, when a lean-to vehicle is traveling in the left side area of ​​the center lane, a portion of the vehicle traveling in the center lane is located on the right side area of ​​the center lane and a portion of the vehicle traveling in the center-left lane is located on the left side area of ​​the center-left lane, and when a portion of the vehicle traveling in the center lane is located on the left side area of ​​the center-left lane and a portion of the vehicle traveling in the center-left lane is located on the right side area of ​​the center-left lane, the presence or absence of an alarm can be differentiated depending on the situation. With the above configuration, the image processing device for a lean-to vehicle can capture and recognize two vehicles located in front of the lean-to vehicle, whether the lean-to vehicle is traveling in the right side area or the left side area, and output an alarm according to the vehicle width of the lean-to vehicle. Therefore, when a lean-to vehicle, which has a slimmer body than a car, is traveling in the same lane as a car, the image capture, recognition, and alarm can be adapted to situations where the vehicle is traveling in the right side area or the left side area of ​​the lane. As a result, it is possible to provide an image processing device for a lean vehicle that is more adaptable to situations in which a lean vehicle, which has a slimmer body than an automobile, travels in the right side area and left side area of ​​a lane while traveling in the same lane as an automobile, and that is highly mountable in a lean vehicle, which has a slimmer body than an automobile. Note that the image processing device for a lean vehicle may be an image processing device dedicated to lean vehicles, in which the warning device is configured in advance according to the width of the lean vehicle. The image processing device for a lean vehicle may also be an image processing device for both lean vehicles and automobiles, in which the warning device is configured to adapt to the width of the lean vehicle or the width of an automobile by switching. The switching may be performed by a user operation or automatically based on a captured image.

[0016] (3) An image processing device for a lean vehicle according to one embodiment of the present invention may have the following configuration in addition to the configuration described in (1) or (2) above: The imaging device includes the monocular camera that captures, at regular time intervals, a plurality of images including the left or right side and rear of an automobile traveling ahead of the lean vehicle, and the arrival time estimation device includes an algorithm that acquires the plurality of consecutive images including the left or right side and rear of the automobile captured by the imaging device at the regular time intervals, and estimates the arrival time required for the automobile to reach the lean vehicle based on the plurality of consecutive images including the left or right side and rear of the automobile and the regular time intervals.

[0017] For example, if a leaning vehicle is traveling in the right side area of ​​the center lane of a five-lane road, the imaging device captures images including the rear and right sides of the vehicle traveling in the center lane. For example, if a leaning vehicle is traveling in the left side area of ​​the center lane of a five-lane road, the imaging device captures images including the rear and left sides of the vehicle traveling in the center lane. According to the above configuration, in such a scene, the imaging device captures images including the left or right side and rear of the vehicle traveling in the center lane at regular time intervals. Therefore, the rate of change in size of the vehicle in multiple images including the left or right side and rear of the vehicle is the rate of change in size of the vehicle in images taken at regular time intervals. Therefore, compared to when the imaging device captures images including the left or right side and rear of the vehicle at irregular time intervals, the rate of change in size of the vehicle in multiple images including the left or right side and rear of the vehicle can be easily used to estimate the arrival time. This reduces the processing load of the arrival time estimation device, and, for example, allows for a simple and compact heat dissipation structure of the arrival time estimation device, thereby making the arrival time estimation device more compact. Therefore, it is possible to improve the mountability of the image processing device for lean vehicles in lean vehicles, which have slimmer bodies than automobiles. Also, since the arrival time is estimated based on multiple images including the left or right side and rear of the automobile and a fixed time interval at which the multiple images are captured, it is possible to make the estimation of arrival time suitable for situations specific to lean vehicles traveling in the right side area of ​​the lane and in the left side area.

[0018] (4) An image processing device for a lean vehicle according to one embodiment of the present invention may have the following configuration in addition to any of the configurations (1) to (3) above: The imaging device includes the monocular camera that captures a plurality of color images including a left side or right side and a rear of an automobile traveling ahead of the lean vehicle, and the arrival time estimation device includes an algorithm that acquires the plurality of color images including the left side or right side and the rear of the automobile captured by the imaging device, and estimates the arrival time required for the automobile to reach the lean vehicle based on the acquired plurality of color images including the left side or right side and the rear of the automobile.

[0019] According to this configuration, for example, when a leaning vehicle is traveling in the right side area of ​​the center lane of a five-lane road, the imaging device captures a color image including the rear and right side of the vehicle traveling in the center lane. For example, when a leaning vehicle is traveling in the left side area of ​​the center lane of a five-lane road, the imaging device captures a color image including the rear and left side of the vehicle traveling in the center lane. The arrival time estimation device uses a color image including at least the rear of the vehicle traveling in the center lane as well as the left or right side of the vehicle traveling in the center lane to estimate the arrival time, thereby increasing the amount of information about the vehicle traveling in the center lane. This increases the design flexibility of the arrival time estimation algorithm. For example, in an algorithm that estimates the arrival time based on the size (horizontal, vertical, and diagonal width) of the rear of the vehicle in an image, using a color image including the rear and side of the vehicle makes it easy to recognize the size of the rear of the vehicle. For example, in an algorithm that estimates the arrival time based on the distance between multiple feature points of the vehicle in an image, using a color image including the rear and side of the vehicle increases the freedom of feature point selection. For example, in an algorithm that estimates arrival time using a deep learning algorithm to estimate the rate of change in the size of a vehicle in an image from image information, using a color image that includes the rear and side of the vehicle increases the amount of information, thereby improving estimation accuracy. In this way, in usage scenarios specific to lean vehicles traveling in the right side area of ​​a lane and in the left side area of ​​a lane, arrival time is estimated based on a color image that includes at least the rear of the vehicle traveling in the center lane, as well as the left or right side of the vehicle traveling in the center lane. This increases the design flexibility of the arrival time estimation algorithm. This allows for the miniaturization of the hardware of the arrival time estimation device. Therefore, the installability of lean vehicle image processing devices in lean vehicles, which have slimmer bodies than automobiles, is further improved.

[0020] (5) The image processing device for a lean vehicle according to one embodiment of the present invention may have the following configuration in addition to any one of the configurations (1) to (4) above. The arrival time estimation device acquires a plurality of consecutive images including the left or right side and rear of the automobile captured by the imaging device when the automobile is traveling in front of the lean vehicle, and includes: (I) an algorithm that acquires the size of the rear of the automobile in the images based on the plurality of consecutive images including the left or right side and rear of the automobile, and estimates the arrival time until the automobile reaches the lean vehicle; (II) an algorithm that acquires the distance between a plurality of feature points on the left or right side and rear of the automobile in the images based on the plurality of consecutive images including the left or right side and rear of the automobile, and estimates the arrival time until the automobile reaches the lean vehicle; or (III) an algorithm that estimates the arrival time until the automobile reaches the lean vehicle based on the plurality of consecutive images including the left or right side and rear of the automobile, using a deep-learned algorithm to estimate the rate of change of the size of the automobile having the left or right side and rear in the images.

[0021] For example, if a leaning vehicle is traveling in the right side area of ​​the center lane of a five-lane road, the imaging device captures an image including the rear and right side of the vehicle traveling in the center lane. For example, if a leaning vehicle is traveling in the left side area of ​​the center lane of a five-lane road, the imaging device captures an image including the rear and left side of the vehicle traveling in the center lane. With the above configuration, in such a scene, the algorithm (I) for estimating the arrival time from the size (horizontal, vertical, and diagonal width) of the rear of the vehicle in the image can easily recognize the size of the rear of the vehicle by using an image including the rear and side of the vehicle traveling in the center lane. Furthermore, the algorithm (II) for estimating the arrival time from the distance between multiple feature points of the vehicle in the image can increase the flexibility of feature point selection by using an image including the rear and side of the vehicle traveling in the center lane. Furthermore, in the algorithm (III) described above, which estimates arrival time using a deep learning algorithm to estimate the rate of change in the size of a vehicle in an image from image information, using images including the rear and side of a vehicle traveling in the center lane increases the amount of information, thereby improving estimation accuracy. In this manner, in usage scenarios specific to lean vehicles traveling in the right side area of ​​a lane and in the left side area, the arrival time is estimated based on an image that includes at least the rear of the vehicle traveling in the center lane as well as the left or right side of the vehicle traveling in the center lane, thereby increasing the design flexibility of the arrival time estimation algorithm. Note that, since the image includes at least the rear of the vehicle traveling in the center lane as well as the left or right side of the vehicle traveling in the center lane, the amount of information about the vehicle traveling in the center lane is large. This further increases the design flexibility of the arrival time estimation algorithm, thereby enabling the hardware of the arrival time estimation device to be miniaturized. Therefore, the installability of lean vehicle image processing devices for lean vehicles, which have slimmer bodies than regular cars, is further improved.

[0022] In the present invention and its embodiments, a lean vehicle may be a vehicle having a body frame that leans left when turning left and leans right when turning right. In the present invention and its embodiments, a lean vehicle may be, for example, a motorcycle or a three-wheeled motor vehicle. Motorcycles include scooters, mopeds, and motorized bicycles. The width of a lean vehicle is, for example, less than 1 meter. On the other hand, the width of a car is, for example, 2 to 2.5 meters.

[0023] In the present invention and embodiments, the monocular camera has a single imaging element. An imaging device including a monocular camera has an optical device such as a lens and a single imaging element. An imaging device including a monocular camera may also have a single lens. The imaging element may be an image sensor such as a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. In the present invention and embodiments, the imaging device may be disposed in the center of the vehicle in the left-right direction. However, the location of the imaging device is not limited to this.

[0024] In the present invention and embodiments, the arrival time estimation device includes a processor and a memory. The memory includes, for example, a random access memory (RAM). The memory may also include a read-only memory (ROM). The processor includes any circuit capable of processing images, such as a field-programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), or a system on chip (SoC). An SoC has a configuration in which the processor and memory are integrated.

[0025] In the present invention and its embodiments, a lane refers to a driving lane. A lane may or may not have marks (e.g., white lines) indicating the edges of the lane. The width of a lane is, for example, 3.5 to 4 m. In the present invention and its embodiments, the right side area, left side area, and center area correspond to the three areas obtained by dividing a lane into thirds in the left-right direction. In the present invention and its embodiments, a five-lane road is a straight road.

[0026] In the present invention and embodiments, a lean vehicle traveling in the right side area of ​​a lane means that the lean vehicle travels so that the left-right center of the lean vehicle is located within the right side area. A lean vehicle traveling in the left side area of ​​a lane means that the lean vehicle travels so that the left-right center of the lean vehicle is located within the left side area. In the present invention, a lean vehicle traveling in the right side area (left side area) of a lane includes traveling so that the entire lean vehicle is located within the right side area (left side area) in a plan view. In the present invention, a lean vehicle traveling in the right side area (left side area) of a lane may also include traveling so that part of the lean vehicle is located outside the right side area (left side area).

[0027] In the present invention and embodiments, the configurations of the imaging device and the arrival time estimation device are defined using a situation in which the longitudinal distance between the five automobiles and the lean vehicle is the distance required to capture an image so that the TTC is 3 seconds, assuming that the time from image capture to completion of image processing is 0.5 seconds and that the object approaches the lean vehicle at a relative speed of 15 km / h. "The distance required to capture an image so that the TTC is 3 seconds, assuming that the time from image capture to completion of image processing is 0.5 seconds and that the object approaches the lean vehicle at a relative speed of 15 km / h" can be rephrased as follows: That is, this distance is the distance required to capture an image so that the TTC is 3 seconds, assuming that the lean vehicle image processing device is configured to perform image processing so that the time from image capture to completion of image processing is 0.5 seconds and that the object is approaching the lean vehicle at a relative speed of 15 km / h. The TTC is the time it takes for the object to reach the lean vehicle. Specifically, this distance is 14.58335 m (= 15 km / h × (3.0 s + 0.5 s) ÷ 3600 × 1000). "An object approaches the lean vehicle at a relative speed of 15 km / h" means that both the object and the lean vehicle are traveling, with the lean vehicle traveling at a speed 15 km / h faster than the object, or that the object is stationary and the lean vehicle is traveling at a speed of 15 km / h. The "time from capturing an image to completing image processing" is a hypothetical time and does not need to be the same as the time actually required for image processing by the lean vehicle image processing device. The "time from capturing an image to completing image processing" may be, for example, the time from when the imaging device captures an image to when the image is processed and the arrival time is estimated. Generally, it takes about 0.5 seconds from capturing an image to estimating the TTC. The "time from capturing an image to completing image processing" may be, for example, the time from when an image capturing device captures an image to when a command to output an alarm to an alarm device (described later) is output.

[0028] In the present invention and embodiments, "the rear of the vehicle is included in the image" in the configuration "when a lean-to vehicle is traveling in the right side area of ​​the center lane of a five-lane road, the horizontal angle of view is configured so that an image of at least the rear of the vehicle traveling in the center lane and the right side of the vehicle traveling in the center lane is included" means that the entire rear of the vehicle is included in the image. Furthermore, "the right side of the vehicle is included in the image" means that, if the right side of the vehicle is flat, the entire right side of the vehicle is included in the image. In other words, "the right side of the vehicle is included in the image" means that at least a portion of the right side of the vehicle is included in the image. In the present invention and embodiments, "included in the image" means appearing in the image. By interpreting these definitions in a similar manner, the meaning of "the left side of the vehicle is included in the image" in the configuration "when a lean-to vehicle is traveling in the left side area of ​​the center lane of a five-lane road, the horizontal angle of view is configured so that at least the rear of the vehicle traveling in the center lane and the left side of the vehicle traveling in the center lane are included in the image" can also be interpreted.

[0029] In the present invention and embodiments, a vehicle traveling ahead of a lean vehicle refers to a vehicle located ahead of the lean vehicle when the forward direction of the lean vehicle is defined as the forward direction. In the present invention and embodiments, a vehicle traveling ahead of a lean vehicle refers to a vehicle traveling directly or almost directly in front of the lean vehicle. A vehicle traveling ahead of a lean vehicle is not limited to a vehicle traveling ahead of the lean vehicle.

[0030] The image processing device for a lean vehicle of the present invention and embodiments may or may not include an alarm device that outputs an alarm based on the arrival time of an object to reach the lean vehicle, as estimated by the arrival time estimation device. In the present invention and embodiments, if the image processing device for a lean vehicle includes an alarm device, the use of the arrival time estimated by the arrival time estimation device is not limited to the alarm device. Regardless of whether the image processing device for a lean vehicle includes an alarm device, the arrival time estimated by the arrival time estimation device may be used, for example, to control the vehicle speed of the lean vehicle. More specifically, for example, the brake device of the lean vehicle may be activated based on the arrival time estimated by the arrival time estimation device without the driver's brake operation.

[0031] In the present invention and embodiments, the warning device that outputs a warning based on the arrival time of an object estimated by the arrival time estimation device to reach the lean vehicle may output a warning to alert the driver to the automobile, or may output a warning to alert the driver to an object other than the automobile. In the present invention and embodiments, the warning output by the warning device is a warning to alert the driver of the lean vehicle. The warning is a warning to at least one of the driver's hearing, sight, or touch. An auditory warning is a sound warning. A visual warning is, for example, a light warning. A tactile warning is, for example, a vibration warning. The vibration may be, for example, a grip vibration or a seat vibration. The means for generating the vibration may be a dedicated means for the warning device, or may be a brake or a drive source.

[0032] In the present invention and embodiments, the phrase "two automobiles are traveling straight ahead of a lean vehicle, side by side with a gap in the left-right direction, and the size of the gap is such that there is no room for the width of the lean vehicle" means that the size of the gap is smaller than or equal to the width of the lean vehicle. In other words, the size of the gap is such that the lean vehicle cannot pass through the gap under normal traffic conditions. By interpreting it in a similar way to this definition, the meaning of "two automobiles are traveling straight ahead of a lean vehicle, side by side with a gap in the left-right direction, and the size of the gap is such that there is no room for the width of the automobiles, but there is room for the width of the lean vehicle" can also be interpreted.

[0033] In the present invention or embodiments, a monocular camera that captures multiple images at a fixed time interval may be able to change the fixed time interval for capturing images. That is, the monocular camera that captures multiple images at a fixed time interval may be configured to be able to switch between capturing multiple images at a first fixed time interval and capturing multiple images at a second fixed time interval that is different from the first time interval. For example, the monocular camera may be configured to be able to switch between capturing multiple images at 30 frames per second (FPS) and capturing multiple images at 60 FPS.

[0034] In the present invention or embodiments, a color image is an image in which each pixel of the image contains information of multiple colors (e.g., red, green, and blue). A color image does not include a monochromatic image that contains, for example, only red information.

[0035] In the present invention or embodiments, (I) an algorithm that acquires the size of the rear of the vehicle based on a plurality of consecutive images including the left or right side and rear of the vehicle and estimates the arrival time until the vehicle reaches a lean vehicle may be an algorithm such as the following: That is, an algorithm that acquires the size of the rear of the vehicle in a plurality of consecutive images, derives a rate of change in the size of the rear of the vehicle, and estimates the arrival time based on the derived rate of change. In the present invention or embodiments, (II) an algorithm that acquires the distances between a plurality of feature points on the left or right side and rear of the vehicle based on a plurality of consecutive images including the left or right side and rear of the vehicle and estimates the arrival time until the vehicle reaches a lean vehicle may be an algorithm such as the following: That is, the algorithm may acquire the positions of multiple feature points on the left or right side and rear of the vehicle for each of multiple consecutive images, derive distances between the multiple feature points in each image based on the acquired positions of the feature points, and estimate a time it takes for a vehicle traveling in the center lane to reach a lean vehicle based on the derived distances. The multiple feature points on the left or right side and rear of the vehicle may include at least one feature point on the left or right side of the vehicle. The multiple feature points on the left or right side and rear of the vehicle may include at least one feature point on the rear of the vehicle. The multiple feature points on the left or right side and rear of the vehicle may include at least one feature point on a boundary between the left or right side and rear of the vehicle. When the number of feature points on the left or right side and rear of the vehicle is two, the distance between the multiple feature points on the left or right side and rear of the vehicle is the distance between these two feature points. If the number of feature points on the left or right side and rear of the automobile is three or more, the distance between the feature points on the left or right side and rear of the automobile may be the distance between any two of these three or more feature points.Furthermore, if the number of feature points on the left or right side and rear of the vehicle is three or more, the distance between the two feature points may be obtained for each of two or more sets of two or more different combinations of two feature points among these three or more feature points. In the present invention or embodiments, (III) an algorithm for estimating the arrival time until a vehicle reaches a lean vehicle based on a plurality of consecutive images including the left or right side and rear of the vehicle using a deep learning algorithm to estimate the rate of change in size of the vehicle having the left or right side and rear of the vehicle based on the plurality of consecutive images including the left or right side and rear of the vehicle may be the following algorithm. That is, the deep learning algorithm may be an algorithm that is deep learned based on data of a large number of images captured by an imaging device, includes a trained model that receives as input a plurality of consecutive images captured by the imaging device and outputs the rate of change in size of the vehicle in the images, and estimates the arrival time based on the rate of change in size of the vehicle output from the trained model.

[0036] The situation in which the lean-vehicle image processing device of the present invention is used is not limited to a situation in which a lean-vehicle is traveling in the right or left side area of ​​a center lane of a five-lane road. The lean-vehicle image processing device may also be used in the following driving situations, for example. The driving situation may be, for example, a situation in which a lean-vehicle is traveling on a road other than a five-lane road. A road other than a five-lane road may be a road with six or more lanes, a two-lane road, or a road with one lane in each direction. The driving situation may also be, for example, a situation in which a lean-vehicle is traveling in the center area of ​​a certain lane. The driving situation may also be, for example, a situation in which a lean-vehicle is traveling in the right or left side area of ​​a lane other than the center lane of a five-lane road.

[0037] In the present invention and embodiments, "at least one (one) of a plurality of options" includes all possible combinations of the plurality of options. "At least one (one) of a plurality of options" may be any one of the plurality of options or all of the plurality of options. For example, "at least one of A, B, and C" may be only A, only B, only C, A and B, A and C, B and C, or A, B, and C.

[0038] In this specification, for example, "1 to 10" means 1 or more and 10 or less. The same definition applies to numbers other than 1 and 10.

[0039] In the present invention and embodiments, the words including, comprising, having, and their derivatives are used herein to encompass the listed items and equivalents thereof as well as additional items.

[0040] Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification and claims have the same meaning as commonly understood by those skilled in the art to which this invention belongs. Terms, such as those defined in commonly used dictionaries, should be interpreted to have a meaning consistent with the meaning in the context of the relevant technology and this disclosure, and should not be interpreted in an idealized or overly formal sense.

[0041] In this specification, the term "may" is non-exclusive. "may" means "may, but is not limited to." In this specification, "may" implicitly includes the possibility that "may not." In this specification, a configuration described as "may" at least achieves the above-mentioned effect obtained by the configuration of claim 1.

[0042] Before describing embodiments of the present invention in detail, it is to be understood that the invention is not limited to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The present invention is capable of embodiments other than those described below. The present invention is also capable of embodiments incorporating various variations of the embodiments described below.

[0043] According to the present invention, an image processing device for a lean vehicle can be provided that can adapt to situations in which a lean vehicle, which has a slimmer body than an automobile, is traveling in the right side area and left side area of ​​a lane while traveling in the same lane, and that is highly mountable on a lean vehicle, which has a slimmer body than an automobile.

[0044] FIG. 1 is a diagram for explaining the configuration of an image processing device for a lean vehicle according to a first embodiment. FIG. 2 is a diagram for explaining the configuration of an image processing device for a lean vehicle according to a second embodiment. FIG. 3 is a diagram for explaining the configuration of an image processing device for a lean vehicle according to a third embodiment. FIG. 4 is a diagram for explaining the configuration of an image processing device for a lean vehicle according to a fourth embodiment. FIG. 5 is a diagram for explaining the configuration of an image processing device for a lean vehicle according to a fifth embodiment. FIG. 6 is a diagram for explaining the configuration of an image processing device for a lean vehicle according to a sixth embodiment. FIG. 7 is a diagram for explaining the configuration of an image processing device for a lean vehicle according to a seventh embodiment.

[0045] Arrows F, Re, L, and R shown in the drawings represent the forward direction, the backward direction, the left direction, and the right direction, respectively.

[0046] [First Embodiment] An image processing device 10 for a lean vehicle according to a first embodiment of the present invention will be described below with reference to Fig. 1. The image processing device 10 for a lean vehicle is mounted on a lean vehicle 1. In Fig. 1, the lean vehicle 1 is a motorcycle, but the lean vehicle 1 is not limited to a motorcycle. The image processing device 10 for a lean vehicle includes an imaging device 11 that captures images, and an arrival time estimation device 12 that processes the images captured by the imaging device 11 and estimates the arrival time of an object to reach the lean vehicle 1. The imaging device 11 includes a monocular camera.

[0047] FIG. 1 includes plan views showing two examples (CASE 1 and CASE 2) of a situation in which a lean vehicle 1 and five automobiles 61 to 65 are traveling on a five-lane road. The five-lane road includes a left-most lane 51, a center-left lane 52, a center lane 53, a center-right lane 54, and a right-most lane 55. In CASE 1 and CASE 2, the lean vehicle 1 travels in the center lane 53, and the five automobiles 61 to 65 travel in a line in the left-right direction ahead of the lean vehicle 1. In CASE 1, the lean vehicle 1 travels in a right-side area 50R of the center lane 53, and in CASE 2, the lean vehicle 1 travels in a left-side area 50L of the center lane 53. The right side area 50R is an area including the right end of a single lane 50 (e.g., center lane 53) when the single lane 50 is divided into thirds in the left-right direction, and the left side area 50L is an area including the left end of the single lane 50. The area between the right side area 50R and the left side area 50L is a center area 50C. The width of the lean vehicle 1 is very small compared to the width of an automobile 60 (e.g., automobiles 61-65). The width of the lean vehicle 1 is very small compared to the width of the lane 50. Therefore, the lean vehicle 1 has a high degree of freedom in its left-right traveling position within the center lane 53, and can move left and right within the center lane 53, and can travel in either the right side area 50R or the left side area 50L within the center lane 53.

[0048] In Cases 1 and 2, the distance D in the longitudinal direction between the five automobiles 61-65 and the lean vehicle 1 is the distance required to capture an image so that image processing is completed at a time to collision (TTC) of 3 seconds, assuming that the time Ti from capturing an image to completing image processing is 0.5 seconds and that the object approaches the lean vehicle 1 at a relative speed Vr of 15 km / h. This distance is approximately 15 m. Note that, hereinafter, this distance may be referred to as the "specific distance."

[0049] When a lean vehicle 1 is traveling in the right side area 50R or left side area 50L of the center lane 53 of a five-lane road, and five automobiles 61 to 65 are traveling in a line in the left-right direction ahead of the lean vehicle 1, and the longitudinal distance D between the five automobiles 61 to 65 and the lean vehicle 1 is the specific distance described above, the imaging device 11 and the arrival time estimation device 12 have the following configuration.

[0050] The monocular camera of the imaging device 11 is configured with a horizontal angle of view such that, while the lean-in vehicle 1 is traveling in the right side area 50R of the center lane 53, the left edge 61BL of the rear face of the automobile 61 traveling in the leftmost lane 51 is not included in the captured image PR, but at least the rear face 63B of the automobile 63 traveling in the center lane 53 and the right side 63R of the automobile 63 traveling in the center lane 53 are included in the image PR. The monocular camera of the imaging device 11 is configured with a horizontal angle of view such that, while the lean-in vehicle 1 is traveling in the left side area 50L of the center lane 53, the right edge 65BR of the rear face of the automobile 65 traveling in the rightmost lane 55 is not included in the captured image PL, but the rear face 63B of the automobile 63 traveling in the center lane 53 and the left side 63L of the automobile 63 traveling in the center lane 53 are included in the image PL.

[0051] The arrival time estimation device 12 estimates the arrival time Ta required for the vehicle 63 traveling in the center lane 53 to reach the lean vehicle 1 based on an image PR that includes at least a rear surface 63B of the vehicle 63 traveling in the center lane 53 and a right side surface 63R of the vehicle 63 traveling in the center lane 53 while the lean vehicle 1 is traveling in the right side area 50R of the center lane 53. At this time, the arrival time estimation device 12 estimates the arrival time Ta based at least on an image PR captured by the imaging device 11 when the distance D is the above-mentioned specific distance. The arrival time estimation device 12 estimates the arrival time Ta required for the vehicle 63 traveling in the center lane 53 to reach the lean vehicle 1 based on an image PL that includes at least a rear surface 63B of the vehicle 63 traveling in the center lane 53 and a left side surface 63L of the vehicle 63 traveling in the center lane 53 while the lean vehicle 1 is traveling in the left side area 50L of the center lane 53. At this time, the arrival time estimation device 12 estimates the arrival time Ta based at least on the image PL captured by the imaging device 11 when the distance D is the above-mentioned specific distance.

[0052] 1, the angle θmin is a schematic representation of the minimum horizontal angle of view of the monocular camera of the imaging device 11, and the angle θmax is a schematic representation of the maximum horizontal angle of view of the monocular camera of the imaging device 11, for conceptual understanding.

[0053] In FIG. 1, the symbols "o", "△", and "x" attached to the automobiles 61 to 65 have the following meanings: The symbol "o" indicates that the entire rear of the automobile is imaged. The symbol "x" indicates that the rear of the automobile is not imaged at all, or that only a portion of the rear of the automobile is imaged. The symbol "△" indicates that the rear of the automobile is not imaged at all, or that only a portion of the rear of automobile 60 is imaged, or that the entire rear of the automobile is imaged.

[0054] The lean vehicle 1 or the lean vehicle image processing device 10 may or may not have a display device that displays the image captured by the imaging device 11. When the image is displayed, for example, an object (e.g., a car) recognized in the image may be surrounded by a frame, as shown in the two lower diagrams of FIG.

[0055] According to the first embodiment, an image processing device 10 for a lean vehicle can be realized that can adapt to scenes in which a lean vehicle 1, which has a slimmer body than an automobile 60, is traveling in the right side area 50R and left side area 50L of the lane 50 while traveling in the same lane 50 as the automobile 60, and that is highly mountable on a lean vehicle 1, which has a slimmer body than an automobile 60.

[0056] Second Embodiment Next, an image processing device 10 for a lean vehicle according to a second embodiment of the present invention will be described with reference to Fig. 2. The second embodiment has all of the configurations of the first embodiment.

[0057] 2, the lean vehicle image processing device 10 further includes a warning device 13 that outputs a warning to urge the driver to pay attention to the vehicle 60, based on the arrival time Ta that the vehicle 60 takes to reach the lean vehicle 1, estimated by the arrival time estimation device 12. The warning device 13 outputs a warning according to the vehicle width of the lean vehicle 1, as will be described below.

[0058] As shown in CASE 3 of FIG. 2 , when two automobiles 60 are traveling straight ahead of the lean vehicle 1, side by side with a gap G in the left-right direction, and the size of the gap G is too small for the width of the automobiles 60 but large enough for the width of the lean vehicle 1, the warning device 13 is configured not to output an alarm even if the lean vehicle 1 travels straight towards and approaches the center between the two automobiles 60. As shown in CASE 4 of FIG. 2 , when two automobiles 60 are traveling straight ahead of the lean vehicle 1, side by side with a gap G in the left-right direction, and the size of the gap G is too small for the width of the lean vehicle 1, the warning device 13 is configured to output an alarm when the lean vehicle 1 travels straight towards and approaches the center between the two automobiles 60. Note that in CASE 4 of FIG. 2 , the gap G is slightly larger than the width of the lean vehicle 1, but may be smaller than the width of the lean vehicle 1.

[0059] 2, when two automobiles 60 are traveling straight ahead of an automobile 66 (60) equipped with an alarm device, side by side with a gap G in the left-right direction, and the size of the gap G is too small for the width of the automobiles 66 but large enough for the width of the lean-to vehicle 1, the alarm device of the automobile 66 will output an alarm when the automobile 66 travels straight towards and approaches the center between the two automobiles 60. Although not shown, when two automobiles 60 are traveling straight ahead of an automobile 66 equipped with an alarm device, side by side with a gap G in the left-right direction, and the size of the gap G is too small for the width of both the automobile 66 and the lean-to vehicle 1, the alarm device of the automobile 66 will output an alarm when the automobile 66 travels straight towards and approaches the center between the two automobiles 60.

[0060] EXAMPLE 1 and EXAMPLE 2 in Fig. 2 are specific examples of CASE 2. EXAMPLE 3 and EXAMPLE 4 in Fig. 2 are specific examples of CASE 4.

[0061] In EXAMPLE 1 and EXAMPLE 3, a lean vehicle 1 is traveling in a right side area 50R of a center lane 53 of a five-lane road, and two automobiles 63, 64 traveling in the center lane 53 and center right lane 54 are located in front of the lean vehicle 1. In EXAMPLE 1, a portion of the automobile 63 is located on the left side area 50L of the center lane 53, and a portion of the automobile 64 is located on the right side area 50R of the center right lane 54. In EXAMPLE 3, a portion of the automobile 63 is located on the right side area 50R of the center lane 53, and a portion of the automobile 64 is located on the left side area 50L of the center right lane 54.

[0062] In EXAMPLE 2 and EXAMPLE 4, the lean vehicle 1 is traveling in the left side area 50L of the center lane 53 of a five-lane road, and two automobiles 63, 62 traveling in the center lane 53 and center-left lane 52 are located in front of the lean vehicle 1. In EXAMPLE 2, a portion of the automobile 63 is located in the right side area 50R of the center lane 53, and a portion of the automobile 62 is located on the left side area 50L of the center-left lane 52. In EXAMPLE 4, a portion of the automobile 63 is located on the left side area 50L of the center lane 53, and a portion of the automobile 62 is located on the right side area 50R of the center-left lane 52.

[0063] In the second embodiment, if the lean vehicle 1 has a display device that displays an image captured by the imaging device 11, and a frame surrounding the vehicle in the image captured by the imaging device 11 is displayed together with the captured image, the color of the frame surrounding the vehicle that is the subject of the alarm may be different when the alarm device 13 outputs an alarm and when the alarm device 13 does not output an alarm.

[0064] Third Embodiment Next, a lean vehicle image processing device 10 according to a third embodiment will be described with reference to Fig. 3. The second embodiment has all of the configurations of the first embodiment. The third embodiment may also have the configuration of the second embodiment.

[0065] In the lean-vehicle image processing device 10 of the third embodiment, when the lean-vehicle 1 is traveling in the right side area 50R of the lane 50, the imaging device 11 captures multiple images PR including a right side 60R and a rear 60B of an automobile 60 traveling in front of the lean-vehicle 1 on the lane 50 at regular time intervals ΔT. FIG. 3 shows three consecutive images PR1, PR2, and PR3 among the multiple images PR captured by the imaging device 11 at regular time intervals ΔT while the lean-vehicle 1 is traveling in the right side area 50R of the lane 50. Specifically, images PR1, PR2, and PR3 represent images PR captured at times TR1, TR2, and TR3, respectively. Time TR2 is later than time TR1, and time TR3 is later than time TR2. Furthermore, the time interval between time TR1 and time TR2, and the time interval between time TR2 and time TR3 are both ΔT. In the lean vehicle image processing device 10 of the third embodiment, when the lean vehicle 1 is traveling in the right side area 50R of the lane 50, the arrival time estimation device 12 estimates the arrival time Ta until the automobile 60 reaches the lean vehicle 1 based on the fixed time interval ΔT and multiple images PR including the right side 60R and rear 60B of the automobile 60 traveling ahead of the lean vehicle 1 on the lane 50, which are captured at fixed time intervals ΔT.

[0066] In the lean-vehicle image processing device 10 of the third embodiment, when the lean-vehicle 1 is traveling in the left side area 50L of the lane 50, the imaging device 11 captures, at a constant time interval ΔT, a plurality of images PL including a left side 60L and a rear 60B of an automobile 60 traveling in front of the lean-vehicle 1 on the lane 50. FIG. 3 shows three consecutive images PL1, PL2, and PL3 among the plurality of images PL captured by the imaging device 11 at a constant time interval ΔT when the lean-vehicle 1 is traveling in the left side area 50L of the lane 50. Specifically, the images PL1, PL2, and PL3 represent images PL captured at times TL1, TL2, and TL3, respectively. Time TL2 is later than time TL1, and time TL3 is later than time TL2. Furthermore, the time interval between time TL1 and time TL2 and the time interval between time TL2 and time TL3 are both ΔT. In the image processing device 10 for a lean vehicle of the third embodiment, when the lean vehicle 1 is traveling in the left side area 50L of the lane 50, the arrival time estimation device 12 estimates the arrival time Ta until the automobile 60 reaches the lean vehicle 1 based on the fixed time interval ΔT and multiple images PL including the left side 60L and rear 60B of the automobile 60 traveling in front of the lean vehicle 1 on the lane 50, which are captured at fixed time intervals ΔT.

[0067] In this way, the arrival time estimation device 12 of the third embodiment acquires multiple consecutive images (multiple images PR or multiple images PL) including the left side 60L or right side 60R and rear 60B of the automobile 60 captured by the imaging device 11 at a fixed time interval ΔT, and includes an algorithm that estimates the arrival time Ta until the automobile 60 reaches the lean vehicle 1 based on the multiple consecutive images (multiple images PR or multiple images PL) including the left side 60L or right side 60R and rear 60B of the automobile 60 and the fixed time interval ΔT.

[0068] In the lean vehicle image processing device 10 of the third embodiment, the time interval ΔT at which images are captured by the imaging device 11 may be changeable. That is, the imaging device 11 may be able to switch at least between capturing images at a first constant time interval and capturing images at a second constant time interval. Alternatively, in the lean vehicle image processing device 10 of the third embodiment, the time interval ΔT at which images are captured by the imaging device 11 may not be changeable.

[0069] [Fourth embodiment] Next, a lean vehicle image processing device 10 according to a fourth embodiment will be described with reference to Fig. 4. The fourth embodiment has all of the configurations of the first embodiment. The fourth embodiment may also have at least one of the configurations of the second embodiment and the third embodiment.

[0070] In the lean vehicle image processing device 10 of the fourth embodiment, the imaging device 11 captures color images as images PR and PL, as shown in Fig. 4. In Fig. 4, the color density of the rear surface 63B, right side surface 60R, and left side surface 60L of the automobile 60 is lighter than in Fig. 3, etc., to indicate that the captured images PR (PR1 to PR3) and PL (PL1 to PL3) are color images.

[0071] In the lean-vehicle image processing device 10 of the fourth embodiment, when the lean-vehicle 1 is traveling in the right side area 50R of the lane 50, the imaging device 11 continuously captures multiple images PR including a right side 60R and a rear 60B of an automobile 60 traveling in front of the lean-vehicle 1 on the lane 50. As described in the third embodiment, FIG. 4 shows three images PR1, PR2, and PR3 captured consecutively among the multiple images PR. However, in the fourth embodiment, as described above, the images PR1, PR2, and PR3 are color images. Furthermore, in the fourth embodiment, the time interval between time TR1 and time TR2 and the time interval between time TR2 and time TR3 may be the same or different. In the image processing device 10 for lean vehicles of the fourth embodiment, when the lean vehicle 1 is traveling in the right side area 50R of the lane 50, the arrival time estimation device 12 estimates the arrival time Ta until the automobile 60 reaches the lean vehicle 1 based on multiple images PR (images PR1 to PR3, etc.) including the right side 60R and rear 60B of the automobile 60 traveling in front of the lean vehicle 1 on the lane 50, and the time interval between capturing the multiple images PR.

[0072] In the lean-vehicle image processing device 10 of the fourth embodiment, when the lean-vehicle 1 is traveling in the left side area 50L of the lane 50, the imaging device 11 continuously captures a plurality of images PL including a left side 60L and a rear 60B of an automobile 60 traveling in front of the lean-vehicle 1 on the lane 50. As described in the third embodiment, FIG. 4 shows three images PL1, PL2, and PL3 captured consecutively among the plurality of images PL. However, in the fourth embodiment, as described above, the images PL1, PL2, and PL3 are color images. In the fourth embodiment, the time interval between time TL1 and time TL2 and the time interval between time TL2 and time TL3 may be the same or different. In the image processing device 10 for a lean vehicle of the fourth embodiment, when the lean vehicle 1 is traveling in the left side area 50L of the lane 50, the arrival time estimation device 12 estimates the arrival time Ta for the automobile 60 to reach the lean vehicle 1 based on multiple images PL (images PL1 to PL3, etc.) captured consecutively, including the left side 60L and rear 60B of the automobile 60 traveling in front of the lean vehicle 1 on the lane 50, and the time interval between capturing the multiple images PL.

[0073] In this way, the arrival time estimation device 12 of the fourth embodiment acquires multiple color images (multiple images PR or multiple images PL) including the left side 60L or right side 60R and rear 60B of the automobile 60 captured by the imaging device 11, and includes an algorithm that estimates the arrival time Ta until the automobile 60 reaches the lean vehicle 1 based on the acquired multiple color images (multiple images PR or multiple images PL) including the left side 60L or right side 60R and rear 60B of the automobile 60.

[0074] Fifth Embodiment Next, a lean vehicle image processing device 10 according to a fifth embodiment will be described with reference to FIG. 5. The fifth embodiment has all of the configurations of the first embodiment. The fifth embodiment may also have at least one of the configurations of the second to fourth embodiments.

[0075] In the lean-vehicle image processing device 10 of the fifth embodiment, when the lean-vehicle 1 is traveling in the right side area 50R of the lane 50, the imaging device 11 continuously captures a plurality of images PR including a right side 60R and a rear 60B of an automobile 60 traveling in front of the lean-vehicle 1 on the lane 50. As described in the third embodiment, Fig. 5 shows three images PR1, PR2, and PR3 that are continuously captured among the plurality of images PR. However, in the fifth embodiment, the time interval between time TR1 and time TR2 and the time interval between time TR2 and time TR3 may be the same or different. In the image processing device 10 for lean vehicles of the fifth embodiment, when the lean vehicle 1 is traveling in the right side area 50R of the lane 50, the arrival time estimation device 12 acquires the width (WR1, WR2, WR3, etc.) of the automobile 60 as the size of the automobile 60 in the image PR based on each of multiple images PR including the right side 60R and rear 60B of the automobile 60 traveling in front of the lean vehicle 1 in the lane 50, and estimates the arrival time Ta for the automobile 60 to reach the lean vehicle 1 based on the width of the automobile in the multiple images PR and the time interval between capturing the multiple images PR.

[0076] In the lean-vehicle image processing device 10 of the fifth embodiment, when the lean-vehicle 1 is traveling in the left side area 50L of the lane 50, the imaging device 11 continuously captures a plurality of images PL including a left side 60L and a rear 60B of an automobile 60 traveling in front of the lean-vehicle 1 on the lane 50. As described in the third embodiment, Fig. 5 shows three images PL1, PL2, and PL3 that are continuously captured among the plurality of images PL. However, in the fifth embodiment, the time interval between time TL1 and time TL2 and the time interval between time TL2 and time TL3 may be the same or different. In the image processing device 10 for lean vehicles of the fifth embodiment, when the lean vehicle 1 is traveling in the left side area 50L of the lane 50, the arrival time estimation device 12 acquires the width (WL1, WL2, WL3, etc.) of the automobile 60 as the size of the automobile 60 in the image PL based on each of multiple images PL including the left side 60L and rear 60B of the automobile 60 traveling in the lane 50, and estimates and outputs the arrival time Ta for the automobile 60 to reach the lean vehicle 1 based on the width of the automobile in the multiple images PL and the time interval between capturing the multiple images PL.

[0077] In this way, the arrival time estimation device 12 of the fifth embodiment acquires multiple consecutive images (multiple images PR or multiple images PL) including the left side 60L or right side 60R and rear 60B of the automobile 60 captured by the imaging device 11 when the automobile 60 is traveling in front of the lean vehicle 1, and based on the multiple consecutive images (multiple images PR or multiple images PL) including the left side 60L or right side 60R and rear 60B of the automobile 60, acquires the size of the rear of the automobile 60 in the image PR or image PL, and includes an algorithm that estimates the arrival time Ta for the automobile 60 to reach the lean vehicle 1.

[0078] In the fifth embodiment, the arrival time estimation device 12 acquired the width of the automobile 60 as the size of the automobile 60 in the image, but it may also acquire, for example, the height (vertical length) of the rear surface 60B of the automobile 60 as the size of the rear surface 60B of the automobile 60 in the image, the area of ​​the rear surface 60B of the automobile 60 in the image, etc.

[0079] [Sixth embodiment] Next, a lean vehicle image processing device 10 according to a sixth embodiment will be described with reference to Fig. 6. The sixth embodiment has all of the configurations of the first embodiment. The sixth embodiment may also have at least one of the configurations of the second to fourth embodiments.

[0080] In the lean-vehicle image processing device 10 of the sixth embodiment, when the lean-vehicle 1 is traveling in the right side area 50R of the lane 50, the imaging device 11 continuously captures a plurality of images PR including a right side 60R and a rear 60B of an automobile 60 traveling in front of the lean-vehicle 1 on the lane 50. As described in the third embodiment, Fig. 6 shows three images PR1, PR2, and PR3 that are continuously captured among the plurality of images PR. However, in the sixth embodiment, the time interval between time TR1 and time TR2 and the time interval between time TR2 and time TR3 may be the same or different. In the lean-vehicle image processing device 10 of the sixth embodiment, when the lean-vehicle 1 is traveling in the right side area 50R of the lane 50, the arrival time estimation device 12 acquires distances between multiple feature points of the automobile 60 for each of multiple images PR including the right side 60R and rear 60B of the automobile 60 traveling in the lane 50. At this time, the arrival time estimation device 12 may acquire positions of the multiple feature points of the automobile 60 for each of the multiple images PR and derive the distances between the multiple feature points based on the positions of the multiple feature points acquired. Figure 6 illustrates a case where the feature points of the automobile 60 are the upper end F1 and lower end F2 of the boundary line between the right side 60R and rear 60B of the automobile 60, and the distances between the multiple feature points of the automobile 60 are the distances between the upper end F1 and the lower end F2 (HR1, HR2, HR3, etc.). Then, the arrival time estimation device 12 estimates and outputs the arrival time Ta for the automobile 60 to arrive at the lean vehicle 1 based on the distance between the multiple feature points in each of the multiple images PR and the time interval between capturing the multiple images PR.

[0081] In the lean-vehicle image processing device 10 of the sixth embodiment, when the lean-vehicle 1 is traveling in the left side area 50L of the lane 50, the imaging device 11 continuously captures a plurality of images PL including a left side 60L and a rear 60B of an automobile 60 traveling in front of the lean-vehicle 1 on the lane 50. As described in the third embodiment, Fig. 6 shows three images PL1, PL2, and PL3 that are continuously captured among the plurality of images PL. However, in the sixth embodiment, the time interval between time TL1 and time TL2 and the time interval between time TL2 and time TL3 may be the same or different. In the lean-vehicle image processing device 10 of the sixth embodiment, when the lean-vehicle 1 is traveling in the left side area 50L of the lane 50, the arrival time estimation device 12 acquires distances between multiple feature points of the automobile 60 for each of multiple images PL including the left side 60L and rear 60B of the automobile 60 traveling ahead of the lean-vehicle 1 on the lane 50. At this time, the arrival time estimation device 12 may acquire positions of the multiple feature points of the automobile 60 for each of the multiple images PL and derive the distances between the multiple feature points based on the positions of the multiple acquired feature points. Figure 6 illustrates a case where the feature points of the automobile 60 are the upper end F3 and lower end F4 of the boundary line between the rear 60B and left side 60L of the automobile 60, and the distances between the multiple feature points of the automobile 60 are the distances (HL1, HL2, HL3, etc.) between the upper end F3 and the lower end F4. Then, the arrival time estimation device 12 estimates and outputs the arrival time Ta for the automobile 60 to arrive at the lean vehicle 1 based on the distance between the multiple feature points in each of the multiple images PL and the time interval between capturing the multiple images PL.

[0082] In this way, the arrival time estimation device 12 of the sixth embodiment acquires multiple consecutive images (multiple images PR or multiple images PL) including the left side 60L or right side 60R and rear 60B of the automobile 60 captured by the imaging device 11 when the automobile 60 is traveling in front of the lean vehicle 1, and based on the multiple consecutive images (multiple images PR or multiple images PL) including the left side 60L or right side 60R and rear 60B of the automobile 60, acquires the distance between multiple feature points on the left side 60L or right side 60R and rear 60B of the automobile 60 in the image PR or image PL, and includes an algorithm to estimate the arrival time Ta until the automobile 60 reaches the lean vehicle 1.

[0083] In the sixth embodiment, the multiple feature points of the automobile 60 in the images PR and PL are the upper and lower ends of the boundary line between the right side 60R and the rear side 60B or the left side 60L of the automobile 60, but this is not limited to this. At least one of the multiple feature points of the automobile 60 in the images PR and PL may be another point on the rear side 60B, the right side 60R, or the left side 60L of the automobile 60, or on the boundary line. The number of feature points may also be three or more. When the number of feature points is three or more, the arrival time estimation device 12 may obtain the distances between two or more pairs of two feature points that are different combinations from the three or more feature points, and estimate the arrival time for the automobile 60 to arrive at the lean-to vehicle 1 based on these distances.

[0084] [Seventh embodiment] Next, a lean vehicle image processing device 10 according to a seventh embodiment will be described with reference to Fig. 7. The seventh embodiment has all of the configurations of the first embodiment. The seventh embodiment may also have at least one of the configurations of the second to fourth embodiments.

[0085] As shown in FIG. 7 , the arrival time estimation device 12 of the lean vehicle image processing device 10 of the seventh embodiment has a trained model 21 and an arrival time derivation unit 22 .

[0086] The trained model 21 may be a model that has undergone deep learning based on data of a large number of images captured by the imaging device 11. The trained model 21 is a model that receives as input a plurality of consecutive images P (a plurality of images PR or a plurality of images PL) captured by the imaging device 11, and outputs a rate of change E in the size of the automobile 60 between the plurality of images P. The rate of change E in the size of the automobile 60 in the image P may be a rate of change in the width of the rear surface 60B of the automobile 60. The rate of change E in the size of the automobile 60 in the image P may be a rate of change in the height (vertical length) of the rear surface 60B of the automobile 60. The rate of change E in the size of the automobile 60 in the image P may be a rate of change in the area of ​​the rear surface 60B of the automobile 60 in the image P. The arrival time derivation unit 22 derives an arrival time Ta for the automobile 60 to reach the lean vehicle 1 based on the rate of change E output from the trained model 21.

[0087] In the seventh embodiment, when a lean-to vehicle 1 is traveling in the right side area 50R of a lane 50, the imaging device 11 continuously captures multiple images PR including a right side 60R and a rear side 60B of an automobile 60 traveling in front of the lean-to vehicle 1 in the lane 50, and inputs the images into the trained model 21. As described in the third embodiment, FIG. 7 shows three consecutively captured images PR1, PR2, and PR3 among the multiple images PR. However, in the seventh embodiment, the time interval between time TR1 and time TR2 and the time interval between time TR2 and time TR3 may be the same or different. The trained model 21 outputs a rate of change E in the size of the automobile 60 between the multiple images PR based on the input multiple images PR. The output change rate E is, for example, a change rate ER1 of the size of the automobile 60 in image PR2 relative to the size of the automobile 60 in image PR1, a change rate ER2 of the size of the automobile 60 in image PR3 relative to the size of the automobile 60 in image PR2, etc. Then, the arrival time derivation unit 22 derives the arrival time Ta for the automobile 60 to arrive at the lean vehicle 1 based on the change rate E output from the trained model 21 and the time intervals at which the multiple images PR were captured.

[0088] In the seventh embodiment, when a lean-to vehicle 1 is traveling in the left side area 50L of a lane 50, the imaging device 11 continuously captures multiple images PL including the left side 60L and rear 60B of an automobile 60 traveling in front of the lean-to vehicle 1 on the lane 50, and inputs the images into the trained model 21. As described in the third embodiment, FIG. 7 shows three consecutively captured images PL1, PL2, and PL3 among the multiple images PL. However, in the seventh embodiment, the time interval between time TL1 and time TL2 and the time interval between time TL2 and time TL3 may be the same or different. The trained model 21 outputs the rate of change E in size of the automobile 60 between the multiple images PL based on the input multiple images PL. The output change rate E is, for example, a change rate EL1 of the size of the automobile 60 in image PL2 relative to the size of the automobile 60 in image PL1, a change rate EL2 of the size of the automobile 60 in image PL3 relative to the size of the automobile 60 in image PL2, etc. The arrival time derivation unit 22 then derives an arrival time Ta for the lean vehicle 1 to reach the automobile 60 traveling in the lane 50 based on the change rate E output from the trained model 21 and the time intervals at which the multiple images PL are captured.

[0089] In this way, the arrival time estimation device 12 of the seventh embodiment acquires multiple consecutive images (multiple images PR or multiple images PL) including the left side 60L or right side 60R and rear 60B of the automobile 60 captured by the imaging device 11 when the automobile 60 is traveling in front of the lean vehicle 1, and includes an algorithm that estimates the arrival time Ta until the automobile 60 reaches the lean vehicle 1 using a deep learning algorithm to estimate the rate of change E in the size of the automobile 60 having the left side 60L or right side 60R and rear 60B in the image PR or image PL based on the multiple consecutive images (multiple images PR or multiple images PL) including the left side 60L or right side 60R and rear 60B of the automobile 60.

[0090] The arrival time estimation device 12 may perform, for example, the following processing to improve the accuracy of the change rate E output from the trained model 21. For example, when the leaning vehicle 1 is traveling in the right side area 50R of the lane 50, the arrival time estimation device 12 inputs two images PR and enlarges the image PR captured earlier of the two images PR based on the change rate E output from the trained model 21. The trained model 21 then learns the error obtained by comparing the automobile 60 in the enlarged image PR with the automobile 60 in the image PR captured later of the two images PR. In other words, the trained model 21 is adjusted to reduce this error. This error may be obtained, for example, from the difference in pixel values ​​(e.g., RGB values) of each pixel of the automobile 60 when the enlarged image PR and the image PR captured later are superimposed. More specifically, the error may be, for example, the mean square value or absolute mean value of the difference in pixel values ​​of each pixel. The arrival time estimation device 12 similarly causes the trained model 21 to learn an error when the lean vehicle 1 is traveling in the left side area 50L of the lane 50. Such error learning may be performed both when the trained model 21 is used and when the trained model 21 is generated. Furthermore, such error learning may be performed only when the trained model 21 is generated, rather than when the trained model 21 is used.

[0091] Here, unlike the seventh embodiment, consider a case where the trained model 21 receives an image as input and outputs the distance between the lean vehicle 1 and the automobile 60, and the arrival time derivation unit 22 derives the arrival time Ta based on the distance between the lean vehicle 1 and the automobile 60 output from the trained model 21. In this case, when generating the trained model by deep learning, a correct value of the distance between the lean vehicle 1 and the automobile 60 is required as training data. Therefore, it is necessary to prepare correct values ​​of the distance between the lean vehicle 1 and the automobile 60 in a large number of situations where the surrounding environment and the inclination angle of the lean vehicle 1 are different. In contrast, in the seventh embodiment, as described above, the trained model 21 receives an image as input and outputs the rate of change E, and the arrival time derivation unit 22 derives the arrival time Ta based on the rate of change E output from the trained model 21. In this case, the rate of change E is information obtained from the image, so a correct value of the rate of change E is not required. Therefore, the cost and labor required to generate the trained model can be reduced compared to when the trained model 21 takes an image as input and outputs the distance between the lean vehicle 1 and the automobile 60.

[0092] The right side area in this specification is included in the right driving area in the specification of Japanese Patent Application No. 2023-099900, which is the basic application of the present application. The left side area in this specification is included in the left driving area in the specification of the same application. The central area in this specification is included in the central driving area in the specification of the same application. The automobile in this specification corresponds to a four-wheeled vehicle in the specification of the same application.

[0093] 1: lean vehicle, 10: image processing device for lean vehicle, 11: imaging device, 12: arrival time estimation device, 16: warning device, 21: trained model, 22: arrival time derivation unit, 51: leftmost lane, 52: center left lane, 53: center lane, 54: center right lane, 55: rightmost lane, 50L: left side area, 50C: center area, 50R: right side area, 61 to 65: automobile, 63R: right side, 63L: left side, 63B: rear, E: rate of change, P, PR, PL: image, Ta: arrival time

Claims

1. An image processing device for a lean vehicle that is mounted on a lean vehicle and includes an imaging device that captures images, and an arrival time estimation device that processes the images captured by the imaging device to estimate the arrival time of an object to reach the lean vehicle, wherein, when a single lane is divided into thirds in the left-right direction, the area including the right end of the single lane is defined as the right side area, the area including the left end of the single lane is defined as the left side area, and the area between the right side area and the left side area is defined as the center area, and when each lane of a five-lane road is defined from left to right as the leftmost lane, center left lane, center lane, center right lane, and rightmost lane, When the lean vehicle is traveling in the right side area or left side area of ​​the center lane of the five-lane road, and five automobiles are traveling in a line left and right in front of the lean vehicle, and the longitudinal distance between the five automobiles and the lean vehicle is a distance necessary to capture an image so that the image processing is completed when the TTC (Time To Collision) is 3 seconds, assuming that the time from capturing image to completion of image processing is 0.5 seconds and that an object approaches the lean vehicle at a relative speed of 15 km / h, the imaging device and the arrival time estimation device are configured as follows, and are configured to estimate the arrival time for the automobile traveling in the center lane to reach the lean vehicle based on the image that includes the left or right side of the automobile traveling in the center lane, both when the lean vehicle is traveling in the right side area of ​​the center lane of the five-lane road and when the lean vehicle is traveling in the left side area of ​​the center lane of the five-lane road. The imaging device includes a monocular camera having a left-right angle of view configured such that, when the lean-to vehicle is traveling in the right side area of ​​the center lane of the five-lane road, the left edge of the rear of the automobile traveling in the leftmost lane is not included in the image, but at least the rear of the automobile traveling in the center lane and the right side of the automobile traveling in the center lane are included in the image, and, when the lean-to vehicle is traveling in the left side area of ​​the center lane of the five-lane road, the right edge of the rear of the automobile traveling in the rightmost lane is not included in the image, but at least the rear of the automobile traveling in the center lane and the left side of the automobile traveling in the center lane are included in the image. The arrival time estimation device is configured to estimate the arrival time until the automobile traveling in the center lane reaches the lean vehicle based on the image that includes at least the rear of the automobile traveling in the center lane and the right side of the automobile traveling in the center lane while the lean vehicle is traveling in the right side area of ​​the center lane of the five-lane road, and to estimate the arrival time until the automobile traveling in the center lane reaches the lean vehicle based on the image that includes at least the rear of the automobile traveling in the center lane and the left side of the automobile traveling in the center lane while the lean vehicle is traveling in the left side area of ​​the center lane of the five-lane road.

2. The image processing device for a lean vehicle further includes an alarm device that outputs an alarm based on the arrival time for an object to reach the lean vehicle estimated by the arrival time estimation device, wherein the alarm device is configured to not output the alarm even if the lean vehicle travels straight toward the center between the two automobiles and approaches them when two automobiles are traveling straight ahead in front of the lean vehicle, side by side, with a gap between them in the left and right direction, and the size of the gap is too large for the width of the automobiles to allow for left and right clearance, but is large enough for the width of the lean vehicle to allow for left and right clearance; and is configured to output the alarm when the lean vehicle travels straight toward the center between the two automobiles and approaches them when two automobiles are traveling straight ahead in front of the lean vehicle, side by side, with a gap between them in the left and right direction, and the size of the gap is too large for the width of the lean vehicle to allow for left and right clearance; and outputs the alarm according to the width of the lean vehicle.

3. The image processing device for a lean vehicle as described in claim 1 or 2, characterized in that the imaging device includes a monocular camera that captures multiple images including the left or right side and rear of the automobile at a fixed time interval when the automobile is traveling in front of the lean vehicle, and the arrival time estimation device includes an algorithm that acquires the multiple consecutive images including the left or right side and rear of the automobile captured by the imaging device at the fixed time interval, and estimates the arrival time until the automobile reaches the lean vehicle based on the multiple consecutive images including the left or right side and rear of the automobile and the fixed time interval.

4. An image processing device for a lean vehicle as described in any one of claims 1 to 3, characterized in that the imaging device includes a monocular camera that captures multiple color images including the left or right side and rear of the automobile when an automobile is traveling in front of the lean vehicle, and the arrival time estimation device includes an algorithm that acquires the multiple color images including the left or right side and rear of the automobile captured by the imaging device, and estimates the arrival time for the automobile to reach the lean vehicle based on the acquired multiple color images including the left or right side and rear of the automobile.

5. The arrival time estimation device includes: when an automobile is traveling in front of the lean vehicle, the arrival time estimation device acquires a plurality of consecutive images including the left side or right side and rear of the automobile captured by the imaging device; (I) an algorithm for acquiring the size of the rear of the automobile in the images based on the plurality of consecutive images including the left side or right side and rear of the automobile, and estimating the arrival time until the automobile reaches the lean vehicle; (II) an algorithm for acquiring the distance between a plurality of feature points on the left side or right side and rear of the automobile in the images based on the plurality of consecutive images including the left side or right side and rear of the automobile, and estimating the arrival time until the automobile reaches the lean vehicle; or (III) an algorithm for estimating the arrival time until the automobile reaches the lean vehicle using a deep learning algorithm to estimate the rate of change in size of the automobile having the left side or right side and rear in the images based on the plurality of consecutive images including the left side or right side and rear of the automobile.

5. The image processing device for a lean vehicle according to claim 1,

Images