Vehicle Interruption Prediction System

Abstract

To provide a system for predicting cut-in driving by other vehicles which uses a monocular camera image as main input information with reduced burden on hardware.SOLUTION: A region recognized as a vehicle is extracted as a rectangular frame from an image of a monocular camera which images an area in front of a vehicle. The rectangular frame is classified into a left-side rectangular frame (Li) positioned in a left-side region in a width direction of the image, and a right-side rectangular frame (Ri) positioned in a right-side region. In a case where two or more rectangular frames are extracted in at least one of the left-side region and the right-side region, successively from a rectangular frame of which the longitudinal direction coordinate is lowest (most proximal) with respect to the rectangular frames, calculations are made as to (a) an overlap degree which is a ratio of an overlap area with the next rectangular frame to a possible maximum area, (b) a distance difference rate which is a ratio of the longitudinal direction coordinate of a lower side with the next rectangular frame and (c) an influence degree which is the ratio of the longitudinal direction coordinate of the lower side to a disappear point coordinate, thereby determining a possibility of cut-in driving by a vehicle corresponding to the rectangular frames.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a vehicle intrusion prediction system.

Background Art

[0002] As part of vehicle driving support and preventive safety, a system for predicting the intrusion of other vehicles from adjacent lanes into the front of the host vehicle has been proposed. For example, Patent Document 1 discloses a vehicle control device configured to set an intrusion possibility index value based on driving environment information such as the number of lanes acquired by a camera sensor or a laser sensor, the number of vehicles traveling in adjacent lanes, the distance to a traffic signal, and traffic jam information acquired by a navigation device, and execute control to set a target inter-vehicle distance for inter-vehicle control with a preceding vehicle according to the index value.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The intrusion prediction based on the driving environment information as described above requires ranging means such as a laser sensor or a stereo camera and integrated processing of their recognition results, and has a problem that it cannot be used with a monocular camera.

[0005] The present invention has been made in view of the above actual situation, and an object thereof is to provide a vehicle intrusion prediction system that accurately uses a monocular camera image as input information.

Means for Solving the Problems

[0006] To solve the above problems, a vehicle intrusion prediction system according to the present invention includes a monocular camera disposed to image the front of the vehicle, An image processing unit that processes an image captured by the monocular camera, and is provided with the image processing unit, extracts a region recognized as a vehicle from the image as a rectangular frame, classifies the rectangular frame into a left rectangular frame located in the left region in the width direction of the image and a right rectangular frame located in the right region, when two or more rectangular frames are extracted in at least one of the left region and the right region, for the two or more rectangular frames, in order from the rectangular frame with the lowest (nearest) vertical coordinate, (a) the degree of overlap, which is the ratio of the possible maximum area of the overlapping area with the next rectangular frame, (b) the distance difference ratio, which is the ratio of the vertical coordinate of the lower side with the next rectangular frame, (c) the degree of influence, which is the ratio of the vertical coordinate of the lower side to the vanishing point coordinate, are calculated, and based on them, the possibility of interruption by the vehicle corresponding to the rectangular frame is determined, and is configured to execute the process.

Advantages of the Invention

[0007] As described above, the vehicle interruption prediction system according to the present invention extracts a region recognized as a vehicle from an image of a monocular camera as a rectangular frame, focuses on the fact that the vertical coordinate of the lower side of the rectangular frame reflects distance information, and performs interruption prediction based on the degree of overlap and the distance difference ratio, which are indicators reflecting the positional relationship of vehicles in adjacent lanes, and the degree of influence, which is an indicator reflecting the distance from the host vehicle. With this configuration, using the image of the monocular camera as input information, there is an advantage that an interruption prediction system can be constructed with high accuracy.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In FIG. 1, an interrupt prediction system 100 for a vehicle according to an embodiment of the present invention mainly includes a front camera 20 that images the front of the vehicle and an image processing unit 10.

[0010] The front camera 20 is a monocular camera equipped with an image sensor (such as a CMOS or CCD imaging device) that images a field-of-view image formed through an optical system, and is preferably disposed at the upper part of the front windshield (windshield) inside the vehicle cabin.

[0011] The image processing unit 10 includes a program that executes a process (11) of extracting a region recognized as a vehicle (moving body) from an input image captured by the front camera 20 as a rectangular frame (Ci, Li, Ri), and a process of classifying the extracted rectangular frame into a central region, a left region, and a right region in the width direction, and a program that executes processes (14L, 14R) of calculating the degree of overlap, the distance difference rate, and the influence degree described later and determining the interrupt possibility, and is configured as an electronic control unit (ECU) incorporating these programs.

[0012] In addition, the image processing unit 10 is preferably configured to acquire detection information of a positioning system 21 such as a GNSS sensor, map information 22, and vehicle information 23 such as vehicle speed, steering angle, and yaw rate through an in-vehicle network in order to reflect road curvature and road gradient in the processing as described later.

[0013] The vehicle extraction process (11) is executed using an AI model (machine learning model), for example, a vehicle detection model obtained by supervised learning using a dataset and correct labels (rectangular frames for vehicles, trucks, buses, motorcycles, etc.) for each data. Since the correct label is created as a rectangular frame including hidden parts such as other vehicles, it is possible to detect a vehicle that appears partially in the image.

[0014] The classification process (12C, 12L, 12R) of the extracted rectangular frames is performed on the premise that the front camera 20 is attached to the center or substantially the center in the width direction of the vehicle. As shown in FIG. 2, a rectangular frame including the center line (center coordinates) c in the width direction of the input image 200 on its lower side is classified as the central rectangular frame Ci, and among the remaining rectangular frames, a rectangular frame whose position coordinates (center coordinates) in the width direction are on the left side with respect to the center line c is classified as the left rectangular frame Li, and a rectangular frame on the right side is classified as the right rectangular frame Ri.

[0015] At this time, in the input image 200, even without performing lane recognition in particular, the central rectangular frame Ci located in the central region in the width direction can be regarded as the preceding vehicle in the own lane, the left rectangular frame Li located in the left region can be regarded as the preceding vehicle in the left adjacent lane, and the right rectangular frame Ri located in the right region can be regarded as the preceding vehicle in the right lane.

[0016] As is clear from FIG. 2, the lower side of the rectangular frame on the road plane in the input image 200 that has been distortion-corrected has distance information in the vertical direction. The pixel coordinates of the input image data (200) are composed of an x coordinate corresponding to the width direction position with the positive direction being to the right from the upper left origin and a y coordinate corresponding to the vertical direction position with the positive direction being downward from the origin. Since the number of pixels X in the width direction and the number of pixels Y in the vertical direction of the input image data (200) are known, by taking the difference (Y - y) from the number of pixels Y in the vertical direction, the pixel distance corresponding to the distance from the own vehicle (for example, h Li ,h l2 ,h C1 ) can be obtained.

[0017] Note that the central rectangular frame Ci is considered only when the pixel distance (h C1 ) from the host vehicle is on the proximal side where the threshold value or less, and otherwise, the rectangular frame on the left side with respect to the center line c may be classified into the left rectangular frame Li, and the rectangular frame on the right side may be classified into the right rectangular frame Ri.

[0018] The central rectangular frame Ci, the left rectangular frame Li, and the right rectangular frame Ri are arranged in the order of the rectangular frame with the lowest (nearest) vertical coordinate from the rectangular frame corresponding to the distance from the host vehicle, such as Ci = C1, C2, ···, Li = L1, L2, ···, Ri = R1, R2, ···.

[0019] As described later, the target for interrupt prediction is a rectangular frame located in the foreground or middle ground, and the rectangular frame in the background area is not the target for significant prediction. Therefore, the case where the pixel distance (h Li , h l2 , h C1 ) from the host vehicle is equal to or greater than a predetermined threshold value that can be regarded as the background area may be excluded in advance. This threshold value is obtained from the threshold value of the influence degree (R) described later.

[0020] Since the rectangular frame has a width corresponding to the vehicle width, even if the road has a certain curvature, it can be handled by extracting the central rectangular frame Ci by a straight line (center line c) as shown in FIG. 2, and the central rectangular frame Ci located in the foreground or middle ground is not misclassified to the left or right side.

[0021] On the other hand, as shown in FIG. 8, the road curvature can also be reflected in the center line c'. For example, in the input image 800, the left and right lane dividing lines mL and mR can be detected by lane recognition, and the center line c' can be dynamically set based on them.

[0022] Alternatively, the road curvature can be obtained from the detection information of the positioning system 21 and the map information 22, or from the steering angle (or yaw rate) of the vehicle. Assuming that the vehicle is in a steady circular turn, the steady circular turning radius can be obtained by the equation of motion and reflected in the center line c' of the input image 800 by affine transformation.

[0023] In addition, if no two or more rectangular frames are detected in either the left rectangular frame Li or the right rectangular frame Ri, the processing for one frame ends at that point. If two or more rectangular frames are detected in either the left rectangular frame Li or the right rectangular frame Ri, the loop processing (13L to 16L, 13R to 16R) of the rectangular frames is entered.

[0024] At this time, if at least one central rectangular frame Ci is detected in block 12C, in block 13C, the pixel distance (h C1 ) of the lowermost and nearest central rectangular frame C1 in the input image 200 is determined as the threshold for the end condition of the loop processing (13L to 16L, 13R to 16R) of the left rectangular frame Li and the right rectangular frame Ri described below.

[0025] Specifically, in the input image 200 shown in FIG. 2, although three rectangular frames L1, L2, and L3 are detected in the left region, only two rectangular frames L1 and L2 exist on the proximal side (the lower side is the lower side and the pixel distance is smaller than that of the central rectangular frame C1) of the nearest central rectangular frame C1. Therefore, the loop processing (13L to 16L) is executed for these rectangular frames L1 and L2.

[0026] Also, although two rectangular frames R1 and R2 are detected in the right region, only one rectangular frame R1 exists on the proximal side (the lower side is the lower side) of the nearest central rectangular frame C1. Therefore, the loop processing (13R to 16R) is executed only for this rectangular frame R1.

[0027] In the loop processing (13L to 16L, 13R to 16R), first, in blocks 14L and 14R, (a) the degree of overlap, (b) the distance difference rate, and (c) the influence degree, which are indicators for predicting interruptions, are calculated.

[0028] (a) The degree of overlap (IoMaxI; Intersection over Maximum Intersection) is an indicator that reflects the actual distance difference between the vehicle on the proximal side (the first vehicle) on each of the left and right sides and its preceding vehicle (the second vehicle). As shown in FIG. 3, the overlapping area (I ij) of the maximum volume or area (MaxI ij ) with respect to the next rectangular frame Lj is calculated, for example, by the following formula. Degree of polymerization: IoMaxI = (I ij ) / (MaxI ij ) However, for the pixel coordinates of the rectangular frames Li and Lj Li(x i1 , y i1 ; x i2 , y i2 ), Lj(x j1 , y j1 ; x j2 , y j2 ), in MaxI ij = min(x i2 - x i1 , x j2 - x j1 ) * min(y i2 - y i1 , y j2 - y j1 ) I ij = (max(x i1 , x j1 ) - min(x i2 , x j2 )) * (max((y i1 , y j1 ) - min(y i2 , y j2 ))

[0029] (b) The distance difference rate (DR; Distance Rate) is also an index that reflects the actual distance difference between the proximal vehicle (the first vehicle) on each side and its leading vehicle (the second vehicle), and is calculated, for example, by the following formula as the ratio of the vertical coordinates of the lower sides of the rectangular frames Li and Lj. Distance difference rate: DR = {(h Li ) / (h Lj )} 2

[0030] (c) Influence degree (R; or risk probability; Risk) is an index that reflects the magnitude of the impact on the host vehicle in the event of an interruption. The closer the interruption occurs to the host vehicle, the greater the impact, and the farther the position from the host vehicle, the smaller the impact. Therefore, for example, it is calculated by the following formula as the ratio of the vertical coordinate (h Li ) of the lower side of the rectangular frame Li to the vertical coordinate (hv) of the vanishing point VP. Influence degree: R = 1 - {(h Li ) / (hv)} 2

[0031] Note that the vertical coordinate (hv) of the vanishing point VP may be obtained through in-vehicle verification on flat ground, or may be calculated based on the camera mounting pitch angle and the camera vertical field of view angle as follows. Vanishing point height: hv = vertical center coordinate × (camera mounting pitch angle / camera vertical field of view angle)

[0032] Also, in relation to the above, it is also possible to obtain the road gradient based on the detection information of the satellite positioning system 21 and the map information 22, and to dynamically set the vanishing point height (hv) based on the road gradient. Furthermore, when reflecting the road curvature described above on the center line c′, it is also possible to correct it according to the vanishing point height (hv) set based on the road gradient.

[0033] Next, in blocks 15L and 15R, three indicators, namely (a) degree of overlap, (b) distance difference rate, and (c) influence degree, which are indicators for predicting interruptions, are each compared with their respective threshold values. Basically, when all indicators are equal to or greater than the threshold values, it is determined that there is a possibility of interruption (YES), and it is counted in the interruption prediction information of block 17.

[0034] If any of the indicators is less than the threshold value, it is not counted in the interruption prediction information, returns to blocks 13L and 13R, and based on the pixel distance (h C1 ) of the nearest central rectangular frame C1, it is confirmed whether the end condition is met. If the end condition is not met, the loop process of the next rectangular frame (13L to 16L, 13R to 16R) is performed.

[0035] When the end condition based on the pixel distance (h C1 ) of the nearest central rectangular frame C1 is satisfied, or when the central rectangular frame Ci is not detected in block 12C and the loop processing (13L~16L, 13R~16R) of all the rectangular frames Li and Ri on each of the left and right sides is completed, the process proceeds to the interrupt prediction information output in block 17.

[0036] Note that when it is determined that there is a possibility of an interrupt (YES) on either the left or right side, the loop processing (13L~16L, 13R~16R) in the current frame may be terminated immediately, and the process may immediately proceed to the interrupt prediction information output in block 17.

[0037] In the interrupt prediction information output 17, when there is counted interrupt prediction information, the interrupt prediction information designating either the left or right lane is output to the vehicle's HMI device 30. Alternatively, an interrupt prediction flag may be set when the interrupt prediction information is counted, and the interrupt prediction information may be output when the interrupt prediction flag continues for a predetermined number of frames or a predetermined time.

[0038] In the HMI device 30, a notification for alerting the driver to an interrupt is made by designating either the left or right lane through display on the screen, display on the HUD, etc., or by sound. Note that the interrupt prediction system 100 according to the present invention can be mounted on a drive recorder. In that case, the HMI device 30 is constituted by a display screen of the drive recorder or the like.

[0039] Also, when the vehicle is following a preceding vehicle while maintaining a predetermined set inter-vehicle time by adaptive cruise control (ACC), the set inter-vehicle time can be temporarily shortened to suppress an interrupt, so as to avoid a speed change due to the interrupt.

[0040] The interruption prediction according to the present invention is effective when driving on an arterial road or a highway with two or more lanes on one side at a medium speed or a high speed. On the contrary, it is not suitable for situations where the vehicle is driving at a low speed such as in traffic jams. Therefore, when it is detected from the detection information of the positioning system 21, the lane recognition from the map information 22 or the image of the front camera 20 that the vehicle is driving on a road with two or more lanes on one side, and the vehicle speed of the host vehicle is medium speed or higher (for example, 50 km / h or higher), the interruption prediction process may be executed, or the interruption prediction information may be output.

[0041] Next, FIGS. 4 to 7 show examples (input images) of interruption prediction according to respective situations, and determination examples corresponding thereto will be described below. Note that the input image size in each example is 1920×1080 pixels, and the threshold values of the degree of overlap (IoMaxI), the distance difference rate (DR), and the influence degree (R) in each example are all set to 0.4.

[0042] In the input image 400 shown in FIG. 4, the central rectangular frame and the right rectangular frame on the center line c are not detected, and two rectangular frames L1 and L2 are detected on the left side of the center line c. In this example, rectangular frame L1(x 11 ,y 11 ;x 12 ,y 12 )=(30,402;480,762), rectangular frame L2(x j1 ,y 21 ;x 22 ,y 22 )=(289,433;609,689), and in the rectangular frame L1, the degree of overlap with the rectangular frame L2: IoMaxI = 0.597, the distance difference rate: DR = 0.661, and the influence degree: R = 0.663, and it is determined that the possibility of interruption is high because all are equal to or higher than the threshold value of 0.4. This is a situation where the leading vehicle L1 is in front of the leading vehicle L2 in the left adjacent lane and immediately behind the leading vehicle L2, and it is predicted that the leading vehicle L1 is likely to cut in front of the host vehicle to avoid the leading vehicle L2.

[0043] <00> Similarly, neither the central rectangular frame nor the right rectangular frame is detected in the input image 500 shown in FIG. 5, and two rectangular frames L1 and L2 are detected on the left side. In this example, Rectangular frame L1(x 11 ,y 11 ;x 12 ,y 12 ) = (440, 423; 705, 635), Rectangular frame L2(x j1 ,y 21 ;x 22 ,y 22 ) = (0, 510, 288; 870), and in the rectangular frame L1, the overlap degree with the rectangular frame L2: IoMaxI = 0, the distance difference rate: DR = 0.223, and the influence degree: R = 0.849. Since the overlap degree and the distance difference rate are less than the threshold value of 0.4, it is determined that the possibility of intrusion is low. The leading vehicle L1 is traveling at a position close to the host vehicle, but an appropriate inter-vehicle distance is ensured between the leading vehicle L1 and the leading vehicle L2, and it is a situation where the possibility of intrusion is predicted to be low.

[0044] Similarly, two rectangular frames L1 and L2 are detected on the left side in the input image 600 shown in FIG. 6. In this example, Rectangular frame L1(x 11 ,y 11 ;x 12 ,y 12 ) = (775, 460; 875, 540), Rectangular frame L2(x j1 ,y 21 ;x 22 ,y 22 ) = (727, 487; 827, 567), and in the rectangular frame L1, the overlap degree with the rectangular frame L2: IoMaxI = 0.345, the distance difference rate: DR = 0.903, and the influence degree: R = 0.098. Since the overlap degree and the influence degree are less than the threshold value of 0.4, it is determined that the possibility of intrusion is low. The leading vehicle L1 is traveling at a position relatively close to the leading vehicle L2, but it is far ahead of the host vehicle, and even if it changes lanes in front of the host vehicle lane, the host vehicle will not be affected, so it is not counted as a target of intrusion prediction that requires attention.

[0045] In the input image 700 shown in FIG. 7, the central rectangular frame C1 on the center line c is detected, and two rectangular frames L1 and L2 are detected on the left side of the center line c in the same manner as in FIG. 4. Also in this example, the degree of overlap, the distance difference rate, and the influence degree between the rectangular frame L2 in the left rectangular frame L1 are all 0.4 or more, and in terms of calculation, it is a situation where the possibility of interruption is counted. However, the preceding vehicle C1 is traveling behind the preceding vehicle L2, and it cannot be said that it is a situation where an unreasonable cut-in occurs.

[0046] Therefore, if the start condition of the loop processing (13L to 16L, 13R to 16R) is limited to the case where two or more rectangular frames are detected on the nearer side (the lower side of the lower side) than the nearest central rectangular frame C1 in either the left or right region, it is possible to avoid the cut-in prediction determination in the situation as shown in FIG. 7.

[0047] Also, in such a case, it is also effective to set the threshold value of the duration of the above-described cut-in prediction flag. If it continues for a predetermined time or more even in the situation as shown in FIG. 7, it can be said that the possibility of executing a lane change becomes high.

[0048] In the above embodiment, the case where it is determined that there is a possibility of cut-in by the vehicle corresponding to the rectangular frame when the degree of overlap (IoMaxI), the distance difference rate (DR), and the influence degree (R) are each equal to or greater than the respective threshold values has been described. However, among these indexes, since both the degree of overlap and the distance difference rate are indexes that reflect the actual distance difference between the first vehicle and its preceding vehicle (the second vehicle), it is also possible to perform cut-in prediction based on either one and the influence degree.

[0049] That is, the system can also be configured to determine that there is a possibility of cut-in by the vehicle corresponding to the rectangular frame when the degree of overlap (IoMaxI) or the distance difference rate (DR) is equal to or greater than the threshold value and the influence degree (R) is equal to or greater than the threshold value.

[0050] Alternatively, the system can also be configured to determine that there is a possibility of cut-in by the vehicle corresponding to the rectangular frame when the distance difference rate (DR) is equal to or greater than the threshold value and the influence degree (R) is equal to or greater than the threshold value.

[0051] Although some embodiments of the present invention have been described above, it should be noted that the present invention is not limited to the above-described embodiments, and various modifications and changes can be made based on the technical idea of the present invention.

Explanation of Reference Numerals

[0052] 10 Image processing unit 20 Front camera (monocular camera) 21 Positioning system 22 Map information 23 Vehicle information 30 HMI device 100 Interruption prediction system

Claims

1. A vehicle intrusion prediction system, comprising: A monocular camera arranged to image the front of the vehicle; An image processing unit that processes the image captured by the monocular camera; The image processing unit is configured to: Extract, as a rectangular frame, a region recognized as a vehicle from the image; Classify the rectangular frame into a left rectangular frame located in the left region in the width direction of the image and a right rectangular frame located in the right region; When two or more rectangular frames are extracted in at least one of the left region and the right region, For the two or more rectangular frames, in order from the rectangular frame with the lowest (nearest) vertical coordinate, (a) The degree of overlap, which is the ratio of the possible maximum area of the overlapping area with the next rectangular frame; (b) The distance difference ratio, which is the ratio of the vertical coordinates of the lower sides of the rectangular frames; (c) The influence degree, which is the ratio of the vertical coordinate of the lower side to the vanishing point coordinate; Calculate these values, and based on them, determine the possibility of intrusion by the vehicle corresponding to the rectangular frame. A vehicle intrusion prediction system configured to execute such a process.

2. The image processing unit is configured to: Classify the rectangular frame into a central rectangular frame located in the central region in the width direction of the image, a left rectangular frame located in the left region, and a right rectangular frame located in the right region; When two or more rectangular frames are extracted in at least one of the left region and the right region and there is no central rectangular frame in the lower (nearer) part in the vertical coordinate of the image, For the two or more rectangular frames, in order from the rectangular frame with the lowest (nearest) vertical coordinate, Calculate the degree of overlap, the distance difference ratio, and the influence degree. The vehicle intrusion prediction system according to Claim 1.

3. When the degree of overlap, the distance difference ratio, and the influence degree are each equal to or greater than their respective threshold values, it is determined that there is a possibility of intrusion by the vehicle corresponding to the rectangular frame. The vehicle intrusion prediction system according to Claim 1 or 2.

4. The classification of the central rectangular frame, the left rectangular frame, and the right rectangular frame by the image processing unit includes the process of extracting, as the central rectangular frame, a rectangular frame whose center coordinate in the width direction of the host vehicle on the image is included in the width coordinate of the lower side, and classifying a rectangular frame whose width coordinate of the lower side is on the left side of the center line of the host vehicle as the left rectangular frame and a rectangular frame on the right side as the right rectangular frame. The vehicle intrusion prediction system according to Claim 2 or 3.

5. The image processing unit is configured to execute including a process of estimating a radius of curvature of a road based on at least one of matching of a satellite positioning system and map information, recognition of road division lines in the image, a steering angle, and a yaw rate, and correcting the own vehicle center line on the image based on the radius of curvature. The vehicle intrusion prediction system according to claim 4.

6. The image processing unit is configured to execute including a process of correcting the own vehicle center line based on a road gradient. The vehicle intrusion prediction system according to claim 5.

7. When it is determined by the image processing unit that there is a possibility of intrusion by a vehicle corresponding to the left or right rectangular frame and the vehicle speed is medium speed or higher, it is configured to notify the occupant. The vehicle intrusion prediction system according to any one of claims 1 to 6.

8. When the degree of polymerization or the distance difference rate is equal to or greater than a threshold value and the degree of influence is equal to or greater than a threshold value, it is determined that there is a possibility of intrusion by the vehicle corresponding to the rectangular frame. The vehicle intrusion prediction system according to claim 1 or 2.

9. The image processing unit is configured to execute the process in time series and notify the occupant when the state determined to have a possibility of intrusion continues for a predetermined time. The vehicle intrusion prediction system according to any one of claims 1 to 8.

Images