External world recognition device and external world recognition method

Abstract

The purpose of the present invention is to provide an external environment recognition device which, when a cutting-in vehicle cuts in between a preceding vehicle and a host vehicle, recognizes the cutting-in vehicle as a vehicle on the basis of side surface information of the cutting-in vehicle. The external environment recognition device is characterized by having: a side surface extraction unit which, on the basis of the parallax between left and right images captured of a portion of another vehicle approaching the host vehicle, extracts side surface information of the other vehicle from the images; a rear surface search unit that searches for the rear surface position of the other vehicle on the basis of the side surface information; a shape fitting unit which, on the basis of the side surface information and the rear surface position, fits a rectangular parallelepiped shape that approximates the other vehicle in the images; and an other vehicle recognition unit that recognizes the other vehicle as the vehicle in front of the host vehicle by using the rectangular parallelepiped shape.

Description

[Technical field]

[0001] The present invention relates to an external environment recognition device and an external environment recognition method used during vehicle control such as following a preceding vehicle. [Background technology]

[0002] In recent years, automobiles equipped with adaptive cruise control (ACC), which keeps a safe distance from the vehicle ahead according to the vehicle speed, are becoming more and more popular. As a prior art document related to this ACC, for example, there is Patent Document 1.

[0003] The abstract of Patent Document 1 states that the problem is to "provide an object detection method and object detection device that quickly determine whether or not a vehicle is cutting in," and states that the solution is to "provide an object detection method and object detection device that quickly determine whether or not a vehicle is cutting in." The abstract also states that the solution is to "provide an object detection device 100 that includes stereo cameras 10, 11 that capture images of the surroundings of a vehicle 50, and a controller 20. The controller 20 extracts a plurality of feature points from the images, and extracts a plurality of feature points that constitute a moving object from the plurality of feature points. The controller 20 calculates the variance of motion vectors of the plurality of feature points that constitute a moving object from a plurality of images captured at different times. If the variance of the components of the motion vector in the road width direction is equal to or greater than a threshold, the controller 20 determines that the moving object may cut in to the lane in which the vehicle is traveling."

[0004] Furthermore, paragraph 0014 of Patent Document 1 explains that "the optical flow calculation unit 22 calculates an optical flow from feature points of a past image and feature points of a current image. Optical flow is a vector that represents the movement of an object in a stereo image. The optical flow calculation unit 22 detects, as related feature points, current feature points that correspond to the same object in real space as an object corresponding to a past feature point. The optical flow calculation unit 22 calculates a combination of mutually related past feature points and current feature points as an optical flow."

[0005] In this way, the object detection device of Patent Document 1 recognizes another vehicle that cuts in front of the host vehicle by utilizing optical flow in stereo images. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2020-3971 A Summary of the Invention [Problem to be solved by the invention]

[0007] Paragraph 0030 of Patent Document 1 states that "When the cut-in determination unit 28 determines that there is a possibility that another vehicle will cut in, the vehicle control unit 30 brakes the vehicle so that the occupants do not feel uncomfortable. Also, the vehicle control unit 30 may brake the vehicle so that the occupants do not feel uncomfortable even when the cut-in determination unit 28 determines that another vehicle will cut in." That is, in Patent Document 1, when there is a possibility that another vehicle will cut in or when it is determined that another vehicle will cut in, the vehicle control unit 30 brakes the vehicle regardless of the manner of the cut-in. However, even if there is an approaching vehicle, if it is not possible to accurately determine whether or when there will be contact based on the contour shape or speed vector of the approaching vehicle, it is clear that the control will give the occupants a sense of discomfort.

[0008] Furthermore, in paragraph 0004 of Patent Document 1, it is stated that "When speed control is performed using a preceding vehicle following control or the like, if the cut-in determination is delayed, sudden deceleration may occur, which may cause passengers to feel uncomfortable. " However, there is no specific mention of how to achieve control to follow the preceding vehicle without causing discomfort to the occupants.

[0009] Therefore, the present invention aims to provide an external environment recognition device and an external environment recognition method that recognizes a cutting-in vehicle as a vehicle based on side information of the cutting-in vehicle when the cutting-in vehicle cuts in between the preceding vehicle and the own vehicle. It also aims to provide an external environment recognition device and an external environment recognition method that brakes the own vehicle if there is a possibility of contact with the cutting-in vehicle, and switches the vehicle to be followed in the preceding vehicle following control from the previous preceding vehicle to the cutting-in vehicle if there is no possibility of contact, thereby allowing the vehicle to continue stable driving. [Means for solving the problem]

[0010] An external environment recognition device comprising: a side extraction unit that extracts side information of another vehicle from an image based on a parallax between left and right images of a portion of another vehicle approaching a host vehicle; a rear search unit that searches for a rear position of the other vehicle based on the side information; a shape fitting unit that fits a rectangular parallelepiped shape that approximates the other vehicle in the image based on the side information and the rear position; and an other vehicle recognition unit that recognizes the other vehicle as a vehicle ahead of the host vehicle using the rectangular parallelepiped shape. Effect of the Invention

[0011] According to the present invention, when a vehicle cuts in between a preceding vehicle and the own vehicle, the vehicle can be recognized as a vehicle based on side information of the cutting in. If there is a possibility of contact with the cutting in, the own vehicle is braked, whereas if there is no possibility of contact, the vehicle to be followed in the preceding vehicle following control is switched from the previous preceding vehicle to the cutting in, thereby allowing the vehicle to continue stable driving. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a front view of a vehicle equipped with an external environment recognition device according to an embodiment of the present invention; [Diagram 2] Functional block diagram of an external environment recognition device according to an embodiment. [Figure 3A] A bird's-eye view of a scene where a vehicle cuts in between the vehicle ahead and the vehicle in question [Figure 3B]The environment ahead of the vehicle when a car cuts in between the vehicle ahead and the vehicle itself [Figure 4] An example of UD space after voting for disparity values [Figure 5A] An explanatory diagram of three-dimensional object candidates extracted from the UD space in Fig. 4 [Figure 5B] An explanatory diagram of a detection frame set in an image based on the three-dimensional object candidate in FIG. 5A. [Figure 6] Functional block diagram of the preceding vehicle identification unit [Figure 7] Functional block diagram of side surface extraction unit [Figure 8] An explanatory diagram of the line segments parallel to the vehicle path extracted from the UD space in Figure 4. [Figure 9] Illustration of the procedure for extracting side candidate lines [Figure 10] Functional block diagram of right-angle fitting [Figure 11] Illustration of side end fitting procedure [Figure 12] Illustration of fitting procedure for rear side [Figure 13A] An explanatory diagram of the side-based rear search procedure when capturing an image of a part of a cutting-in vehicle [Figure 13B] An explanatory diagram of the side-based rear search procedure when capturing an image of the entire cutting-in vehicle [Figure 14] An illustration of how the vehicle outline appears on the image [Figure 15] Illustration of the back-based side-searching procedure [Figure 16] Detailed shape fitting procedure diagram [Figure 17] Diagram of the procedure for tracking feature points within a contour and estimating speed [Figure 18] Functional block diagram of the cutting-in vehicle determination unit [Figure 19] Functional block diagram of the collision tracking determination unit [Figure 20] An overhead view of an example of an environment in which collision tracking determination is performed. [Figure 21] Processing flow chart of an external environment recognition device according to an embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the external environment recognition device of the present invention will be described with reference to the drawings.

[0014] <External world recognition device 100> FIG. 1 is a front view of a host vehicle V0 to which an external environment recognition device 100 according to an embodiment of the present invention is attached, and FIG. 2 is a functional block diagram of the external environment recognition device 100 of FIG.

[0015] As shown in the front view of FIG. 1, the external environment recognition device 100 is attached to the upper inner surface of the windshield of the host vehicle V0, and is a device for recognizing the environment ahead of the host vehicle V0 during vehicle control such as adaptive cruise control (ACC).

[0016] As shown in the functional block diagram of FIG. 2, the external environment recognition device 100 includes a stereo camera unit 1, a parallax voting unit 2, a back surface extraction unit 3, a preceding vehicle identification unit 4, a contact tracing determination unit 5, a side surface extraction unit 6, a right-angle shape fitting unit 7, and an intrusion vehicle determination unit 8. In addition, among the components of the external environment recognition device 100, each unit other than the stereo camera unit 1 is specifically a computer unit equipped with hardware such as a calculation device such as a CPU, a storage device such as a semiconductor memory, and a communication device. The calculation device executes a program stored in the storage device to realize each of the above-mentioned functions, and the details of each unit will be described in sequence below while appropriately omitting such well-known technologies in the computer field.

[0017] <Stereo camera unit 1> The stereo camera unit 1 is composed of a left camera 1L and a right camera 1R facing the front of the vehicle, and an image processing unit. The image processing unit performs a well-known stereo matching process on the left image PL and the right image PR captured by the image pickup elements of each camera to generate a parallax image (information in which the parallax values ​​D of the left and right images are arranged in two dimensions), and outputs this as three-dimensional information in front of the vehicle. In this embodiment, the stereo camera unit 1 is used to generate the three-dimensional information, but instead of the stereo camera unit 1, other types of front sensors such as a monocular camera or Lidar may be used to generate the three-dimensional information.

[0018] <Parallax voting section 2> The disparity voting unit 2 votes for the disparity value D, which is the result of stereo matching in the stereo camera unit 1, in a disparity voting space (hereinafter referred to as "UD space") with the vertical axis representing the disparity value D and the horizontal axis representing the image horizontal coordinate U. By applying the principle of triangulation, the disparity value D can be converted into the depth distance to the captured object, so that the vertical axis of the UD space shown in FIG. 4 and other figures indicates not only the magnitude of the disparity value D but also the near or far depth distance.

[0019] For example, as illustrated in the overhead view of Figure 3A, if a cutting-in vehicle V2 cuts in between the host vehicle V0, which is undergoing control to follow the preceding vehicle, and the preceding vehicle V1, which is the current vehicle to be followed, from the left lane, the environment in front of the host vehicle V0 will appear as illustrated in Figure 3B.

[0020] Fig. 4 shows a UD space in which the disparity value D of the disparity image output by the stereo camera unit 1 is voted in the environment of Fig. 3B. The disparity voting unit 2 completes the UD space as shown in Fig. 4 by voting the disparity value D for each pixel row in the UD space while scanning the disparity value D vertically, one pixel row at a time (or multiple pixel rows at a time) in sequence from one end of the disparity image. When the disparity image includes a three-dimensional object, the disparity value D is approximately the same in the portion of the three-dimensional object at the same depth distance, so that in the vertical axis direction of each row in the UD space, the votes for the disparity value D are concentrated near a certain value. In Fig. 4, the vote concentration of the disparity value D is displayed with high luminance (dark white). In the UD space of Fig. 4 generated under the environment of Fig. 3B, vote concentrations of the disparity value D corresponding to the depth of each occur at the positions of the horizontal axis (image horizontal coordinate U) corresponding to the back of the preceding vehicle V1, the side of the cutting-in vehicle V2, and the back of the cutting-in vehicle V2, respectively, resulting in high-luminance areas (dark white areas) being expressed in three places in the UD space. Note that, as shown in Fig. 3B, a curb is installed on the right side of the host vehicle V0, so a vote concentration (inclined light white area) of the disparity value D caused by the curb is also expressed on the right side of the UD space in Fig. 4.

[0021] <Backside extraction part 3> The back surface extraction unit 3 detects a three-dimensional object based on the UD space (Fig. 4) obtained by the parallax voting unit 2. For a wide three-dimensional object that moves parallel to the host vehicle V0, such as a preceding vehicle V1 or an oncoming vehicle (not shown), the vote concentrations of the parallax value D are aligned horizontally in the UD space, so the back surface extraction unit 3 can detect a three-dimensional object candidate (back surface) by scanning the UD space horizontally and extracting a high-brightness area that is long in the horizontal direction. Below, an overview of the processing by the back surface extraction unit 3 will be explained using Figs. 5A and 5B.

[0022] Fig. 5A shows an area where vote concentrations (dark white) of disparity value D are continuous in the horizontal direction, extracted as a solid object candidate by the back surface extraction unit 3 scanning the UD space in Fig. 4 in the horizontal direction. Note that the UD space in Fig. 5A originally includes the luminance information shown in Fig. 4, but in order to emphasize the subject of explanation in Fig. 5A, the luminance information is omitted in Fig. 5A and only information not in Fig. 4 is displayed. This treatment is the same in UD space such as Figure 8 described later.

[0023] When the three-dimensional object candidates shown in FIG. 5A are extracted, the back surface extraction unit 3 can correctly extract three-dimensional objects such as the back surface of the preceding vehicle V1 moving parallel to the host vehicle V0, as shown in the detection frame in the image in FIG. 5B. On the other hand, for a three-dimensional object such as the cutting-in vehicle V2 moving in a direction not parallel to the host vehicle V0, the back surface extraction unit 3 may erroneously detect the back surface and side surface of the cutting-in vehicle V2 in the image as independent three-dimensional objects. In addition, when the traveling direction (appearance) of the cutting-in vehicle V2 gradually changes in each frame continuously captured by the stereo camera unit 1, the width and ratio of the two three-dimensional object candidates (see FIG. 5A) caused by the cutting-in vehicle V2 gradually change, so that the back surface extraction unit 3 extracts two three-dimensional objects whose size, position, attitude, speed, etc. are unstable.

[0024] Here, the back surface extraction unit 3 tracks the detection frame (dotted line in FIG. 5B) of the three-dimensional object extracted in each frame of the continuous captured images in a time series, and when it is tracked as a stable horizontal position and depth, it is determined to be a stable three-dimensional object. Then, only the object that can be tracked stably as a three-dimensional object in a time series is used for the subsequent processing, and the object that is not stable in a time series is excluded as a noise factor. For this reason, the detection frame for an unstable object such as the side of the cutting-in vehicle V2 is not stable in a time series and is not used as a three-dimensional object. Also, as for the back surface, the width is unstable to a certain extent, and it may not be detected, or even if it can be tracked to a certain extent, the position, attitude, speed, etc. are unstable, and it is often not the correct target of warning or control. In order to suppress false warnings and erroneous control, it is necessary to design it so that control is not performed on objects with unstable positions and speeds.

[0025] In this way, in the case of three-dimensional object extraction by the back surface extraction unit 3, it was difficult to accurately recognize the cutting-in vehicle V2, whose back surface and part of the side surface are captured simultaneously, and in particular, because the cutting-in vehicle V2 cuts in at a steep angle from close range during traffic congestion, etc., and therefore cannot be captured in its entirety.

[0026] <Preceding vehicle identification unit 4> The preceding vehicle identification unit 4 selects three-dimensional objects that can be stably extracted in time series from among the three-dimensional object candidates extracted by the back surface extraction unit 3 (for example, three-dimensional object candidates caused by the preceding vehicle V1 in FIG. 5A), and further performs a process of determining whether or not one of these is a vehicle based on its size and appearance in the image. To perform this process, the preceding vehicle identification unit 4 has a tracking unit 41, a vehicle back surface candidate selection unit 42, a vehicle identification unit 43, and a position and speed estimation unit 44, as shown in FIG. 6. Each unit will be described in turn below.

[0027] First, the tracking unit 41 performs a tracking process on a three-dimensional object whose position and size are stable over time in continuously captured images, and calculates the position and speed in a simplified manner.

[0028] Next, the vehicle back surface candidate selection unit 42 selects, from the three-dimensional objects being tracked, three-dimensional objects having a size and speed that could possibly be that of a vehicle, and selects vehicle back surface candidates.

[0029] The vehicle identification unit 43 performs an identification process on the selected three-dimensional object candidates that are the size of the rear face of a vehicle, and identifies whether they are cars or not. Specifically, the identification process is performed for each frame, and a three-dimensional object that has a high identification score for several consecutive frames is finally identified as a vehicle.

[0030] The position and speed estimation unit 44 estimates with high accuracy the position and speed of a three-dimensional object identified as a vehicle.

[0031] 3B, other vehicles traveling in the same lane as the host vehicle V0, or other vehicles traveling in an adjacent lane but at a certain distance and whose sides are not easily visible, can be stably captured from the rear and can be identified as vehicles by the preceding vehicle identification unit 4. In addition, for the cutting-in vehicle V2 at a long distance, it was also possible to handle the cutting-in vehicle V2 using the output of the preceding vehicle identification unit 4, since the possibility of a collision is low even if there is some recognition delay.

[0032] However, when a vehicle V2 cuts in at a steep angle from a close distance during a traffic jam, the back surface extraction unit 3 may not be able to extract the vehicle as a three-dimensional object and may not be able to identify it properly, so the vehicle ahead identification method based on the back surface of the vehicle ahead identification unit 4 often does not work well. In order to accurately recognize the vehicle V2 cutting in under such circumstances, the external environment recognition device 100 of this embodiment has an additional side surface extraction unit 6 and a right-angle shape fitting unit 7, which will be described below.

[0033] <Side extraction part 6> The side surface extraction unit 6 searches for side surface candidates of the cutting-in vehicle V2 by using the UD space exemplified in Fig. 4. To execute this process, the side surface extraction unit 6 has a side surface candidate line search unit 61, a vehicle candidate determination unit 62, a tracking unit 63, and a side surface speed analysis unit 64, all of which are shown in Fig. 7. Each unit will be described in turn below.

[0034] First, the side surface candidate line search unit 61 searches for a line segment extending along the host vehicle path R0 (a line segment within a predetermined angle with respect to the host vehicle path R0 on the overhead view) as shown in Fig. 8 from the high luminance area on the UD space in Fig. 4. In the example of Fig. 8, a line segment caused by the side of the cutting-in vehicle V2 is searched for on the left side of the host vehicle path R0, and a line segment caused by a curb is searched for on the right side of the host vehicle path R0.

[0035] In the search process here, the search target is a line segment inclined at about ±50° with respect to the vehicle's path R0 so that the side of the cutting-in vehicle V2 that cuts into the vehicle's path R0 can be searched. Note that this angle of "50°" does not refer to the angle in the UD space of FIG. 4 or the angle on the actual image as in FIG. 3B, but refers to the actual relative attitude of the vehicle on the overhead view (see FIG. 3A). The reason for limiting the angle of the line segment to be searched is that even in the process of searching the back, a certain degree of rear inclination is supported. In addition, for vehicles that move in a direction at an angle of 90° with respect to the vehicle V0 on the overhead view, such as completely jumping out from the side, separate measures have been taken as a response to head-on accidents, so they are not searched for in this embodiment. Note that, in the following, it is assumed that the cutting-in vehicle V2, including motorcycles, is detected, and line segments present on the side of the vehicle are searched for, for example, with a length of 1.5 m to 15 m. In the UD space, there are many three-dimensional objects other than the vehicle on the side, such as road shoulders and building walls, so multiple sides can be searched. However, when considering the safety of the vehicle V0, it is sufficient to consider only the closest object.

[0036] For this reason, the side surface candidate line search unit 61 detects the approximate lateral positions where the nearby cutting-in vehicle V2 is likely to cut in, separately for the left and right sides, from the UD space in FIG. 4, according to the procedure shown in FIG.

[0037] First, as shown in Fig. 9(1), many lines parallel to the vehicle's path R0 on the overhead view are set in the UD space (Fig. 4). Next, as shown in Fig. 9(2), a histogram is generated that accumulates the number of votes for the disparity value D on each line. At a position where a three-dimensional object with sides (cutting-in vehicle V2) is located (near the dashed-dotted arrow in Fig. 9(1)), the accumulated number of votes on the line is greater than anywhere else even if the angle of the side is slightly off. Therefore, in the histogram, a peak (maximum value) of the accumulated number of votes occurs at the position on the horizontal axis (abscissa of the image U) that corresponds to this line.

[0038] In FIG. 9(3), a peak with an accumulated number of votes equal to or greater than a threshold value is searched for from the vehicle path R0 (center) outward. For example, in an environment in which the side of the cutting-in vehicle V2, the road shoulder, and a wall are present, in that order, on the left side of the vehicle path R0, three peaks will be lined up on the left side of the histogram. However, since the process here is for searching for a three-dimensional object that comes into contact with the vehicle V0 or a three-dimensional object that directly affects the vehicle behavior of the vehicle V0, when a peak close to the center of the histogram (a peak caused by the side of the cutting-in vehicle V2) is extracted, the search process outward from that point may be stopped, and it may be determined that the peak is highly likely to be a three-dimensional object with a side, and a detailed analysis of this peak may be performed.

[0039] Next, in FIG. 9(4), a line segment (a dashed line arrow in FIG. 9(1)) that corresponds to the approximate position of the line segment extending along the vehicle's path R0 in the UD space is identified from the position of the peak of the histogram selected in FIG. 9(3). After that, in FIG. 9(5), a number of line segments with different positions and angles are set around the line segment selected in FIG. 9(4) as a reference, and a histogram is generated by accumulating the number of votes for the disparity value D for each line segment, as in FIG. 9(2). Then, the line with the largest number of accumulated votes on the line (for example, the dashed line arrow in FIG. 9(5)) is extracted as a candidate side line.

[0040] Next, the vehicle candidate determination unit 62 provisionally determines the length of the side based on the extracted side candidate line. For example, if the vote value on the side candidate line is greater than or equal to a certain threshold three times in a row, the first vote is set as the starting point of the line, and if the vote value falls below the threshold twice along the way, the line is considered to be continuing, the length is checked by tracing the line, and if the length falls below the threshold three times in a row, the point before the three consecutive votes is registered as the end point of the line. If the length of the line obtained in this way is 1.5 m or more, which corresponds to the side length of a two-wheeled vehicle, and 15 m or less, which corresponds to the side length of a large truck, the line is extracted as a vehicle candidate. On the other hand, if the length is outside this range, it is excluded from the vehicle side candidate line as being noise, a road shoulder, a wall, or the like. Furthermore, if it is determined to be a vehicle side candidate line, it is assumed that the length is 1 m or more and that the effective parallax is about 40% or more, and it is confirmed that the vote amount on the line changes depending on the depth, but is 40% of the vote amount assuming a height of 1 m. Some areas that fall below the threshold are acceptable, and if the average exceeds the 40% threshold, the vehicle is considered a candidate. In situations where there are not enough votes, the object is determined to be a low object from the road surface, such as a road shoulder, and is not a vehicle, including motorcycles.

[0041] The tracking unit 63 performs tracking of the side candidate lines selected by the vehicle candidate determination unit 62. Once tracking of the side candidate lines has begun, processing is performed to reduce interruptions in tracking due to tracking failures by including line segments that do not satisfy the length or height criteria set by the vehicle candidate determination unit 62 as tracking targets.

[0042] The side speed analysis unit 64 calculates the speed (depth direction speed, lateral movement direction speed) of the side candidate line tracked by the tracking unit 63. Furthermore, the side speed analysis unit 64 performs stabilization in a time series manner, and judges for each frame whether the extracted side candidate line is approaching the host vehicle V0 and whether it is an approaching side even when the lateral movement of the host vehicle behavior is taken into account.

[0043] <Right-angle fitting part 7> The right-angle fitting unit 7 precisely extracts the contour shape of the cutting-in vehicle V2 while reducing the computational processing load by fitting a right-angle shape only to the side candidate lines extracted by the side extraction unit 6. The right-angle fitting unit 7 also calculates the position and speed of the cutting-in vehicle V2, and determines whether the cutting-in vehicle V2 should be monitored in order to continue safe and comfortable driving of the host vehicle V0. Here, the cutting-in vehicle V2 that is determined to be a monitoring target mainly includes two types: (1) a cutting-in vehicle V2 that may come into contact with the host vehicle V0 unless the host vehicle V0 takes measures such as emergency braking or emergency avoidance, and (2) a cutting-in vehicle V2 that may approach to a distance less than a safe distance unless the host vehicle V0 decelerates. 10, the right-angle shape fitting unit 7 has a side leading edge fitting unit 71, a side trailing edge fitting unit 72, a side-based back surface searching unit 73, a detection base line switching unit 74, a back surface-based side surface searching unit 75, a detailed shape fitting unit 76, a feature point tracking / speed estimation unit 77, and a vehicle identification unit 78. Each unit will be described in turn below.

[0044] Side end fitting 71 While the side surface extraction unit 6 roughly extracts the side length of the cutting-in vehicle V2, the side surface tip fitting unit 71 searches for the tip of the cutting-in vehicle V2 in detail. To this end, the side surface tip fitting unit 71 executes a method of finding the tip based on distance using parallax, or a method of finding the tip based on texture.

[0045] First, a method for re-searching the front end of the cutting-in vehicle V2 based on the distance using parallax will be described with reference to FIG. 11. In this method, as shown in FIG. 11, the depth of the front end of the cutting-in vehicle V2 on the original image is measured using the parallax image with the side end position (initial value) extracted in the UD space as a reference. At this time, the vehicle body front end position is specified by using the difference in depth between the vehicle body and the road surface in the parallax image. As shown in the upper diagram of FIG. 11, when checking the depth on the dashed dotted line that crosses near the bumper of the cutting-in vehicle V2 on the original image (e.g., corresponding to a height of 50 cm from the bottom end of the cutting-in vehicle V2), a parallax value D equivalent to the depth distance to the side of the vehicle body is calculated on the left side of the dashed dotted line, whereas a parallax value D equivalent to the depth distance to the road surface that is farther away than the side of the vehicle body is calculated on the right side of the dashed dotted line, so that there is a large difference in the parallax value D (depth distance) between the left and right of a certain position. By extracting the boundary where this difference occurs as a more accurate position of the vehicle's front end, the accurate front end of the cutting-in vehicle V2 can be searched for.

[0046] On the other hand, when the above method cannot be used, for example, when only a small part of the cutting-in vehicle V2 is included in the imaging range and the texture of the cutting-in vehicle V2 is insufficient, or when it is nighttime and the texture of the cutting-in vehicle V2 is dark and cannot be clearly imaged, normal stereo matching cannot be performed, and invalid parallax occurs frequently, so that fitting of the tip position described above may not be performed. In such cases, not only the parallax value but also the texture information is effectively utilized. Specifically, a method of correcting the tip of the vehicle body by utilizing a method of identifying the boundary position from the difference between the road surface texture and the vehicle body texture may be used.

[0047] The method for separating by texture may be a method for extracting boundaries based on differences in brightness or color information, a method for dividing similar textures into small regions, or a method for utilizing semantic segmentation. A method for improving stability by using both parallax and brightness may also be utilized.

[0048] Rear side fitting 72 In the side rear end fitting unit 72, the rear end position is fitted again to the line segment of the side of the cutting-in vehicle V2 that has already been extracted. Even if the cutting-in vehicle V2 cuts in front of the host vehicle V0, it basically cuts in while running parallel to the host vehicle V0. Therefore, if the rear end of the cutting-in vehicle V2 comes into the image captured by the stereo camera unit 1 of the host vehicle V0 to some extent, the parallax of at least the rear end (rear) part of the vehicle should be visible as parallax vote values ​​(high brightness area) that are aligned almost horizontally in the UD space. Of course, if the angle of cutting in is too sharp, this is not the case and the voting line may be slightly inclined, but basically the vehicle rear parallax is aligned horizontally in the UD space.

[0049] As shown in the lower diagram of Fig. 12, an analysis is performed in which the voting values ​​are accumulated horizontally for the rear end of the candidate line on the side of the vehicle that was extracted in advance, and a histogram is generated. When generated as a histogram, the rear part where the voting values ​​accumulate horizontally becomes the peak of the accumulated value. For this reason, this peak part is extracted as the detailed rear position. As a result of fitting, the extension of the candidate line segment on the side and the intersection of the histogram peak are extracted as white square points, and this is used as the rear end fitting result. If the cut-in angle is sharp and the histogram peaks are relatively dispersed, the current correction value can be treated as provisional, with emphasis placed on correction in later processing.

[0050] Side-based rear search section 73 The side-based rear search unit 73 searches for a rear line segment starting from the position of the rear end of the side surface in the UD space extracted by the side rear end fitting unit 72. For example, FIG. 13A is an explanatory diagram of the side-based rear search procedure when a part of the cutting-in vehicle V2 is imaged, and FIG. 13B is an explanatory diagram of the side-based rear search procedure when the entire cutting-in vehicle V2 is imaged. As shown in these figures, the angle is changed to search while assuming some angular deviation of the side line segment centered on the position that is perpendicular to the inclination of the side surface in the real space (on the bird's-eye view). Line scanning is performed by changing the angle and vertical position centered on the rear end position of the side surface of each view, which is the starting point for starting the search. In the line scan, the vote values ​​on the line are accumulated in the UD space to generate a histogram. Therefore, the accumulated vote values ​​are highest on the line drawn along the actual rear surface of the vehicle, so the vehicle rear position can be searched for by identifying the line with the highest accumulated value from the histogram. Since the position of the rear edge of the side surface itself may contain an error, by searching while changing the position and angle using this position as a reference, the error of the rear edge position of the side surface itself can be absorbed, and even if the angle of the side surface is misaligned, it is possible to correctly and stably search for the line segment of the back surface in UD space. It is also possible to confirm that the rear edge of the side surface is almost not within the screen by checking that the accumulated value of the histogram does not become high even if the position of the rear edge of the side surface is at the edge of the screen.

[0051] Detection base line switching unit 74 The detection base line switching unit 74 determines whether it is more stable to perform detection processing of the cutting-in vehicle V2 shown in the image based on the back or on the side. For more stable detection of the cutting-in vehicle V2, it is better to use a longer line segment on the image as the base line, and by using the longer base line as a clue to search for the other line segment that is difficult to see in the perpendicular direction, the vehicle's outline shape (length and position of the back and side) can be stably detected. Therefore, the detection base line switching unit 74 determines whether to use the back or side as the base.

[0052] Here, the left diagram of FIG. 14 is shown as an example of a situation where it is more stable to detect the contour of the cutting-in vehicle V2 based on the side, and the right diagram of FIG. 14 is shown as an example of a situation where it is more stable to detect the contour of the cutting-in vehicle V2 based on the back. As shown in the upper left diagram of FIG. 14, when the cutting-in vehicle V2 cuts in just before the host vehicle V0, the length of the side of the cutting-in vehicle V2 is longer than the length of the back in the image captured by the stereo camera unit 1 (lower left diagram of FIG. 14). Therefore, it is considered that the contour detection of the cutting-in vehicle V2 is stable if the side is used as the base. On the other hand, as shown in the upper right diagram of FIG. 14, when the cutting-in vehicle V2 cuts in from a distance, the length of the back of the cutting-in vehicle V2 is longer than the length of the side in the image captured by the stereo camera unit 1 (lower right diagram of FIG. 14). Therefore, it is considered that the contour detection of the cutting-in vehicle V2 is stable if the back is used as the base.

[0053] As shown in FIG. 14, if the exact contour shape (length of the back and length of the side) of the cutting-in vehicle V2 is known on the image captured by the stereo camera unit 1 of the vehicle V0 (see the lower left or right diagram of FIG. 14), it is possible to calculate which of the side and the back is longer on the image. However, before the back and the side are specified, a process is required to select either the back or the side as the base line. For this reason, the detection base line switching unit 74 estimates the rough center of gravity position of the cutting-in vehicle V2 at the time when only the back of the cutting-in vehicle V2 is specified or when only the side of the cutting-in vehicle V2 is specified, and selects which of the estimated center of gravity and the angle of the stereo camera unit 1 is used as the base for extracting the contour shape. In this case, the center of gravity position may be estimated using a standard vehicle size that serves as a reference from the extraction results of the side and the back, and their relative positions. Note that this only roughly determines whether the back or side should be used as the base to stabilize the overall processing, so if the back and side are roughly the same length, the results will not differ significantly regardless of which is used as the base for the search.

[0054] Contour detection performance deteriorates significantly when, as shown in FIG. 15, the side of the cutting-in vehicle V2 is not clearly visible and the back surface is extracted based on the side surface. In this case, the back surface is searched for based on the unstable side surface extraction result, making it difficult to finally obtain a stable contour shape (length of the back surface and side surface). Conversely, for a cutting-in vehicle V2 whose entire back surface is not captured in the image as shown in FIG. 13A, or a cutting-in vehicle V2 whose side surface appears longer than its back surface due to a sharp cutting-in angle as shown in FIG. 13B, if the back surface is used as a base, the side surface will be extracted from the unstable back surface, and as a result, the searched side surface will also tend to be unstable and unreliable.

[0055] For this reason, the detection base line switching unit 74 may estimate the approximate center of gravity position of the cutting-in vehicle V2 under various conditions, such as assuming that the cutting-in vehicle V2 is about the same size as a general passenger car, or assuming that the cutting-in vehicle V2 is traveling parallel to the host vehicle V0, and then select whether the back or side is to be used as the base line based on the angle calculation result between the center of gravity position and the installation position of the stereo camera unit 1 of the host vehicle V0. Alternatively, the detection base line switching unit 74 may switch the base line for the detection process using the extracted length of the side or back, and the angle between the center of gravity and the host vehicle V0.

[0056] In addition, in a case where a cutting-in vehicle V2 that has begun to be tracked gradually moves in front of the vehicle and eventually becomes the preceding vehicle V1 of the vehicle V0, the detection base line switching unit 74 may switch from side-based detection processing to rear-based detection processing at a certain timing.

[0057] -Rear base side search section 75 The rear-based side search unit 75 uses the searched rear inner end position to search for the side line segment in the UD space. As shown in the lower diagram of FIG. 15, the side is searched for by changing the angle in the UD space, assuming some angle deviation of the rear line segment, centered on the position that is perpendicular to the inclination of the rear of the cutting-in vehicle V2 on the bird's-eye view. Furthermore, taking into account the positional deviation of the rear inner end, the lateral position is also changed and searched for. Line scanning is performed by changing the angle and position with the angle at the center being a right angle to the rear, centered on the rear inner end in the lower diagram of FIG. 15, which is the reference position for starting the search. In line scanning, a line is drawn in the UD space, and the voting values ​​on the line are accumulated to generate a histogram. Therefore, the line drawn to trace the side part of the vehicle with a high voting value has the highest cumulative value, and the vehicle side position can be searched for. Since the rear inner position itself may have an error, by searching by changing the position and angle, it is possible to absorb the deviation of the rear inner position and stably search the side line segment correctly in the UD space.

[0058] Detailed shape fitting part 76 In determining a collision with the cutting-in vehicle V2, it is important to accurately estimate the rear corner position of the cutting-in vehicle V2. This is because, although a collision or contact with the cutting-in vehicle V2 can occur at the rear, rear corner, side, or front corner, and while a collision or contact with the rear can be avoided by using conventional technology to avoid rear-end collision with the preceding vehicle V1, a collision or contact with the rear corner, side, or front corner cannot be avoided by the conventional technology, so that accurately estimating the positions of the side and front corners based on an accurately estimated rear corner position is effective in avoiding a collision or contact with the rear corner, side, or front corner of the cutting-in vehicle V2.

[0059] Although the general position of the side can be estimated by any of the above methods, if the cutting in vehicle V2 is a vehicle with rounded corners, or if the vehicle has a body shape or color that makes it difficult to calculate parallax, the length of the side line segment (i.e., the positions of the front and rear corners) may not be accurately estimated. Therefore, the detailed shape fitting unit 76 assumes that the cutting in vehicle V2 has a rectangular outline on the overhead view, fits the vehicle's outer frame, and also accurately extracts the positions of the rear corners.

[0060] FIG. 16 is a diagram for explaining an example of a procedure for fitting an outline frame by the detailed shape fitting unit 76. First, as shown in FIG. 16(a), the parallax of the side imaged by the stereo camera unit 1 is used to calculate the normal at each point on the side, and the parallax of the rear imaged by the stereo camera unit 1 is used to calculate the normal at each point on the rear. This allows a large number of normals to be drawn on the side and rear of the cutting-in vehicle V2. Next, as shown in FIG. 16(b), a perpendicular line on the overhead view to the rear normal and a perpendicular line on the overhead view to the side normal are calculated, pairs of side perpendicular lines and rear perpendicular lines are selected in sequence, and the intersections of each pair are calculated. Then, a point where the intersections are concentrated near the rear corner of the cutting-in vehicle V2 is calculated, and this part is extracted as the rear end corner. This method allows the position of the rear corner to be accurately extracted while removing the influence of missing parallax and errors in the length of the line segment as noise.

[0061] Alternatively, instead of Fig. 16(b), the position of the rear corner may be accurately extracted by executing the process of Fig. 16(c). That is, as shown in Fig. 16(c), a large number of normal lines obtained in Fig. 16(a) and a rectangle prepared in advance (for example, a rectangle corresponding to the outline of a general passenger vehicle) are set, and the rectangle is rotated so that the total distance at which the normal lines intersect with the rectangle is the shortest (so that the total square error is the smallest), and the angle at which the rectangle best fits the cutting-in vehicle V2 is estimated. This makes it possible to accurately estimate the rear end corner and the leading edge.

[0062] Feature point tracking / velocity estimation part 77 The detection process described above has made it possible to estimate the contour shape (length and position of the back and sides) of the cutting-in vehicle V2. However, in a situation where only a part of the side or back of the cutting-in vehicle V2 is captured in the image, after fitting the contour shape by the above method, the vehicle size is often detected as larger as the cutting-in vehicle V2 enters the screen. For this reason, there is a possibility that the error in the estimated speed of the cutting-in vehicle V2 is large in a speed estimation method using the center of gravity or a speed estimation method assuming that the entire vehicle body is visible. Therefore, in order to stably estimate the speed of the cutting-in vehicle V2 that has entered the image captured by the stereo camera unit 1, a method of tracking characteristic parts of the vehicle body in the image (such as corners of parts) is effective, and in order to extract the accurate vehicle speed, it is considered effective to use the original image, which is the image with the highest resolution. A general UD space often has a lower resolution than the original image, and it may be difficult to calculate an accurate speed in terms of resolution. However, if the resolution is the same as the original image, there is no effect of the resolution. However, to calculate the speed more accurately, it is better to use the corners of the image, etc., as this allows for a stable calculation of the speed regardless of whether the part captured on the image is part of the image or not.

[0063] 17, the feature point tracking / speed estimation unit 77 tracks feature points between successive image frames of the cutting-in vehicle V2 inside a rectangular parallelepiped equivalent to the rectangle fitted by the detailed shape fitting unit 76. In particular, by preferentially using feature points that can be stably tracked across multiple frames, noise components that could only be tracked in one frame are removed, and from among the tracked feature points, those with a moving distance that deviates significantly are removed as noise components, while only the feature points that could be stably tracked are used to calculate the speed of the cutting-in vehicle V2.

[0064] Since the vehicle contour and the three-dimensional positions of the feature points can be measured from the parallax images of the stereo camera unit 1, in FIG. 17, the movement directions (depth direction, lateral direction) of the three-dimensional points on the vehicle contour are expressed as velocity vectors. In this way, since multiple feature points are calculated, multiple velocity vectors are calculated with each position on the surface of the vehicle contour as the origin. If the vehicle body is considered to be a rigid body, even if the origin is different in three-dimensional space, the same speed will be obtained in the same direction. Although the obtained velocity vector contains noise components, a stable velocity vector is calculated by removing outliers and then using the average value.

[0065] In addition, the system calculates speed vectors by tracking feature points on the background and white lines in the image, and calculates the relative speed vector, absolute speed vector (the direction in which the vehicle is moving relative to stationary objects), and position within the lane, which are necessary to determine whether the cutting-in vehicle V2 poses a risk of collision or contact with the host vehicle V0, such as whether the cutting-in vehicle V2 is approaching a white line, whether it is approaching across a white line, or whether the other vehicle is approaching the host vehicle regardless of the behavior of the host vehicle V0.

[0066] ·Vehicle Identification Unit 78 When the feature point tracking / speed estimation unit 77 completes the estimation of the speed vector, it can be determined whether the three-dimensional object candidate has a rectangular parallelepiped shape and is approaching the host vehicle V0, so that the three-dimensional object candidate can be identified as the cutting-in vehicle V2 with a certain degree of accuracy. However, in order to further improve the detection accuracy of the cutting-in vehicle V2, the vehicle identification unit 78 is configured to identify whether the cutting-in vehicle V2 is a four-wheeled vehicle or a two-wheeled vehicle based on information such as the rear width and side length of the cutting-in vehicle V2. This prevents a stationary object from being erroneously detected as a cutting-in vehicle candidate in an environment where errors are likely to occur in the host vehicle behavior and the movement of other vehicle candidates, such as a situation in which the host vehicle V0 approaches a wall while traveling around a curve.

[0067] <Cut-in vehicle determination section 8> The cutting-in vehicle determination unit 8 utilizes the information obtained in the right-angle shape fitting unit 7 in a time series manner to comprehensively determine whether the cutting-in vehicle V2 should be subject to control with respect to the host vehicle V0. To execute the above processes, the cutting-in vehicle determination unit 8 has a white line information unit 81, a same lane merging determination unit 82, a complex congested environment determination unit 83, a preceding vehicle candidate determination unit 84, and a contact possibility determination unit 85, all of which are shown in Fig. 18. Each unit will be described in turn below.

[0068] ·White Line Information Department 81 First, the white line information unit 81 observes the results of white line recognition as one piece of information for making a comprehensive judgment. Specifically, the white line information unit 81 grasps the left and right white lines of the lane in which the host vehicle V0 is traveling, and also grasps whether the host vehicle's behavior is stable within the left and right white lines. Furthermore, if there is a vehicle traveling parallel to the vehicle, it also grasps whether the vehicle traveling parallel to the vehicle is traveling within the white lines while complying with traffic rules. Then, if both the host vehicle and the other vehicle are traveling stably within the white lines, it activates the same lane merging determination unit 82. On the other hand, if not, it activates the congested complex environment determination unit 83.

[0069] ·Same lane merging judgment unit 82 The same lane merging determination unit 82 uses the position and presence of white lines to determine whether a vehicle will cut in to the lane in which the host vehicle V0 is traveling. The same lane merging determination unit 82 extracts, as candidates for cutting in vehicles, vehicles that are likely to cut in to the lane in which the host vehicle is traveling, that are already approaching by slightly straddling the white line, or vehicles that are likely to come into contact within a predetermined time. Conversely, for vehicles that do not cut in to the white line, even if they are approaching the host vehicle V0 to a certain extent, it is determined that they have no intention of cutting in. However, if the lateral position of the host vehicle or another vehicle within the white line is changing quickly, the possibility of contact may be determined by predicting the degree of this change even if the vehicle does not deviate from the lane.

[0070] ·Congested and Complex Environment Judgment Unit 83 The congested and complex environment determination unit 83 determines that the driving environment is one in which the rule of driving within the white lines is not observed and the white lines are either blurred or non-existent, and does not use the results of white line recognition even if they are available. This is because if neither other vehicles nor the subject vehicle V0 observe the rule of driving within the white lines, there is no point in using the results of white line recognition. In such a driving environment, it is basically determined whether or not control should be performed based on the behavior of the subject vehicle V0 and the behavior of the opposing vehicle.

[0071] ·Leading vehicle candidate determination unit 84 When the host vehicle V0 performs adaptive cruise control (ACC), the preceding vehicle candidate determination unit 84 determines whether a nearby vehicle can be a candidate for the preceding vehicle V1 to be followed, and if so, sets the vehicle as a candidate for the preceding vehicle V1. In particular, as shown in FIG. 3A and FIG. 3B, when the host vehicle V0 is automatically following the preceding vehicle V1, if a cutting-in vehicle V2 moving at a speed slower than the preceding vehicle V1 cuts in on the host vehicle's path, it is important to quickly switch the vehicle to be followed from the preceding vehicle V1 to the cutting-in vehicle V2 in order to continue safe preceding vehicle following control. If the switching of the vehicle to be followed is delayed under such circumstances, the host vehicle V0 may rear-end the low-speed cutting-in vehicle V2 without decelerating, and even if the host vehicle V0 avoids a rear-end collision by decelerating to a certain extent, if the host vehicle V0 approaches the cutting-in vehicle V2 excessively, it may cause anxiety to the occupants of the host vehicle V0, and the driver may perform operations such as braking or avoidance by himself despite the automatic following operation.

[0072] Therefore, for example, when the driving area is judged as one in which driving within the white lines is required, the preceding vehicle candidate determination unit 84 uses the cutting-in vehicle V2 crossing the white lines and cutting in to the own lane as one of the conditions for determining a preceding vehicle candidate, but conversely, when the driving area is judged as one in which driving within the white lines is not required, the condition is not used.

[0073] ·Contact possibility determination unit 85 The contact possibility determination unit 85 performs contact determination using the contour shapes of the vehicle V0 and the other vehicle. In the conventional contact determination method using only the rear surface of the other vehicle, it was difficult to determine the possibility of contact with the tip of the other vehicle or the possibility of contact between the sides. Therefore, the contact possibility determination unit 85 of this embodiment accurately estimates the contour shape of the other vehicle, and performs contact determination between the contour shapes by assuming that the shape of the vehicle V0 is known. Then, in an area where the rule of driving within the white lines is observed, an alarm or control is issued early against a vehicle that cuts in and approaches the white lines. On the other hand, in an area where the rule of driving within the white lines is not observed, whether or not the vehicle cuts in the white lines is not used, and an alarm or control is not issued until the vehicle is about to come into contact. This is because the vehicle V0 and the other vehicle often drive close to coming into contact on a daily basis, so it is difficult to determine whether the driving is intentional or whether there is a risk of contact unintentionally. For this reason, an alarm or control is not issued until the vehicle is about to come into contact.

[0074] <Contact tracing judgment unit 5> The contact tracing determination unit 5 performs determination of warning and control for contact, collision, etc. with the host vehicle V0 for objects that are identified as vehicles and whose position and speed change in a way that is possible for a vehicle. Also, the objects are used as tracking targets while the preceding vehicle automatic following ACC is in use. To execute this process, the contact tracing determination unit 5 has a collision determination unit 51, a preceding vehicle tracking unit 52, a tracking target change unit 53, and an integrated determination unit 54, as shown in FIG. 19. Each unit will be described in order below.

[0075] 20 shows a situation in which a preceding vehicle V1 and a host vehicle V0 are present in the same lane. First, the collision determination unit 51 and the preceding vehicle tracking unit 52 will be described with reference to this diagram. 20 is moving slower than the host vehicle V0 or is stopped, there is a possibility that the host vehicle V0 will rear-end collide with the preceding vehicle V1. If the distance between the preceding vehicle V1 and the host vehicle V0 is sufficiently large, the driver can apply the brakes to slow down or stop the vehicle, and a warning would be sufficient as a collision prevention measure, with no need for emergency braking.

[0076] Therefore, the collision judgment unit 51 uses the relative speed and distance from the position and speed of the vehicle recognized by the preceding vehicle identification unit 4 to judge how many seconds it will take for the vehicle to come into contact with the host vehicle V0, and decides to issue an alarm if the situation is such that the rear-end collision can be avoided by the driver applying the brakes, but on the other hand, decides to issue an emergency brake if the timing is such that the rear-end collision cannot be avoided by the driver's operation.

[0077] Next, the preceding vehicle tracking unit 52 determines the control amount of the accelerator, brake, etc. of the host vehicle V0 so that the host vehicle V0 can smoothly follow the preceding vehicle V1 based on the relative speed and distance between the preceding vehicle V1 and the host vehicle V0 when the driver is executing ACC, so that the distance between the preceding vehicle V1 and the host vehicle V0 is maintained at a constant distance set by the driver and below the upper limit speed.

[0078] When a preceding vehicle candidate determination unit 84 determines that a cutting-in vehicle V2 has cut in between the preceding vehicle V1 and the vehicle itself V0 during preceding vehicle following, a tracking target change unit 53 determines the timing to change the vehicle to be followed.

[0079] Conversely, for a vehicle approaching from a close distance, only the side of the vehicle may be captured in the image of the stereo camera unit 1, or even if the rear of the vehicle is captured, only a small portion of the rear of the vehicle may be captured in the image. In such cases, in scenes where the side of the vehicle is captured in a large area, the detection of the approaching vehicle and the speed estimation may become unstable in conventional methods, and it may often be determined that control cannot be used.

[0080] For this reason, in this embodiment, a side-based cutting-in vehicle detection method beginning with side surface extraction unit 6 is used to stably detect the vehicle body contour and estimate the speed of a vehicle whose side appears large. Based on the information obtained by these methods and using the judgment results of cutting-in vehicle judgment unit 8, the integrated judgment unit 54 determines the timing of warning and the content of control, including emergency braking and avoidance, in response to a cutting-in vehicle V2 at close range.

[0081] In addition, in the integrated judgment unit 54, when the driver has permitted control to avoid the obstacle within the lane or the roadway, based on the information obtained above, an integrated judgment may be made as to whether the driver should only step on the brakes, or whether avoidance while steering and braking should be performed after checking the availability of three-dimensional objects within the lane or the availability of three-dimensional objects on the roadway including the shoulder. In addition to avoidance within the white lines, when the vehicle can detect the shoulder and sense the entire circumference of the vehicle, the integrated judgment unit also performs evacuation to the shoulder. When a vehicle traveling slightly diagonally ahead of the vehicle V0 is on the right front and there are no vehicles running around, a judgment is made to perform evasive steering while crossing the white line on the left side, and evacuation is performed by simultaneously sensing vehicles running in the left lane and the shoulder.

[0082] Vehicle control unit 9 The vehicle control unit 9 performs sound and vehicle speed control, emergency braking, steering, etc., according to the warning and control content determined above.

[0083] <Figure 21 Processing flow> Hereinafter, a processing procedure performed by the external environment recognition device 100 having the above-mentioned configuration will be described with reference to the flowchart of FIG.

[0084] First, in step S1, the left and right cameras of the stereo camera unit 1 capture a left image PL and a right image PR of the area ahead of the host vehicle V0 (see FIG. 3B).

[0085] Next, in step S2, the image processing unit of the stereo camera unit 1 performs parallelization of the left and right images and sensitivity correction, and then performs stereo matching to generate a parallax image.

[0086] In step S3, the disparity voting unit 2 uses the disparity image obtained in step S2 to vote for a disparity value D in the UD space (see FIG. 4). In this disparity value voting, when there is a disparity with the same depth and the same horizontal coordinate on the image, that is, when there is a three-dimensional object standing vertically on the road surface reflected in the image, the disparity votes tend to gather in the same place in the UD space. In other words, by using this UD space, the disparity vote value is accumulated in the place where the three-dimensional object exists. Therefore, by searching for a place with a high vote value in the UD space, the presence of the three-dimensional object can be detected. The surrounding environment is sensed using this UD space.

[0087] In step S4, the back surface extraction unit 3 horizontally scans the UD space to search for three-dimensional object candidates in order to extract the back surface of the three-dimensional object using the results of voting in the UD space (see FIG. 5A). Since the search is performed horizontally, a three-dimensional object whose back surface extends horizontally as viewed from the camera can be detected stably. In particular, since the vehicles are often facing the same direction, the back surface of the vehicle can be detected by this process, and the vehicle back surface candidates extracted by this process are used for emergency braking of the preceding vehicle V1 and following the preceding vehicle V1. In this process, three-dimensional objects such as two-wheeled vehicles, pedestrians, and utility poles are also extracted, and three-dimensional objects with a certain width or more are extracted once, and then, for running objects such as pedestrians, four-wheeled vehicles, and two-wheeled vehicles, the type identification process is performed.

[0088] In step S5, the preceding vehicle identification unit 4 performs an identification process on the three-dimensional object candidates extracted in step S4 that are within the vehicle's expected width, and identifies whether they are vehicles or not. Furthermore, if a lane recognition result is obtained, this result is used in combination to determine whether the object is a vehicle in an adjacent lane or the preceding vehicle V1. Furthermore, for objects within the expected width for a two-wheeled vehicle among the three-dimensional object candidates, a determination is simultaneously made as to whether they are two-wheeled vehicles or not. Similarly, the process is performed on pedestrians, and by identifying the type of moving object that is of concern, particularly when driving a vehicle, collisions, etc., reliability is improved.

[0089] The conventional three-dimensional object detection and vehicle detection by steps S4 and S5 described above are basically a method of searching for a wall facing the host vehicle V0, and the processing is simplified by capturing the three-dimensional object. This method itself is a very efficient method for detecting a three-dimensional object present in the host vehicle's path R0 and applying emergency braking, or from the viewpoint of following the preceding vehicle V1. However, the drawback is that it is insufficient for handling a wall that comes into contact with the side of the host vehicle V0, the side of a cutting-in vehicle, etc., and it is difficult to accurately capture the inclination and position of a three-dimensional object.

[0090] Previously, this method was sufficient for basic functions such as vehicles and pedestrians that were relatively tall and were controlled only when they entered the vehicle's path R0, but in recent years, this method was insufficient when considering not only the incorporation of basic functions but also control of the driving environment around the path. For this reason, in the present invention, in order to perform contact judgment with the cutting-in vehicle V2 and switching judgment of the preceding vehicle to be followed when cutting in, which are difficult to handle in the conventional processing (steps S4 and S5) that searches for the rear side, the following steps S6 to S8 are executed in parallel with the above steps S4 and S5.

[0091] In the case of a cutting-in vehicle V2 traveling close by in an adjacent lane, the side of the vehicle is largely visible from the host vehicle V0, so it is often more stable to extract the side rather than the back. In particular, there are cases where part or all of the back of the vehicle is not visible from the camera, and the back extraction process (step S4), which searches laterally, cannot stably and accurately extract the side of the cutting-in vehicle V2.

[0092] Therefore, in step S6, the side surface extraction unit 6 performs a side surface search and extracts the side surface of the cutting-in vehicle V2 (see Fig. 8 and Fig. 9). Furthermore, since the cutting-in vehicle V2 is the search target in this step, the side surface extraction unit 6 judges whether the vehicle is a three-dimensional object with a certain height, has a certain depth length or more, and is approaching the vehicle's own vehicle, thereby judging whether the vehicle is a candidate for cutting in.

[0093] In step S7, if the cutting-in vehicle candidate is actually the cutting-in vehicle V2, the right-angle shape fitting unit 7 utilizes the existence of a back line segment at a right angle to the side line segment to perform a refitting process (see Figs. 11 and 12) of the leading edge and rear end positions of the side line segment, and further a back search process (see Figs. 13A and 13B) to search for a more stable right-angle shape. This makes it possible to capture a stable outline shape for a vehicle that is missing only the side or part of the back relative to the host vehicle V0, or a vehicle whose side appears larger than the back, which was previously difficult to do, and makes it possible to obtain information for accurately determining contact, switching of the preceding vehicle to be followed, etc.

[0094] Furthermore, regarding the vehicle speed, the movement of characteristic points within this contour shape is searched for on the image to further improve accuracy (see Figure 17). This is particularly necessary when there is a difference in resolution between the parallax image and the original image. Image information processed for post-processing, such as parallax images, often has a lower resolution than the original image. For this reason, accuracy is high if the original image with its high resolution is used for speed calculation. For this reason, the original image is used to calculate the flow within the extracted contour shape to calculate the relative speed, and by taking into account the contour shape, the speed in the depth direction using parallax and the speed in the lateral direction are calculated simultaneously. Furthermore, the speed of stationary objects in the background is calculated in the same way, and the moving speed of the cutting-in vehicle V2 relative to the stationary objects (absolute speed is also calculated). In step S8, the cutting-in vehicle determination unit 8 uses the relative position, speed, absolute position, speed, white line position, vehicle behavior, etc. of the cutting-in vehicle candidate to determine whether the cutting-in vehicle V2 will affect the driving of the host vehicle V0. If there is a possibility of contact, etc., depending on the angle of contact and taking into account the contour shapes of each vehicle, it determines whether the host vehicle V0 should apply emergency braking, or whether it should avoid the vehicle within the lane by steering, or whether it should consider avoidance candidates including the shoulder of the road. In addition, if the host vehicle is following the preceding vehicle, it determines when to switch the target vehicle to follow, even if there is no risk of contact or collision.

[0095] In step S9, the contact tracing determination unit 5 uses the preceding vehicle identification information, the vehicle behavior, and information on the vehicle determined to be the cutting-in vehicle V2 to determine a collision or contact with the vehicle V0. By using the contour information of the vehicle V0 and the contour information of the other vehicle while taking into account the vehicle behavior, it is possible to accurately determine a contact accident, including contact, collision of the vehicle body corner and side, in addition to a head-on collision. It is also possible to suppress false alarms and erroneous control.

[0096] In addition, even if the vehicle V0 travels close to the point of contact and becomes the new leading vehicle V1 for the vehicle V0, the timing of switching to the leading vehicle V1 is appropriately adjusted so that the automatic following driving of the vehicle V0 itself does not become a speed that makes the driver feel uneasy. By switching the leading following vehicle to the cutting-in vehicle V2 at a safe timing, an effort is made to ensure a safe vehicle distance. Even if the driving of the other vehicle itself is like that, the vehicle V0 moves as quickly and accurately as possible to a safe relative distance and position, making it possible to provide safety and security to the driver. By performing a collision and tracking judgment that utilizes the outline shapes, which is simple but takes advantage of the right-angle shapes of the vehicle V0 and the other vehicle, it is possible to stably perform appropriate control of the vehicle V0 against the cutting-in vehicle V2 that passes close to the side of the vehicle V0, which was previously difficult.

[0097] In step S10, the vehicle control unit 9 issues a warning, performs control, or switches the target vehicle to be followed based on the result of the determination in step S9.

[0098] As described above, according to the external environment recognition device of this embodiment, when a vehicle cuts in between the preceding vehicle and the own vehicle, the vehicle can be recognized as a vehicle based on the side information of the cut-in vehicle. Also, if there is a possibility of contact with the cut-in vehicle, the own vehicle is braked, whereas if there is no possibility of contact, the vehicle to be followed in the preceding vehicle following control is switched from the previous preceding vehicle to the cut-in vehicle, thereby allowing the vehicle to continue stable driving. [Explanation of symbols]

[0099] 100...external environment recognition device, 1...stereo camera unit, 1L...left camera, 1R...right camera, 2...parallax voting unit, 3...back surface extraction unit, 4...preceding vehicle identification unit, 41...tracking unit, 42...vehicle back surface candidate selection unit, 43...vehicle identification unit, 44...position and speed estimation unit, 5...contact tracing determination unit, 51...collision determination unit, 52...preceding vehicle tracking unit, 53...tracking target change unit, 54...integration determination unit, 6...side surface extraction unit, 61...side surface candidate line search unit, 62...vehicle candidate determination unit, 63...tracking unit, 64... Side speed analysis unit, 7...right angle shape fitting unit, 71...side leading edge fitting unit, 72...side trailing edge fitting unit, 73...side-based rear search unit, 74...detection base line switching unit, 75...rear-based side search unit, 76...detailed shape fitting unit, 77...feature point tracking / speed estimation unit, 78...vehicle identification unit, 8...cutting in vehicle determination unit, 81...white line information unit, 82...same lane merging determination unit, 83...congested complex environment determination unit, 84...preceding vehicle candidate determination unit, 85...contact possibility determination unit, 9...vehicle control unit, V0...own vehicle, V1...preceding vehicle, V2...cutting in vehicle

Claims

1. Based on the parallax between left and right images of a part of another vehicle approaching a host vehicle, a side surface extraction unit for extracting side surface information from the image; a rear surface search unit that searches for a rear surface position of the other vehicle based on the side surface information; Based on the side information and the rear position, a rectangular object that is close to the other vehicle in the image is a shape fitting unit that fits a body shape; Using the rectangular parallelepiped shape, the vehicle is recognized as a vehicle ahead of the host vehicle. In an external environment recognition device having a recognition unit, The other vehicle recognition unit, A relative attitude between the vehicle ahead and the host vehicle, or a side of the vehicle ahead in the image Based on the estimated back length, The side is searched based on the side of the preceding vehicle, or the side is searched based on the rear of the preceding vehicle. An external environment recognition device characterized by switching between searching and not searching.

2. Based on the parallax between left and right images capturing a part of another vehicle approaching the own vehicle, a side surface extraction unit for extracting side surface information from the image; a rear surface search unit that searches for a rear surface position of the other vehicle based on the side surface information; Based on the side information and the rear position, a rectangular object that is close to the other vehicle in the image is a shape fitting unit that fits a body shape; Using the rectangular parallelepiped shape, the vehicle is recognized as a vehicle ahead of the host vehicle. In an external environment recognition device having a recognition unit, The side information includes a position and a posture of the side of the other vehicle, and is the length in the depth direction, The side surface extraction unit extracts the left and right images in a disparity voting space having two axes, that is, a disparity value and a horizontal image coordinate. The external environment recognition is characterized in that the side information of the other vehicle is extracted by voting the parallax of the image. Device.

3. Based on the parallax between left and right images capturing a part of another vehicle approaching the own vehicle, a side surface extraction unit for extracting side surface information from the image; a rear surface search unit that searches for a rear surface position of the other vehicle based on the side surface information; Based on the side information and the rear position, a rectangular object that is close to the other vehicle in the image is a shape fitting unit that fits a body shape; Using the rectangular parallelepiped shape, the vehicle is recognized as a vehicle ahead of the host vehicle. In an external environment recognition device having a recognition unit, The side information includes a position and a posture of the side of the other vehicle, and is the length in the depth direction, The side surface extraction unit is By analyzing the image laterally and identifying locations of different textures, or The disparity between the left and right images is analyzed in the horizontal direction, and a position where the disparity value changes suddenly is identified. An external environment recognition device characterized by extracting the position of the side tip of the vehicle ahead with high accuracy.

4. Based on the parallax between left and right images capturing a part of another vehicle approaching the own vehicle, a side surface extraction unit for extracting side surface information from the image; a rear surface search unit that searches for a rear surface position of the other vehicle based on the side surface information; Based on the side information and the rear position, a rectangular object that is close to the other vehicle in the image is a shape fitting unit that fits a body shape; Using the rectangular parallelepiped shape, the vehicle is recognized as a vehicle ahead of the host vehicle. In an external environment recognition device having a recognition unit, The side information includes a position and a posture of the side of the other vehicle, and is the length in the depth direction, The side surface extraction unit is Assuming that the rear surface of the vehicle in front is perpendicular to the side surface of the vehicle in front, the parallax of the rear surface is calculated by dividing the parallax of the rear surface by the side surface. By voting for the position with the most votes, the rear end of the side is identified. An external environment recognition device characterized by extracting the position of the rear end of the side of the vehicle ahead with high accuracy.

5. The external environment recognition device according to claim 2, The side surface extraction unit is On the assumption that the rear surface of the vehicle in front is perpendicular to the side surface of the vehicle in front, in the disparity voting space, The disparity of the back positions arranged in a straight line is searched, and the line with the most votes is selected as the back candidate line. An external environment recognition device characterized by extracting the above-mentioned.

6. Based on the parallax between left and right images capturing a part of another vehicle approaching the own vehicle, a side surface extraction unit for extracting side surface information from the image; a rear surface search unit that searches for a rear surface position of the other vehicle based on the side surface information; Based on the side information and the rear position, a rectangular object that is close to the other vehicle in the image is a shape fitting unit that fits a body shape; Using the rectangular parallelepiped shape, the vehicle is recognized as a vehicle ahead of the host vehicle. In an external environment recognition device having a recognition unit, The other vehicle recognition unit recognizes a straight line perpendicular to a normal line of a side surface of the forward vehicle and a The rear angle of the vehicle ahead is accurately calculated by calculating the intersection point of the straight line perpendicular to the normal line of the rear surface of the vehicle ahead. An external environment recognition device characterized by calculating

7. Based on the parallax between left and right images capturing a part of another vehicle approaching the own vehicle, a side surface extraction unit for extracting side surface information from the image; a rear surface search unit that searches for a rear surface position of the other vehicle based on the side surface information; Based on the side information and the rear position, a rectangular object that is close to the other vehicle in the image is a shape fitting unit that fits a body shape; Using the rectangular parallelepiped shape, the vehicle is recognized as a vehicle ahead of the host vehicle. In an external environment recognition device having a recognition unit, The other vehicle recognition unit uses a plurality of frames of the image, By tracking the forward vehicle within the rectangular parallelepiped shape, a relative velocity vector of the forward vehicle is calculated. Calculate and By calculating the velocity vector of a stationary object outside the rectangular parallelepiped shape, An external environment recognition device characterized by estimating absolute velocity.

8. During vehicle following control, the system detects the vehicle ahead based on three-dimensional information obtained from the vehicle's front sensor. An external environment recognition method for recognizing a vehicle cutting in front of a vehicle, comprising: The depth distance indicated by the three-dimensional information is expressed as a voting space having two axes, the depth distance and the image abscissa. a voting step of voting for A line segment within a predetermined angle to the path of the vehicle on the overhead view is searched for within the voting space. A search step; a side surface extraction step of extracting a side surface of the cutting-in vehicle from the line segment within the predetermined angle that has been searched for; , a back surface extraction step of extracting a back surface of the cutting-in vehicle based on the extracted side surface; a cutting-in vehicle recognition step of recognizing the cutting-in vehicle based on the extracted side and rear faces; An external environment recognition method comprising:

9. The method for recognizing an external environment according to claim 8, In the side surface extraction step, the line segment closest to the host vehicle among the line segments within the predetermined angle is as the side of the cutting-in vehicle.

10. The method for recognizing an external environment according to claim 8, a contact determination step for determining a possibility of contact with the cutting-in vehicle; a braking command step of commanding the host vehicle to brake if there is a possibility of contact; a step of changing the vehicle to be followed if there is no possibility of braking; The external environment recognition method further comprises:

Images