External environment recognition device

Abstract

To improve the accuracy of identifying a three-dimensional object by properly pairing a back face and side faces of the three-dimensional object.SOLUTION: A surrounding environment recognition apparatus includes: a position derivation unit which derives a three-dimensional position by block in a captured image; a grouping unit which groups blocks of which differences of three-dimensional positions fall within a predetermined range to specify three-dimensional object candidates; a pairing unit which projects the three-dimensional object candidates on a horizontal plane, discriminates a back face from side faces on the basis of angles with respect to a depth direction, and pairs the side faces and back face to specify a pair; and a pair prediction unit which predicts future positions of the pair. The grouping unit specifies three-dimensional object candidates (220h, 220i) on the basis of the future positions of the pair (232, 234) predicted in the past by the pair prediction unit.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to an exterior environment recognition device that identifies a three-dimensional object that exists in the traveling direction of a host vehicle. [Background technology]

[0002] Patent Document 1 discloses a technology for detecting a preceding vehicle located in front of the vehicle and reducing damage caused by a collision with the preceding vehicle, as well as a technology for performing following control to maintain a safe distance between the vehicle and the preceding vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 3349060 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to reduce damage from a collision between the vehicle and a preceding vehicle and to achieve control to follow the preceding vehicle, the vehicle first identifies a three-dimensional object present in the traveling direction and determines whether the three-dimensional object is a specific object such as a preceding vehicle. In order to appropriately identify the three-dimensional object, the vehicle, for example, groups distance image blocks whose three-dimensional positions are adjacent to each other as the back surface and side surface of the three-dimensional object, respectively. The vehicle then pairs the grouped back surface and side surface, and identifies them as a pair representing the same three-dimensional object.

[0005] However, if noise occurs due to mismatching of relative distances in the distance image, the host vehicle cannot properly group the rear or side of a three-dimensional object, and as a result, it becomes difficult to stably identify the object as a three-dimensional object.

[0006] In view of these problems, the present invention aims to provide an exterior environment recognition device that can improve the accuracy of identifying three-dimensional objects by appropriately grouping the back and side surfaces of three-dimensional objects. [Means for solving the problem]

[0007] In order to solve the above problems, the vehicle exterior environment recognition device of the present invention includes a position derivation unit that derives a three-dimensional position for each block in a captured image, a grouping unit that identifies a solid object candidate by grouping blocks whose three-dimensional position difference is within a predetermined range, a pairing unit that projects the solid object candidate onto a horizontal plane, distinguishes between a back surface and a side surface based on an angle with respect to the depth direction, and pairs the back surface and the side surface to identify pairs, and a pair prediction unit that predicts a future position of the pair, and the grouping unit predicts a future position of the pair by inputting a prediction result of the pair ... Back position By grouping blocks whose distance from the predicted back surface line is within a certain range, Identifying potential solid objects Then, the pair prediction unit groups blocks whose distance from a predicted side line indicating the position of the side of the pair predicted in the past is within a predetermined range, thereby identifying candidates of a solid object corresponding to the side. .

[0008] The pairing unit may correct the position of the back surface of the pair based on the position of the back surface of the pair previously predicted by the pair prediction unit, and may correct the position of the side surface of the pair based on the position of the side surface of the pair previously predicted by the pair prediction unit, and the pair prediction unit may predict a future predicted back surface line based on the position of the back surface of the pair corrected by the pairing unit, and may predict a future predicted side line based on the position of the side surface of the pair corrected by the pairing unit. Effect of the Invention

[0009] According to the present invention, by appropriately pairing the rear surface and the side surface of a three-dimensional object, it is possible to improve the accuracy of identifying the three-dimensional object. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the connections of the exterior environment recognition system. [Diagram 2] FIG. 2 is a functional block diagram showing a schematic function of the vehicle exterior environment recognition device. [Diagram 3] FIG. 3 is a flowchart showing the flow of the vehicle exterior environment recognition method. [Figure 4] FIG. 4 is an explanatory diagram for explaining a luminance image and a distance image. [Diagram 5] FIG. 5 is an explanatory diagram for explaining the operation of the grouping unit. [Figure 6] FIG. 6 is an explanatory diagram for explaining the operation of the pairing unit. [Figure 7] FIG. 7 is an explanatory diagram for explaining the determination by the pairing unit as to whether pairing is possible. [Figure 8] FIG. 8 is an explanatory diagram for explaining the correction of the back surface and the side surface. [Figure 9] FIG. 9 is an explanatory diagram illustrating the operation of the pair prediction unit. [Figure 10] FIG. 10 is an explanatory diagram for explaining the operation of the grouping unit. [Figure 11] FIG. 11 is an explanatory diagram for explaining the operation of the grouping unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings. The dimensions, materials, and other specific values ​​shown in the embodiment are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In this specification and drawings, elements having substantially the same functions and configurations are given the same reference numerals to avoid repeated explanations, and elements not directly related to the present invention are not shown.

[0012] (Exterior environment recognition system 100) 1 is a block diagram showing connections in a vehicle exterior environment recognition system 100. The vehicle exterior environment recognition system 100 is provided in a vehicle 1, and includes an imaging device 110, an exterior environment recognition device 120, and a vehicle control device .

[0013] The imaging device 110 is composed of an imaging element such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor). The imaging devices 110 are arranged in the traveling direction of the vehicle 1 so that the optical axes of the two imaging devices 110 are approximately parallel to each other and spaced apart in the horizontal direction. The imaging device 110 captures an image of the external environment in front of the vehicle 1, and generates a luminance image (color image or monochrome image) that includes at least luminance information. The imaging device 110 continuously generates luminance images of three-dimensional objects present in a detection area in front of the vehicle 1, for example, at every 1 / 60th of a second frame (60 fps). Here, the three-dimensional objects recognized by the external environment recognition device 120 include not only independently existing objects such as bicycles, pedestrians, vehicles, traffic lights, road signs, guardrails, and buildings, but also objects that can be identified as parts of a vehicle, such as the back and side of a vehicle, and bicycle wheels. Here, the rear surface of the vehicle refers to the surface facing the vehicle 1 in front of the vehicle 1, and does not refer to the rear surface of the vehicle itself.

[0014] The vehicle exterior environment recognition device 120 acquires luminance images from each of the two imaging devices 110, and generates a distance image using so-called pattern matching. Based on the luminance image and the distance image, the vehicle exterior environment recognition device 120 groups blocks that are located above the road surface, have the same color value, and have adjacent three-dimensional position information as solid objects. Then, the vehicle exterior environment recognition device 120 identifies which specific object the solid object is. For example, the vehicle exterior environment recognition device 120 identifies the solid object as a leading vehicle. Furthermore, when the vehicle exterior environment recognition device 120 identifies the leading vehicle in this way, it performs a reduction control of collision damage between the vehicle 1 and the leading vehicle, and also performs a following control to make the vehicle 1 follow the leading vehicle. The specific operation of the vehicle exterior environment recognition device 120 will be described in detail later.

[0015] The vehicle control device 130 is composed of an ECU (Electronic Control Unit) and the like, and receives operational inputs from the driver via a steering wheel 132, an accelerator pedal 134, and a brake pedal 136, and transmits the inputs to a steering mechanism 142, a drive mechanism 144, and a braking mechanism 146 to control the host vehicle 1. The vehicle control device 130 also controls the steering mechanism 142, the drive mechanism 144, and the braking mechanism 146 in accordance with instructions from the exterior environment recognition device 120.

[0016] (External environment recognition device 120) 2 is a functional block diagram showing a schematic function of the vehicle exterior environment recognition device 120. As shown in FIG. 2, the vehicle exterior environment recognition device 120 includes an I / F unit 150, a data storage unit 152, and a central control unit 154.

[0017] The I / F unit 150 is an interface for two-way information exchange with the imaging device 110 and the vehicle control device 130. The data storage unit 152 is composed of a RAM, a flash memory, a HDD, etc., and stores various information necessary for the processing of each functional unit described below.

[0018] The central control unit 154 is configured with a semiconductor integrated circuit including a central processing unit (CPU), a ROM in which programs and the like are stored, a RAM as a work area, and the like, and controls the I / F unit 150, the data holding unit 152, and the like through a system bus 156. In this embodiment, the central control unit 154 also functions as a position derivation unit 170, a grouping unit 172, a pairing unit 174, a pair prediction unit 176, and a three-dimensional object identification unit 178. The external environment recognition method will be described in detail below, taking into account the operation of each functional unit of the central control unit 154.

[0019] (How to recognize the environment outside the vehicle) FIG. 3 is a flowchart showing the flow of the vehicle exterior environment recognition method. The vehicle exterior environment recognition device 120 executes the vehicle exterior environment recognition method at every predetermined interruption time. In the vehicle exterior environment recognition method, the position derivation unit 170 derives a three-dimensional position for each block in the luminance image acquired from the imaging device 110 (position derivation process S200). The grouping unit 172 groups the blocks together to identify a solid object candidate (grouping process S202). The pairing unit 174 projects the solid object candidate onto a horizontal plane, distinguishes between a back surface and a side surface based on the angle with respect to the depth direction, and pairs the back surface and the side surface to identify a pair (pairing process S204). The pair prediction unit 176 predicts the future position of the pair (pair prediction process S206). The solid object identification unit 178 identifies which specific object the solid object constituting the pair and other solid objects correspond to (solid object identification process S208).

[0020] (Position derivation process S200) 4 (FIGS. 4A to 4C) are explanatory diagrams for explaining a luminance image and a distance image. The position derivation unit 170 acquires a plurality of luminance images (two in this case) captured at the same timing with different optical axes by the imaging device 110. Here, it is assumed that the position derivation unit 170 acquires, as the luminance image 212, a first luminance image 212a captured by the imaging device 110 located relatively to the right of the vehicle 1 as shown in FIG. 4A, and a second luminance image 212b captured by the imaging device 110 located relatively to the left of the vehicle 1 as shown in FIG. 4B.

[0021] 4A and 4B, it can be seen that the image positions of three-dimensional objects included in the first luminance image 212a and the second luminance image 212b differ in the horizontal direction due to the difference in the imaging position of the imaging device 110. Here, the horizontal direction indicates the horizontal direction of the screen of the captured image, and the vertical direction indicates the vertical direction of the screen of the captured image.

[0022] The position derivation unit 170 generates a distance image 214, as shown in FIG. 4C, capable of identifying the distance to the imaging target, based on the first luminance image 212a shown in FIG. 4A and the second luminance image 212b shown in FIG. 4B, which are acquired by the position derivation unit 170.

[0023] Specifically, the position derivation unit 170 derives disparity information including disparity and an image position indicating the position of an arbitrary block in an image by using so-called pattern matching. Then, the position derivation unit 170 searches for a block corresponding to a block arbitrarily extracted from one luminance image (here, the first luminance image 212a) from the other luminance image (here, the second luminance image 212b). Here, the block is represented by, for example, an array of 4 pixels horizontally by 4 pixels vertically. Moreover, pattern matching is a method of searching for a block corresponding to a block arbitrarily extracted from one luminance image from the other luminance image.

[0024] For example, functions for evaluating the degree of match between blocks in pattern matching include SAD (Sum of Absolute Difference), which takes the difference in brightness, SSD (Sum of Squared Intensity Difference), which uses the squared difference, and NCC (Normalized Cross Correlation), which takes the similarity of the variance value obtained by subtracting the average value from the brightness of each pixel.

[0025] The position derivation unit 170 performs such a block-by-block disparity derivation process for all blocks displayed in a detection area of, for example, 600 pixels x 200 pixels. Here, the position derivation unit 170 processes the block as 4 pixels x 4 pixels, but may process the block as any number of pixels.

[0026] The position derivation unit 170 converts the parallax information for each block in the distance image 214 into a three-dimensional position in real space including a horizontal distance x, a height y, and a relative distance z, using a so-called stereo method. Here, the stereo method is a method of deriving the relative distance z of a block to the imaging device 110 from the parallax in the distance image 214 of the block by using a triangulation method. At this time, the position derivation unit 170 derives the height y of the block from the road surface based on the relative distance z of the block and the detected distance on the distance image 214 between the block and a point on the road surface at the same relative distance z as the block. Then, the position derivation unit 170 again associates the derived three-dimensional position with the distance image 214. Since various known techniques can be applied to the process of deriving the relative distance z and the process of specifying the three-dimensional position, a description thereof will be omitted here.

[0027] (Grouping process S202) Grouping unit 172 identifies solid object candidates by grouping blocks that are located above the road surface, have the same color value, and have a difference in three-dimensional position within a predetermined range in distance image 214. Specifically, grouping unit 172 groups blocks whose difference in horizontal distance x, height y, and relative distance z in distance image 214 are within a predetermined range (e.g., 0.1 m) on the assumption that they correspond to the same specified object. In this way, a virtual block group is generated.

[0028] 5 (FIGS. 5A to 5C) are explanatory diagrams for explaining the operation of grouping unit 172. Here, it is assumed that position derivation unit 170 generates distance image 214 as shown in FIG. 5A, for example. Grouping unit 172 groups blocks from distance image 214. In this way, a plurality of grouped block groups are extracted as shown in FIG. 5B. Grouping unit 172 sets the outline of all the grouped blocks in FIG. 5B, for example, a rectangular frame (surface) consisting of horizontal and vertical lines, or lines extending in the depth direction and vertical lines, as solid object candidate 220 (220a, 220b, 220c, 220d, 220e).

[0029] When the solid object candidates 220a, 220b, 220c, 220d, and 220e on the distance image 214 in FIG. 5B are represented on a two-dimensional horizontal plane indicated by the horizontal distance x and the relative distance z, they become the solid object candidates 220a, 220b, 220c, 220d, and 220e in FIG. 5C.

[0030] (Pairing process S204) In FIG. 5C, for example, solid object candidate 220a and solid object candidate 220b are different solid object candidates. However, solid object candidate 220a constitutes the back surface of the preceding vehicle, and solid object candidate 220b constitutes the side surface of the preceding vehicle. Therefore, solid object candidate 220a and solid object candidate 220b should be recognized as a pair that constitutes an integrated, identical solid object. Similarly, solid object candidate 220d and solid object candidate 220e should also be recognized as the same solid object. Therefore, pairing unit 174 pairs solid object candidates 220 that should be the back surface and side surface of the same solid object.

[0031] Fig. 6 is an explanatory diagram for explaining the operation of the pairing unit 174. Fig. 6 shows a three-dimensional object candidate 220 projected onto a horizontal plane, and is represented only by the horizontal distance x and the relative distance z.

[0032] The pairing unit 174 projects the three-dimensional object candidate 220 onto a horizontal plane, distinguishes between a back face and a side face based on the angle with respect to the depth direction, and if the relationship between the back face and the side face satisfies a predetermined condition, pairs the back face and the side face to identify the pair.

[0033] Specifically, the pairing unit 174 first derives the angle of the approximation line of the solid object candidate 220 with respect to the axis of the relative distance corresponding to the depth direction. If the absolute value of the derived angle is 45 degrees or more and less than 135 degrees, the pairing unit 174 determines that the solid object candidate 220 is the back surface. On the other hand, if the absolute value of the derived angle is 0 degrees or more and less than 45 degrees, or 135 degrees or more and 180 degrees or less, the pairing unit 174 determines that the solid object candidate 220 is the side surface.

[0034] Here, for example, assume that the grouping unit 172 has derived three-dimensional object candidates 220f and 220g as a result of grouping, as shown in FIG. 6A. In this example, as shown in FIG. 6B, the absolute value of the angle of the approximation line of the three-dimensional object candidate 220f is equal to or greater than 45 degrees and less than 135 degrees, so the pairing unit 174 determines that the three-dimensional object candidate 220f is a rear surface. On the other hand, the absolute value of the angle of the approximation line of the three-dimensional object candidate 220g is equal to or greater than 0 degrees and less than 45 degrees, so the pairing unit 174 determines that the three-dimensional object candidate 220g is a side surface. In this way, the grouping unit 172 can roughly distinguish between a rear surface and a side surface.

[0035] The pairing unit 174 determines whether pairing is possible for all combinations of the rear surface and the side surface thus distinguished in a brute force manner. Therefore, the pairing unit 174 repeats the pairing determination for one rear surface with all the side surfaces as many times as the number of rear surfaces, so the number of times of processing is the number of rear surfaces x the number of side surfaces. At this time, the pairing unit 174 may execute pairing if the distance between the rear surface and the side surface is within an appropriate distance (e.g., 2 m) as part of the vehicle, the speed of both the rear surface and the side surface is stable, the length of each of the rear surface and the side surface is a predetermined value (e.g., 1 m) or more, and the center position is within the detection area. By adding such a precondition for pairing, the pairing unit 174 can appropriately limit the objects to be paired, and the processing load can be reduced.

[0036] FIG. 7 (FIG. 7A to FIG. 7D) is an explanatory diagram for explaining the judgment of the pairing by the pairing unit 174. The pairing unit 174 judges whether pairing is possible depending on whether the relationship between the back surface 222 and the side surface 224 satisfies a predetermined condition. As a result, when it is judged that the predetermined condition is satisfied, the pairing unit 174 specifies the back surface 222 and the side surface 224 of the same solid object as a pair. Here, the back surface 222 corresponding to the solid object candidate 220f and the side surface 224 corresponding to the solid object candidate 220g are listed, and the possibility of pairing is judged. In addition, four items, "distance between end points", "velocity vector distance", "intersection distance", and "angle formed" are explained as the predetermined condition. Note that it is not necessary to adopt all four items as the predetermined condition, and one or more items may be adopted as appropriate.

[0037] First, as shown in Fig. 7A, pairing unit 174 identifies end points of rear surface 222 and side surface 224 that are closer to each other in a horizontal plane, and derives the distance between the identified end points. Then, if the distance between the end points is, for example, less than 1 m, pairing unit 174 determines that the condition of "distance between end points" is satisfied.

[0038] Next, the pairing unit 174 derives the velocity vector of the rear surface 222 and the velocity vector of the side surface 224 in the horizontal plane, as shown in Fig. 7B. Then, the pairing unit 174 derives the difference vector of the velocity vectors, that is, the distance between the two velocity vectors when the starting points of the two velocity vectors are aligned. Then, if the velocity vector distance y is less than 10km / h, for example, the pairing unit 174 determines that the condition of "velocity vector distance" is satisfied.

[0039] 7C, the pairing unit 174 derives the distance between the intersection of the approximation line of the rear surface 222 and the approximation line of the side surface 224, and the end point of the rear surface 222 on the intersection side. If the intersection distance is, for example, less than 1 m, the pairing unit 174 determines that the condition of "intersection distance" is satisfied.

[0040] 7D, the pairing unit 174 derives the angle formed by an approximate straight line of the rear surface 222 and an approximate straight line of the side surface 224 on the horizontal plane. If the formed angle v is close to 90 degrees, for example, in the range of 70 to 110 degrees, the pairing unit 174 determines that the condition of "formed angle" is satisfied.

[0041] The pairing unit 174 pairs the rear surface 222 and the side surface 224 when the four conditions of "distance between end points," "velocity vector distance," "intersection distance," and "angle formed" are satisfied.

[0042] Next, the pairing unit 174 corrects the paired back surface 222 and side surface 224 using the pair of the back surface 222 and side surface 224 generated in the previous frame.

[0043] 8 (FIGS. 8A and 8B) are explanatory diagrams for explaining the correction of the rear surface 222 and the side surface 224. In FIG. 8, the rear surface 222 and the side surface 224 are shown, as in FIG. 7. The pairing unit 174 first specifies actual measurement values ​​related to the intersections and end points of the rear surface 222 and the side surface 224 paired in the frame.

[0044] Specifically, as shown in Fig. 8A, the pairing unit 174 sets the actual measurement value of the intersection between the approximation line of the rear surface 222 and the approximation line of the side surface 224 as the actual intersection, the actual measurement value of the end point on the rear surface 222 on the intersection side as the actual intersection side end point, the actual measurement value of the end point opposite to the intersection side of the rear surface 222 as the actual rear end point, and the actual measurement value of the end point opposite to the intersection side of the side surface 224 as the actual side end point. Note that, in deriving the actual measurement value corresponding to the intersection, the pairing unit 174 does not use the actual intersection between the rear surface 222 and the side surface 224 as it is, but uses the midpoint between the intersection and the end point on the intersection side of the rear surface 222, i.e., the actual intersection side end point. This is because the accuracy of identifying the rear surface 222 is higher than that of the side surface 224, and the end points of the rear surface 222 are more reliable.

[0045] Here, the midpoint is found by averaging the horizontal distance and the relative distance of each point. The midpoint does not necessarily have to be the average, and may be weighted according to its reliability. For example, if the reliability of the end point of the back surface 222 is high, the corrected intersection may be found by setting a point closer to the end point on the side of the actual measurement intersection than the actual measurement intersection as the midpoint.

[0046] Next, the pairing unit 174 corrects the actual measurement value and derives the corrected intersection, the corrected back end point, and the corrected side end point. Specifically, the pairing unit 174 sets the midpoint between the predicted intersection and the midpoint between the actual measurement intersection and the actual measurement intersection side end point as the corrected intersection, the midpoint between the predicted back end point and the actual measurement back end point as the corrected back end point, and the midpoint between the predicted side end point and the actual measurement side end point as the corrected side end point. Here, the predicted intersection, the predicted back end point, and the predicted side end point are predicted by the pair prediction unit 176 in the previous frame as the intersection, the back end point, and the side end point in the current frame. Note that, if the predicted intersection, the predicted back end point, and the predicted side end point have not been derived in the previous frame, the pairing unit 174 uses the actual measurement intersection, the actual measurement back end point, and the actual measurement side end point as they are as the predicted intersection, the predicted back end point, and the predicted side end point.

[0047] In this way, the pairing unit 174 identifies the corrected intersection, the corrected rear end point, and the corrected side end point as shown in Fig. 8B. The vehicle exterior environment recognition device 120 determines the line connecting the corrected intersection and the corrected rear end point as the corrected rear face 222, and the line connecting the corrected intersection and the corrected side end point as the corrected side face 224. The vehicle exterior environment recognition device 120 uses the corrected intersection, the corrected rear end point, the predicted side end point, the rear face 222, and the side face 224 to perform control to reduce collision damage between the host vehicle 1 and the preceding vehicle, and also performs tracking control to make the host vehicle 1 follow the preceding vehicle.

[0048] (Pair prediction process S206) The pair prediction unit 176 predicts the future position of the pair of the back surface 222 and the side surface 224 paired by the pairing unit 174, here, the position in the next frame.

[0049] FIG. 9 is an explanatory diagram for explaining the operation of the pair prediction unit 176. FIG. 9 is shown only by the horizontal distance x and the relative distance z. The pair prediction unit 176 uses the derived corrected intersection, corrected back end point, and corrected side end point, and the corrected intersection, corrected back end point, and corrected side end point derived multiple times in the past, and derives the next predicted intersection, predicted back end point, and predicted side end point using, for example, a Kalman filter. At this time, the pair prediction unit 176 predicts the next predicted intersection, predicted back end point, and predicted side end point taking into account the relative speed of the back and side, the angular speed around the Y axis, and the egomotion.

[0050] (Three-dimensional object identification process S208) The three-dimensional object identification unit 178 identifies which specific object the three-dimensional object formed by the pair of the back surface 222 and the side surface 224 paired by the pairing unit 174 and other three-dimensional objects correspond to. For example, if the three-dimensional object formed by the pair of the back surface 222 and the side surface 224 has a size, shape, and relative speed that are typical of a vehicle, and has a light source such as a brake lamp or a high-mounted stop lamp at a predetermined position behind it, the three-dimensional object identification unit 178 identifies the three-dimensional object as a preceding vehicle.

[0051] As described with reference to FIG. 9, the pair prediction unit 176 predicts the rear surface 222 and the side surface 224 in the next frame. Then, as described with reference to FIG. 5, the grouping unit 172 groups adjacent blocks, and as described with reference to FIGS. 6 and 7, the pairing unit 174 pairs the grouped solid object candidates 220. Then, as described with reference to FIG. 8, the pairing unit 174 corrects the positions of the rear surface 222 and the side surface 224 using the predicted rear surface 222 and the side surface 224. In this way, the pairing unit 174 can appropriately correct the positions of the rear surface 222 and the side surface 224 by using the rear surface 222 and the side surface 224 predicted by the pair prediction unit 176.

[0052] However, even if pairing is performed appropriately in pairing section 174, unless grouping is performed appropriately in grouping section 172 at the preceding stage, the accuracy of identifying a three-dimensional object will not improve.

[0053] Fig. 10 is an explanatory diagram for explaining the operation of grouping unit 172. Fig. 10 is expressed only by horizontal distance x and relative distance z. For example, assume that blocks as shown in Fig. 10 are extracted from distance image 214. Grouping unit 172 groups blocks whose three-dimensional position difference in distance image 214 is within a predetermined range. However, when noise occurs due to mismatching of relative distance in distance image 214, grouping unit 172 may not be able to appropriately group back surface 222 and side surface 224.

[0054] For example, in the example of FIG. 10, the grouping unit 172 identifies three solid object candidates 220h, 220i, and 220j. Here, the solid object candidate 220h corresponds to the back surface 222, and the solid object candidate 220i corresponds to the side surface 224. On the other hand, the solid object candidate 220j is noise and should not be considered as a solid object candidate. Depending on the position of the block, the grouping unit 172 may identify the solid object candidate 220j including such noise. In that case, the pairing unit 174 cannot appropriately pair the back surface 222 and the side surface 224, and it becomes difficult for the exterior environment recognition device 120 to stably identify them as solid objects.

[0055] Therefore, the grouping unit 172 identifies the three-dimensional object candidate 220 by using the back surface 222 and the side surface 224 in the next frame predicted by the pair prediction unit 176 in the previous frame.

[0056] 11 (FIGS. 11A and 11B) are explanatory diagrams for explaining the operation of the grouping unit 172. The grouping unit 172 gives priority to groups of blocks close to the back surface 222 and the side surface 224 predicted by the pair prediction unit 176.

[0057] Specifically, as shown in FIG. 11A, the grouping unit 172 generates a predicted back surface line 232 corresponding to the back surface 222, which connects the predicted intersection point predicted by the pair prediction unit 176 and the predicted back surface end point. Next, the grouping unit 172 groups blocks whose distance from the predicted back surface line 232 is within a predetermined range indicated by a dashed line in the figure and whose three-dimensional position difference is within a predetermined range. In this way, the grouping unit 172 can identify a solid object candidate 220h corresponding to the back surface 222. The distance from a block to the predicted back surface line 232 indicates the distance between the block and the intersection point of the perpendicular line and the predicted back surface line 232 by dropping a perpendicular line from the block to the predicted back surface line 232.

[0058] 11B, the grouping unit 172 generates a predicted side line 234 that connects the predicted intersection and predicted side end point predicted by the pair prediction unit 176 and corresponds to the side surface 224. Next, the grouping unit 172 groups blocks whose distance from the predicted side surface line 234 is within a predetermined range indicated by a dashed line in the figure and whose three-dimensional position difference is within a predetermined range. In this way, the grouping unit 172 can identify a solid object candidate 220i that corresponds to the side surface 224.

[0059] In this embodiment, the grouping unit 172 groups blocks whose distance from the predicted flank line 234 is within a predetermined range, but this is not the only case and various statistical methods can be used. For example, the grouping unit 172 can use the least square distance or M-estimation to group only blocks whose variance is within a predetermined range.

[0060] In addition, the grouping unit 172 may process a block located at the intersection of the back surface 222 and the side surface 224 as a block belonging to both the back surface 222 and the side surface 224, or as a block belonging to only one of the back surface 222 and the side surface 224.

[0061] With this configuration, even if noise occurs due to mismatching of relative distances in distance image 214, grouping unit 172 can appropriately identify three-dimensional object candidate 220h corresponding to back surface 222 and three-dimensional object candidate 220i corresponding to side surface 224. As a result, pairing unit 174 can appropriately pair grouped three-dimensional object candidates 220h, 220i, and can improve the accuracy of identifying three-dimensional objects.

[0062] Here, the pair prediction unit 176 predicts the position of the pair of the back surface 222 and the side surface 224, but the grouping unit 172 performs grouping of the back surface 222 and grouping of the side surface 224 independently. Therefore, the grouping unit 172 can appropriately identify the back surface 222 and the side surface 224 independently.

[0063] Also provided are a program that causes a computer to function as the vehicle exterior environment recognition device 120, and a storage medium, such as a computer-readable flexible disk, a magneto-optical disk, a ROM, a CD, a DVD, or a BD, on which the program is recorded. Here, the program refers to a data processing means written in any language or description method.

[0064] Although the preferred embodiment of the present invention has been described above with reference to the accompanying drawings, it goes without saying that the present invention is not limited to such an embodiment. It is clear that a person skilled in the art can think of various modifications or alterations within the scope of the claims, and it is understood that these also naturally belong to the technical scope of the present invention.

[0065] For example, in the above embodiment, an example has been described in which the preceding vehicle is located to the right of the host vehicle 1 on the screen, and the grouping unit 172 identifies the left side surface 224 of the preceding vehicle. However, this is not the only case, and when the preceding vehicle is located to the left of the host vehicle 1 on the screen, the grouping unit 172 identifies the right side surface 224 of the preceding vehicle.

[0066] In the above embodiment, the three-dimensional object candidate 220 is specified as either the back surface 222 or the side surface 224, but there are cases where it is not possible to clearly specify whether the three-dimensional object candidate 220 is the back surface 222 or the side surface 224 until the pairing unit 174 specifies it. In this case, the grouping unit 172 may process the two three-dimensional object candidates 220 that can be paired as the left surface and the right surface, or as the first surface and the second surface, respectively.

[0067] It should be noted that the steps of the vehicle exterior environment recognition method in this specification do not necessarily need to be processed in chronological order according to the order described in the flowchart, and may include parallel or subroutine processing. [Industrial Applicability]

[0068] The present invention can be used in an exterior environment recognition device that identifies a three-dimensional object that exists in the traveling direction of a host vehicle. [Explanation of symbols]

[0069] 110 Imaging device 120 External environment recognition device 170 Position derivation part 172 Grouping Section 174 Pairing Section 176 Pair Prediction Section 222 Back 224 Side 232 Predicted back line 234 Predicted Side Lines

Claims

1. a position derivation unit that derives a three-dimensional position of each block in a captured image; a grouping unit that groups blocks whose difference in three-dimensional position is within a predetermined range to identify solid object candidates; a pairing unit that projects the three-dimensional object candidate onto a horizontal plane, classifies the three-dimensional object candidate into a back surface and a side surface based on an angle with respect to a depth direction, and identifies pairs by pairing the back surface and the side surface; a pair prediction unit for predicting a future position of the pair; Equipped with The grouping unit includes: the pair prediction unit groups blocks whose distance from a predicted back surface line indicating the position of the back surface of the pair predicted in the past is within a predetermined range, thereby identifying the solid object candidate corresponding to the back surface; An exterior environment recognition device that identifies the three-dimensional object candidate corresponding to the side by grouping blocks whose distance from a predicted side line indicating the position of the side of the pair previously predicted by the pair prediction unit is within a predetermined range.

2. The pairing unit, correcting the position of the back surface of the pair based on the position of the back surface of the pair previously predicted by the pair prediction unit; correcting the positions of the side faces of the pair based on the positions of the side faces of the pair previously predicted by the pair prediction unit; The pair prediction unit predicting the predicted back surface line in the future based on the back surface positions of the pair corrected by the pairing unit; The vehicle exterior environment recognition device according to claim 1 , wherein the pairing unit predicts the predicted side line in the future based on the positions of the side surfaces of the pair corrected.

Images