Outdoor environment recognition device

Abstract

To provide a vehicle surroundings recognition device which improves accuracy in determining a preceding vehicle.SOLUTION: A vehicle surroundings recognition device performs processing including a step S206 of using semantic segmentation to identify a vehicle class in luminance images captured by an imaging apparatus, a step S202 of generating a depth map from a plurality of luminance images, a step S208 of identifying an ineffective pixel corresponding to the vehicle class, of which the parallax is ineffective, in the depth map, a step S210 of identifying an effective pixel vertically lower than the ineffective pixel, of which the parallax is effective, a step S212 of interpolating the parallax of the ineffective pixel with the parallax of the identified effective pixel, and a step S214 of identifying an object on the basis of the depth map interpolated with the parallax.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an out-of-vehicle environment recognition device that identifies a specific object existing in the traveling direction of a host vehicle.

Background Art

[0002] Conventionally, as disclosed in Patent Document 1, there is known a technique for detecting a preceding vehicle located in front of a host vehicle and reducing damage caused by a collision with the preceding vehicle or performing follow-up control to keep the inter-vehicle distance from the preceding vehicle at a safe distance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to realize the above-described follow-up control for a preceding vehicle, a host vehicle determines whether a three-dimensional object located in front is a specific object such as a preceding vehicle. For example, a host vehicle uses two imaging devices separated in the horizontal direction, recognizes the shape and relative distance of a three-dimensional object in a luminance image and a distance image, and determines the three-dimensional object in the image as a preceding vehicle.

[0005] However, if raindrops or the like exist in the detection area of one of the two imaging devices, pattern matching of the image is not appropriately performed in the process of generating the distance image, leading to a decrease in the determination accuracy of the preceding vehicle. In this case, a host vehicle can perform monocular tracking to identify the preceding vehicle from the image of one of the two imaging devices that is not affected by raindrops. However, if raindrops or the like exist in the detection areas of both imaging devices, the determination accuracy of the preceding vehicle may decrease even by monocular tracking.

[0006] In view of such problems, an object of the present invention is to provide an out-of-vehicle environment recognition device capable of improving the determination accuracy of a preceding vehicle.

Means for Solving the Problems

[0007] To solve the above problems, the out-of-vehicle environment recognition device of the present invention includes one or more processors and one or more memories connected to the processors. The processors cooperate with a program included in the memories and use semantic segmentation to identify the vehicle class in a luminance image captured by an imaging device, generate a distance image from a plurality of the luminance images, identify invalid pixels in the distance image where the parallax corresponding to the vehicle class is invalid, identify valid pixels with valid parallax vertically below the invalid pixels, interpolate the parallax of the invalid pixels with the parallax of the identified valid pixels, and identify a specific object based on the distance image with interpolated parallax.

Effects of the Invention

[0008] According to the present invention, it is possible to improve the determination accuracy of a preceding vehicle.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Dimensions, materials, and other specific numerical values shown in such embodiments are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In the present specification and drawings, elements having substantially the same function and configuration are denoted by the same reference numerals to omit redundant description, and elements not directly related to the present invention are not shown.

[0011] (Vehicle exterior environment recognition system 100) FIG. 1 is a block diagram showing the connection relationship of the vehicle exterior environment recognition system 100. The vehicle exterior environment recognition system 100 includes an imaging device 110, a vehicle exterior environment recognition device 120, and a vehicle control device 130.

[0012] The imaging device 110 is configured to include an image sensor such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor). The imaging device 110 can image the out-of-vehicle environment in front of the host vehicle 1 and generate a luminance image including at least luminance information, for example, a color image or a monochrome image. Also, the imaging devices 110 are arranged at a distance from each other in a substantially horizontal direction such that the optical axes of the two imaging devices 110 are substantially parallel on the traveling direction side of the host vehicle 1. The imaging device 110 continuously generates a luminance image obtained by imaging a three-dimensional object existing in the detection region in front of the host vehicle 1, for example, for each frame of 1 / 60 second.

[0013] Also, the out-of-vehicle environment recognition device 120 recognizes the out-of-vehicle environment through the luminance image acquired from the imaging device 110 and the distance image based on the two luminance images. Then, the out-of-vehicle environment recognition device 120 performs speed control and steering angle control in the traveling of the host vehicle 1 based on the recognized out-of-vehicle environment and the traveling state of the host vehicle 1. The out-of-vehicle environment recognition device 120 will be described in detail later.

[0014] The vehicle control device 130 is composed of an ECU (Electronic Control Unit), receives an operation input from the driver through the steering wheel 132, the accelerator pedal 134, and the brake pedal 136, and controls the steering mechanism 142, the drive mechanism 144, and the braking mechanism 146 with reference to the information generated by the out-of-vehicle environment recognition device 120.

[0015] (Out-of-vehicle environment recognition device 120) FIG. 2 is a functional block diagram showing the schematic functions of the out-of-vehicle environment recognition device 120. As shown in FIG. 2, the out-of-vehicle environment recognition device 120 includes an I / F unit 150, a data holding unit 152, and a central control unit 154.

[0016] The I / F unit 150 is an interface for bidirectional information exchange between the imaging device 110 and the vehicle control device 130. The data holding unit 152 is composed of a RAM, a flash memory, an HDD, etc., and holds various information necessary for the processing of each of the following functional modules.

[0017] The central control unit 154 is composed of a semiconductor integrated circuit including a processor, a ROM storing programs, etc., and a RAM as a work area, etc., and controls the I / F unit 150, the data holding unit 152, etc. through the system bus 156. Also, in the present embodiment, in the central control unit 154, the processor functions as functional modules such as an image acquisition unit 160, a distance image generation unit 162, a monocular tracking unit 164, a class identification unit 166, an invalid pixel identification unit 168, a valid pixel identification unit 170, a disparity interpolation unit 172, a three-dimensional object identification unit 174, a specific object determination unit 176, and a specific object tracking unit 178 in cooperation with the programs included in the ROM.

[0018] The central control unit 154 recognizes the shape and relative distance of three-dimensional objects in the luminance image and the distance image through such functional modules and determines the three-dimensional objects as preceding vehicles. The central control unit 154 can reduce damage caused by a collision with a preceding vehicle or perform follow-up control to keep the inter-vehicle distance from a preceding vehicle at a safe distance.

[0019] Hereinafter, an out-of-vehicle environment recognition method that extracts three-dimensional objects in front of the host vehicle 1 and determines specific objects such as preceding vehicles, which is characteristic of the present embodiment, will be described in detail based on the operations of the respective functional modules of the central control unit 154.

[0020] (Out-of-vehicle environment recognition method) FIG. 3 is a flowchart showing the flow of the vehicle external environment recognition method. The vehicle external environment recognition device 120 executes the vehicle external environment recognition method at every predetermined interruption time. In the vehicle external environment recognition method, first, the image acquisition unit 160 acquires a plurality of luminance images (S200). The distance image generation unit 162 generates a distance image from two luminance images (S202). The monocular tracking unit 164 identifies a three-dimensional object from one luminance image capable of appropriately extracting a specific object (S204). The class identification unit 166 identifies the vehicle class in the luminance image using semantic segmentation (S206). The invalid pixel identification unit 168 identifies invalid pixels in the distance image where the parallax included in the vehicle class is invalid (S208). The valid pixel identification unit 170 identifies valid pixels where the parallax is valid vertically below the invalid pixels (S210). The parallax interpolation unit 172 interpolates the parallax of the invalid pixels with the parallax of the identified valid pixels (S212). The three-dimensional object identification unit 174 identifies a three-dimensional object based on the luminance image and the distance image (S214). The specific object determination unit 176 determines a specific object (S216). The specific object tracking unit 178 performs tracking control on the determined specific object (S218).

[0021] Hereinafter, each process of the vehicle external environment recognition method will be described in detail, and processes unrelated to the features of the present embodiment will be omitted.

[0022] (Image acquisition process S200) FIGS. 4 and 5 are explanatory diagrams for explaining the luminance image, and FIG. 6 is an explanatory diagram for explaining the distance image. The image acquisition unit 160 acquires a plurality of, here two, luminance images 180 imaged with different optical axes by the imaging device 110. Here, as the luminance images 180, the image acquisition unit 160 acquires a first luminance image 180a imaged by the imaging device 110 located relatively on the right side of the host vehicle 1 shown in FIG. 4 and a second luminance image 180b imaged by the imaging device 110 located relatively on the left side of the host vehicle 1 shown in FIG. 5.

[0023] Referring to FIGS. 4 and 5, it can be understood that due to the difference in the imaging positions of the imaging device 110, the image positions of the three-dimensional objects included in the images are different in the horizontal direction between the first luminance image 180a and the second luminance image 180b. Here, the horizontal direction indicates the horizontal direction of the captured image screen, and the vertical direction indicates the vertical direction of the captured image screen.

[0024] (Distance image generation process S202) Based on the first luminance image 180a shown in FIG. 4 and the second luminance image 180b shown in FIG. 5, which are acquired by the image acquisition unit 160, the distance image generation unit 162 generates a distance image 182 capable of specifying the distance of the imaging target, as shown in FIG. 6.

[0025] Specifically, the distance image generation unit 162 uses so-called pattern matching to derive parallax information including the parallax and the image position indicating the position within the image of an arbitrary block. Specifically, a block corresponding to a block arbitrarily extracted from one luminance image (here, the first luminance image 180a) is searched for in the other luminance image (here, the second luminance image 180b). Here, the block is represented by an array of, for example, 4 pixels horizontally × 4 pixels vertically. Also, pattern matching is a method of searching for a block corresponding to a block arbitrarily extracted from one luminance image in the other luminance image.

[0026] For example, as functions for evaluating the degree of coincidence between blocks in pattern matching, there are methods such as SAD (Sum of Absolute Difference) that takes the difference in luminance, SSD (Sum of Squared intensity Difference) that uses the squared difference, and NCC (Normalized Cross Correlation) that takes the similarity of the variance value obtained by subtracting the average value from the luminance of each pixel.

[0027] The distance image generation unit 162 performs such parallax derivation processing for each block on all the blocks mapped in a detection area of, for example, 600 pixels × 200 pixels. Here, the block is 4 pixels × 4 pixels, but the number of pixels within the block can be arbitrarily set.

[0028] However, although the distance image generation unit 162 can derive parallax for each block which is a detection resolution unit, it cannot recognize what part of a three-dimensional object the block is. Therefore, the parallax information is derived independently in units of the detection resolution in the detection area, for example, in block units, rather than in units of three-dimensional objects. Here, for convenience of explanation, the blocks for which parallax has been derived are represented by black dots.

[0029] The distance image generation unit 162 converts the parallax information for each block in the distance image 182 into a relative distance using a so-called stereo method, and derives three-dimensional position information, that is, relative distance, height, and horizontal position. Here, the stereo method is a method of deriving the relative distance of a block with respect to the imaging device 110 from the parallax of the block by using triangulation. Note that in the distance image 182, parallax and relative distance may be used in the sense of the same depth direction distance.

[0030] The vehicle exterior environment recognition device 120 recognizes the vehicle exterior environment through the luminance image 180 and the distance image 182 derived in this way, and identifies, for example, a three-dimensional object in front of the host vehicle 1 as a specific object such as a preceding vehicle. However, in a predetermined environment such as rainy weather, the detection area of the imaging device 110 may be covered with foreign matters such as raindrops.

[0031] FIG. 7 and FIG. 8 are explanatory diagrams for explaining the luminance image, and FIG. 9 is an explanatory diagram for explaining the distance image. For example, when raindrops adhere to the windshield located in the detection range of one of the two imaging devices 110, the portion corresponding to the raindrops in the luminance image 180 captured by that imaging device 110 becomes unclear. For example, it is assumed that the portion 184a of the first luminance image 180a shown in FIG. 7 becomes unclear due to raindrops.

[0032] Then, even if the portion 184b corresponding to the portion 184a is clearly obtained in the second luminance image 180b shown in FIG. 8, in the generated distance image 182, as shown in FIG. 9, pattern matching cannot be performed in the portion 184c corresponding to the portion 184a. Therefore, the three-dimensional object that should originally exist cannot be detected. Thus, when the three-dimensional object cannot be effectively detected in this way, monocular tracking may be performed.

[0033] (Monocular Tracking Process S204) The monocular tracking unit 164 selects one luminance image 180 from the luminance images 180a and 180b captured by the two imaging devices 110 in which a specific object can be appropriately extracted, and extracts a three-dimensional object from the selected luminance image 180. Specifically, the monocular tracking unit 164 identifies a three-dimensional object whose size, shape, color arrangement, etc. are similar to those of the specific object within the range where the specific object identified in the previous frame is predicted to exist in the current frame in one of the luminance images 180. Since various existing techniques can be applied to such monocular tracking, the detailed description thereof is omitted here.

[0034] Even if the three-dimensional object becomes unclear in one luminance image 180 due to monocular tracking, the three-dimensional object can be clearly identified using the other luminance image 180, so it is possible to suppress a decrease in the determination accuracy of the specific object and a decrease in the performance of the following tracking control described later.

[0035] However, in a predetermined environment such as rainy weather, when raindrops adhere to the front glass located within the detection range of the imaging device 110 in both of the two imaging devices 110, the portions corresponding to the raindrops in the luminance image 180 captured by the imaging device 110 become unclear in both cases. Then, not only pattern matching becomes impossible, but there is also a possibility that the three-dimensional object cannot be identified by monocular tracking. Therefore, semantic segmentation is used to identify the three-dimensional object.

[0036] Semantic segmentation is a deep learning algorithm that associates a label, category, or class with each pixel in the luminance image 180 based on its meaning. With such semantic segmentation, the vehicle exterior environment recognition device 120 can integrally recognize an aggregate of pixels that form a characteristic class, such as a vehicle class, even if a part of one or both of the luminance images 180a and 180b is unclear. Since various existing technologies can be applied to semantic segmentation, a detailed description thereof is omitted here, and only the configuration necessary for the implementation of this embodiment will be described.

[0037] (Class identification process S206) The class identification unit 166 identifies a predetermined class in the luminance image 180 using semantic segmentation. Here, an example in which the class identification unit 166 is realized by the central control unit 154 has been described, but the present invention is not limited to this case, and it may be realized by a separate AI (Artificial Intelligence), a computing engine specialized for DNN (Deep Neural Network), or FPGA (Field Programmable Gate Array).

[0038] FIG. 10 and FIG. 11 are explanatory diagrams for explaining an example to which semantic segmentation is applied. The class identification unit 166 generates a class image 190 classified into classes as shown in FIG. 11 using semantic segmentation for the first luminance image 180a shown in FIG. 10, for example. Here, each pixel of the first luminance image 180a is associated with one of a plurality of classes.

[0039] Specifically, the class identification unit 166 derives probabilities corresponding to a plurality of classes for each pixel of the first luminance image 180a. Hereinafter, the probability corresponding to an arbitrary class is referred to as the corresponding probability. Then, the class identification unit 166 associates the class with the highest corresponding probability with an arbitrary pixel of the first luminance image 180a. For example, the pixels belonging to the road surface 192a, the lane boundary 192b, and the vehicle 192c in the first luminance image 180a of FIG. 10 are classified into the road surface class 194a, the lane boundary class 194b, and the vehicle class 194c in the class image 190 of FIG. 11, respectively.

[0040] Here, as shown in FIG. 10, even if the portion 184a of the first luminance image 180a is blurred by raindrops, as shown in FIG. 11, the portion 184d of the class image 190 is appropriately classified into the vehicle class 194c.

[0041] Note that the class identification unit 166 uses the average value or the median value of the corresponding probabilities of all the pixels included in the aggregate as the corresponding probability of the aggregate of the pixels classified into the same class.

[0042] (Invalid Pixel Identification Process S208) The invalid pixel identification unit 168 compares the class image 190 with the distance image 182 and identifies the invalid pixels of the distance image 182 corresponding to the vehicle class of the class image 190. Here, a pixel for which a valid parallax is derived effectively by pattern matching is referred to as a valid pixel, and a pixel for which a valid parallax is not derived effectively by pattern matching is referred to as an invalid pixel.

[0043] FIGS. 12 and 13 are explanatory diagrams for explaining invalid pixels. In the class image 190 of FIG. 12, it is assumed that the vehicle class 194c is identified. Further, in the distance image 182 shown in FIG. 13, it is assumed that due to some cause such as raindrops, there are no valid pixels in the region 196 corresponding to the vehicle class 194c, and all the pixels have become invalid pixels. Therefore, the invalid pixel identification unit 168 identifies all the pixels in the region 196 of the distance image 182 corresponding to the vehicle class 194c in the class image 190 as invalid pixels.

[0044] Here, first, the vehicle class 194c is specified, and then the invalid pixels corresponding to the vehicle class 194c are specified. However, not limited to such a case, first, the invalid pixels may be specified from all the pixels of the distance image 182, and it may be specified that the pixel of the class image 190 corresponding to the invalid pixel is the vehicle class 194c.

[0045] (Valid pixel specifying process S210) FIGS. 14 and 15 are explanatory diagrams for explaining the processing of the valid pixel specifying unit 170, and FIG. 16 is a flowchart showing the processing of the valid pixel specifying unit 170. The valid pixel specifying unit 170 specifies valid pixels having a valid parallax vertically below the invalid pixels for each of the invalid pixels specified by the invalid pixel specifying unit 168. Hereinafter, the processing of the valid pixel specifying unit 170 will be described with reference to FIGS. 14 to 16.

[0046] First, the valid pixel specifying unit 170 uses, as a search start point, a pixel 202b at the boundary between the vehicle class 194c and the road surface class 194a vertically below the pixel 202a of the class image 190 shown in FIG. 15 corresponding to the invalid pixel 200 of the distance image 182 shown in FIG. 14.

[0047] Specifically, as shown in FIG. 16, the valid pixel specifying unit 170 extracts one invalid pixel 200 among the invalid pixels specified by the invalid pixel specifying unit 168 (S300). The valid pixel specifying unit 170 extracts, as a search pixel, a pixel one vertically below the pixel 202a of the class image 190 corresponding to the extracted invalid pixel 200 (S302), and determines whether the search pixel is the vehicle class 194c (S304). As a result, if the search pixel is the vehicle class 194c (YES in S304), the valid pixel specifying unit 170 repeats from the process of extracting, as a search pixel, a pixel one vertically below (S302). On the other hand, if the search pixel is not the vehicle class 194c (NO in S304), the valid pixel specifying unit 170 determines that the search pixel, for example, the pixel 202b has reached the boundary between the vehicle class 194c and the road surface class 194a, and stores the position information of the pixel 202b as a search start point (S306).

[0048] Subsequently, the valid pixel specifying unit 170 specifies the valid pixel 204 in FIG. 14 corresponding to the pixel 202c in FIG. 15 located in the horizontal direction from the search start point in the class image 190. When the valid pixel specifying unit 170 recognizes a tire in the vehicle class 194c, the height to be searched is updated vertically downward, for example, from the pixel 202d to the pixel 202e.

[0049] Specifically, as shown in FIG. 16, the valid pixel specifying unit 170 extracts, as a new search pixel, a search pixel determined to be the boundary between the vehicle class 194c and the road surface class 194a, for example, a pixel one horizontal right direction of the pixel 202b (S308), and determines whether either the search pixel or a pixel one vertically downward thereof is the vehicle class 194c (S310).

[0050] As described above, since the search start point is the road surface class 194a, the search pixel in the horizontal direction thereof should originally be the road surface class 194a. However, when the vehicle class 194c includes a vehicle tire, the search pixel at that position and the pixel further vertically downward become the vehicle class 194c. Here, if both the search pixel and the pixel one vertically downward thereof are the vehicle class 194c, the valid pixel specifying unit 170 determines that the search pixel is located on the tire. In this case, since the boundary between the vehicle class 194c and the road surface class 194a should be located further vertically downward, the valid pixel specifying unit 170 updates the height to be searched vertically downward.

[0051] In step S310, if both the search pixel, for example, pixel 202d and the pixel one vertically below it are of vehicle class 194c (YES in S310), the valid pixel identification unit 170 extracts the pixel one vertically below pixel 202d as the search pixel (S312), and determines whether the search pixel is of vehicle class 194c (S314). As a result, if the search pixel is of vehicle class 194c (YES in S314), the valid pixel identification unit 170 repeats from the process of extracting the pixel one vertically below as the search pixel (S312). On the other hand, if the search pixel is not of vehicle class 194c (NO in S314), the valid pixel identification unit 170 determines that the search pixel, for example, pixel 202e is at the boundary between vehicle class 194c and road surface class 194a, and updates the search height (S316). Thus, the search in the horizontal right direction will resume from pixel 202e in FIG. 15.

[0052] Subsequently, the valid pixel identification unit 170 determines whether the search pixel is of vehicle class 194c or road surface class 194a (S318). As a result, if the search pixel is of vehicle class 194c or road surface class 194a (YES in S318), the valid pixel identification unit 170 determines whether the pixel of the distance image 182 corresponding to the search pixel is a valid pixel (S320). As a result, if the pixel of the distance image 182 corresponding to the search pixel, for example, pixel 202c is a valid pixel 204 (YES in S320), the valid pixel identification unit 170 associates the valid pixel 204 with the invalid pixel 200 as an interpolation target (S322).

[0053] On the other hand, if the pixel of the distance image 182 corresponding to the search pixel is not a valid pixel (NO in S320), the valid pixel identification unit 170 repeats from the process of extracting the pixel one horizontally to the right as the search pixel (S308). Also, if the search pixel is neither of vehicle class 194c nor of road surface class 194a (NO in S318), the valid pixel identification unit 170 gives up the search in the horizontal right direction and searches in the horizontal left direction, assuming that other three-dimensional objects such as guardrails have been identified before the valid pixel is identified.

[0054] Subsequently, the valid pixel specifying unit 170 reads out the position information of the pixel 202b which is the search start point in order to search in the horizontal left direction (S324). The valid pixel specifying unit 170 extracts, as a new search pixel, a search pixel determined to be the boundary between the vehicle class 194c and the road surface class 194a, for example, a pixel one horizontal left of the pixel 202b (S326), and determines whether either the search pixel or a pixel one vertically below it is of the vehicle class 194c (S328). As a result, if both the search pixel and the pixel one vertically below it are of the vehicle class 194c (YES in S328), the valid pixel specifying unit 170 extracts, as a new search pixel, a pixel one vertically below the search pixel (S330), and determines whether the search pixel is of the vehicle class 194c (S332). As a result, if the search pixel is of the vehicle class 194c (YES in S332), the valid pixel specifying unit 170 repeats from the process of extracting a pixel one vertically below as the search pixel (S330). On the other hand, if the search pixel is not of the vehicle class 194c (NO in S332), the valid pixel specifying unit 170 determines that the search pixel is the boundary between the vehicle class 194c and the road surface class 194a, and updates the search height (S334). Thus, the search in the horizontal left direction will resume from the pixel vertically below.

[0055] Subsequently, the valid pixel specifying unit 170 determines whether the search pixel is of vehicle class 194c or road surface class 194a (S336). As a result, if the search pixel is of vehicle class 194c or road surface class 194a (YES in S336), the valid pixel specifying unit 170 determines whether the pixel of the distance image 182 corresponding to the search pixel is a valid pixel (S338). As a result, if the pixel of the distance image 182 corresponding to the search pixel is a valid pixel (YES in S338), the valid pixel specifying unit 170 associates the valid pixel with the invalid pixel 200 as an interpolation target (S322). On the other hand, if the pixel of the distance image 182 corresponding to the search pixel is not a valid pixel (NO in S338), the valid pixel specifying unit 170 repeats from the process of extracting one pixel in the horizontal left direction as the search pixel (S326). Also, if the search pixel is neither of vehicle class 194c nor road surface class 194a (NO in S336), the valid pixel specifying unit 170 gives up the search in the horizontal left direction, assuming that another three-dimensional object such as a guardrail has been specified before the valid pixel is specified, and proceeds to step S340.

[0056] Subsequently, the valid pixel specifying unit 170 determines whether the search process for all the invalid pixels specified by the invalid pixel specifying unit 168 has been completed (S340). As a result, if the search process for all the invalid pixels has not been completed (NO in S340), the valid pixel specifying unit 170 repeats from the process of extracting one invalid pixel for which the search process has not yet been executed (S300). On the other hand, if all the invalid pixels have been processed (YES in S340), the valid pixel specifying unit 170 ends the valid pixel specifying process.

[0057] If the effective pixel identification unit 170 randomly searches for the horizontal effective pixels of the vehicle class 194c without identifying the boundary between the vehicle class 194c and the road surface class 194a, pixels indicating a white line or the like farther away than the preceding vehicle will be identified, and the parallax of the vehicle will not be appropriately shown. Here, the horizontal parallax of the road at the boundary between the vehicle class 194c and the road surface class 194a is theoretically equal to the parallax of the entire vehicle class 194c. Here, by using the boundary between the vehicle class 194c and the road surface class 194a directly below the invalid pixel 200 as the search starting point, it becomes possible to appropriately derive the parallax of the vehicle class 194c.

[0058] Also, when the effective pixel identification unit 170 recognizes the tire in the vehicle class 194c, the search height is updated vertically downward, for example, from the pixel 202d to the pixel 202e in FIG. 15. With such a configuration, since the effective pixels can be identified from the horizontal direction of the grounding position of the vehicle with respect to the road surface, it becomes possible to more appropriately derive the parallax of the vehicle class 194c.

[0059] Here, an example has been described in which the search is abandoned when the pixel reaches a class that is not the vehicle class 194c or the road surface class 194a in the horizontal rightward and leftward searches. However, not limited to such a case, a predetermined search distance may be set in the horizontal rightward and leftward directions, and the search may be abandoned when the search distance is reached.

[0060] (Disparity interpolation process S212) The disparity interpolation unit 172 interpolates the invalid pixels using the disparities of the effective pixels identified by the effective pixel identification unit 170. That is, the effective pixel identification unit 170 overwrites the disparity (relative distance) of the associated effective pixel for each invalid pixel identified by the invalid pixel identification unit 168. With such a configuration, it becomes possible to associate an assumed valid disparity with an invalid pixel having an invalid disparity.

[0061] However, since the valid pixels are identified by the complicated valid pixel identification process S210, they are liable to be affected by factors such as the derivation accuracy of the vehicle class 194c and the road surface class 194a, and their parallax may fluctuate or noise may be mixed in. Therefore, the parallax interpolation unit 172 may associate the parallax of the valid pixels associated with the invalid pixels not directly, but the parallax after stabilization by a plurality of low-pass filters or the like.

[0062] (Stereo object identification process S214) The stereo object identification unit 174 uses the distance image 182 with interpolated parallax to first identify the road surface in front of the host vehicle 1. Then, the stereo object identification unit 174 identifies a stereo object having a height vertically above the identified road surface. Specifically, the stereo object identification unit 174 regards a block whose height from the road surface is equal to or more than a predetermined distance (for example, 0.3 m) as a candidate for a stereo object protruding in the height direction from the road surface. The stereo object identification unit 174 groups blocks having the same relative distance from the host vehicle 1 among a plurality of blocks that are candidates for stereo objects having a height vertically above the road surface, and identifies them as stereo objects.

[0063] (Specific object determination process S216) The specific object determination unit 176 determines whether the stereo object identified by the stereo object identification unit 174 is a predetermined specific object such as a preceding vehicle, a pedestrian, a building, or the like.

[0064] (Specific object tracking process S218) The specific object tracking unit 178 estimates the position of the specific object in the current frame from the position and speed of the specific object determined by the specific object determination unit 176 in the previous frame. If the specific object determined by the specific object determination unit 176 in the current frame exists at the estimated position, the specific object tracking unit 178 associates it as the same specific object. In this way, the specific object tracking unit 178 can track the transition of the specific object in front of the host vehicle 1 and perform follow-up control.

[0065] In addition, there are also provided a program that causes a computer to function as the vehicle external environment recognition device 120, and a computer-readable storage medium such as a flexible disk, a magneto-optical disk, a ROM, a CD, a DVD, a BD, etc. that records the program. Here, the program refers to data processing means described in an arbitrary language and description method.

[0066] As described above, the preferred embodiments of the present invention have been described with reference to the accompanying drawings. Needless to say, the present invention is not limited to such embodiments. It is obvious that those skilled in the art can conceive of various modification examples or correction examples within the scope described in the claims, and it is naturally understood that they also belong to the technical scope of the present invention.

[0067] For example, in the above-described embodiment, the monocular tracking unit 164 identifies a three-dimensional object from one luminance image that can appropriately extract a specific object. When the three-dimensional object still cannot be identified, an example of improving the determination accuracy of the preceding vehicle by the class identification unit 166, the invalid pixel identification unit 168, the valid pixel identification unit 170, and the parallax interpolation unit 172 has been described. However, not limited to such a case, the determination accuracy of the preceding vehicle may be improved only by the class identification unit 166, the invalid pixel identification unit 168, the valid pixel identification unit 170, and the parallax interpolation unit 172 without providing the monocular tracking unit 164.

[0068] In addition, in the above-described embodiment, an example of independently executing monocular tracking by the monocular tracking unit 164 and parallax interpolation by the class identification unit 166, the invalid pixel identification unit 168, the valid pixel identification unit 170, and the parallax interpolation unit 172 to identify a specific object has been described. However, not limited to such a case, a specific object may be identified using a calculation result obtained by weighted averaging monocular tracking and parallax interpolation.

[0069] For example, the disparity derived by monocular tracking is defined as the monocular disparity, and the probability that the monocular disparity is reliable is defined as the monocular probability. Also, the disparity used for disparity interpolation is defined as the interpolation disparity, and the above-mentioned corresponding probability is used as the probability that the disparity is reliable. Then, as a weighted average, the result of adding the monocular disparity × monocular probability and the interpolation disparity × corresponding probability and dividing by the monocular probability + corresponding probability is defined as the weighted average of the disparity. By using such a weighted average, monocular tracking and disparity interpolation can be combined based on their reliability, so that it is possible to improve the determination accuracy of the preceding vehicle.

[0070] Note that each step of the method for recognizing the out-of-vehicle environment in this specification does not necessarily need to be processed in time series according to the order described as a flowchart, and may include parallel or subroutine processing.

Explanation of Signs

[0071] 110 Imaging device 120 Out-of-vehicle environment recognition device 162 Distance image generation unit 164 Monocular tracking unit 166 Class identification unit 168 Invalid pixel identification unit 170 Valid pixel identification unit 172 Disparity interpolation unit 180 Luminance image 180a First luminance image 180b Second luminance image 182 Distance image 190 Class image 194a Road surface class 194b Lane boundary line class 194c Vehicle class

Claims

1. One or more processors, One or more memories connected to the processor, Comprising, The processor cooperates with a program included in the memory, Using semantic segmentation to identify the vehicle class in the luminance image captured by the imaging device, Generating a distance image from a plurality of the luminance images, Identifying invalid pixels in the distance image where the parallax corresponding to the vehicle class is invalid, Identifying valid pixels with valid parallax vertically below the invalid pixels, Interpolating the parallax of the invalid pixels with the parallax of the identified valid pixels, Identifying a specific object based on the distance image with interpolated parallax, An out-of-vehicle environment recognition device that executes a process including.

2. The processor, Using semantic segmentation to identify the road surface class in the luminance image, A point located at the boundary between the vehicle class and the road surface class vertically below the invalid pixel and belonging to the road surface class is used as a search starting point, Identifying the valid pixels in the horizontal direction from the search starting point, The out-of-vehicle environment recognition device according to claim 1, which executes a process including.

Images