Vehicle External Environment Recognition Device

Abstract

To provide a vehicle external environment recognition apparatus for improving the detection accuracy of floating objects.SOLUTION: A vehicle external environment recognition apparatus is provided with an imaging device, one or more processors, and one or more memories connected to the processors. The processor cooperates with a program contained in the memory to perform processing including generating a range image from a plurality of brightness images captured by the imaging device, identifying floating object classes in the brightness images using semantic segmentation, and invalidating the parallax of floating pixels which are pixels corresponding to the floating object class among the pixels of the range image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an exterior environment recognition device that identifies a specific object that exists in the traveling direction of a vehicle. [Background technology]

[0002] Conventionally, there is known a technology, such as that disclosed in Patent Document 1, that detects a preceding vehicle located in front of the vehicle, reduces damage caused by a collision with the preceding vehicle, and performs tracking control to maintain a safe distance between the vehicle and the preceding vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 3349060 Summary of the Invention [Problem to be solved by the invention]

[0004] In cold regions or at high altitudes, masses of water vapor may float above the road. Furthermore, exhaust gases emitted from the exhaust pipe of a preceding vehicle may not immediately disperse and may remain on the road. A vehicle recognizes floating objects such as water vapor and exhaust gases as three-dimensional objects through images. The vehicle may mistakenly identify the floating objects as specific objects, such as a preceding vehicle or a pedestrian. Therefore, the vehicle may execute stop control or deceleration control to avoid a collision with the floating object mistakenly identified as a specific object. This may impair the vehicle's comfortable driving experience and may also result in a collision between the vehicle and a following vehicle.

[0005] In view of the above problems, the present invention aims to provide an exterior environment recognition device that can improve the accuracy of detecting floating objects. [Means for solving the problem]

[0006] In order to solve the above problems, the vehicle exterior environment recognition device of the present invention includes one or more processors and one or more memories connected to the processors, and the processors cooperate with a program stored in the memory to generate a range image from a plurality of luminance images captured by an imaging device, and to use semantic segmentation to generate a depth image in the luminance image. Pixels Floating Object Class Whether or not to respond to and determining the pixel of the distance image. of Among them, the disparity of the floating pixels which are pixels corresponding to the floating object class , and the disparity of a neighboring pixel that is a pixel that does not correspond to the floating object class and is a pixel whose distance from the floating pixel is within a predetermined range. and invalidating the [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the accuracy of detecting floating matter. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing the connections of the exterior environment recognition system. [Figure 2] FIG. 2 is a functional block diagram showing a schematic function of the vehicle exterior environment recognition device. [Figure 3] FIG. 3 is a flowchart showing the flow of the vehicle exterior environment recognition method. [Figure 4] FIG. 4 is an explanatory diagram for explaining the luminance image. [Figure 5] FIG. 5 is an explanatory diagram for explaining the luminance image. [Figure 6] FIG. 6 is an explanatory diagram for explaining a distance image. [Figure 7] FIG. 7 is an explanatory diagram for explaining the necessity of semantic segmentation. [Figure 8] FIG. 8 is an explanatory diagram for explaining the necessity of semantic segmentation. [Figure 9] FIG. 9 is an explanatory diagram for explaining the floating object class. [Figure 10]FIG. 10 is an explanatory diagram for explaining the processing of the second parallax nullifying unit. [Figure 11] FIG. 11 is an explanatory diagram for explaining the processing of the second parallax nullifying unit. [Figure 12] FIG. 12 is an explanatory diagram for explaining the operation of the invalidation canceling unit. [Figure 13] FIG. 13 is an explanatory diagram for explaining the operation of the invalidation canceling unit. DETAILED DESCRIPTION OF THE INVENTION

[0009] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Dimensions, materials, and other specific values ​​shown in the embodiments are merely examples for facilitating understanding of the invention and, unless otherwise specified, do not limit the present invention. In this specification and drawings, elements having substantially the same functions and configurations are designated by the same reference numerals to avoid redundant explanation, and elements not directly related to the present invention are not shown.

[0010] (Exterior environment recognition system 100) 1 is a block diagram showing the connections of the vehicle exterior environment recognition system 100. The vehicle exterior environment recognition system 100 includes an imaging device 110, an exterior environment recognition device 120, and a vehicle control device .

[0011] The imaging device 110 includes an imaging element such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor). The imaging device 110 captures an image of the external environment ahead of the vehicle 1 and can generate a luminance image containing at least luminance information, for example, a color (RGB) image or a monochrome (grayscale) image. The imaging devices 110 are arranged in the traveling direction of the vehicle 1, spaced apart in a substantially horizontal direction so that the optical axes of the two imaging devices 110 are substantially parallel. The imaging devices 110 continuously generate luminance images of three-dimensional objects present in a detection area ahead of the vehicle 1, for example, at 1 / 60-second frames.

[0012] The vehicle exterior environment recognition device 120 recognizes the environment outside the vehicle through the brightness image acquired from the imaging device 110 and the distance image generated from the two brightness images. The vehicle exterior environment recognition device 120 then controls the speed and steering angle of the vehicle 1 based on the recognized vehicle exterior environment and the traveling conditions of the vehicle 1. The vehicle exterior environment recognition device 120 will be described in detail later.

[0013] The vehicle control device 130 is composed of an ECU (Electronic Control Unit) and receives driver inputs via a steering wheel 132, an accelerator pedal 134, and a brake pedal 136, and controls a steering mechanism 142, a drive mechanism 144, and a braking mechanism 146 by referring to information generated by the exterior environment recognition device 120.

[0014] (External environment recognition device 120) 2 is a functional block diagram showing the general functions of the vehicle exterior environment recognition device 120. As shown in FIG. 2, the vehicle exterior environment recognition device 120 includes an I / F unit 150, a data storage unit 152, and a central control unit 154.

[0015] The I / F unit 150 is an interface for two-way information exchange with the imaging device 110 and the vehicle control device 130. The data storage unit 152 is composed of RAM, flash memory, HDD, etc., and stores various information required for processing by each of the functional modules described below.

[0016] The central control unit 154 is configured with a semiconductor integrated circuit including a processor, a ROM storing programs, etc., a RAM as a work area, etc., and controls the I / F unit 150, the data storage unit 152, etc. via a system bus 156. In this embodiment, the processor in the central control unit 154 works in cooperation with the programs stored in the ROM to function as functional modules such as an image acquisition unit 160, a distance image generation unit 162, a specified object tracking unit 164, a first class identification unit 166, a second class identification unit 168, a first parallax invalidation unit 170, a second parallax invalidation unit 172, an invalidation cancellation unit 174, a three-dimensional object identification unit 176, and a specified object determination unit 178.

[0017] Through these functional modules, the central control unit 154 recognizes the shape and relative distance of a three-dimensional object in the brightness image and the distance image, and determines that the predetermined three-dimensional object is a specific object such as a preceding vehicle. The central control unit 154 can reduce damage caused by a collision with a preceding vehicle, and can perform tracking control to maintain a safe distance from the preceding vehicle.

[0018] However, a three-dimensional object recognized through a brightness image or a distance image is not necessarily a specific object with which a collision must be avoided. For example, although floating objects such as water vapor or exhaust gases remaining on the road are displayed in a brightness image, the vehicle 1 does not need to avoid such floating objects. Even if the vehicle 1 were to rush into a floating object, no problem would arise. Here, floating objects include water vapor, which is water vapor that is liquid water that has turned into gas, and smoke, which is a collection of solid or liquid particles that is generated when flammable materials burn.

[0019] If the central control unit 154 erroneously determines that such a floating object is a specific object, it may activate, for example, AEB (Automatic Emergency Braking) and execute stop control or deceleration control of the host vehicle 1. This may impair the comfortable driving of the host vehicle 1 and may also cause a following vehicle to collide with the host vehicle 1. Therefore, in this embodiment, the detection accuracy of floating objects is improved and the host vehicle 1 is appropriately controlled.

[0020] The following describes a vehicle exterior environment recognition method that is characteristic of this embodiment, which extracts three-dimensional objects ahead of the vehicle 1 and determines specific objects such as a preceding vehicle and floating objects such as water vapor. In addition, the operation of each functional module of the central control unit 154 in the vehicle exterior environment recognition method will be described in detail.

[0021] (How to recognize the environment outside the vehicle) FIG. 3 is a flowchart showing the flow of the vehicle exterior environment recognition method. The vehicle exterior environment recognition device 120 executes the vehicle exterior environment recognition method at predetermined interruption times. In the vehicle exterior environment recognition method, first, the image acquisition unit 160 acquires multiple luminance images (S200). The distance image generation unit 162 generates a distance image from two luminance images (S202). The specific object tracking unit 164 identifies the position in the current frame of a three-dimensional object determined to be a specific object in the previous frame (S204). The first class identification unit 166 identifies multiple predetermined classes in the luminance image using semantic segmentation (S206). The second class identification unit 168 identifies a floating object class in the luminance image using semantic segmentation (S208). The first parallax invalidation unit 170 invalidates the parallax of floating pixels, which are pixels corresponding to the floating object class, among the pixels in the distance image (S210). The second parallax nullifying unit 172 nullifies the parallax of neighboring pixels of the distance image, which are pixels whose distance to the floating pixel is within a predetermined range (S212). The nullification cancellation unit 174 cancels the nullification of the parallax of the neighboring pixels when the neighboring pixels satisfy a predetermined cancellation condition (S214). The three-dimensional object identifying unit 176 identifies a three-dimensional object based on the brightness image and the distance image in which the parallax of some pixels has been nullified (S216). The specified object determining unit 178 determines whether the three-dimensional object is a specified object (S218).

[0022] Each step of the exterior environment recognition method will be described in detail below, and a description of steps unrelated to the features of this embodiment will be omitted.

[0023] (Image acquisition process S200) 4 and 5 are explanatory diagrams for explaining a luminance image, and Fig. 6 is an explanatory diagram for explaining a distance image. The image acquisition unit 160 acquires a plurality of images captured by the imaging device 110 with different optical axes, here two luminance images 180. Here, it is assumed that the image acquisition unit 160 acquires, as luminance images 180, a first luminance image 180a captured by the imaging device 110 located relatively to the right of the host vehicle 1 as shown in Fig. 4, and a second luminance image 180b captured by the imaging device 110 located relatively to the left of the host vehicle 1 as shown in Fig. 5.

[0024] 4 and 5, it can be seen that the image positions of three-dimensional objects included in the first luminance image 180a and the second luminance image 180b differ in the horizontal direction due to differences in the imaging positions of the imaging device 110. Here, horizontal refers to the horizontal direction of the captured image screen, and vertical refers to the vertical direction of the captured image screen.

[0025] (Distance image generation process S202) The distance image generation unit 162 generates a distance image 182, such as that shown in FIG. 6, that can identify the distance to the imaging target, based on the first luminance image 180a shown in FIG. 4 and the second luminance image 180b shown in FIG. 5, both of which are acquired by the image acquisition unit 160.

[0026] Specifically, the distance image generation unit 162 uses so-called pattern matching to derive disparity information including disparity and an image position indicating the position of an arbitrary block within the image. Then, the distance image generation unit 162 searches the other luminance image, here the second luminance image 180b, for a block corresponding to a block arbitrarily extracted from one luminance image, here the first luminance image 180a. The block is represented, for example, by an array of 4 pixels horizontally by 4 pixels vertically. Here, pixels included in a block from which disparity information has been derived are considered valid pixels and have the derived disparity information. On the other hand, pixels included in a block from which disparity information has not been derived do not have disparity information.

[0027] For example, functions used to evaluate the degree of match between blocks in pattern matching include SAD (Sum of Absolute Difference), which takes the difference in brightness, SSD (Sum of Squared Intensity Difference), which uses the square of the difference, and NCC (Normalized Cross Correlation), which takes the similarity of the variance value obtained by subtracting the average value from the brightness of each pixel.

[0028] Distance image generation unit 162 performs this type of block-by-block parallax derivation processing for all blocks displayed in a detection area of, for example, 600 pixels by 200 pixels. Here, the blocks are assumed to be 4 pixels by 4 pixels, but the number of pixels in a block can be set arbitrarily.

[0029] However, although distance image generation unit 162 can derive disparity for each block, which is the detection resolution unit, it cannot recognize what kind of three-dimensional object the block is part of. Therefore, disparity information is derived independently not for each three-dimensional object, but for each detection resolution unit in the detection area, for example, for each block. In Figure 6, for convenience of explanation, the blocks for which disparity is derived are represented by black dots.

[0030] Distance image generation unit 162 converts the parallax information for each block in distance image 182 into relative distance (z) using a so-called stereo method, and derives three-dimensional position information, i.e., horizontal distance (x), vertical distance (y), and relative distance (z). Here, stereo is a method of deriving the relative distance of a block to imaging device 110 from the parallax of that block by using triangulation. Note that in distance image 182, parallax and relative distance may be used to mean the same distance in the depth direction.

[0031] The vehicle exterior environment recognition device 120 recognizes the vehicle exterior environment based on the brightness image 180 and distance image 182 thus derived, and identifies, for example, a three-dimensional object ahead of the vehicle 1 as a specific object such as a leading vehicle.

[0032] (Specific object tracking process S204) The specified object tracking unit 164 estimates the position in the current frame from the position and speed in the distance image 182 of a three-dimensional object that the specified object determination unit 178, which will be described later, determined to be a specified object in the previous frame. Specifically, if there is a three-dimensional object in the vicinity of the estimated position that is similar in size, shape, color arrangement, etc. to the specified object, the specified object tracking unit 164 associates that three-dimensional object with the specified object determined in the previous frame as the same specified object. In this way, the specified object tracking unit 164 can track the movement of the specified object ahead of the vehicle 1 and perform tracking control. Since various existing technologies, such as technologies using machine learning, can be applied to the processing by the specified object tracking unit 164, detailed explanations thereof will be omitted here.

[0033] (First class identification process S206) The first class identification unit 166 identifies a plurality of predetermined classes in the intensity image 180 using semantic segmentation.

[0034] Here, semantic segmentation is a deep learning algorithm that associates a label, category, or class with each pixel in the intensity image 180 based on the meaning of the pixel as an image, and a confidence level that indicates the likelihood of that label, category, or class.

[0035] Such semantic segmentation enables the exterior environment recognition device 120 to integrally recognize a collection of pixels forming a characteristic class, for example, a vehicle class indicating a vehicle, even if one or both of the luminance images 180a, 180b are partially blurred. Since various existing technologies can be applied to semantic segmentation, detailed explanations thereof will be omitted here, and only the configuration necessary to realize this embodiment will be described.

[0036] 7 and 8 are explanatory diagrams illustrating the necessity of semantic segmentation. The first class identification unit 166 uses semantic segmentation on the first luminance image 180a shown in Fig. 7, for example, to generate a class image 190 classified into classes as shown in Fig. 8. Here, each pixel of the first luminance image 180a is associated with one of a plurality of classes selected from, for example, road surface, lane boundary line, vehicle, sidewalk, fence, pole, cone, shrub, and sky.

[0037] Specifically, the first class identification unit 166 derives a probability that each pixel in the first luminance image 180a corresponds to one of multiple classes. Hereinafter, the probability that each pixel corresponds to a class is referred to as reliability. The first class identification unit 166 then associates the class with the highest reliability for a given pixel in the first luminance image 180a. For example, the pixel with the highest reliability, which is the road surface 192a, the pixel with the highest reliability, which is the lane boundary line 192b, and the pixel with the highest reliability, which is the vehicle 192c in the first luminance image 180a in FIG. 7, are classified into the road surface class 194a, the lane boundary line class 194b, and the vehicle class 194c in the class image 190 in FIG. 8, respectively.

[0038] Here, even if the portion 184a of the first luminance image 180a is blurred by raindrops as shown in Fig. 7, the first class identification unit 166 appropriately classifies the portion 184d of the class image 190 into the vehicle class 194c by semantic segmentation as shown in Fig. 8. Therefore, the first class identification unit 166 can appropriately extract the preceding vehicle.

[0039] The first class identification unit 166 may use the average or median of the reliabilities of all pixels included in the collection as the reliability of the entire collection of pixels classified into the same class.

[0040] (Second class identification process S208) The second class identification unit 168 uses semantic segmentation to identify a specific class in the intensity image 180, here a floating object class indicating a specific object, that is, a floating object, based on the confidence that it is a floating object. Here, the second class identification unit 168 is the same as the first class identification unit 166 in that it uses semantic segmentation, but differs in that it narrows down the classes to be identified to the floating object class.

[0041] FIG. 9 is an explanatory diagram illustrating floating object classes. The second class identification unit 168 distinguishes between two classes for any pixel: whether the pixel is in the floating object class or a class different from the floating object class. For example, in the example of FIG. 9, the second class identification unit 168 identifies a floating object class 194d, as indicated by cross-hatching in the figure, and classifies all other areas as a class 194e, which is different from the floating object class 194d. Unlike the case of distinguishing between multiple classes as in FIG. 8, the second class identification unit 168 distinguishes only whether the pixel is in the floating object class 194d, and therefore can identify floating objects with higher accuracy than the first class identification unit 166.

[0042] Note that an example is described here in which first class identification unit 166 and second class identification unit 168 function independently, and floatable matter is identified using floatable matter class 194d identified by second class identification unit 168. However, if floatable matter class 194d is included in the identification targets of first class identification unit 166 and the identification accuracy is sufficient, floatable matter class 194d identified by first class identification unit 166 may be applied to this embodiment. In this case, the processing of second class identification unit 168 can be omitted, thereby reducing the processing load.

[0043] In addition, here, an example has been described in which the specific object tracking unit 164, the first class identification unit 166, and the second class identification unit 168 are realized by the central control unit 154, but this is not limiting, and they may be realized by a separate calculation engine specialized in AI (Artificial Intelligence), DNN (Deep Neural Network), or the like, or an FPGA (Field Programmable Gate Array). Module Not limited to the above, other functions shown in this embodiment Module This can also be achieved using a separate calculation engine or FPGA specialized for AI, DNN, etc.

[0044] (First parallax invalidation process S210) The first parallax nullifying unit 170 nullifies the distance image 182 Among the pixels, the second class identification unit 168 invalidates the parallax of floating pixels that are identified as the floating object class 194d. Here, invalidating the parallax means that when a valid value is derived as the parallax of the pixel, the pixel is not recognized as parallax in identifying the processing target of the vehicle exterior environment recognition method. However, since the parallax of the floating pixels is used in the second parallax invalidation unit 172 described later, the parallax itself is not deleted.

[0045] Here, by invalidating the parallax for a pixel identified by the second class identification unit 168 as being of the floating object class 194d, it is determined that no three-dimensional object exists at the position of that pixel. Therefore, the central control unit 154 will not mistakenly determine that a floating object such as water vapor or exhaust gas is a specific object such as a preceding vehicle. Furthermore, the central control unit 154 will not execute stop control or deceleration control to avoid a collision with a floating object that has been mistakenly determined as a specific object. This configuration makes it possible to maintain a comfortable driving of the host vehicle 1. Furthermore, following vehicles will not collide with the host vehicle 1.

[0046] (Second parallax invalidation process S212) However, floating objects are not necessarily identified as floating object class 194d because their density is not uniform and some parts are transparent to the background. For example, even if a floating object has a low density and is transparent to the background, it may be identified as class 194e, which is different from floating object class 194d. If a floating object is classified as class 194e, which is different from floating object class 194d, the area of ​​class 194e, which is different from floating object class 194d, may be determined to be an object for which collision should be avoided, and stop control or deceleration control of the host vehicle 1 may be executed. Therefore, the exterior environment recognition device 120 also identifies pixels located near floating pixels as floating objects.

[0047] The second parallax invalidating unit 172 invalidates the parallax of a neighboring pixel, which is a pixel whose distance to the floating pixel corresponding to the floating object class 194d is within a predetermined range. However, since the parallax of the neighboring pixel is used in the invalidation canceling unit 174 described later, the parallax itself is not deleted.

[0048] 10 and 11 are explanatory diagrams for explaining the processing of second parallax nullifying section 172. For example, when pixels having parallax in distance image 182 are plotted on a zx plane defined by relative distance (z) and horizontal distance (x), the result is as shown in Fig. 10. Note that in Fig. 10, pixels are indicated by white or black circles.

[0049] Here, it is assumed that floating objects are present in distance image 182. In Fig. 10, floating pixels 196 corresponding to floating object class 194d, indicated by black circles, have parallax in distance image 182. Furthermore, floating pixels 196 are concentrated within a predetermined range on the zx plane in Fig. 10.

[0050] Furthermore, pixels that are not classified into the floating object class 194d exist near the floating pixel 196 in the zx plane. These pixels are not located near other specific objects such as vehicles, but are located only near the floating pixel 196, and are therefore highly likely to be floating objects. Therefore, the second parallax invalidating unit 172 determines pixels that are within a predetermined range of distance from the floating pixel 196 as neighboring pixels 198 that are highly likely to be floating objects, and invalidates the parallax of these pixels, as with the floating pixel 196.

[0051] 10, the second parallax invalidating unit 172 derives a minimum value Zmin of the relative distance (z) between the host vehicle 1 and the floating pixel 196 and a maximum value Zmax of the relative distance (z) between the host vehicle 1 and the floating pixel 196. Then, the second parallax invalidating unit 172 determines pixels that are not floating pixels 196 and that are located between the minimum value Zmin and the maximum value Zmax as neighboring pixels 198, and invalidates the parallax.

[0052] Alternatively, the following processing may be performed. That is, as shown in Fig. 11, the second parallax nullifying unit 172 derives a minimum value Zmin of the relative distance (z) between the host vehicle 1 and the floating pixel 196, a maximum value Zmax of the relative distance (z) between the host vehicle 1 and the floating pixel 196, a left end Xmin of the horizontal distance (x) between the host vehicle 1 and the floating pixel 196, and a right end Xmax of the horizontal distance (x) of the floating pixel 196. Then, the second parallax nullifying unit 172 determines, as neighboring pixels 198, pixels that are not floating pixels 196 and that are located between the minimum value Zmin and the maximum value Zmax and between the left end Xmin and the right end Xmax, and that are not floating pixels 196, and nullifies the parallax.

[0053] Alternatively, a pixel existing within a range in which the floating pixel 196 is located with respect to a three-dimensional position including a horizontal distance (x), a vertical distance (y), and a relative distance (z) may be set as the neighboring pixel 198. The second parallax nullifying unit 172 may determine, for example, the Euclidean distance (√(horizontal distance 2 + vertical distance 2 +Relative Distance 2 )) is included in a predetermined range, the pixel is determined to be a neighboring pixel 198, and the parallax is invalidated. Alternatively, a pixel included within a predetermined distance (for example, 2 m) from the center position of the range in which the floating pixel 196 is located may be determined to be a neighboring pixel 198. Alternatively, the average value of all the floating pixels 196, that is, a pixel included within a predetermined distance (for example, 2 m) from the center of gravity of the floating pixel 196 may be determined to be a neighboring pixel 198.

[0054] Here, not only the floating pixel 196 corresponding to the floating object class 194d but also the neighboring pixel 198 located nearby are considered to be floating objects. In this way, the parallax of the neighboring pixel 198 that should be judged as a floating object can be invalidated, so that the exterior environment recognition device 120 will no longer mistakenly judge floating objects such as water vapor or exhaust gas as specific objects such as a preceding vehicle.

[0055] For convenience of explanation, an example has been described in which the processing of the second parallax nullifying unit 172 starts after the processing of the first parallax nullifying unit 170 is completed. However, in reality, the first parallax nullifying unit 170 sequentially extracts each pixel in the detection area, and if there is a pixel corresponding to the floating object class 194d, the first parallax nullifying unit 170 designates the pixel as a floating pixel 196 and invalidates the parallax. The second parallax nullifying unit 172 sequentially extracts each pixel located within a predetermined neighboring range of the floating pixel 196 whose parallax has been invalidated, and if there is a pixel whose distance from the floating pixel 196 is within a predetermined range and which is not a floating pixel 196, the second parallax nullifying unit 172 designates the pixel as a neighboring pixel 198 and invalidates the parallax. In other words, the processing of the first parallax nullifying unit 170 and the processing of the second parallax nullifying unit 172 are performed in parallel, and the processing of the second parallax nullifying unit 172 is executed depending on the result of the processing of the first parallax nullifying unit 170. Here, the neighborhood range is a range of a predetermined number of horizontal pixels and a predetermined number of vertical pixels on the luminance image 180, centered on the floating pixel 196 for which parallax is nullified.

[0056] (Invalidation cancellation process S214) However, if the parallax is simply disabled for pixels located near the floating pixel 196, the following phenomenon may occur.

[0057] The second parallax nullifying unit 172 may mistakenly regard a three-dimensional object that is close to the floating pixel 196 but is not actually a floating object as a floating object. Also, although a leading vehicle exists in the depth direction of the floating object, the second parallax nullifying unit 172 may mistakenly regard the pixel as a floating object.

[0058] In this case, the second parallax invalidation unit 172 invalidates the parallax of the pixel that should be considered a specific object. As a result, the exterior environment recognition device 120 is unable to perform such processing on the specific object that should be used to mitigate damage caused by a collision with a leading vehicle or to perform tracking control to maintain a safe distance from the leading vehicle. Therefore, the invalidation cancellation unit 174 strictly determines whether the neighboring pixel 198 is another specific object.

[0059] The invalidation cancellation unit 174 determines whether the neighboring pixel 198 invalidated by the second parallax invalidation unit 172 satisfies a predetermined cancellation condition, and cancels the invalidation of the parallax of the neighboring pixel 198 that satisfies the cancellation condition. In other words, the parallax of that pixel is used effectively.

[0060] For example, even if a pixel is determined to be a neighboring pixel 198, if there is a possibility that the pixel corresponds to a specific object such as a preceding vehicle, the neighboring pixel 198 Parallax This is because it is more important to avoid a situation where the host vehicle 1 mistakes a floating object for a leading vehicle and collides with it than to avoid a situation where the host vehicle 1 mistakes a floating object for a leading vehicle and executes stop control or deceleration control.

[0061] 12 and 13 are explanatory diagrams illustrating the operation of the invalidation cancellation unit 174. As shown in FIG. 12, the second class identification unit 168 identifies a floating object class 194d, which is indicated by cross-hatching in the figure. The first parallax invalidation unit 170 identifies a floating pixel 196 corresponding to the floating object class 194d. Then, the second parallax invalidation unit 172 identifies a neighboring pixel 198 whose distance from the floating pixel 196 is within a predetermined range. However, the neighboring pixel 198 is merely a pixel located in the vicinity of the floating pixel 196, and is not the floating pixel 196 itself. Furthermore, the neighboring pixel 198 may be a pixel identified as the vehicle class 194c by the first class identification unit 166.

[0062] Therefore, even if the second parallax invalidating unit 172 has identified a neighboring pixel 198, if the neighboring pixel 198 is a pixel identified as the vehicle class 194c by the first class identifying unit 166, the invalidation cancellation unit 174 cancels the invalidation of the parallax of the neighboring pixel 198. In the example of Fig. 12, if the neighboring pixel 198 overlaps with the vehicle class 194c, the invalidation cancellation unit 174 cancels the invalidation of the parallax of the neighboring pixel 198.

[0063] As described above, the specific object tracking unit 164 estimates the position of a three-dimensional object in the current frame from the position and speed of the same three-dimensional object determined to be a vehicle in the previous frame, as shown in Fig. 13. The specific object tracking unit 164 recognizes the preceding vehicle existing at the estimated position as a rectangular area 200 consisting of a vertical line passing through the horizontal end of the preceding vehicle and a horizontal line passing through the vertical end. This is to allow the exterior environment recognition device 120 to appropriately recognize the size of the preceding vehicle and to reduce the processing load by targeting a simple shape.

[0064] Then, even if the second parallax invalidating unit 172 identifies a pixel as a neighboring pixel 198, if the neighboring pixel 198 is a pixel within the area 200 determined by the specific object tracking unit 164 to be a preceding vehicle, the invalidation canceling unit 174 cancels the invalidation of the parallax of the neighboring pixel 198. In the example of Fig. 13, if the neighboring pixel 198 overlaps with the area 200 determined to be a preceding vehicle, the invalidation canceling unit 174 cancels the invalidation of the parallax of the neighboring pixel 198.

[0065] Such an invalidation canceling unit 174 198 The cancellation conditions for canceling the invalidation of disparity can be either that the nearby pixel 198 is identified as a vehicle class 194c using semantic segmentation, or that the nearby pixel 198 is included in the position in the current frame of a three-dimensional object that was determined to be a vehicle in the previous frame, or both.

[0066] With this configuration, the vehicle exterior environment recognition device 120 can reduce damage caused by a collision with a specific object without mistaking a specific object such as a leading vehicle for a floating object.

[0067] (Three-dimensional object identification process S216) The three-dimensional object identification unit 176 first identifies the road surface ahead of the vehicle 1 using the brightness image 180 and the distance image 182 in which the floating pixels 196 and nearby pixels 198 have been disabled. The three-dimensional object identification unit 176 then identifies a three-dimensional object located vertically above the identified road surface. Specifically, the three-dimensional object identification unit 176 identifies blocks located at a vertical distance from the road surface that is equal to or greater than a predetermined distance (e.g., 0.3 m) as candidates for a three-dimensional object protruding vertically from the road surface. Of the multiple blocks identified as candidates for a three-dimensional object located vertically above the road surface, the three-dimensional object identification unit 176 groups blocks that are at the same relative distance from the vehicle 1 and identifies them as three-dimensional objects.

[0068] (Specific object determination process S218) The specified object determination unit 178 determines whether the three-dimensional object identified by the three-dimensional object identification unit 176 is a predetermined specified object, such as a preceding vehicle, a pedestrian, a building, etc. Since various existing technologies can be applied to determine whether an object is a specified object, detailed description thereof will be omitted here.

[0069] The vehicle exterior environment recognition device 120 and vehicle exterior environment recognition method described above improve the detection accuracy of floating objects and prevent the vehicle 1 from being mistaken for a specific object and executing stop control or deceleration control. Furthermore, the second parallax invalidation unit 172 can further improve the detection accuracy of floating objects by determining that a neighboring pixel 198 located near the floating pixel 196 is also a floating object. Furthermore, if there is a possibility that the neighboring pixel 198 is not a floating object, the invalidation cancellation unit 174 does not determine that pixel as a floating object, thereby reducing damage caused by a collision with a specific object.

[0070] Also provided are a program that causes a computer to function as the exterior environment recognition device 120, and a computer-readable storage medium, such as a flexible disk, optical magnetic disk, ROM, CD, DVD, or BD, on which the program is recorded. Here, the program refers to a data processing means written in any language or description method.

[0071] While the preferred embodiments of the present invention have been described above with reference to the accompanying drawings, it goes without saying that the present invention is not limited to such embodiments. It is clear that those skilled in the art can conceive of various modifications and alterations within the scope of the claims, and it is understood that such modifications and alterations also fall within the technical scope of the present invention.

[0072] For example, in the above-described embodiment, the invalidation cancellation unit 174 cancels the invalidation of the disparity of the neighboring pixel 198 when the neighboring pixel 198 is identified as a vehicle class 194c using semantic segmentation or when the neighboring pixel 198 is included in the position in the current frame of a three-dimensional object determined to be a vehicle in the previous frame. However, the cancellation condition is not limited to such cases. In addition to or instead of the above, the invalidation cancellation unit 174 may also be when the neighboring pixel 198 is determined to be an edge. The invalidation cancellation unit 174 detects the size of the edge of the neighboring pixel 198 in the luminance image 180 using a unidirectional filter, for example, one or both of a horizontal filter and a vertical filter, and determines that the edge is an edge of a specific object, such as a vehicle, if the size of the edge is equal to or greater than a threshold. The invalidation cancellation unit 174 cancels the invalidation of the disparity of the neighboring pixel 198 determined to be an edge of a specific object.

[0073] Specifically, for example, assume that a 3-row, 3-column array of {{-1,0,1},{-2,0,2},{-1,0,1)}} is used as the horizontal filter Dx, and a 3-row, 3-column array of {{-1,-2,-1},{0,0,0},{1,2,1}}} is used as the vertical filter Dy. Here, the horizontal edge size is derived by multiplying the edge of a predetermined 3-row, 3-column block L by the horizontal filter Dx (Dx×L), and the vertical edge size is derived by multiplying it by the vertical filter Dy (Dy×L). Then, the invalidation cancellation unit 174 calculates the root mean square of the value Lx obtained by multiplying the block L by the horizontal filter Dx and the value Ly obtained by multiplying the block L by the vertical filter Dy. Square root value (√(Lx 2 +Ly 2 If )) is equal to or greater than the threshold, the invalidation of the disparity of the pixels included in that block L is canceled.

[0074] In this way, if a horizontal edge or a vertical edge is present, it is possible to use the parallax to determine that the block L is not likely to be a floating object. Since various existing techniques can be applied to determine such edges, detailed explanations thereof will be omitted here.

[0075] Furthermore, the cancellation condition may further refer to the reliability of semantic segmentation, and may be determined based on whether the reliability that the neighboring pixel 198 is a floating object is less than a predetermined threshold. As described above, in semantic segmentation, the reliability is included in the input data to softmax or sigmoid during the calculation of the second class identification unit 168, or in the intermediate data after softmax or sigmoid. For example, the invalidation cancellation unit 174 determines whether the reliability of the neighboring pixel 198 in the luminance image 180 is equal to or less than a threshold. The invalidation cancellation unit 174 cancels the invalidation of the disparity of the neighboring pixel 198 whose reliability is determined to be equal to or less than the threshold.

[0076] In this way, it becomes possible to use the parallax by assuming that pixels with low reliability as floating object class 194d are highly likely not to be floating objects.

[0077] It should be noted that the steps of the vehicle exterior environment recognition method in this specification do not necessarily need to be processed in chronological order according to the order described in the flowchart, and may include parallel or subroutine processing. [Explanation of symbols]

[0078] 110 Imaging device 120 External environment recognition device 160 Image acquisition unit 162 Range image generation unit 164 Specific Item Tracking Department 166 First Class Specific Division 168 Second Class Specific Division 170 First parallax nullifying unit 172 Second parallax nullifying unit 174 Revocation of invalidation 176 Three-dimensional object identification part 178 Specific object determination section 180 luminance images 180a First luminance image 180b Second luminance image 182 Range Images 190 Class Images 194c vehicle class 194d Floating Class 196 floating pixels 198 neighboring pixels

Claims

1. one or more processors; one or more memories coupled to the processor; Equipped with The processor operates in conjunction with a program contained in the memory; generating a distance image from a plurality of luminance images captured by an imaging device; using semantic segmentation to identify whether pixels in the intensity image correspond to a floating object class; invalidating the parallax of a floating pixel, which is a pixel corresponding to the floating object class among the pixels of the distance image, and the parallax of a neighboring pixel, which is a pixel not corresponding to the floating object class and whose distance to the floating pixel is within a predetermined range; An exterior environment recognition device that performs processing including the steps of:

2. The processor: canceling the invalidation of the disparity of the neighboring pixel when the neighboring pixel satisfies a predetermined cancellation condition; The vehicle exterior environment recognition device according to claim 1 , which executes a process including:

3. The vehicle exterior environment recognition device according to claim 2 , wherein the cancellation condition is that the vehicle is identified to a vehicle class using semantic segmentation.

4. The processor: Identifying a three-dimensional object based on the luminance image and the distance image in which the parallax of some of the pixels has been nullified; determining whether the three-dimensional object is a vehicle; Including, The vehicle exterior environment recognition device according to claim 2 or 3, wherein the cancellation condition is that the three-dimensional object determined to be a vehicle in the previous frame is included in the position in the current frame.

5. The vehicle exterior environment recognition device according to claim 2 , wherein the cancellation condition is that the vehicle is determined to be an edge.

6. The vehicle exterior environment recognition device according to claim 2 , wherein the cancellation condition is that a confidence level of a floating object in semantic segmentation is less than a predetermined threshold.

Images