External environment recognition device

Abstract

To suppress a deviation in a relative distance of a three-dimensional object.SOLUTION: A vehicle exterior environment recognition apparatus 120 includes: a monocular distance calculator 168 which calculates a monocular distance being a relative distance of a three-dimensional object from a luminance image generated by an imaging unit 110; a relaxation distance calculator 162 which calculates a relaxation distance being a relative distance of the three-dimensional object having a more lenient threshold for determining a degree of image matching than a stereo distance being a relative distance of the three-dimensional object calculated on the basis of the degree of image matching between the luminance images generated by two imaging units 110; and an updated distance calculator 170 which calculates an updated distance being the updated relative distance of the three-dimensional object by mixing the monocular distance and the relaxation distance at a predetermined ratio.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an exterior environment recognition device capable of deriving a relative distance to an imaged three-dimensional object. [Background technology]

[0002] There is known a technology for recognizing the external environment of a vehicle, such as the relative distance to a three-dimensional object (e.g., a preceding vehicle) located in the vehicle's direction of travel, using images captured by two imaging devices mounted on the vehicle (e.g., Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-34664 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology for recognizing the vehicle's external environment, if one of the two imaging devices cannot capture a clear image of a three-dimensional object, the speed and relative distance of the three-dimensional object are derived using only the other imaging device. However, in this case, since the relative distance is expressed as the integral value of the speed, if the relative distance of the three-dimensional object is derived using only one imaging device for a relatively long period of time, an error in the relative distance accumulates, and in some cases, the deviation in the relative distance becomes large. If cruise control or the like is performed using such a relative distance, the vehicle may be accelerated unintentionally and may erroneously approach the vehicle ahead.

[0005] Therefore, an object of the present invention is to provide an exterior environment recognition device that can suppress deviations in the relative distances of three-dimensional objects. [Means for solving the problem]

[0006] In order to solve the above problems, the vehicle exterior environment recognition device of the present invention includes a monocular distance derivation unit that derives a monocular distance, which is a relative distance of a three-dimensional object, based on a luminance image of one imaging device, and a stereo distance deriving unit that derives a stereo distance, which is a relative distance of a three-dimensional object, based on an image matching degree between luminance images of two imaging devices. is a different parameter from the stereo distance. The threshold for determining image matching is looser than that for the relative distance of three-dimensional objects. Parameter indicating and an updated distance derivation unit that mixes the monocular distance and the relaxed distance at a predetermined ratio to derive an updated distance that is an updated relative distance of the three-dimensional object.

[0007] The image capturing apparatus may also include a stereo distance derivation unit that derives a stereo distance, and a switching control unit that switches between stereo control using the stereo distance and monocular control using at least the monocular distance.

[0008] The image processing device may also have an object frame derivation unit that derives an object frame indicating an area occupied by a three-dimensional object in the luminance image, wherein the monocular distance derivation unit derives the monocular distance based on the object frame, and the relaxation distance derivation unit derives the relaxation distance for blocks located within the area corresponding to the object frame.

[0009] The relaxation distance derivation unit may derive the relaxation distance for blocks located within a predetermined range based on the monocular distance.

[0010] Also, The apparatus includes an object frame derivation unit that derives an object frame that indicates an area occupied by a three-dimensional object in the luminance image, and the relaxation distance derivation unit extracts blocks located within the area corresponding to the object frame; The updated distance derivation part is Extracted by the relaxation distance calculation part If the number of blocks is equal to or greater than a predetermined number, the monocular distance and the relaxation distance are mixed together to obtain an updated distance. Extracted by the relaxation distance calculation part If the number of blocks is less than the predetermined extraction number, the monocular distance may be set as the updated distance. Effect of the Invention

[0011] According to the present invention, it is possible to suppress deviations in the relative distances of three-dimensional objects. [Brief description of the drawings]

[0012] [Figure 1]FIG. 1 is a block diagram showing the connections of the exterior environment recognition system. [Diagram 2] 2A and 2B are diagrams for explaining a luminance image and a distance image. Fig. 2A shows an example of the luminance image. Fig. 2B shows an example of the distance image. [Diagram 3] FIG. 3 is a functional block diagram showing a schematic function of the vehicle exterior environment recognition device. [Figure 4] Fig. 4 is a diagram showing an example of a luminance image. Fig. 4A shows an example of a luminance image of the imaging device on the right side. Fig. 4B shows an example of a luminance image of the imaging device on the left side. [Diagram 5] FIG. 5 is a diagram showing an example of a distance image derived based on the luminance image of FIG. 4A and the luminance image of FIG. 4B. [Figure 6] FIG. 6 is a diagram for explaining a method of deriving the speed of a three-dimensional object in monocular control. [Figure 7] FIG. 7 is a diagram for explaining derivation of the distance image and the relaxed distance image. [Figure 8] FIG. 8 is a diagram showing an example of a relaxation distance image derived based on FIGS. 4A and 4B. [Figure 9] Fig. 9 is a diagram illustrating a relaxed distance derivation unit and an updated distance derivation unit. Fig. 9A to Fig. 9C show an example of the process of the relaxed distance derivation unit. Fig. 9D shows an example of the process of the updated distance derivation unit. [Figure 10] FIG. 10 is a flowchart illustrating the flow of operations of the central control unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. The dimensions, materials, and other specific numerical values ​​shown in the embodiment are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In this specification and drawings, elements having substantially the same functions and configurations are given the same reference numerals to avoid repeated explanations, and elements not directly related to the present invention are not shown.

[0014] 1 is a block diagram showing connections in a vehicle exterior environment recognition system 100. The vehicle exterior environment recognition system 100 includes an imaging device 110, an exterior environment recognition device 120, and a vehicle control device (ECU: Engine Control Unit) .

[0015] The imaging device 110 includes imaging elements such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor), and can capture an external environment in front of the vehicle 1 and generate a luminance image (color image or monochrome image) including at least luminance information. The imaging devices 110 are arranged in the traveling direction of the vehicle 1, spaced apart from each other in a substantially horizontal direction so that the optical axes of the two imaging devices 110 are substantially parallel. The imaging device 110 continuously generates luminance images of three-dimensional objects present in a detection area in front of the vehicle 1, for example, at every 1 / 60th of a second frame (60 fps). Here, the three-dimensional objects recognized by the imaging device 110 include not only independently existing objects such as bicycles, pedestrians, vehicles, traffic lights, roads (paths of travel), road signs, guardrails, and buildings, but also objects that can be identified as parts of the objects, such as bicycle wheels.

[0016] Moreover, the vehicle exterior environment recognition device 120 obtains luminance images from each of the two imaging devices 110, and derives disparity information including disparity and a screen position indicating a position of an arbitrary block in a screen by using so-called pattern matching, in which a block corresponding to a block (for example, an array of 4 horizontal pixels by 4 vertical pixels) arbitrarily extracted from one luminance image is searched for in the other luminance image. Here, horizontal indicates the horizontal direction of the screen of the captured image, and vertical indicates the vertical direction of the screen of the captured image. As this pattern matching, it is considered to compare luminance (Y) in an arbitrary block unit between a pair of images. For example, there are methods such as SAD (Sum of Absolute Difference) which takes the difference in luminance, SSD (Sum of Squared Intensity Difference) which uses the difference by squaring, and NCC (Normalized Cross Correlation) which takes the similarity of the variance value obtained by subtracting the average value from the luminance of each pixel. The vehicle exterior environment recognition device 120 performs such a block-by-block disparity derivation process for all blocks displayed in the detection area (for example, 600 pixels by 200 pixels). Here, the block is 4 pixels x 4 pixels, but the number of pixels in a block can be set arbitrarily.

[0017] However, although the vehicle exterior environment recognition device 120 can derive disparity for each block, which is a detection resolution unit, it cannot recognize what object the block is a part of. Therefore, disparity information is derived independently for each detection resolution unit (for example, block unit) in the detection area, not for each object.

[0018] Furthermore, the exterior environment recognition device 120 can convert the parallax information for each block in the distance image 128 into three-dimensional position information using a so-called stereo method, thereby deriving the relative distance to the vehicle 1 for each block. Here, the stereo method is a method of deriving the relative distance of an object to the imaging device 110 from the parallax of the object by using a triangulation method.

[0019] Here, an image in which the relative distance derived from the parallax information is associated with each block is called a distance image to distinguish it from the above-mentioned luminance image. Note that the distance image is not limited to an image in which the relative distance is associated with each block, but may be an image in which the parallax information before being converted into the relative distance is associated with each block.

[0020] FIG. 2 is an explanatory diagram for explaining the luminance image 126 and the distance image 128. FIG. 2A shows an example of the luminance image 126. FIG. 2B shows an example of the distance image 128. For example, it is assumed that the luminance image 126 as shown in FIG. 2A is generated for the detection area 124 through two imaging devices 110. However, here, for ease of understanding, only one of the two luminance images 126 is shown in schematic form. The vehicle exterior environment recognition device 120 obtains the parallax for each block from such a luminance image 126, and forms a distance image 128 as shown in FIG. 2B. Each block in the distance image 128 is associated with the parallax (relative distance) of that block. Here, for convenience of explanation, the blocks from which the parallax (relative distance) is derived are represented by black dots.

[0021] The vehicle exterior environment recognition device 120 uses the luminance value (color value) based on the luminance image 126 and the three-dimensional position information in real space including the relative distance from the vehicle 1 calculated based on the distance image 128, groups blocks having the same color value and close three-dimensional position information as objects, and identifies which specific object (for example, a preceding vehicle or a bicycle) the object in the detection area in front of the vehicle 1 corresponds to. Furthermore, when the vehicle exterior environment recognition device 120 identifies a three-dimensional object in this way, it avoids a collision with the three-dimensional object (collision avoidance control) or controls the vehicle 1 to keep a safe distance from the preceding vehicle (cruise control).

[0022] 1, the vehicle control device 130 receives operational inputs from the driver via a steering wheel 132, an accelerator pedal 134, and a brake pedal 136, and transmits the inputs to a steering mechanism 142, a drive mechanism 144, and a braking mechanism 146 to control the host vehicle 1. The vehicle control device 130 also controls the steering mechanism 142, the drive mechanism 144, and the braking mechanism 146 in accordance with instructions from the exterior environment recognition device 120.

[0023] 3 is a functional block diagram showing a schematic function of the vehicle exterior environment recognition device 120. As shown in FIG. 3, the vehicle exterior environment recognition device 120 includes an I / F unit 150, a data storage unit 152, and a central control unit 154.

[0024] The I / F unit 150 is an interface for two-way information exchange with the imaging device 110 and the vehicle control device 130. The data storage unit 152 is composed of a RAM, a flash memory, a HDD, etc., and stores various information necessary for the processing of each functional unit described below.

[0025] The central control unit 154 is configured with a semiconductor integrated circuit including a central processing unit (CPU), a ROM in which programs and the like are stored, a RAM as a work area, and the like, and controls the I / F unit 150, the data holding unit 152, and the like through a system bus 156. In this embodiment, the central control unit 154 also functions as a stereo distance derivation unit 160, a relaxed distance derivation unit 162, a switching control unit 164, an object frame derivation unit 166, a monocular distance derivation unit 168, and an updated distance derivation unit 170.

[0026] Fig. 4 is a diagram showing an example of a luminance image 126. Fig. 4A shows an example of the luminance image 126 of the imaging device 110 on the right side. Fig. 4B shows an example of the luminance image 126 of the imaging device 110 on the left side.

[0027] It is assumed that the imaging device 110 on the right side can clearly image the detection area 124, as shown by the solid line in Fig. 4A. On the other hand, it is assumed that the imaging device 110 on the left side cannot clearly image the detection area 124, as shown by the dashed line in Fig. 4B, due to the presence of raindrops or the like in the detection area 124. In other words, it is assumed that the imaging device 110 on the left side produces a blurred luminance image 126 (an image in which a three-dimensional object is not clearly imaged) as shown in Fig. 4B.

[0028] Fig. 5 is a diagram showing an example of a distance image 128 derived based on the luminance image 126 of Fig. 4A and the luminance image 126 of Fig. 4B. In the distance image 128, the image matching degree between the luminance images 126 of the left and right imaging devices 110 is equal to or greater than a threshold, and the maximum disparity is derived for each block. If the image matching degree satisfies this condition, the disparity is actually derived, and if the image matching degree does not satisfy this condition, it is determined that the disparity could not be derived.

[0029] In Fig. 5, the blocks from which the parallax is derived are indicated by hatching. If raindrops are present in the detection area 124 of either the left or right imaging device 110, the degree of match between the left and right luminance images 126 decreases, and as shown in Fig. 5, the number of blocks from which the parallax can actually be derived in the distance image 128 decreases.

[0030] When the number of blocks for which the disparity cannot be derived increases, it becomes difficult to identify a three-dimensional object from the distance image 128. For example, in the example of FIG. 5, it is difficult to identify the preceding vehicle in FIG. 4A and FIG. 4B. As a result, it becomes impossible to accurately derive the relative distance of a three-dimensional object, such as a preceding vehicle, to the own vehicle 1. Hereinafter, the relative distance of a three-dimensional object to the own vehicle 1 may be referred to as the object distance.

[0031] Therefore, when the number of blocks for which disparity can actually be derived in the distance image 128 falls below a predetermined number of blocks, the vehicle exterior environment recognition device 120 switches from stereo control to monocular control. Stereo control is to control the vehicle 1 using the stereo distance. The stereo distance is the relative distance of a three-dimensional object extracted based on the distance image. Monocular control is to control the vehicle 1 using at least the monocular distance. The monocular distance is the relative distance of a three-dimensional object extracted based on the luminance image of one (one) of the imaging devices 110.

[0032] In the case of monocular control, the vehicle exterior environment recognition device 120 uses the luminance image 126 of the imaging device 110 that can capture images relatively clearly (for example, the one whose luminance value of the luminance image 126 is higher overall) between the left and right imaging devices 110.

[0033] FIG. 6 is a diagram for explaining a method of deriving the speed of a three-dimensional object in monocular control. As shown in FIG. 6, in monocular control, an object frame 200 indicating an area occupied by a three-dimensional object (a preceding vehicle in FIG. 6) is derived in the luminance image 126 of one imaging device 110. The vehicle exterior environment recognition device 120 acquires the width of the object frame 200 as indicated by an arrow 202 in FIG. 6. The vehicle exterior environment recognition device 120 then derives the speed of the three-dimensional object by regarding the change in width as a change in relative distance based on the difference between the width of the object frame 200 a predetermined frame (e.g., 100 frames) before the current frame and the relative distance at that time (the stereo distance immediately before switching when switching to monocular control for the first time). The vehicle exterior environment recognition device 120 then integrates the derived speed of the three-dimensional object over time to derive the distance of the three-dimensional object.

[0034] Hereinafter, the speed of a three-dimensional object derived based on a luminance image of one imaging device 110 may be referred to as a monocular speed. Also, the distance of a three-dimensional object derived based on a luminance image of one imaging device 110 may be referred to as a monocular distance.

[0035] In this way, in the exterior environment recognition device 120, when the luminance image 126 of one of the two imaging devices 110 is blurred, the speed and distance of a three-dimensional object are derived using only the luminance image 126 of the other imaging device 110, thereby preventing a decrease in accuracy of the speed and distance of the three-dimensional object.

[0036] However, if the monocular speed and monocular distance are continuously derived over a relatively long period of time (e.g., 1000 frames), errors in the monocular speed or monocular distance occurring in each frame may accumulate. This is presumably because once an error occurs, a new monocular distance including an error is derived based on the monocular distance including an error. For example, assuming that the monocular distance deviates by 1 mm in one frame, the deviation in the monocular distance may be 10 cm to 20 cm while the host vehicle 1 moves 100 m.

[0037] If the deviation in the monocular distance becomes large, when cruise control or the like is performed using the monocular distance, the vehicle 1 may be accelerated unintentionally, and the vehicle 1 may mistakenly approach a preceding vehicle that is actually moving at a constant speed.

[0038] Therefore, the vehicle exterior environment recognition device 120 of this embodiment uses a new parameter called a relaxation distance. The relaxation distance indicates the relative distance of a three-dimensional object with a looser threshold for determining the image matching degree than the stereo distance. The image matching degree is an index showing the degree of matching between a block arbitrarily extracted from one luminance image and a block arbitrarily extracted from the other luminance image in pattern matching. The image matching degree is a higher value as the images of the blocks match each other. For example, when a method of taking the difference in luminance is adopted as pattern matching, the image matching degree is a higher value as the difference in luminance is smaller. The stereo distance is derived based on the disparity between blocks whose image matching degree is equal to or greater than a predetermined threshold and whose image matching degree is the maximum value. A loose threshold indicates that the threshold for determining the disparity from which the relative distance of a three-dimensional object is derived from the image matching degree is lower than the original threshold for determining the disparity from which the stereo distance is derived from the image matching degree. The threshold for the relaxation distance is lower than that for the stereo distance. Lowering the threshold corresponds to relaxing the conditions for determining the disparity from which the relative distance of the three-dimensional object is derived. The relaxed distance is the relative distance of the three-dimensional object derived from the disparity determined using a lower threshold than the stereo distance. By using a lower threshold, the disparity is more easily determined, and as a result, the relative distance of the three-dimensional object is more easily derived. Therefore, the relaxed distance is derived using more blocks than the stereo distance.

[0039] For example, in pattern matching, an arbitrary reference block serving as a reference for comparison is extracted from one luminance image. Next, an arbitrary target block to be compared is extracted from the other luminance image. Next, the image matching degree of the target block with respect to the reference block is derived. Then, the reference block is held and the extraction of a new target block and the derivation of the image matching degree are repeated until the extraction of the target block is completed for all blocks in the detection area of ​​the other luminance image. Then, the image matching degree is obtained for each target block with respect to one reference block.

[0040] Furthermore, the disparity is derived from the positions of the base block and the target block. Therefore, in pattern matching, for one base block, the disparity with respect to the base block can be derived for each target block.

[0041] FIG. 7 is a diagram for explaining derivation of the distance image 128 and the relaxed distance image. FIG. 7 shows an example of a diagram created by plotting a combination of image matching and disparity for each target block with respect to one reference block. Since disparity is related to the position of the reference block and the position of the target block, the disparity on the horizontal axis of FIG. 7 corresponds to the position of the target block relative to the reference block. A solid line 210 in FIG. 7 indicates a first threshold. The first threshold corresponds to an original threshold for determining the disparity from which the stereo distance is derived from the image matching. A dashed line 212 in FIG. 7 indicates a second threshold lower than the first threshold. The second threshold corresponds to a threshold for determining the disparity from which the relaxed distance is derived from the image matching. The relaxed distance image shows an image in which the relaxed distance derived from the disparity determined using the second threshold is associated with each block.

[0042] The stereo distance derivation unit 160 extracts one reference block in one luminance image, and derives an image matching degree between the reference block and the target block for each target block in the other luminance image. Next, the stereo distance derivation unit 160 derives a disparity at which the image matching degree is equal to or greater than a predetermined first threshold and is a maximum value. Next, the stereo distance derivation unit 160 derives disparity information by associating the derived disparity with the position of the reference block. The stereo distance derivation unit 160 derives this disparity information for each reference block. Then, the stereo distance derivation unit 160 converts each derived disparity information into a stereo distance to derive a distance image.

[0043] For example, in the case of the example of the transition of the image matching degree with respect to the disparity shown by the solid line 220, since there is a portion where the image matching degree is equal to or greater than the first threshold (solid line 210), the disparity P1 corresponding to the maximum value of the image matching degree is derived. In this case, disparity information including the disparity P1 is derived for one reference block, and the stereo distance is derived from the disparity information. However, in the example of the transition of the image matching degree with respect to the disparity shown by the dashed line 222, since the image matching degree is less than the first threshold over the entire range of the horizontal axis, the condition that the image matching degree is equal to or greater than the first threshold is not satisfied, and the disparity from which the stereo distance is derived is not derived. This corresponds to the fact that the disparity is not derived in the block of the distance image corresponding to the position of one reference block. As described above, if raindrops or the like are present in the detection area 124 of one of the imaging devices 110, the image matching degree decreases in each block as shown by the dashed line 222. As a result, in the distance image in this case, there are many blocks from which the disparity is not derived (see FIG. 5).

[0044] Therefore, as shown by the dashed-dotted line 212 in FIG. 7, a second threshold lower than the first threshold is newly set. The relaxed distance derivation unit 162 of the vehicle exterior environment recognition device 120 extracts one reference block in one luminance image, and derives the image matching degree between the reference block and the target block for each target block in the other luminance image. Next, the relaxed distance derivation unit 162 derives a disparity at which the image matching degree is equal to or greater than a second threshold lower than the first threshold and is a maximum value. Hereinafter, the disparity derived by the relaxed distance derivation unit 162 may be referred to as a relaxed disparity. Next, the relaxed distance derivation unit 162 derives relaxed disparity information by associating the derived relaxed disparity with the position of the reference block. The relaxed distance derivation unit 162 derives this relaxed disparity information for each reference block. Then, the relaxed distance derivation unit 162 converts each derived relaxed disparity information into a relaxed distance for each block, and derives a relaxed distance image. Since the relaxation distance is derived for each block based on the relaxed parallax, hereinafter, the relaxation distance for each block may be referred to as a block relaxation distance.

[0045] For example, in the case of the example of the dashed-dotted line 222 where no disparity was derived in the distance image, since there is a portion where the image matching degree is equal to or greater than the second threshold, a relaxed disparity P2 corresponding to the maximum image matching degree is derived. In this case, relaxed disparity information including the relaxed disparity P2 is derived for one reference block, and a block relaxation distance is derived from the relaxed disparity information. Although the relaxed disparity P2 may include an error with respect to the disparity P1 of the distance image, it is estimated that it is derived relatively close to the disparity P1 of the distance image.

[0046] In addition, since the threshold for image matching is lower in the relaxed distance image than in the distance image, the accuracy of the relaxed parallax is lower than in the distance image, but the number of blocks from which the relaxed parallax (block relaxation distance) is actually derived can be increased.

[0047] In addition, in the case of the example of the change in image matching rate with respect to disparity shown by the two-dotted dashed line 224, since the image matching rate is less than the second threshold value over the entire range of the horizontal axis, the condition that the image matching rate is greater than or equal to the second threshold value is not met, and no relaxed disparity is derived.

[0048] Fig. 8 is a diagram showing an example of a relaxed distance image derived based on Fig. 4A and Fig. 4B. In Fig. 8, the blocks from which the relaxed parallax is derived are indicated by hatching. The relaxed parallax of the relaxed distance image of Fig. 8 is derived in a wider range of the image compared to the parallax of the distance image of Fig. 5. For this reason, the relaxed distance image of Fig. 8 includes more relaxed parallax (in other words, block relaxed distances) related to a three-dimensional object (a preceding vehicle) than the distance image of Fig. 5.

[0049] The switching control unit 164 of the exterior environment recognition device 120 controls switching between stereo control and monocular control. When the number of blocks in which disparity is derived in the distance image 128 is less than a predetermined number of blocks, the switching control unit 164 switches to monocular control. Also, when the number of blocks in which disparity is derived in the distance image 128 is equal to or greater than the predetermined number of blocks, the switching control unit 164 switches from monocular control to stereo control.

[0050] The object frame derivation section 166 derives an object frame 200 that indicates the area occupied by a three-dimensional object in the brightness image 126 of one image capture device 110.

[0051] The monocular distance derivation unit 168 derives the monocular distance, which is the relative distance of a three-dimensional object based on the luminance image 126 of one imaging device 110, based on the object frame 200.

[0052] In the case of monocular control, the updated distance derivation unit 170 mixes the monocular distance and the relaxed distance at a predetermined ratio to derive an updated distance that is an updated relative distance of a three-dimensional object.

[0053] Fig. 9 is a diagram illustrating the relaxed distance derivation section 162 and the updated distance derivation section 170. Fig. 9A to Fig. 9C show an example of the process of the relaxed distance derivation section 162. Fig. 9D shows an example of the process of the updated distance derivation section 170.

[0054] 9A to 9D show schematic plan views of the vehicle 1 viewed from above. In the drawings, the upward direction is the traveling direction of the vehicle 1. In the drawings, two imaging devices 110 are disposed inside the windshield 230 of the vehicle 1. In the drawings, a raindrop 232 is attached to the windshield 230 at a position in front of the right imaging device 110. In other words, in the drawings, the luminance image 126 of the right imaging device 110 is blurred.

[0055] Hatched squares 240 in Figs. 9A to 9D indicate block relaxation distances. Fig. 9A shows an example of a three-dimensional arrangement of blocks associated with block relaxation distances. A dashed line 242 in Fig. 9A shows an example of the width of a preceding vehicle. As described above, the relaxed parallax is derived with a relaxed threshold value, and therefore is less reliable than the parallax of the distance image. In addition, the block relaxation distance is derived from the relaxed parallax, and therefore is less reliable than the relative distance for each block in the distance image. For this reason, as shown in Fig. 9A, many of the blocks associated with block relaxation distances are located near the position of the preceding vehicle, but some of them are located away from the preceding vehicle.

[0056] Next, as shown in Fig. 9B, the relaxation distance derivation unit 162 applies the object frame 200 derived by the object frame derivation unit 166 to the relaxation distance image including a plurality of block relaxation distances. In other words, the relaxation distance derivation unit 162 applies the object frame 200 to the relaxation distance image, thereby associating the position of a three-dimensional object (preceding vehicle) identified from the luminance image of one imaging device 110 with the position of a block associated with the block relaxation distance. Note that Fig. 9B illustrates the position of the horizontal frame of the object frame 200.

[0057] A solid line 250 in FIG. 9B illustrates a position corresponding to the monocular distance derived by the monocular distance derivation unit 168. An arrow 252 in FIG. 9B illustrates the monocular distance. A two-dot chain line 254 and a two-dot chain line 256 in FIG. 9B illustrate a predetermined range (predetermined distance range) based on the monocular distance (solid line 250). For example, the two-dot chain line 254 illustrates a position corresponding to a distance (for example, monocular distance×1.1) that is longer than the monocular distance by a predetermined percentage (for example, 10%). The two-dot chain line 256 illustrates a position corresponding to a distance (for example, monocular distance×0.9) that is shorter than the monocular distance by a predetermined percentage (for example, 10%). Note that the specific numerical values ​​of the predetermined distance range are not limited to those illustrated, and can be appropriately set depending on, for example, the size of a three-dimensional object.

[0058] As shown in FIG. 9B, the relaxed distance derivation unit 162 also applies the monocular distance (solid line 250) and the predetermined range (dashed-dotted lines 254, 256) to the relaxed distance image after the relaxed parallax is converted into the relaxed distance.

[0059] As shown in FIG. 9B, the relaxation distance derivation unit 162 derives the relaxation distance for blocks located in an area corresponding to the object frame 200 and within a predetermined range based on the monocular distance.

[0060] 9B, relaxation distance derivation unit 162 extracts block relaxation distances associated with each of a plurality of blocks (rectangles 240) located within an area surrounded by object frame 200, two-dot chain line 254, and two-dot chain line 256. This removes the block relaxation distances located away from the solid object (leading vehicle), i.e., noise.

[0061] Next, as shown in Fig. 9C, the relaxation distance derivation unit 162 derives a representative value of the extracted block relaxation distances as the relaxation distance. A dashed line 260 in Fig. 9C illustrates a position corresponding to the relaxation distance. An arrow 262 in Fig. 9C indicates the relaxation distance.

[0062] The relaxation distance is, for example, the average value of the extracted block relaxation distances. Note that the relaxation distance is not limited to this example. For example, the relaxation distance may be a value obtained by weighting the extracted block relaxation distances according to the image matching degree. In this case, the weight of the block relaxation distance with a high image matching degree is increased, and the weight of the block relaxation distance with a low image matching degree is decreased, and the average is obtained. In this aspect, the relaxation distance can be made closer to a more likely relative distance.

[0063] Also, for example, a histogram of the extracted block relaxation distances may be created, and the block relaxation distance corresponding to the peak of the histogram may be set as the relaxation distance.Also, the extracted block relaxation distances may be sorted in ascending or descending order, and the median of the sorted block relaxation distances may be set as the relaxation distance.

[0064] Next, as shown in Fig. 9D, the update distance derivation unit 170 derives an update distance, which is an updated relative distance of a solid object, by mixing the monocular distance and the relaxed distance. The dashed line 270 in Fig. 9D illustrates a position corresponding to the derived update distance. Also, the arrow 272 in Fig. 9D indicates the derived update distance.

[0065] The monocular distance and the relaxation distance are mixed so that the monocular distance is greater than the relaxation distance. For example, the mixture ratio of the monocular distance and the relaxation distance is 7 to 1. In this case, the updated distance derivation unit 170 derives the updated distance by the following formula (1). (Object distance) = (Monocular distance) × 7 / 8 + (Relaxation distance) × 1 / 8 (1)

[0066] Note that the specific numerical values ​​of the mixture ratio of the monocular distance and the relaxation distance are not limited to the exemplified values, and may be appropriately set depending on the external environment of the host vehicle 1 or the driving scene.

[0067] Mixing the relaxed distance with the monocular distance corresponds to calibrating the monocular distance, which may accumulate errors, with the relaxed distance based on the relaxed parallax with a relaxed threshold. Since the monocular distance is calibrated, a more likely updated distance can be derived.

[0068] Although the relaxed distance is a value derived with a relaxed threshold, it is derived based on two image capturing devices 110 and is therefore more likely than the monocular distance, which accumulates errors. In the exterior environment recognition device 120, the more likely relaxed distance is mixed with the monocular distance, so that the accumulation of errors can be suppressed even if the derivation of the updated distance is continued for a long period of time.

[0069] Furthermore, when the number of blocks to be used to derive the relaxed distance (the number of blocks extracted by dividing the object frame 200 and the predetermined distance range) is less than the predetermined extraction number, the updated distance derivation unit 170 may use the monocular distance as the updated distance without mixing the monocular distance and the relaxed distance. This is because when the number of target blocks is small, it is estimated that the ratio of noise contained in the relaxed distance is high, and there is a risk of mixing noise into the monocular distance.

[0070] Fig. 10 is a flowchart for explaining the flow of operations of the central control unit 154. The central control unit 154 repeats the example process of Fig. 10 at each predetermined interrupt timing that occurs in a predetermined control period. In the flowchart of Fig. 10, the flow of operations related to the derivation of the updated distance is explained, and explanations of operations unrelated to the derivation of the updated distance are omitted.

[0071] First, the stereo distance derivation unit 160 acquires the luminance images 126 from the left and right imaging devices 110 (S100). Next, the stereo distance derivation unit 160 derives the disparity for each block when the image matching degree between the left and right luminance images 126 is equal to or greater than a first threshold. Then, the stereo distance derivation unit 160 derives the relative distance for each block from the disparity, and generates the distance image 128 (S110).

[0072] Next, the relaxed distance derivation unit 162 derives a relaxed parallax for each block when the image matching degree of the left and right luminance images 126 acquired in step S100 becomes equal to or greater than a second threshold value. Then, the relaxed distance derivation unit 162 derives a block relaxed distance for each block from the relaxed parallax, and generates a relaxed distance image (S120).

[0073] Next, switching control unit 164 determines whether the number of blocks from which disparity is derived in distance image 128 is less than a predetermined number of blocks (S130). If the number of blocks from which disparity is derived is equal to or greater than the predetermined number of blocks (NO in S130), switching control unit 164 determines that switching to monocular control is not necessary, and proceeds to the process of step S140.

[0074] In step S140, stereo distance derivation unit 160 identifies a three-dimensional object based on distance image 128 (S140). Next, stereo distance derivation unit 160 derives the stereo distance of the three-dimensional object based on distance image 128 (S150). Then, updated distance derivation unit 170 sets the stereo distance as an updated distance (S160), and the series of processes ends.

[0075] Furthermore, if the number of blocks for which the disparity is derived is less than the predetermined number of blocks (YES in S130), the switching control unit 164 determines that it is necessary to switch to monocular control, and proceeds to the process of step S200.

[0076] In step S200, object frame derivation unit 166 derives object frame 200 using luminance image 126 of one imaging device 110 (S200). Next, monocular distance derivation unit 168 derives the monocular speed of the three-dimensional object according to the distance before the change and the change in the width of object frame 200 (S210). Next, monocular distance derivation unit 168 integrates the monocular speed of the three-dimensional object over time to derive the monocular distance of the three-dimensional object (S220).

[0077] Next, the relaxation distance derivation unit 162 applies the object frame 200 to the relaxation distance image including a plurality of block relaxation distances. The relaxation distance derivation unit 162 extracts blocks (block relaxation distances) that are within the object frame 200 and within a predetermined range based on the monocular distance (S230). Next, the relaxation distance derivation unit 162 determines whether the number of extracted blocks is a predetermined extraction number (S240).

[0078] If the number of blocks is equal to or greater than the predetermined extraction number (YES in S240), the relaxation distance derivation unit 162 derives a representative value of the block relaxation distances corresponding to the extracted blocks, and sets the representative value as the relaxation distance (S250).

[0079] Next, the updated distance derivation unit 170 mixes the monocular distance and the relaxed distance at a predetermined mixing ratio to derive the updated distance (S260), and ends the series of processes.

[0080] Furthermore, if the number of blocks is less than the predetermined extraction number (NO in S240), updated distance derivation section 170 sets the monocular distance derived in step S220 as the updated distance (S270), and ends the series of processes.

[0081] As described above, in the case of monocular control, the vehicle exterior environment recognition device 120 of this embodiment derives an updated distance, which is an updated relative distance of a three-dimensional object, by mixing the monocular distance with the relaxation distance. As a result, in the vehicle exterior environment recognition device 120 of this embodiment, the monocular distance is calibrated with the relaxation distance and the relative distance of the three-dimensional object is updated.

[0082] Therefore, in the vehicle exterior environment recognition device 120 of this embodiment, it is possible to suppress deviations in the relative distances of three-dimensional objects.

[0083] In addition, in the case of monocular control, the update distance derivation unit 170 may compare the monocular distance with the relaxation distance, and when the monocular distance is less than the relaxation distance, the monocular distance is set as the update distance, and when the monocular distance is equal to or greater than the relaxation distance, the update distance may be derived by mixing the monocular distance and the relaxation distance. In other words, of the monocular distance and the relaxation distance, the one closer to the vehicle 1 is more reliable. In this embodiment, it is possible to prevent the vehicle 1 from unintentionally approaching the three-dimensional object by determining that the derived object distance is farther away than the actual relative distance of the three-dimensional object.

[0084] Although the embodiment of the present invention has been described above with reference to the attached drawings, it goes without saying that the present invention is not limited to such an embodiment. It is clear that a person skilled in the art can think of various modified or altered examples within the scope of the claims, and it is understood that such examples also naturally belong to the technical scope of the present invention. [Explanation of symbols]

[0085] 110 Imaging device 120 External environment recognition device 126 Luminance Images 128 Range Images 160 Stereo Distance Derivation Unit 162 Relaxation distance derivation part 164 Switching control section 166 Object frame extraction section 168 Monocular distance derivation unit 170 Update distance derivation part 200 Object Frame

Claims

1. a monocular distance derivation unit that derives a monocular distance, which is a relative distance of a three-dimensional object, based on a luminance image of one imaging device; a relaxed distance derivation unit that derives a relaxed distance, which is a parameter different from a stereo distance that is a relative distance of the three-dimensional object derived based on an image matching degree between luminance images of two image capturing devices, and which is a parameter indicating the relative distance of the three-dimensional object having a relaxed threshold value for determining the image matching degree compared to the stereo distance; an update distance derivation unit that derives an update distance, which is an updated relative distance of the three-dimensional object, by mixing the monocular distance and the relaxation distance at a predetermined ratio; An exterior environment recognition device comprising:

2. a stereo distance derivation unit for deriving the stereo distance; a switching control unit that switches between stereo control using the stereo distance and monocular control using at least the monocular distance; The vehicle exterior environment recognition device according to claim 1 .

3. an object frame derivation unit that derives an object frame indicating an area occupied by the three-dimensional object in the brightness image; the monocular distance derivation unit derives the monocular distance based on the object frame; The vehicle exterior environment recognition device according to claim 1 , wherein the relaxation distance deriving unit derives the relaxation distance for a block located within a region corresponding to the object frame.

4. The vehicle exterior environment recognition device according to claim 1 , wherein the relaxation distance deriving unit derives the relaxation distance for a block located within a predetermined range based on the monocular distance.

5. An object frame derivation unit that derives an object frame indicating an area occupied by the three-dimensional object in the brightness image, The relaxation distance derivation unit extracts a block located within a region corresponding to the object frame, The vehicle exterior environment recognition device according to any one of claims 1 to 4, wherein the update distance derivation unit mixes the monocular distance and the relaxation distance to set the update distance if the number of blocks extracted by the relaxation distance derivation unit is equal to or greater than a predetermined extraction number, and sets the monocular distance to the update distance if the number of blocks extracted by the relaxation distance derivation unit is less than the predetermined extraction number.

Images