External perception device

Abstract

To provide an environment recognition device which can correctly calculate a distance to an object regardless of a use environment.SOLUTION: An environment recognition device is mounted on a vehicle. The environment recognition device comprises: an image acquisition unit which acquires an image of a landmark and an object; a landmark information acquisition unit which acquires three-dimensional information of the landmark; and a distance estimation unit which estimates a distance to the object on the basis of the three-dimensional information of the landmark and the sizes of the landmark and the object in the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an external environment recognition device. [Background technology]

[0002] Conventionally, systems have been known that use cameras to detect pedestrians and other obstacles crossing in front of a vehicle and, if necessary, warn the driver or apply automatic braking. In particular, from the perspective of realizing autonomous driving and preventing traffic accidents, there has been considerable interest in automatic braking systems that detect pedestrians and other obstacles and apply braking control. Automatic braking systems must accurately calculate the distance from the vehicle to the pedestrian. Furthermore, automatic braking systems must be able to determine the distance to the pedestrian over a wider viewing angle to respond to pedestrians suddenly stepping out into the road.

[0003] For example, Patent Document 1 describes a method for calculating the distance to a pedestrian using a stereo camera with a field-of-view overlapping region and a field-of-view non-overlapping region. Specifically, Patent Document 1 describes a stereo camera device that calculates the distance to a moving object in a monocular region using the road surface height calculated from parallax information. The stereo camera device obtains coordinates by subtracting (lowering) the height of the stairs from the vertical coordinates on an image of the feet of a pedestrian standing on stairs. The stereo camera device then extends these obtained coordinates to the stereo region and determines the distance to the pedestrian using the parallax information of the road surface at that position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-96777 Summary of the Invention [Problem to be solved by the invention]

[0005] The stereo camera device described in Patent Document 1 calculates the parallax of the road surface based on texture information of the road surface in the acquired image, and uses this parallax information to calculate the distance to the pedestrian. However, in usage environments where the road surface texture is not clear, such as at night or in rain, it may be difficult to obtain the parallax information of the road surface, and the distance to the pedestrian may not be calculated accurately.

[0006] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an external environment recognition device that can accurately calculate the distance to an object regardless of the usage environment. [Means for solving the problem]

[0007] In order to achieve the above object, the external environment recognition device of the present invention is an external environment recognition device mounted on a vehicle, and is characterized by having an image acquisition unit that acquires images of landmarks and objects, a landmark information acquisition unit that acquires three-dimensional information of the landmarks, and a distance estimation unit that estimates the distance to the object based on the three-dimensional information of the landmarks and the sizes of the landmarks and the object in the images. [Effects of the Invention]

[0008] According to the external environment recognition device of the present invention, the distance to the target object can be accurately calculated regardless of the usage environment. Problems, configurations, and effects other than those described above will become clear from the description of the following embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a functional block diagram showing a schematic configuration of an external environment recognition device according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a plan view showing an example of a state in which the external environment recognition device of FIG. 1 is mounted on a vehicle. [Figure 3] 4 is a flowchart showing landmark information acquisition processing of the external environment recognition device of FIG. 1; [Figure 4]An example of an image acquired by the external environment recognition device in Figure 1 during landmark information acquisition processing. [Figure 5] 4 is a flowchart showing a distance estimation process of the external environment recognition device of FIG. 1; [Figure 6] An example of an image acquired by the external recognition device in Figure 1 during distance estimation processing. [Figure 7] FIG. 4 is a functional block diagram showing a schematic configuration of an external environment recognition device according to a modified example of the first embodiment. [Figure 8] 8 is a flowchart showing a distance estimation process of the external environment recognition device of FIG. 7. [Figure 9] 8A to 8C are diagrams illustrating a proximity determination process of the external environment recognition device in FIG. 7. [Figure 10] 8 is a diagram showing another example of the proximity determination process of the external environment recognition device of FIG. 7. [Figure 11] FIG. 10 is a functional block diagram showing a schematic configuration of an external environment recognition device according to another modified example of the first embodiment. [Figure 12] FIG. 10 is a functional block diagram showing a schematic configuration of an external environment recognition device according to a second embodiment of the present invention. [Figure 13] FIG. 10 is a functional block diagram showing a schematic configuration of an external environment recognition device according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] First Embodiment FIG. 1 is a functional block diagram showing a schematic configuration of an external environment recognition device 1 according to a first embodiment of the present invention. FIG. 2 is a plan view showing an example of a state in which the external environment recognition device 1 of FIG. 1 is mounted on a vehicle V. FIG. 3 is a flowchart showing landmark information acquisition processing of the external environment recognition device 1 of FIG. 1. FIG. 4 is an example of an image acquired by the external environment recognition device 1 of FIG. 1 in the landmark information acquisition processing. FIG. 5 is a flowchart showing distance estimation processing of the external environment recognition device 1 of FIG. 1. FIG. 6 is an example of an image acquired by the external environment recognition device 1 of FIG. 1 in the distance estimation processing.

[0012] The external environment recognition device 1 is mounted on a vehicle V (hereinafter also referred to as the host vehicle V). As shown in FIG. 1, the external environment recognition device 1 includes an image acquisition unit 10, a landmark information acquisition unit 30, and a distance estimation unit 50. Although not shown, the external environment recognition device 1 has a configuration in which a CPU, RAM, ROM, etc. are connected via a bus, and the CPU controls the operation of the entire system by executing various control programs stored in the ROM. In this embodiment, the image acquisition unit 10 acquires images from a pair of cameras 10a and 10b (FIG. 2) functioning as stereo cameras, and the distance estimation unit 50 uses the images to estimate the distance between a pedestrian 200 as an object and the host vehicle V. As used herein, the term "landmark" refers to each three-dimensional object (e.g., reference numerals 100, 101, and 102 shown in FIG. 4) captured by the pair of cameras 10a and 10b. In this specification, the term "three-dimensional information of a landmark" refers to at least one of the height and width of the landmarks 100, 101, and 102 themselves. The "three-dimensional information of the landmark" may include the distance (depth) between the landmark 100, 101, 102 and the vehicle V, in addition to at least one of the height and width of the landmark 100, 101, 102 itself.

[0013] As shown in FIGS. 4 and 6, the image acquisition unit 10 acquires images of landmarks 100, 101, and 102 and a pedestrian (object) 200 from a pair of cameras 10a and 10b. Specifically, the image acquisition unit 10 acquires images captured by the pair of cameras 10a and 10b, including a compound eye region whose fields of view overlap and a monocular region (a left monocular region and a right monocular region) whose fields of view do not overlap. Hereinafter, the compound eye region of the image will be referred to as a compound eye region image CA, and the left monocular region and the right monocular region of the image will be referred to as a left monocular region image LA and a right monocular region image RA, respectively. As shown in FIG. 2, the pair of cameras 10a and 10b are configured as a single stereo camera mounted on a vehicle V. Specifically, the pair of cameras 10a and 10b are configured with CCD or CMOS image sensors and are directed forward of the host vehicle V. Furthermore, the pair of cameras 10a, 10b each capture an image of the area in front of the host vehicle V at a predetermined depression angle (i.e., shooting area), and are installed so that the depression angles overlap. For example, if camera 10a is the left camera and camera 10b is the right camera, the left side of the depression angle of camera 10a overlaps with the right side of camera 10b. As a result, in this embodiment, the central front area of ​​the host vehicle V becomes a stereo area (also referred to as a compound eye area) defined by the shooting areas of camera 10a and camera 10b. Furthermore, the left front area of ​​the host vehicle V is a left monocular area defined by the left side of the shooting area of ​​camera 10b. The right front area of ​​the host vehicle V is a right monocular area defined by the right side of the shooting area of ​​camera 10a. Images captured by the pair of cameras 10a, 10b are input to image acquisition unit 10. The image acquisition unit 10 acquires two images taken by a pair of cameras 10a and 10b, and applies a known parallax calculation algorithm to the acquired left and right images to acquire parallax information.

[0014] 1, the landmark information acquisition unit 30 includes a compound eye region landmark detection unit 32, a size information measurement unit 33, and a landmark information storage unit 36. The landmark information acquisition unit 30 acquires three-dimensional information of landmarks 100, 101, and 102. Specifically, the landmark information acquisition unit 30 acquires three-dimensional information of the landmarks 100, 101, and 102 located in the compound eye region image CA.

[0015] 4, the compound eye area landmark detection unit 32 detects landmarks 100, 101, and 102 from the compound eye area image CA acquired by the image acquisition unit 10. For example, the compound eye area landmark detection unit 32 may analyze the texture of the compound eye area image CA transmitted from the image acquisition unit 10 and thereby detect the landmarks 100, 101, and 102. The compound eye area landmark detection unit 32 may also detect the landmarks 100, 101, and 102 based on disparity information acquired by the image acquisition unit 10. Specifically, the landmarks 100, 101, and 102 may be detected by clustering the disparity information or the like, or the landmarks 100, 101, and 102 may be detected by using a convolutional neural network as statistical machine learning.

[0016] The size information measurement unit 33 measures the actual sizes of the landmarks 100, 101, and 102 included in the three-dimensional information from the compound eye region image CA. Specifically, the size information measurement unit 33 calculates the actual heights of the landmarks 100, 101, and 102 detected by the compound eye region landmark detection unit 32 from the compound eye region image CA. Note that the size information measurement unit 33 may measure not only the actual vertical sizes (actual heights) of the landmarks 100, 101, and 102, but also the actual horizontal sizes (actual widths) of the landmarks 100, 101, and 102. The size information measurement unit 33 may also measure the actual sizes (actual heights and actual widths) of a portion of the landmarks 100, 101, and 102. The landmark information storage unit 36 ​​registers at least one of the size information pieces of the landmarks 100, 101, and 102 measured by the size information measurement unit 33 as landmark information to be used for distance calculation, which will be described later. Specifically, the landmark information storage unit 36 ​​stores texture information of the photographed landmarks 100, 101, and 102 in addition to size information (actual height, actual width, etc.) of the measured landmarks 100, 101, and 102.

[0017] The distance estimation unit 50 estimates the distance to the pedestrian 200 located in the monocular region images (left monocular region image LA, right monocular region image RA). Specifically, the distance estimation unit 50 estimates the distance to the pedestrian 200 included in the monocular region images (left monocular region image LA, right monocular region image RA) acquired by the image acquisition unit 10, using landmarks 100, 101, and 102 detected by the landmark information acquisition unit 30. More specifically, the distance estimation unit 50 estimates the distance from the host vehicle V to the pedestrian 200 based on three-dimensional information about the landmark 100 and the sizes of the landmark 100 and the pedestrian 200 in the image. In this embodiment, a case will be described in which the landmark 100 is used as the landmark for estimating the distance between the pedestrian 200 and the host vehicle V. As shown in FIG. 1 , the distance estimation unit 50 includes a monocular region landmark detection unit 52, a monocular region object detection unit 54, a size estimation unit 58, and a distance calculation unit 60.

[0018] As shown in FIG. 6, the monocular area landmark detection unit 52 detects landmarks 100 from the monocular area images (left monocular area image LA, right monocular area image RA) acquired by the image acquisition unit 10. Specifically, the monocular area landmark detection unit 52 redetects the landmarks 100 based on the texture information of the landmarks 100 stored in the landmark information storage unit 36. For example, the monocular area landmark detection unit 52 may detect the landmarks 100 by template matching using the texture information of the landmarks 100 stored in the landmark information storage unit 36 ​​as a template. Alternatively, the monocular area landmark detection unit 52 may detect the landmarks 100 by tracking processing using a convolutional neural network. Furthermore, the monocular area landmark detection unit 52 calculates the vertical length of the landmark 100 in the image (i.e., image height) from the monocular area images of the landmark 100 (left monocular area image LA, right monocular area image RA) acquired from the image acquisition unit 10.

[0019] The monocular region object detection unit 54 detects an object for distance estimation from the monocular region images (left monocular region image LA, right monocular region image RA) acquired by the image acquisition unit 10. When the detection target is, for example, a pedestrian 200, the monocular region object detection unit 54 performs detection using known statistical machine learning. Specifically, the monocular region object detection unit 54 may detect the pedestrian 200 using classical machine learning such as random forest, support vector machine, or Real Adaboost, or may detect the pedestrian 200 using a convolutional neural network. Furthermore, the monocular region object detection unit 54 calculates the vertical length of the pedestrian 200 in the image (i.e., image height) based on the monocular region image of the pedestrian 200 acquired from the image acquisition unit 10.

[0020] The size estimation unit 58 estimates the size of the pedestrian 200 from the size of the landmark 100 based on the image size of the landmark 100 in the monocular region image (such as the left monocular region image LA) and the image size of the pedestrian 200 in the monocular region image (such as the left monocular region image LA). The size estimation unit 58 estimates the size (such as the height) of the pedestrian 200 using the size (such as the height) of the landmark 100 stored in the landmark information storage unit 36. Specifically, the size estimation unit 58 estimates the height (hereinafter also referred to as body height) of the pedestrian 200 from the actual height of the landmark 100 based on the image height of the landmark 100 and the image height of the pedestrian 200. More specifically, the size estimation unit 58 estimates the body height of the pedestrian 200 by multiplying the actual height of the landmark 100 by the ratio of the image height of the pedestrian 200 to the image height of the landmark 100. The size information of the pedestrian 200 may be estimated only once.

[0021] The distance calculation unit 60 calculates the distance to the pedestrian 200 based on the actual size of the pedestrian 200 estimated by the size estimation unit 58 and the image size of the pedestrian 200 in the monocular region images (left monocular region image LA, right monocular region image RA). Specifically, the distance calculation unit 60 calculates the distance from the host vehicle V to the pedestrian 200 based on the height of the pedestrian 200 estimated by the size estimation unit 58. More specifically, the distance calculation unit 60 calculates the distance between the host vehicle V and the pedestrian 200 based on the height of the pedestrian 200 estimated by the size estimation unit 58 and the image height of the pedestrian 200 detected by the monocular region object detection unit 54.

[0022] Next, with reference to Figs. 3 to 6, an example of the operation of the external environment recognition device 1 of this embodiment will be described, divided into a landmark information acquisition process (Figs. 3 and 4) and a distance estimation process (Figs. 5 and 6). Here, as shown in Fig. 2, the external environment recognition device 1 will be described, which uses a pair of cameras 10a and 10b functioning as stereo cameras installed to monitor the front of the vehicle V. Also, a case will be described in which the distance between the host vehicle V and the pedestrian 200 is calculated based on the height of the pedestrian 200 photographed in the left monocular area image LA shown in Fig. 6.

[0023] First, the landmark information acquisition process will be described with reference to FIGS.

[0024] As shown in FIG. 3, in the landmark information acquisition process, the external environment recognition device 1 sequentially executes an image acquisition process (P101), a parallax calculation process (P102), a three-dimensional object detection process (P103), a three-dimensional information acquisition process (P104), and a landmark registration process (P105).

[0025] In the image acquisition process (P101), the image acquisition unit 10 acquires images captured by a pair of cameras 10a and 10b as shown in FIG. 4, i.e., a compound eye area image CA and a monocular area image (left monocular area image LA, right monocular area image RA).

[0026] Next, in the parallax calculation process (P102), the image acquisition unit 10 calculates the parallax in the compound eye area image CA. For example, a 5×5 pixel window area is set for calculating the parallax. Then, the image from the left camera 10a of the pair of cameras 10a and 10b is used as a reference, and the image from the right camera 10b is scanned in the horizontal direction using the SAD as an evaluation value, thereby calculating the parallax.

[0027] Next, in the three-dimensional object detection process (P103), the compound eye area landmark detection unit 32 uses the parallax calculated in the parallax calculation process (P102) to detect landmarks 100, 101, and 102 as shown in FIG. 4. Specifically, because the areas of the landmarks 100, 101, and 102 are at the same distance, a clustering process is performed on the parallax to roughly extract the areas where the landmarks 100, 101, and 102 exist. Next, because the distance value changes between the boundaries of the landmarks 100, 101, and 102 and the background, a change point in the parallax is detected for the roughly extracted areas to precisely detect the areas where the landmarks 100, 101, and 102 exist. FIG. 4 shows the detection result of the three-dimensional object detection process (P103) for the landmark 100 out of the multiple landmarks 100, 101, and 102. The detection result of the three-dimensional object detection process (P103) is a rectangle B100 as shown in FIG. 4.

[0028] Next, in the three-dimensional information acquisition process (P104), the size information measurement unit 33 calculates the height size (i.e., actual height) of each of the landmarks 100, 101, and 102 detected in the three-dimensional object detection process (P103). Hereinafter, a method for estimating the size (actual height) of the landmark 100 among the multiple landmarks 100, 101, and 102 will be described. The actual height of the landmark 100 is estimated using a rectangle B100, which is the detection result of the three-dimensional object detection process (P103). T100 in FIG. 4 indicates the pixel position of the top edge of the rectangle B100, and L100 in FIG. 4 indicates the pixel position of the bottom edge of the rectangle B100. Furthermore, the median disparity value at the landmark 100 is defined as D_med. D_med is obtained by calculating the median disparity value contained within the rectangle B100. The actual height Height of the landmark 100 is calculated by calculating "Height = BaseLine × abs(T100 - L100) / D_med" (Equation (1)). Here, in Equation (1), BaseLine is the baseline length between the cameras 10a and 10b, and abs represents the absolute value operator.

[0029] Next, in the landmark registration process (P105), the landmark information storage unit 36 ​​registers information on the actual height of the landmark 100 calculated in the three-dimensional information acquisition process (P104). The landmark information storage unit 36 ​​also registers texture information of the compound eye region image CA analyzed by the compound eye region landmark detection unit 32. The landmark information storage unit 36 ​​also stores the compound eye region image CA for the rectangle B100 detected in the three-dimensional object detection process (P103).

[0030] Next, the distance estimation process will be described with reference to FIGS.

[0031] As shown in FIG. 5, in the distance estimation process, the external environment recognition device 1 sequentially executes an image acquisition process (P111), a landmark detection process (P112), a pedestrian detection process (P113), a size estimation process (P115), and a distance estimation process (P116).

[0032] In the image acquisition process (P111), the image acquisition unit 10 acquires images captured by a pair of cameras 10a and 10b as shown in FIG. 6, i.e., a compound eye area image CA and a monocular area image (left monocular area image LA, right monocular area image RA).

[0033] Next, in the landmark detection process (P112), the monocular area landmark detection unit 52 redetects the landmark 100. The landmark 100 was photographed in the compound eye area image CA in FIG. 4, but as the vehicle V moves, it is photographed in the left monocular area image LA in FIG. 6. Here, template matching is performed using the texture information of the landmark 100 stored in the landmark information storage unit 36 ​​as a template, thereby redetecting the landmark 100 photographed in the left monocular area image LA shown in FIG. 6. The area detected in this process is represented by a rectangle B101 as shown in FIG. 6.

[0034] Next, in the pedestrian detection process (P113), the monocular area object detection unit 54 detects the pedestrian 200 captured in the left monocular area image LA in FIG. 6. A convolutional neural network is used for this detection. The convolutional neural network used detects the pedestrian 200 included in a given input image. The convolutional neural network not only identifies the position of the pedestrian 200 but also simultaneously calculates an identification score. The identification score is a score for the type of pedestrian 200, and a high score for a certain type indicates that the area is likely to be that type. In the pedestrian detection process (P113), the left monocular area image LA is input to the convolutional neural network, and only detection results with a high identification score for the pedestrian type are used to detect the pedestrian 200. The detection result is represented by a rectangle B201 as shown in FIG. 6.

[0035] In the size estimation process (P115), the size estimation unit 58 estimates the height of the pedestrian 200 by using information about the landmark 100. In the landmark registration process (P105) described above, the actual height Height of the landmark 100 is stored in the landmark information storage unit 36. In addition, in the landmark detection process (P112), the monocular region landmark detection unit 52 detects the image height IMG_Height_Land (pix) of the landmark 100 (i.e., the height of the rectangle B101 on the image). In addition, in the pedestrian detection process (P113), the monocular region object detection unit 54 detects the image height IMG_Height_Ped (pix) of the pedestrian 200 (i.e., the height of the rectangle B201 on the image). In the size estimation process (P115), the height Height_Ped of the pedestrian 200 is estimated by "Height_Ped = (IMG_Height_Ped / IMG_Height_Land) × Height" (Equation (2)). In other words, the height of the pedestrian 200 is estimated by applying the ratio of the image height (IMG_Height_Land) of the landmark 100 to the image height (IMG_Height_Ped) of the pedestrian 200 to the actual height (Height) of the landmark 100.

[0036] Next, in the distance estimation process (P116), the distance calculation unit 60 calculates the distance from the host vehicle V to the pedestrian 200 using the height (Height_Ped) of the pedestrian 200 and the image height (IMG_Height_Ped(pix)) of the rectangle B201 detected in the pedestrian detection process (P113). The distance Dist from the host vehicle V to the pedestrian 200 is calculated based on "Dist = f × (Height_Ped / IMG_Height_Ped)" (Equation (3)). Here, f is the focal length (pix) of the pair of cameras 10a, 10b. Note that once the height of the pedestrian is calculated in the size estimation process (P115), the distance between the host vehicle V and the pedestrian 200 can be calculated in the distance estimation process (P116) using the image height of the pedestrian 200 in another image frame after the image shown in FIG. 6.

[0037] In this way, the external environment recognition device 1 of this embodiment calculates the distance between the pedestrian 200 and the vehicle V in the monocular area image (for example, the left monocular area image LA) by utilizing the actual height of any landmark 100 captured in the compound eye area image CA. Since the landmark 100 is a three-dimensional object and has texture information, parallax can be acquired more stably than with the road surface. Therefore, even in a situation where it is difficult to acquire parallax with respect to the road surface, such as at night or in rainy weather, the distance to the pedestrian 200 can be calculated with higher accuracy.

[0038] Furthermore, the external environment recognition device 1 of this embodiment measures the actual height of the landmark 100 in the compound eye area image CA and estimates the height of the pedestrian 200 using the measurement result. The lateral size of the pedestrian 200 changes due to hand movements while walking. In contrast, the height of the pedestrian 200 does not change or changes only slightly even when the pedestrian 200 walks. Therefore, by estimating the height of the pedestrian 200 and calculating the distance between the pedestrian 200 and the host vehicle V based on the estimated height, the distance to the pedestrian 200 can be calculated with high accuracy. Furthermore, when a method is adopted to estimate the distance between the pedestrian 200 and the host vehicle V from the pedestrian 200's contact position with respect to the road surface and the camera's mounting orientation, it is necessary to accurately estimate the camera's mounting orientation with respect to the road surface. In contrast, by estimating the height of the pedestrian 200 and calculating the distance, as in this embodiment, the distance between the host vehicle V and the pedestrian 200 can be estimated without being affected by changes in the camera's orientation.

[0039] <Variation 1> Next, a first modification of the first embodiment of the present invention will be described.

[0040] FIG. 7 is a functional block diagram showing a schematic configuration of an external environment recognition device 1d according to Modification 1 of the first embodiment. FIG. 8 is a flowchart showing distance estimation processing of the external environment recognition device 1d of FIG. 7. FIG. 9 is a diagram explaining proximity determination processing of the external environment recognition device 1d of FIG. 7. The external environment recognition device 1d according to Modification 1 differs from the above-described external environment recognition device 1 in that it includes a proximity determination unit 56, which will be described later. Hereinafter, components having the same or similar functions as those of the above-described external environment recognition device 1 will be assigned the same reference numerals, and descriptions thereof will be omitted, and different parts will be described.

[0041] As shown in Fig. 7, the distance estimation unit 50 includes a proximity determination unit 56. The proximity determination unit 56 determines whether the landmark 100 detected by the monocular region landmark detection unit 52 and the pedestrian 200 detected by the monocular region object detection unit 54 are three-dimensionally close to each other. Specifically, as shown in Fig. 9, the proximity determination unit 56 calculates the difference in height between the bottom end of the landmark 100 in the left monocular region image LA (monocular region image) and the bottom end of the pedestrian 200 in the left monocular region image LA (monocular region image). Then, when the difference calculated by the proximity determination unit 56 is within a margin (proximity determination threshold) δ described below and it is determined that the landmark 100 and the pedestrian 200 are close to each other, the size estimation unit 58 estimates the height of the pedestrian 200.

[0042] As shown in FIG. 8, the external environment recognition device 1d according to the first modification executes a proximity determination process (P114) after a pedestrian detection process (P113) and before a size estimation process (P115) in a distance estimation process. In the proximity determination process (P114), the proximity determination unit 56 determines whether or not the landmark 100 detected by the monocular region landmark detection unit 52 and the pedestrian 200 detected by the monocular region object detection unit 54 are spatially close to each other. For example, in the proximity determination process (P114), the proximity determination unit 56 uses the lower end of the landmark 100. As shown in FIG. 9, the proximity determination unit 56 sets a predetermined margin δ (e.g., 5 pix) in the height direction around the lower end of the landmark 100, and determines whether or not the lower end of the pedestrian 200 is included within the margin δ. If the bottom end of the pedestrian 200 is included within the margin δ, the proximity determination unit 56 determines that the landmark 100 and the pedestrian 200 are present at approximately the same distance as seen from the vehicle V, and determines that the two are in close proximity. If the proximity determination unit 56 determines that the landmark 100 and the pedestrian 200 are in close proximity, in a size estimation process (P115), the size estimation unit 58 estimates the height of the pedestrian 200 by using the actual height of the landmark 100 stored in the landmark information storage unit 36. Note that the above-mentioned margin δ (for example, 5 pix) may be stored in the landmark information storage unit 36.

[0043] In the external environment recognition device 1d of the present modified example 1, the proximity determination unit 56 determines whether or not to use the landmark 100 for estimating the height of the pedestrian 200. When estimating the height of the pedestrian 200, the ratio of the size on the image of the pedestrian 200 and the landmark 100 (i.e., image height) is used. However, when the pedestrian 200 and the landmark 100 are located at positions apart from each other, the ratio of their image heights may deviate from the ratio of their actual heights, and the height of the pedestrian 200 may not be estimated accurately. Therefore, as in the present modified example 1, the proximity determination unit 56 performs proximity determination processing (P114), so that the height of the pedestrian 200 can be estimated more accurately.

[0044] Next, another example of the external environment recognition device 1d according to Modification 1 will be described. FIG. 10 is a diagram showing another example of the proximity determination process (P114) of the external environment recognition device 1d of FIG. 7. In FIG. 10, the landmark registration process (P105) is described in the upper part, and the proximity determination process (P114) is described in the lower part. The example shown in FIG. 10 differs from the external environment recognition device 1d according to Modification 1 described above in terms of the landmark registration process (P105) and the proximity determination process (P114). In the example shown in FIG. 10, the same reference numerals as those in the external environment recognition device 1d according to Modification 1 (FIG. 7) will be used and the description will be given.

[0045] In the example shown in FIG. 10, the proximity determination unit 56 executes the proximity determination process (P114) based on the parallax around the landmark 100 detected by the compound eye region landmark detection unit 32. For example, as shown in the upper part of FIG. 10, the compound eye region landmark detection unit 32 may calculate a change in the tilt of the surface on which the detected landmark 100 is placed, from parallax information around the surface on which the landmark 100 is placed. This identifies the location where the tilt of the surface on which the landmark 100 is placed changes. Thereafter, the size information measurement unit 33 calculates the ratio Plane_rate of the image height (Obj_Height) of the landmark 100 detected by the compound eye region landmark detection unit 32 to the image length (Plane_Distance) on the image from the center position of the landmark 100 to the location where the tilt changes, using "Plane_rate = (Plane_Distance / Obj_Height)" (Equation (4)). Then, the landmark information storage unit 36 ​​stores the value of Plane_rate as the proximity determination radius in addition to the predetermined margin δ, for example (landmark registration process (P105)).

[0046] 10, in the proximity determination process (P114), the proximity determination unit 56 calculates a margin δ2 (Plane_rate×Obj_Height2) using the image height (Obj_Height2) of the landmark 100 redetected by the monocular area landmark detection unit 52. Then, the proximity determination unit 56 determines that the landmark 100 and the pedestrian 200 are in proximity when the pedestrian 200 approaches within δ2 (pix) in the horizontal direction in addition to the vertical margin δ (pix).

[0047] In the size estimation process (P115), the height of the pedestrian 200 is estimated on the assumption that the pedestrian 200 and the landmark 100 are at an equal distance from the host vehicle V. However, if the inclination of the surface on which the landmark 100 is in contact changes midway, referring only to the vertical margin δ in the proximity determination process (P114) may result in a determination that the landmark 100 and the pedestrian 200 are close to each other, even though they are not at an equal distance. Therefore, by calculating the area up to the point where the inclination of the road surface does not change (δ2 in the lower part of FIG. 10) based on the parallax information and performing the proximity determination process (P114) based on this result, it is possible to correctly determine whether the two are at an equal distance. Therefore, the height of the pedestrian 200 can be estimated more accurately.

[0048] <Variation 2> Next, a second modification of the first embodiment of the present invention will be described.

[0049] The external environment recognition device 1e according to the modified example 2 differs from the above-described external environment recognition device 1 in that it includes a landmark information measurement unit 34, which will be described later. Hereinafter, components having the same or similar functions as those of the above-described external environment recognition device 1 will be denoted by the same reference numerals, and descriptions thereof will be omitted, and different parts will be described.

[0050] 11 is a functional block diagram showing a schematic configuration of an external environment recognition device 1e according to Modification 2 of Embodiment 1. As shown in FIG.

[0051] The landmark information measurement unit 34 determines whether the landmarks 100, 101, and 102 detected by the compound eye area landmark detection unit 32 have moved from the compound eye area image CA. Then, the landmark information storage unit 36 ​​stores, among the landmarks 100, 101, and 102 included in the compound eye area image CA, the landmark that has been determined to have moved by the landmark information measurement unit 34 (for example, the landmark 101 in FIG. 4) as a landmark for estimating the distance to the pedestrian 200.

[0052] In the external environment recognition device 1e according to the second modification, not only stationary three-dimensional objects are registered as landmarks, but also moving objects, such as other three-dimensional objects such as pedestrians and vehicles, can be used as landmarks. The landmark 100 shown in FIG. 4 is a stationary object, so if the pedestrian 200, which is the target of distance calculation relative to the host vehicle V, is stationary, the two do not approach each other. For this reason, it is not possible to estimate the height of the stationary pedestrian 200 based on the stationary landmark 100. In contrast, by using a moving object (for example, the landmark 101) as in the second modification, the landmark 101 may approach the pedestrian 200 even when the pedestrian 200, which is the target of distance calculation, is stationary. Therefore, it becomes possible to estimate the height of the stationary pedestrian 200.

[0053] <Variation 3> Next, a third modification of the first embodiment of the present invention will be described.

[0054] The external environment recognition device according to the modified example 3 differs from the above-described external environment recognition device in the function of the landmark information storage unit 36. Hereinafter, the external environment recognition device 1 according to the modified example 3 will be described with reference to FIGS. 1 to 6, with the same reference numerals as those of the above-described external environment recognition device 1 assigned.

[0055] In the third modification, the landmark information storage unit 36 ​​shown in FIG. 1 stores, among the landmarks 100, 101, and 102 that are the targets of measurement by the size information measurement unit 33, landmarks whose actual heights are equal to or greater than a height threshold (e.g., 1 m) (e.g., the landmarks 100 and 102 in FIG. 4) as landmarks for estimating the distance to the pedestrian 200. In this way, in the third modification, among the landmarks 100, 101, and 102 measured in the three-dimensional information acquisition process (P104), only landmarks whose actual heights are equal to or greater than a predetermined height threshold (e.g., 1 m or greater) (e.g., the landmarks 100 and 102 in FIG. 4) are registered in the landmark information storage unit 36 ​​in the landmark registration process (P105). Then, in the landmark detection process (P112), the landmarks (e.g., the landmarks 100 and 102 in FIG. 4) that are registered in the landmark registration process (P105) and have actual heights equal to or greater than the height threshold are redetected.

[0056] If a process such as template matching is used to redetect landmarks with low actual heights in the landmark detection process (P112), a deviation in the detected position may occur. If a deviation in the detected position occurs, the image height of the landmark detected by the monocular area landmark detection unit 52 may change from the true image height of the landmark, which may result in a deviation in the height of the pedestrian 200 estimated in the size estimation process (P115). In this regard, the external environment recognition device according to the third modification uses only landmarks with a predetermined height threshold (e.g., 1 m) or higher. This makes it possible to reduce the deviation of the detected landmark image height from the true image height, thereby enabling a more accurate estimation of the pedestrian's height.

[0057] As another example of the third modification, the landmark information storage unit 36 ​​may store, as landmarks for estimating the distance to the pedestrian 200, a predetermined number of landmarks (e.g., the landmarks 100 and 102) with the highest actual heights (e.g., a number equivalent to 60% of the plurality of landmarks) from among the plurality of landmarks 100, 101, and 102 that were the targets of measurement by the size information measurement unit 33. This makes it possible to hold a large number of options for landmarks to be used for estimating the distance to the pedestrian 200.

[0058] <Variation 4> Next, a fourth modification of the first embodiment of the present invention will be described.

[0059] The external environment recognition device according to Modification 4 differs from the external environment recognition device 1e (FIG. 11) of Modification 2 in the functions of the landmark information measurement unit 34 and the landmark information storage unit 36. Hereinafter, the external environment recognition device according to Modification 4 will be described with reference to FIGS. 3 to 6 and 11, with the same reference numerals as those of the above-described external environment recognition device 1e assigned.

[0060] In the fourth modification, the landmark information measurement unit 34 calculates contrast information (edge ​​components, color, brightness difference, etc.) of the landmarks 100 based on the compound eye region image CA for the landmarks 100, 101, and 102 detected by the compound eye region landmark detection unit 32. Then, the landmark information storage unit 36 ​​stores, among the landmarks 100, 101, and 102 included in the compound eye region image CA, a landmark having contrast information equal to or greater than a predetermined threshold (for example, the landmark 100 in FIG. 4) as a landmark for estimating the distance to the pedestrian 200.

[0061] Specifically, in the fourth modification, the landmark information measurement unit 34 extracts edge components (contrast information) by applying a Sobel filter to each of the landmarks 100, 101, and 102 in the landmark registration process (P105). Then, in this process, the landmark information storage unit 36 ​​registers a landmark (e.g., the landmark 100) whose edge component density (number of edges) is equal to or greater than a predetermined value within the region of the landmarks 100, 101, and 102. Alternatively, a landmark (e.g., the landmark 100) whose color information (contrast information) is equal to or greater than a predetermined value may be registered. Specifically, pixels whose saturation and brightness are equal to or greater than a predetermined value may be counted in a hue region that can be considered to be the red component of the image, and a landmark may be registered if the number of pixels is equal to or greater than a predetermined value. This allows the landmark 100 to be redetected more reliably in the landmark detection process (P112), and prevents other three-dimensional objects from being mistakenly detected as landmarks. This allows the height of the pedestrian 200 to be estimated with higher accuracy.

[0062] Furthermore, in the landmark registration process (P105), the landmark information measurement unit 34 analyzes the brightness difference between the lower ends of the landmarks 100, 101, and 102 and the road surface. Then, in this process, the landmark information storage unit 36 ​​stores a landmark (e.g., landmark 100) among the landmarks 100, 101, and 102, for which the number of pixels where the brightness difference between the lower end of the landmark and the road surface is a predetermined value or more is a certain value or more. This makes it easier for the monocular region landmark detection unit 52 to redetect the same area as that detected by the compound eye region landmark detection unit 32, and enables the height of the pedestrian 200 to be estimated with higher accuracy. Furthermore, if the external environment recognition device according to the fourth modification includes the proximity determination unit 56, by registering a landmark 100 that is highly separable from the road surface when using the lower end of the landmark 100 in the proximity determination process (P114), it is possible to perform proximity determination more accurately.

[0063] Furthermore, the landmark information storage unit 36 ​​can register a specific portion of a landmark in the landmark registration process (P105). Specifically, the landmark information measurement unit 34 measures, for example, a portion of the landmark 100 with a strong edge component as the top end of the landmark. Then, in this process, the landmark information storage unit 36 ​​can register this portion as the top end of the landmark. This makes it easier for the monocular region landmark detection unit 52 to redetect the same portion as the portion detected by the compound eye region landmark detection unit 32. In this way, by registering, for example, a specific portion of the landmark 100 based on the portion with a strong edge component, the monocular region landmark detection unit 52 can stably redetect the landmark, allowing for more accurate estimation of the distance of a pedestrian.

[0064] <Variation 5> Next, a fifth modification of the first embodiment of the present invention will be described.

[0065] The external environment recognition device according to the fifth modification is different from the above-described external environment recognition device 1 in the function of the size estimation unit 58. Hereinafter, the external environment recognition device according to the fifth modification will be described with reference to FIGS. 1 to 10, with the same reference numerals as those of the above-described external environment recognition device 1 assigned.

[0066] In the external environment recognition device according to the fifth modification, when a plurality of landmarks 100, 101, and 102 are registered in the landmark information storage unit 36, the size estimation unit 58 estimates the size of the pedestrian 200 by using the plurality of landmarks. Specifically, when the pedestrian 200 approaches the landmark 100 in a certain image frame, the size estimation unit 58 estimates the height (Height_Ped) of the pedestrian 200. Then, the distance calculation unit 60 calculates the distance between the host vehicle V and the pedestrian 200 based on the Height_Ped. Thereafter, when the pedestrian 200 approaches the landmark 101 in another image frame, the size estimation unit 58 updates the height (Height_Ped) of the pedestrian by "Height_Ped = (Height_Ped + Height_Ped_2) / 2" (Equation (5)). Here, Height_Ped_2 is the height of the pedestrian 200 estimated from the actual height of the landmark 101, the image height of the landmark 101, and the image height of the pedestrian 200. In the size estimation process (P115), the size estimation unit 58 updates the height of the pedestrian 200 each time the pedestrian approaches a plurality of landmarks. This allows the height to be updated so as to approach a more correct value each time the pedestrian approaches each landmark, even if the initial height estimation result contains an error, and allows the distance to the pedestrian to be estimated more accurately.

[0067] Second Embodiment Next, a second embodiment of the present invention will be described.

[0068] 12 is a functional block diagram showing a schematic configuration of an external environment recognition device 1a according to a second embodiment of the present invention. The external environment recognition device 1a according to the second embodiment differs from the above-described external environment recognition device 1d (variation 1) in that it includes a landmark selection unit 40 and a traveling path estimation unit 70, which will be described later. Configurations having the same or similar functions as those of the above-described external environment recognition device 1d are given the same reference numerals as those of the external environment recognition device 1d, and their description will be omitted. Different parts will be described with reference to FIGS. 2 to 6 and 12.

[0069] As shown in FIG. 12, the external environment recognition device 1a includes a landmark selection unit 40 and a traveling path estimation unit 70. The landmark selection unit 40 is included in the landmark information acquisition unit 30. When the landmark information acquisition unit 30 of the external environment recognition device 1a according to the second embodiment acquires three-dimensional information of a plurality of landmarks 100, 101, and 102, it selects a landmark (e.g., the landmark 100) to be used for distance estimation from the plurality of landmarks according to the movement of the vehicle V. Specifically, the landmark selection unit 40 of the landmark information acquisition unit 30 selects the landmark (e.g., the landmark 100) to be used for distance estimation from the plurality of landmarks. More specifically, the landmark selection unit 40 selects the landmark 100 present in the traveling direction of the vehicle V estimated by the traveling path estimation unit 70, which will be described later, as the landmark to be used for estimating the distance to the pedestrian 200. Then, the landmark information storage unit 36 ​​stores the landmark 100 selected by the landmark selection unit 40.

[0070] The traveling path estimation unit 70 estimates the traveling direction of the vehicle V from the movement of the vehicle V (i.e., behavior information). Specifically, the traveling path estimation unit 70 predicts the traveling path of the vehicle V based on the speed, yaw rate, etc. of the vehicle V. For example, the traveling path estimation unit 70 estimates the turning radius of the vehicle V from the vehicle speed and yaw rate values, and sets the trajectory on that circumference as the traveling path. Note that, when the vehicle V is traveling straight, the traveling path estimation unit 70 may estimate the trajectory along the straight traveling direction as the traveling path.

[0071] As described above, in the external environment recognition device 1a according to the second embodiment, in the landmark registration process (P105), only landmarks (e.g., landmark 100) that exist in the traveling direction estimated by the traveling path estimation unit 70 are registered in the landmark information storage unit 36. For example, landmarks 100 that exist within a predetermined distance (20 m) from the traveling path of the vehicle V may be stored in the landmark information storage unit 36. In this way, by selecting landmarks to be registered based on the traveling path information of the vehicle V, it is possible to register only landmarks that are used to estimate the distance to the pedestrian 200 who is at risk of colliding with the vehicle V. As a result, by reducing the number of landmarks to be registered, it is possible to limit memory usage and the number of landmarks detected in the landmark detection process (P112), thereby reducing the processing load on the external environment recognition device 1a.

[0072] <Third embodiment> Next, a third embodiment of the present invention will be described.

[0073] 13 is a functional block diagram showing a schematic configuration of an external environment recognition device 1b according to a third embodiment of the present invention. The external environment recognition device 1b according to the third embodiment differs from the above-described external environment recognition device 1d (variation 1) in that it includes a landmark selection unit 40 and a surrounding environment recognition unit 90, which will be described later. Configurations having the same or similar functions as the above-described external environment recognition device 1d are denoted by the same reference numerals as those of the external environment recognition device 1d, and their description will be omitted. Different parts will be described with reference to FIGS. 2 to 6 and 13.

[0074] As shown in FIG. 13, the external environment recognition device 1b includes a landmark selection unit 40 and a surrounding environment recognition unit 90 that recognizes the surrounding environment of the vehicle V. The landmark selection unit 40 is included in the landmark information acquisition unit 30. The landmark information acquisition unit 30 of the external environment recognition device 1b according to the third embodiment selects a landmark (e.g., landmark 100) that exists in an area recognized as a sidewalk by the surrounding environment recognition unit 90 as a landmark to be used for estimating the distance to the pedestrian 200. Specifically, the landmark selection unit 40 of the landmark information acquisition unit 30 selects the landmark (e.g., landmark 100) to be used for estimating the distance to the pedestrian 200. Then, the landmark information storage unit 36 ​​stores the landmark selected by the landmark selection unit 40.

[0075] The surrounding environment recognition unit 90 estimates type information for each pixel in the images captured by the pair of cameras 10a and 10b acquired by the image acquisition unit 10. Here, type information includes information on objects such as vehicles and pedestrians, as well as information on the road surface, sidewalk, etc. A convolutional neural network is used to estimate type information for each pixel. The surrounding environment recognition unit 90 estimates type information corresponding to each pixel using a model trained using ground truth data to which type information for each pixel is assigned. This allows the surrounding environment recognition unit 90 to recognize areas that are sidewalks from the images captured by the pair of cameras 10a and 10b.

[0076] As described above, in the external environment recognition device 1b according to the third embodiment, in the landmark registration process (P105), the landmark registration process is performed using the type information estimated by the surrounding environment recognition unit 90. Specifically, the landmark information storage unit 36 ​​stores the landmarks 100 that exist on pixels that the surrounding environment recognition unit 90 estimates to be sidewalks. This makes it possible to register only the landmarks 100 that are likely to be in close proximity to the pedestrian 200, thereby reducing the processing load on the external environment recognition device 1b.

[0077] In the above-described embodiment, the pair of cameras 10a and 10b are used as one stereo camera to acquire three-dimensional information (size information) of the landmarks 100, 101, and 102 in the landmark information acquisition unit 30. However, a three-dimensional sensor (e.g., Lidar, millimeter-wave radar, ultrasonic sensor, etc.) may be used to acquire the three-dimensional information (size) of the landmarks. In this case, the pair of cameras 10a and 10b may be installed so as not to have a stereo area. For example, the camera 10a may capture only the left monocular area, and the camera 10b may capture only the right monocular area, and the images captured in these areas may be input to the image acquisition unit 10.

[0078] Furthermore, the distance between the host vehicle V and the pedestrian 200 may be estimated using specific landmarks described on a digital map or the like and three-dimensional information registered in the digital map. In the above embodiment, the height of the pedestrian 200 has been described in detail, but the distance between the host vehicle V and the pedestrian 200 may be calculated based on the width of the pedestrian 200. For example, the distance estimation unit 50 may estimate the distance to the pedestrian 200 based on the actual width of the landmark 100 and the lateral length of the landmark 100 and the pedestrian 200 in the image (i.e., the image width). In addition, a method for estimating the distance to the pedestrian 200 has been described using the pedestrian 200 as an example of an object. However, various objects such as vehicles and animals can be treated as objects of distance estimation, and the distance between the vehicle, animal, etc. and the host vehicle V can be estimated based on the height, etc., of the vehicle, animal, etc.

[0079] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0080] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely realized by hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be realized by software, in which a processor interprets and executes a program that realizes each function. Information such as the program, tape, and file that realizes each function can be stored in a memory, a recording device such as a hard disk or solid state drive (SSD), or a recording medium such as an IC card, SD card, or DVD.

[0081] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0082] 1, 1a, 1b, 1d, 1e External environment recognition device, 10 Image acquisition unit, 10a, 10b Pair of cameras (stereo cameras), 30 Landmark information acquisition unit, 32 Compound eye region landmark detection unit, 33 Size information measurement unit, 34 Landmark information measurement unit, 36 Landmark information storage unit, 40 Landmark selection unit, 50 Distance estimation unit, 52 Monocular region landmark detection unit, 54 Monocular region object detection unit, 56 Proximity determination unit, 58 Size estimation unit, 60 Distance calculation unit, 70 Path estimation unit, 90 Surrounding environment recognition unit, 100, 101, 102 Landmark, 200 Pedestrian (object), CA Compound eye region image, LA Left monocular region image (monocular region image), RA Right monocular region image (monocular region image), V Vehicle, δ Margin (proximity determination threshold)

Claims

1. An external environment recognition device mounted on a vehicle, an image acquisition unit that acquires images of landmarks and objects; a landmark information acquisition unit that acquires three-dimensional information of the landmark; a distance estimation unit that estimates a distance to the object based on three-dimensional information of the landmark and sizes of the landmark and the object in the image, the image acquisition unit acquires an image captured by a stereo camera, the image including a compound eye region where fields of view overlap and a monocular region where fields of view do not overlap; the landmark information acquisition unit acquires three-dimensional information of landmarks located in the compound eye region, the distance estimation unit estimates a distance to the object located in the monocular area; The landmark information acquisition unit a landmark information measurement unit that determines whether the landmark is moving from the compound eye area; A landmark information storage unit that stores landmarks included in the compound eye area that are determined to be moving by the landmark information measurement unit as landmarks for estimating the distance to the object. An external environment recognition device characterized by including:

2. The external environment recognition device according to claim 1, The landmark information acquisition unit a compound eye region landmark detection unit that detects the landmarks from the compound eye region; a size information measurement unit that measures the size of the landmark included in the three-dimensional information from the compound eye region, The distance estimation unit a monocular region landmark detection unit that detects the landmarks from the monocular region; a monocular region object detection unit that detects the object from the monocular region; a size estimation unit that estimates a size of the object from a size of the landmark based on an image size of the landmark in the monocular region and an image size of the object in the monocular region; An external environment recognition device comprising: a distance calculation unit that calculates the distance based on the size of the object estimated by the size estimation unit and the image size of the object in the monocular area.

3. The external environment recognition device according to claim 2, the size information measurement unit calculates an actual height of the landmark from the compound eye region; the monocular area landmark detection unit calculates an image height of the landmark from the monocular area; the monocular area object detection unit calculates an image height of the object from the monocular area; the size estimation unit estimates a height of the object from an actual height of the landmark based on an image height of the landmark and an image height of the object; The external environment recognition device, wherein the distance calculation unit calculates the distance based on the estimated height.

4. The external environment recognition device according to claim 3, The size estimation unit estimates the height of the object by multiplying the actual height of the landmark by a ratio of the image height of the object to the image height of the landmark.

5. The external environment recognition device according to claim 3, the distance estimation unit includes a proximity determination unit that calculates a difference in height between a lower end of the landmark in the monocular area and a lower end of the object in the monocular area; The external environment recognition device, characterized in that the size estimation unit estimates the height of the object when the difference calculated by the proximity determination unit is within a proximity determination threshold.

6. The external environment recognition device according to claim 1, an external environment recognition device, characterized in that, when three-dimensional information of a plurality of landmarks is acquired, the landmark information acquisition unit selects a landmark to be used for estimating the distance from among the plurality of landmarks in accordance with the movement of the vehicle.

7. The external environment recognition device according to claim 6, a path estimation unit that estimates a traveling direction of the vehicle from a movement of the vehicle; The landmark information acquisition unit selects landmarks present in the traveling direction of the vehicle estimated by the traveling path estimation unit as landmarks to be used for estimating the distance to the object.

8. The external environment recognition device according to claim 1, a surrounding environment recognition unit that recognizes the surrounding environment of the vehicle; The landmark information acquisition unit selects landmarks present in an area recognized as a sidewalk by the surrounding environment recognition unit as landmarks to be used for estimating the distance to the object.

Images