Device for detecting an external world and method for detecting an external world

Abstract

An external world detection device includes: an imaging plane estimator that estimates an imaging plane in the first image based on a landmark detection result by the first landmark detection unit; an image transformation estimator that estimates an image transformation parameter that corresponds to an imaging plane in the second image based on information about the imaging plane estimated by the imaging plane estimator and imaging parameters of the plurality of imaging units; a feature collation unit that collates a landmark feature detected from the first image, which undergoes an image transformation using an image transformation parameter by the image transformation estimator, with a landmark feature detected from the second image;and a position estimation unit that estimates a three-dimensional position of the landmark based on a collation result of the feature collation unit.

Description

Technical area

[0001] The present invention relates to an external world recognition device and an external world recognition method. State of the art

[0002] In the realization of automatic driving and advanced driving assistance systems, the importance of a camera that monitors the external world of a vehicle and detects obstacles on a travel route and objects necessary for the vehicle's driving, such as lane information of the travel route, is increasing. Specifically, to improve object detection performance, a plurality of cameras are mounted on a vehicle, and a function of acquiring information around the vehicle from the cameras and recognizing an environmental situation is realized.

[0003] One type of camera that detects the external world is a stereo camera, which uses multiple cameras. In a stereo camera, two cameras are arranged in a vehicle at predetermined intervals. Then, the distance to the detected object can be measured using the parallax of the overlapping area of ​​the images captured by the two cameras.

[0004] Stereo cameras include a parallel stereo camera in which the optical axes of two cameras are installed in parallel, a non-parallel stereo camera in which the optical axes are not installed in parallel, and the like. Furthermore, a camera system using two or more cameras can be referred to as a multi-view stereo system.

[0005] An electronic control device (hereinafter referred to as an electronic control unit (ECU)) mounted on a vehicle can detect a possibility of contact with an object in front of the vehicle by measuring a distance to the object using a plurality of images captured by a plurality of cameras. However, even if two cameras located at different positions detect the same object, the appearance of the object is generally different between the two cameras. In a case where images with different appearances are collated with each other and there is a pattern similar to a pattern with a different appearance between one image and the other image, there is a possibility that the image will be collated at an incorrect position.

[0006] If the image is collated at an incorrect position, the distance to the object measured based on the collation result may be incorrect, resulting in erroneous vehicle control. If the model of the object's shape and plane is known in the real world, the appearance is geometrically preserved in each image. Therefore, it is considered that erroneous collation can be suppressed by predicting and collating the shape, plane, and the like.

[0007] PTL 1 describes that "viewpoint conversion is performed in which at least one of a first image taken by a first camera and a second image taken by a second camera is deformed to convert the first image and the second image into an image from a common viewpoint, and then a plurality of corresponding points are extracted, and geographical calibration is performed on the first camera and the second camera using coordinates of the plurality of corresponding points in the first image and the second image before the viewpoint conversion."

[0008] Furthermore, PTL 2 describes that “Since the stereo camera image information taken by the imaging device is obtained by taking an image of a monitoring target surface such as a road surface or a ground surface in a downward direction, the image information is made to face the monitoring target surface more than the conventional stereo camera image information by the stereo camera installed in a substantially horizontal direction, and the 3D distance image information is further generated based on the parallelized image information obtained by performing the parallelization transformation processing on the downward stereo camera image information.Therefore, the 3D distance image information becomes a bird's-eye view image viewed from top to bottom with respect to the monitoring target surface, so that the distance to the road surface or ground surface can be obtained with high accuracy." Citation listPatent literature PTL 1: JP 2020-12735 A PTL 2: JP 2019-16308 A Summary of the inventionTechnical problem

[0009] Although it is appropriate to change the method for calculating the distance from the own vehicle to the object for each object appearing in the image, the techniques disclosed in PTLs 1 and 2 perform processing under the assumption that a specific object appears at a specific location in the image. For this reason, in a case where an unspecified object appears at a specific location in the image, the distance to the object may be erroneously measured. Furthermore, in the techniques disclosed in PTLs 1 and 2, even if the distance to the object facing the own vehicle can be calculated, the distance of the object (for example, side walls, guardrails) or the like that does not face the own vehicle may be incorrect.

[0010] The present invention has been made in view of such a situation, and an object of the present invention is to enable estimation of a distance to an unspecified landmark even in a case where the landmark appears in two images that look different. Solution to the problem

[0011] An external world recognition device according to the present invention includes: a first landmark detection unit that detects a landmark based on a first image acquired by at least one first imaging unit among a plurality of imaging units in which at least a part of an imaging field of view with respect to an external world overlaps; a second landmark detection unit that detects a landmark based on a second image acquired by a second imaging unit among the plurality of imaging units; an imaging plane estimation unit that estimates an imaging plane in the first image based on a detection result of the landmark by the first landmark detection unit;an image transformation estimation unit that estimates an image transformation parameter corresponding to an imaging plane in the second image based on information about the imaging plane estimated by the imaging plane estimation unit and imaging parameters of the plurality of imaging units; a feature collation unit that collates a feature of the landmark detected from the first image subjected to image transformation using an image transformation parameter by the image transformation estimation unit with a feature of the landmark detected from the second image; and a position estimation unit that estimates a three-dimensional position of the landmark based on a collation processing result of the feature collation unit. Advantageous effects of the invention

[0012] According to the present invention, even in a case where an unspecified landmark appears in two images having different appearances, the distance to the landmark can be estimated.

[0013] Objects, configurations and effects besides the above description will become apparent from the explanation of the following embodiments. Short description of the drawings Fig. 1A is a diagram illustrating a position of a vehicle-mounted camera according to a first embodiment of the present invention. Fig. 1B is a diagram illustrating a world coordinate system based on a vehicle according to the first embodiment of the present invention. Fig. 1C is a diagram illustrating an image coordinate system according to the first embodiment of the present invention. Fig. 2 is a diagram illustrating an example of an image of a foreground of a vehicle taken by a front camera and a left camera according to the first embodiment of the present invention. Fig. 3 is a block diagram illustrating an internal configuration example of an external world recognition device according to the first embodiment of the present invention. Fig. 4 is a block diagram illustrating a hardware configuration example of a computer according to the first embodiment of the present invention. Fig. 5A is a diagram illustrating an example of an image captured by the front camera according to the first embodiment of the present invention, in which another vehicle, a person, a guardrail, and a sidewall are reflected in a straight traveling direction. Fig. 5B is a diagram illustrating an example of an image captured by the front camera according to the first embodiment of the present invention in which another vehicle is reflected obliquely in front of the own vehicle. Fig. 6 is a diagram illustrating an example of processing of a feature matching unit according to the first embodiment of the present invention. Fig. 7 is a diagram illustrating a state of processing in which the feature matching unit according to the first embodiment of the present invention performs template matching using a left camera image and a front camera image. Fig. 8 is a block diagram illustrating an internal configuration example of an external world recognition device according to a second embodiment of the present invention. Fig. 9 is a block diagram illustrating an internal configuration example of an external world recognition device according to a third embodiment of the present invention. Fig. 10 is a diagram illustrating an example of a region where a vehicle and a background are mixed according to the third embodiment of the present invention. Fig. 11 is a block diagram illustrating an internal configuration example of an external world recognition device according to a fourth embodiment of the present invention. Fig. 12 is a block diagram illustrating an internal configuration example of a geometric transformation estimation unit and a feature matching unit according to a modification of the external world recognition device according to the fourth embodiment of the present invention. Fig. 13 is a block diagram illustrating an internal configuration example of an external world recognition device according to a fifth embodiment of the present invention. Description of embodiments

[0014] Embodiments of the present invention will be described below with reference to the accompanying drawings. In the present description and the drawings, components having substantially the same function or configuration are denoted by the same reference numerals, and redundant descriptions are omitted. The present invention is applicable, for example, to a computing device for vehicle control with which an advanced driver assistance system (ADAS) or an in-vehicle electronic control unit (ECU) for autonomous driving (AD) can communicate. [First embodiment]

[0015] Fig. 1A, Fig. 1B and Fig. 1C are diagrams illustrating a position of a camera mounted on a vehicle 1 according to a first embodiment, a world coordinate system based on the vehicle 1, and an image coordinate system.

[0016] Fig. 1A illustrates a configuration example in which a front camera 10, a left camera 20, and a right camera 30 are provided at the front, left side, and right side of the vehicle 1, respectively. In the following description, in a case where the front camera 10, the left camera 20, and the right camera 30 are not distinguished, they are also simply referred to as "cameras."

[0017] In Fig. 1A, an imaging area 10a of the front camera 10, an imaging area 20a of the left camera 20, and an imaging area 30a of the right camera 30 are illustrated by being divided by a one-dot chain line. The imaging areas 10a and 20a partially overlap. Similarly, the imaging areas 10a and 30a partially overlap, and the imaging areas 20a and 30a partially overlap. In these overlapping areas, the same object appears in the images captured by the respective cameras.

[0018] From an image captured by a camera installed in the vehicle 1, an ECU (external world recognition device 100 installed in Fig. 3, which will be described later) mounted on the vehicle 1, acquires object information of an object present around the vehicle 1, such as a pedestrian, another vehicle, a white line, or a road surface, and physical quantities such as a distance from the vehicle 1 to the object, a size of the object, and a speed of the object. Then, another ECU (ECU for automatic driving control, which is not illustrated) determines a control amount of the vehicle 1 based on the acquired physical quantity and performs automatic driving, driving assistance, and the like of the vehicle 1. In the following description, an object (road surface, person, guardrail, white line, etc.) that may affect the driving of the vehicle 1 is referred to as a "landmark."

[0019] Fig. Figure 1B illustrates an example of a world coordinate system with the position of vehicle 1 as the origin. In this world coordinate system, an X-axis is taken in a traveling direction of vehicle 1, a Y-axis is taken in a horizontal direction orthogonal to the X-axis, and a Z-axis is taken in a vertical direction. The position of the landmark other than vehicle 1 is estimated by the world coordinate system. Therefore, the distance from vehicle 1 to the landmark is also estimated.

[0020] There are several methods for estimating the distance from the vehicle 1 to the landmark. As a representative distance estimation method, there is a method for measuring the distance to a landmark using the principle of triangulation in a case where multiple cameras capture images of the same landmark. In this method, any point in the world coordinates existing on the road is represented as a transposed matrix X = (X, Y, Z, 1). T Set in a homogeneous coordinate system, an external parameter matrix related to the rotation angle of the camera is set as R, and an external parameter matrix related to the installation position of the camera is set as T. Then, the matrix of the external parameter matrices R and TP = (R|T), and an internal parameter matrix for managing the internal state, such as the focal length and optical center of the camera, is K.

[0021] Fig. Figure 1C illustrates an example of an image coordinate system for identifying a landmark in an image. In this image coordinate system, the u-axis is taken in the horizontal direction, with the upper left side of the image as the origin, and the v-axis is taken in the vertical direction. Then, the position of the landmark in the image captured by the camera is specified by (u, v).

[0022] Here, assuming that the image coordinates obtained by detecting the transposed matrix X, the transposed matrix u = (u, v, 1) Tare in the homogeneous coordinate system, the scale parameter is s, and the lens of the perspective projection model is used, Expression (1) is fixed for each camera. The scale parameter s is a scalar value. Further, a suffix on a symbol in Expression (1) represents the type of camera. Here, to identify two cameras, a suffix of one camera is set to "0", and a suffix of the other camera is set to "1". Note that in Expression (1), if there is no scale parameter s, the right side of Expression (1) becomes a constant multiple of the left side. Therefore, a scale parameter s is provided so that the right side of Expression (1) does not become a constant multiple of the left side. [Math. 1] S0u0=K0P0XS1u1=K1P1X

[0023] In a case where image coordinates of a camera are given, it is impossible to recover a three-dimensional point of a landmark from the image coordinates without a precondition. However, in a case where there are two cameras showing the same position X, if the same section of position X is represented by image coordinates (u0, u1), the three-dimensional point X can be estimated by solving the least squares method using Expression (2). However, depending on the form of the formula deformation, it is not necessary to limit oneself to the solution using Expression (2). [Math. 2] (R0−K0−1u00R10−K1−1u1)(XS0S1)=−(T0T1)

[0024] Here, the meaning of expression (2) is explained with reference to Fig. 1A described.

[0025] The viewing angle of the front camera 10 is a vertex angle of a triangle at an installation position of the front camera 10, as illustrated in the imaging area 10a. Similarly, the viewing angle of the left camera 20 is a vertex angle of a triangle at an installation position of the left camera 20, as illustrated in the imaging area 20a. The viewing angle of the right camera 30 is a vertex angle of a triangle at an installation position of the right camera 30, as illustrated in the imaging area 30a.

[0026] An area where the angle of view of the front camera 10 and the angle of view of the left camera 20 overlap is located on the left front side of the vehicle 1. Therefore, the distance can be estimated by the principle of triangulation with respect to the landmark that appears in the area where the angles of view of the two cameras overlap. Similarly, the distance can also be estimated for an area where the angle of view of the front camera 10 and the angle of view of the right camera 30 (right front side of the vehicle 1) overlap by the principle of triangulation.

[0027] To efficiently search for a landmark that commonly appears in two images at the same position of the two images, the epipole constraint equation shown in Expression (3) is used. In the case where the image coordinates of the landmark in one image are given, the candidate image coordinate position of the landmark in the other image is located on the straight line, thus providing the constraint that the search is performed on the straight line instead of the entire image. The basis matrix used in the epipole constraint equation is derived from the external parameter matrix and the internal parameter matrix of the two cameras (for example, the front camera 10 and the left camera 20) described above. [Math. 3] u0Fu1=0

[0028] If the preprocessing of parallelization is performed by appropriately using the parameter group described above, a conversion to an image in which any points X in the world coordinates are located at the same height of the image can be performed. It is possible to search for the same position of the landmark by a simple process of searching in the lateral direction of the parallelized image. This is the principle of a parallel stereo camera. In any case, the search for the same position of the landmark from the two images can be estimated from parameters unique to the camera that captures each image, information about installation conditions, and the like. Here, in Fig. 1A, a combination of images from the front camera 10 and the left camera 20, images from the front camera 10 and the right camera 30, and images from the left camera 20 and the right camera 30 are assumed to be both images. Hereinafter, searching the two images to obtain the same landmark position is simply referred to as "collation."

[0029] Note that there are several methods for collating image information, and template matching is a typical method. Template matching is a method in which a periphery of image coordinates of a landmark of interest is cut out from an image in a small area such as a rectangle, and an area near a pixel distribution in the rectangle is searched using an appropriate cost function while moving within the small rectangular area from the other image. As a comparison method, a sum of absolute differences, a sum of squared differences, and the like are known as basic methods.

[0030] As another method for collating image information, there is a method in which image coordinates in a portion with a high image feature, such as a corner of a landmark in an image, are first searched to obtain a feature point, and then image feature amounts around the feature point are vectorized and retained. Scale Invariant Feature Transform (SIFT), Oriented FAST and Rotated Brief (ORB), and the like are known as representative feature quantization methods, and there are also deep learning-based methods. The deep learning-based method is a method for collating feature amounts between two images.

[0031] Here, an image is described in which the same landmark appears in an overlapping region of the viewpoints of two cameras.

[0032] Fig. Fig. 2 is a diagram illustrating an example of an image of the foreground of the vehicle 1 acquired by the front camera 10 and the Fig. 1A. An image captured by the front camera 10 is referred to as a "front camera image," and an image captured by the left camera 20 is referred to as a "left camera image."

[0033] There is a pedestrian crossing in front of vehicle 1, and there is a rectangular parallelepiped-shaped landmark behind it. Both the pedestrian crossing and the landmark on the rectangular parallelepiped are shown in the left camera image and the front camera image. Here, partial areas 21 and 22 of the pedestrian crossing shown in the left camera image are indicated by broken rectangular frames. Similarly, partial areas 11 and 12 of the pedestrian crossing shown in the front camera image are indicated by broken rectangular frames. A location corresponding to the rectangular frame of the left camera image, such as a combination of areas 21 and 11 and a combination of areas 22 and 12, also exists in the front camera image. However, the appearance of the inside of the rectangular frame (part of the pedestrian crossing) is greatly different between the left camera image and the front camera image.

[0034] It is difficult to collate images of the same location shown in two images that are largely different in appearance as described above by the template matching described above, and it is also difficult to collate images using feature points or feature sets. Typically, template matching compares pixels at the same position in an image, but when the appearances are different, matching becomes difficult. Moreover, even if feature points or feature sets are generated between images with different appearances, it is difficult to match the images. Therefore, different shapes are collated with each other, or a shape that should originally be determined to be the same is not found in the image and is not collated.If the matching of the same landmark appearing in different images is not performed correctly, the distance to the landmark may be estimated incorrectly.

[0035] Therefore, the inventor of the present invention investigated how to transform an image to reduce the difference between the landmark at the time of collation between two images with a largely different appearance. Hereinafter, an external world recognition device and a landmark collation method according to each embodiment of the present invention, which are capable of improving the performance of collating the same object shown in two images, will be described with reference to Fig. 3 and subsequent drawings. The function of the external world detection device is implemented by software configured in an ECU mounted on the vehicle 1.

[0036] Fig. 3 is a block diagram illustrating an internal configuration example of the external world recognition device 100 according to the first embodiment. The external world recognition device 100 is, for example, a form of ECU mounted on the vehicle 1. Further, in the following description, the vehicle 1 on which the external world recognition device 100 is mounted will also be referred to as "own vehicle 1" to distinguish it from another vehicle 2.

[0037] The external world recognition device 100 is a device mounted on the vehicle 1. The external world is, for example, the exterior of the vehicle 1 on which the external world recognition device 100 is mounted, and the external world recognition device 100 recognizes various landmarks that exist in the external world. The landmark recognition result includes, for example, information such as an attribute of the landmark, a size of the landmark, and a distance from the vehicle 1 to the landmark.

[0038] Two images captured by the two imaging units 40 and 41 are input to the external world recognition device 100. The imaging unit 40 is, for example, the one shown in Fig. 1A illustrates the left camera 20 and outputs an image 40a captured by the left camera 20 to a landmark detection unit 101.

[0039] The imaging unit 41 is, for example, the one shown in Fig. 1A illustrated front camera 10 and outputs an image 41a captured by the front camera 10 to a landmark detection unit 102.

[0040] The combination of Fig. 1A, the cameras of the imaging units 40 and 41 may be the front camera 10 and the right camera 30 and the left camera 20 and the right camera 30.

[0041] The external world recognition device 100 includes two landmark detection units 101 and 102, a camera parameter 103, a collation condition determination unit 104, a feature collation unit 105, and a position estimation unit 106.

[0042] The landmark detection unit 101 performs landmark detection processing on the image input from the imaging unit 40. A first landmark detection unit (landmark detection unit 101) detects a landmark based on a first image (image 40a) acquired by at least one first imaging unit (imaging unit 40) among a plurality of imaging units in which at least a part of an imaging field of view with respect to the external world overlaps.

[0043] The landmark detection unit 102 performs landmark detection processing on the image input from the imaging unit 41. A second landmark detection unit (landmark detection unit 102) detects a landmark based on a second image (image 41a) acquired by a second imaging unit (imaging unit 41) among the plurality of imaging units.

[0044] The landmark detection processing performed by the landmark detection units 101 and 102 is, for example, a process of detecting the vehicle 2, a white line, a pedestrian, a passable area of ​​the vehicle 1, a guardrail, a side wall, and the like from the input images 40a and 41a. A part of each image shown in the areas 11, 12, 21, and 22 as shown in Fig. 2, is output as a result of landmark detection processing to a subsequent functional unit. Note that the landmark detection units 101 and 102 can also obtain an attribute of a landmark by performing image analysis for each landmark through landmark detection processing. A result of the landmark detection processing by the landmark detection unit 101 is output to an imaging plane estimation unit 111 of the collation condition determination unit 104. A result of the landmark detection processing by the landmark detection unit 102 is output to the feature collation unit 105. In the following description, a result of the landmark detection processing is also referred to as a "landmark detection result."

[0045] Furthermore, the image captured by the imaging unit 40 is input to the collation condition determination unit 104 via the landmark detection unit 101 and further input to the feature collation unit 105 via the collation condition determination unit 104. The image captured by the imaging unit 41 is input to the feature collation unit 105 via the landmark detection unit 102.

[0046] The collation condition determination unit 104 determines a collation condition for the feature collation unit 105 to collate positions where the same landmark detected in images 40a and 41a corresponds to each other. The collation condition determination unit 104 includes an imaging plane estimation unit 111 and a geometric transformation estimation unit 112.

[0047] The imaging plane estimation unit (imaging plane estimation unit 111) estimates the imaging plane in the first image (image 40a) based on the detection result of the landmark by the landmark detection unit (landmark detection unit 101). The imaging plane estimation unit 111 estimates a plane of each landmark in the real world based on the detection result of the landmark input from the landmark detection unit 101. A plane constituting each landmark is called an imaging plane and is expressed by a plane equation of the imaging plane. The imaging plane estimation unit (imaging plane estimation unit 111) estimates the imaging plane based on the size of the landmark and the distance to the landmark. Further, the imaging plane estimation unit (imaging plane estimation unit 111) estimates the imaging plane in the image region where the landmark is imaged.Note that the plane equation is represented by a parameter (α, β, γ, δ), for example. The imaging plane equation estimated by the imaging plane estimation unit 111 is output to the geometric transformation estimation unit 112.

[0048] The image transformation estimation unit (geometric transformation estimation unit 112) estimates an image transformation parameter (geometric transformation parameter) that matches the imaging plane of the second image (image 41a) among the plurality of imaging units based on the information of the imaging plane estimated by the imaging plane estimation unit (imaging plane estimation unit 111) and the imaging parameters (camera parameters 103) of the plurality of imaging units. For example, the geometric transformation estimation unit 112 estimates a geometric transformation parameter between images at locations imaged by the imaging units 40 and 41 based on the equation of the imaging plane input from the imaging plane estimation unit 111 and the camera parameter 103. Examples of the camera parameter 103 include K and P included in Expression (1) described above.However, the camera parameter 103 can be a fixed value.

[0049] Note that the imaging plane estimation unit 111 and the geometric transformation estimation unit 112 may be located on the landmark detection unit 102 side.

[0050] The feature collation unit (feature collation unit 105) collates the feature of the landmark detected from the first image (image 40a) subjected to image transformation using the image transformation parameter (geometric transformation parameter) by the image transformation estimation unit (geometric transformation estimation unit 112) with the feature of the landmark detected from the second image (image 41a). The feature collation unit 105 collates a feature of a transformation result obtained by performing a geometric transformation on the landmark detected by the landmark detection unit 101 with a feature of the detection result of the landmark input by the landmark detection unit 102.Therefore, the feature collation unit 105 converts the shape of the landmark detected from the image 40a using the geometric transformation parameters estimated by the geometric transformation estimation unit 112. Next, the feature collation unit 105 collates the shape of the landmark whose shape has been converted with the shape of the landmark detected from the image 41a and calculates the distance from the host vehicle 1 to the landmark. The distance from the host vehicle 1 to the landmark is output by the feature collation unit 105 as a collation result to the position estimation unit 106. For example, the collation result may be represented by a value where a mismatch is 0% and an exact match is 100% for each image region including the landmark to be collated.Then, when the comparison result is 80% or more, the feature collation unit 105 may determine that the collation result has high reliability and output the collation result with high reliability to the position estimation unit 106.

[0051] The position estimation unit (position estimation unit 106) estimates the three-dimensional position of the landmark based on the result of the collation processing by the feature collation unit (feature collation unit 105). For example, the position estimation unit 106 estimates a three-dimensional position (referred to as a "position estimation result") of the landmark in the world coordinate system (see Fig. 1B) based on a collation result by the feature collation unit 105. At this time, the position estimation unit 106 may calculate the distance from the host vehicle 1 to the landmark and incorporate the distance for each landmark into the position estimation result. The position estimation result is used for vehicle control of the host vehicle 1 by another ECU (for example, an ECU for controlling automatic driving) mounted on the host vehicle 1, or for obtaining depth information of a landmark around the host vehicle 1.

[0052] Next, a hardware configuration of a computer 80 constituting the external world recognition device 100 will be described.

[0053] Fig. 4 is a block diagram illustrating a hardware configuration example of the computer 80. The computer 80 is an example of hardware used as a computer operable as the external world recognition device 100 according to the present embodiment. The external world recognition device 100 according to the present embodiment implements an image processing method executable by the respective functional blocks shown in Fig. 3, in cooperation with each other by the computer 80 (computer) executing a program.

[0054] The computer 80 includes a central processing unit (CPU) 81, a read-only memory (ROM) 82, and a random access memory (RAM) 83, each of which is connected to a bus 84. Furthermore, the computer 80 includes a non-volatile memory 85 and a network interface 86.

[0055] The CPU 81 reads a program code of software for implementing each function according to the present embodiment from the ROM 82, loads the program code into the RAM 83, and executes the program code. Variables, parameters, and the like generated during the arithmetic processing of the CPU 81 are temporarily written into the RAM 83, and these variables, parameters, and the like are appropriately read by the CPU 81. However, a microprocessing unit (MPU) may be used instead of the CPU 81. The function of each functional unit in the external world recognition device 100 is realized by the CPU 81, the ROM 82, and the RAM 83.

[0056] For example, a hard disk drive (HDD), a solid-state drive (SSD), a flexible disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, or a non-volatile memory can be used as the non-volatile memory 85. In addition to an operating system (OS) and various parameters, a program for causing the computer 80 to function is recorded in the non-volatile memory 85. The ROM 82 and the non-volatile memory 85 record programs, data, and the like necessary for the operation of the CPU 81 and are used as an example of a computer-readable non-volatile storage medium that stores a program executed by the computer 80. Various values, such as the camera parameter 103, are stored in the RAM 83 or the non-volatile memory 85 and read appropriately.

[0057] A network interface card (NIC) or the like, for example, is used as the network interface 86, and various data can be transmitted and received to and from external devices via a local area network (LAN), a dedicated line, or the like connected to a port of the NIC.

[0058] Here, the processing performed by the imaging plane estimation unit 111 will be explained with reference to Fig. 5A and Fig. 5B.

[0059] Fig. 5A and Fig. 5B illustrate examples of images captured by the front camera 10.

[0060] Fig. 5A is an image showing another vehicle 2, a person, a guardrail, and a sidewall in a straight direction of travel. In Fig. 5A, an area 51 represents a detection result of a road surface, an area 52 represents a detection result of a guardrail, an area 53 represents a detection result of a person, and an area 54 represents a detection result of a sidewall. In this way, the areas 51 to 54 are used as a detection result of the landmark.

[0061] The imaging plane estimation unit 111 estimates parameters (α, β, γ, δ) such that, for example, the imaging plane follows αX + βY + γZ + δ = 0, which is a plane equation, according to a detection result at certain image coordinates.

[0062] For example, assume that a result of the landmark detection unit 101 detecting a landmark from an image is a road surface such as the area 51. In this case, the imaging plane estimation unit 111 estimates a plane captured at the image coordinates as a vehicle contact surface on which the vehicle 1 travels in contact, Z = 0, that is, (α, β, γ, δ) = (0, 0, 1, 0). Note that in a case where the landmark detection result is a slope, the plane equation is expressed as, for example, hX + Z + k = 0. That is, the imaging plane estimation unit 111 can estimate as (α, β, γ, δ) = (h, 0, 1, k).

[0063] Fig. 5B is an image showing another vehicle 2 obliquely in front of the own vehicle 1. Here, a scene is assumed in which there is a curved road in front of the own vehicle 1. For example, assume that the detection result by the landmark detection unit 101 is another vehicle 2, and the detection result is output as a rectangular parallelepiped-shaped bounding box such as the area 55. In this case, the imaging plane estimation unit 111 estimates a plane equation of each surface with the front surface, the back surface, and the side surface of the rectangular parallelepiped as imaging planes from the shape of the rectangular parallelepiped represented by the area 55.Note that the front surface of the bounding box in the area 55 is the surface on the front of the vehicle 2, the rear surface is the surface on the rear of the vehicle 2, and the side surface is the surfaces on the left and right of the vehicle 2.

[0064] Here, it is assumed that another vehicle 2 detected by the landmark detection unit 101 stands upright on a plane of Z = 0. Therefore, in the imaging plane estimation unit 111, the plane equation of the rectangular parallelepiped-shaped surface represented by the area 55 is expressed, for example, as aX + bY + d = 0. That is, the imaging plane estimation unit 111 estimates the parameters of the plane equation as (α, β, y, δ) = (a, b, 0, d). In the present specification, estimating the parameter of the plane equation is also referred to as "estimating the imaging plane."

[0065] However, the actual coefficient value varies depending on the surface of the rectangular parallelepiped. An estimated distance from the host vehicle 1 to another vehicle 2, output from a sensor, is used as the actual coefficient value. For example, the detection result may include distance estimation information using a known measurement principle capable of estimating the distance from the positions of the road surface and the ground contact surface of another vehicle 2, and the installation position and angle of the monocular camera. In this case, the imaging plane estimation unit 111 may also estimate a plane equation of the side surface of the vehicle 1 or the like based on the distance estimation information included in the detection result.Therefore, the imaging plane estimation unit 111 may also output the estimated plane equation as an estimation result.

[0066] Further, for example, assume a case where the detection result is a pedestrian or an obstacle. In this case, the imaging plane estimation unit 111 cannot uniquely estimate the imaging plane only by outputting a rectangular bounding box as a detection result, as in areas 52 and 53 in Fig. 5A. Therefore, the imaging plane estimation unit 111 assumes that, for example, a plane perpendicular to the traveling direction of the host vehicle 1 is formed. Then, the imaging plane estimation unit 111 expresses the plane equation as X + e = 0. That is, the imaging plane estimation unit 111 can estimate the parameters of the plane equation as (α, β, γ, δ) = (1, 0, 0, e). In addition, when a rectangular parallelepiped-shaped bounding box is used as a detection result for the area 52 in which the guardrail is shown and the area 54 in which the sidewall is shown in Fig. 5A, the imaging plane estimation unit 111 may estimate a plane equation of an imaging plane corresponding to a side surface of the rectangular parallelepiped. Note that the imaging plane estimation unit 111 may estimate the parameters (α, β, γ, δ) of the plane equation for each object, assuming that the object faces perpendicularly to the arrival direction of the light beam to each camera or the optical axis of each camera.

[0067] Incidentally, there may be a case where no attribute is given to a three-dimensional object such as a pole or a tree, and there is a region where the attribute of the detection result is unknown. In this case, the imaging plane estimation unit 111 assumes a three-dimensional object with a predetermined height at the position of a pole or a tree and assumes that a plane perpendicular to the traveling direction of the vehicle 1 is formed. Then, the imaging plane estimation unit 111 can estimate the parameters of the plane equation assuming a rectangular bounding box in a region where the attribute of the detection result is unknown, as if the detection result were a pedestrian.

[0068] Note that, as another example of the method for estimating the imaging plane by the imaging plane estimation unit 111, it is also conceivable to use a quadratic equation or a multi-order equation instead of a simple linear equation with respect to the landmark detection result. In this case, the imaging plane estimation unit 111 divides and approximates the multi-order equation into piecewise linear equations according to the position of interest on the image and estimates the imaging plane. As described above, the imaging plane estimation unit 111 estimates the imaging plane of the landmark based on the landmark detection result.

[0069] Next, a method by which the geometric transformation estimation unit 112 estimates a geometric transformation parameter will be described.

[0070] The geometric transformation estimation unit 112 estimates the geometric transformation parameters of the two images 40a and 41a captured by the two cameras (the imaging units 40 and 41) based on the estimation result of the imaging plane estimated by the imaging plane estimation unit 111 and the camera parameters 103. An example of a method for estimating the geometric transformation parameter will be described.

[0071] First, the estimation result of the imaging plane estimation unit 111 processed for an image (the image 40a acquired by the imaging unit 40) is used. In a case where a specific pixel coordinate in the image acquired by the imaging unit 40 is in focus, the geometric transformation estimation unit 112 transforms a plane equation for the pixel coordinate into a plane transformation matrix. For example, the geometric transformation estimation unit 112 organizes the plane equation for Z and introduces a plane transformation matrix C as shown in Expression (4). Then, the four-dimensional world coordinates (homogeneous coordinates) X are one-dimensionally compressed and organized as X'. [Math. 4] X=(100100−αγ−βγ−δγ001)(XY1)=CX′

[0072] The geometric transformation estimation unit 112 can convert s0u0 and s1u1 expressed in Expression (1) into Expressions (5) and (6) using X'. The geometric transformation estimation unit 112 can derive a projective transformation matrix H (geometric transformation parameter) as in Expression (7) by organizing Expression (5) and Expression (6) simultaneously. Note that the camera parameters 103 input to the collation condition determination unit 104 are used in Expressions (5) to (7) because K and P are included in Expression (1). Expression (7) expresses the relative relationship between the two images 40a and 41a. [Math. 5] s0u0=K0P0X=K0P0CX′ s1u1=K1P1X=K1P1CX′ su0=K0P0C0(P1C1)−1K1−1u1=Hu1

[0073] Note that the geometric transformation estimation unit 112 may perform geometric transformation using affine transformation, shear transformation, or the like with a smaller number of transformation parameters than the projective transformation in some cases, depending on the parallelization processing described above, the camera installation position, and the plane equation. When the number of transformation parameters used for the geometric transformation by the geometric transformation estimation unit 112 is small, the calculation time of the geometric transformation estimation unit 112 is shortened. Therefore, the geometric transformation estimation unit 112 can switch the transformation type according to the number of transformation parameters.In the following processing, various types of transformation processing are collectively referred to as “geometric transformation” as an example of image transformation.

[0074] The feature collation unit 105 collates the images using the estimation result by the geometric transformation estimation unit 112 and the images captured by the imaging units 40 and 41. An example of processing in which the feature collation unit 105 collates two images will be described with reference to Fig. 6 described.

[0075] Fig. Fig. 6 is a diagram illustrating an example of processing of the feature collation unit 105. The images 40a and 41a are generated by the Fig. 3 are input into the feature collation unit 105.

[0076] The geometric transformation parameter is used as an estimation result of the geometric transformation estimation unit 112 for any point in the image 40a. The geometric transformation parameter is expressed as the projective transformation matrix H of the above-described expression (7). In the feature collation unit 105, the image 40a captured by the imaging unit 40 and the geometric transformation parameters estimated by the geometric transformation estimation unit 112 are input, and geometric transformation processing is performed on the location of the image 40a (S1).

[0077] Further, the image 41a captured by the imaging unit 41 is input to the feature collation unit 105 via the landmark detection unit 102. Then, the feature collation unit 105 performs image collation processing between the image 40a subjected to geometric transformation processing in step S1 and the other image 41a input via the landmark detection unit 102 (S2). However, the location to be geometrically transformed is a range around the position, including the position where the same part of the landmark is detected in the images 40a and 41a.

[0078] In image collation processing, template matching between the geometrically transformed portion of image 40a and the corresponding portion of image 41a, extraction processing of image feature points, and extraction processing of feature sets are performed. In the case of projective transformation, since the geometric transformation itself includes a motion component, the collation point is equivalent to being estimated to some extent around the position given by the geometric transformation. Therefore, the feature collation unit 105 can search around the position given by the geometric transformation. After step S2, the processing result is output to the position estimation unit 106.

[0079] Here is an example of a template comparison with reference to Fig. 7 described.

[0080] Fig. Fig. 7 is a diagram illustrating how the feature collation unit 105 performs template matching using the left camera image and the front camera image. Here, it is assumed that the left camera image is the Fig. 7 is image 40a and the front camera image is that in Fig. Figure 41a illustrated in Figure 7.

[0081] As in Fig. As illustrated in Figure 2, there are regions 21 and 22 where feature portions of intersection points (portions where white lines intersect) appear in part of the left camera image and the front camera image. This feature portion also exists in the front camera image as regions 11 and 12, respectively. However, since the shapes of regions 21 and 11 are different, and the shapes of regions 22 and 12 are also different, the shapes of the respective regions cannot be easily matched by the conventional method.

[0082] On the other hand, using the projective transformation (geometric transformation) according to the present embodiment, the landmarks in the areas 21 and 22 included in the left camera image are transformed into the shapes of the areas 71 and 72 located on the bottom of Fig. 7. At this time, the shapes included in regions 71 and 72 are close to the shapes included in regions 11 and 12, respectively. Therefore, the feature collation unit 105 can perform the collation processing more easily than searching the shape before the geometric transformation included in regions 21 and 22 from the front camera image. By geometrically transforming the shape of the region containing the feature portion in this way, an effect can be obtained that the feature collation unit 105 can easily perform the collation processing.

[0083] Note that in a case where the plane equation is constant in a certain image region, the feature collation unit 105 may perform template matching after projective transformation of the region. Alternatively, the feature collation unit 105 may perform projective transformation for each template.

[0084] Furthermore, the process of extracting a feature point in an image and the process of extracting a feature set (referred to as "feature point and feature set extraction") are similar. The feature collation unit 105 may perform a projective transformation on the image once and then extract a feature point and feature set. Alternatively, the feature collation unit 105 may integrate a projective transformation into the process of extracting the feature point and feature set to extract the feature point and feature set.

[0085] The position estimation unit 106 calculates the coordinate point of the landmark in the world coordinate system from the obtained corresponding position as a result of collation by the feature collation unit 105 using the above expression (2).

[0086] With the configuration described above, the external world recognition device 100 can improve the collation accuracy of the landmark appearing in the two images and the estimation accuracy of the distance from the vehicle 1 to the landmark while coping with environmental changes such as the vehicle 1, a pedestrian, and a road during the travel of the vehicle 1.

[0087] In the external world recognition device 100 according to the first embodiment described above, in order to estimate the distance from the host vehicle 1 to the landmark, the plane equation of the imaging plane corresponding to the position in the image 40a is estimated for each landmark according to the driving environment, which dynamically changes with the vehicle's travel. Then, the external world recognition device 100 performs a geometric transformation for each landmark to collate the landmark detected from the image 41a with the feature. Therefore, even in a case where an unspecified landmark appears in two images with different appearances, the external world recognition device 100 can estimate the distance to the landmark.Further, the position estimation unit 106 estimates the position of the landmark whose feature is collated, so that the error in the distance to the landmark can be reduced. [Second embodiment]

[0088] Next, a configuration example and a processing example of an external world recognition device 100A according to a second embodiment of the present invention will be described with reference to Fig. 8 described.

[0089] Fig. 8 is a block diagram illustrating an internal configuration example of an external world recognition device 100A according to the second embodiment.

[0090] The external world recognition device 100A includes landmark detection units 101 and 102, a camera parameter 103, a collation condition determination unit 104A, a feature collation unit 105A, and a position estimation unit 106. The external world recognition device 100A performs imaging plane estimation processing and geometric transformation parameter estimation on each of the two images input from the imaging units 40 and 41. Therefore, the external world recognition device 100A includes a plurality of imaging plane estimation units (imaging plane estimation units 111 and 113) and a plurality of image transformation estimation units (geometric transformation estimation units 112 and 114), each provided for a landmark detected from the first image (image 40a) and a landmark detected from the second image (image 41a). As shown in Fig. 8, the collation condition determination unit 104A included in the external world recognition device 100A includes the imaging plane estimation units 111 and 113 and the geometric transformation estimation units 112 and 114. The imaging plane estimation units 111 and 113 have the same function, and the geometric transformation estimation units 112 and 114 have the same function.

[0091] The landmark detection unit 101 detects a landmark from an image captured by the imaging unit 40. For this detection result, the imaging plane estimation unit 111 estimates the imaging plane, and the geometric transformation estimation unit 112 estimates the geometric transformation parameters.

[0092] Similarly, the landmark detection unit 102 detects a landmark from an image captured by the imaging unit 41. Based on the landmark detection result, the imaging plane estimation unit 113 estimates the imaging plane, and the geometric transformation estimation unit 114 estimates the geometric transformation parameter.

[0093] The feature collation unit 105A receives two results by the geometric transformation estimation units 112 and 114. Therefore, the function of the feature collation unit 105A differs from the function of the feature collation unit 105 according to the Fig. 3 illustrated first embodiment.

[0094] Several methods are adopted for implementing the function of the feature collation unit 105A. In one method, the feature collation unit 105A obtains a collation result between a geometric transformation result of the image 40a using the geometric transformation parameter estimated by one geometric transformation estimation unit 112 and the image 41a. Further, the feature collation unit 105A obtains a geometric transformation result of the image 41a using the geometric transformation parameter estimated by the other geometric transformation estimation unit 114 and a collation result with the image 40a. Then, the two collation results obtained by the feature collation unit 105A are compared.In a case where the two collation results reveal that the same landmark is collated at the same location, both of the collation results can be considered to have high reliability.

[0095] Therefore, the feature collation unit (feature collation unit 105A) acquires a collation result between a result of image transformation (geometric transformation) of the first image (image 40a) using the image transformation parameter (geometric transformation parameter) estimated by an image transformation estimation unit (geometric transformation estimation unit 112) and the second image (image 41a), acquires a collation result between a result of geometric transformation of the second image (image 41a) using the image transformation parameter (geometric transformation parameter) estimated by the other image transformation estimation unit (geometric transformation estimation unit 114) and the first image (image 40a), compares the respective collation results, and in one case,By collating the landmark detected from the first image (Figure 40a) and the landmark detected from the second image (Figure 41a) at the same three-dimensional position, the collation result is highly reliable. The reliability imparted to the collation result here can be represented by a value where, for each landmark detected from each image, a mismatch is set to 0% when no collation is performed at the same three-dimensional position, and a mismatch is set to 100% when collation is performed at the same three-dimensional position. Then, the feature collation unit 105 can output a collation result with high reliability to the position estimation unit 106 as long as the reliability is 80% or more.

[0096] Further, the feature collation unit 105A may collate the estimated imaging plane. Therefore, the feature collation unit 105A compares the imaging plane estimated by the imaging plane estimation unit 111 with the imaging plane estimated by the imaging plane estimation unit 113. At this time, in a case where the imaging plane estimated by the imaging plane estimation unit 113 is used as a reference and the imaging planes estimated by the imaging plane estimation unit 111 are different, the feature collation unit 105A determines that a collation error occurs because the reliability of the collation result is low. Then, the feature collation unit 105A can cope with the fact that no feature collation result is output to the position estimation unit 106 with respect to a landmark that is the target of the estimated imaging plane.As described above, the feature collation unit 105A outputs the collation result to the position estimation unit 106 while retaining only the collation result with high reliability, or determines the collation result with low reliability as noise and does not output the collation result. Therefore, the position estimation unit 106 can estimate the position and distance of the landmark only by restricting the collation result to the landmark with the high reliability.

[0097] In the external world recognition device 100A according to the second embodiment described above, the collation condition determination unit 104A includes the imaging plane estimation units 111 and 113 and the geometric transformation estimation units 112 and 114. Then, the feature collation unit 105A can obtain the reliability of the collation result by collating the estimation result with one of the estimation results output from the geometric transformation estimation units 112 and 114 as a reference. Therefore, it is easy to determine whether the landmark has been correctly collated based on the reliability of the collation result by the feature collation unit 105A, and the position estimation unit 106 can also accurately estimate the position of the landmark. [Third Embodiment]

[0098] Next, a configuration example and a processing example of an external world recognition device 100B according to a third embodiment of the present invention will be described with reference to Fig. 9 and Fig. 10 described.

[0099] In a case where landmarks having a plurality of attributes are mixed in the image area designated by the landmark detection unit 101, several methods can be considered so that a Fig. 9 accurately detects the landmark with each attribute. For example, upon receiving the detection result of the landmark detected by the landmark detection unit 101, the collation condition determination unit 104B may preferably measure the landmark that is the foreground of the landmark to be collated from the attribute of the landmark detected by the landmark detection unit 101 and the plane equation estimated by the imaging plane estimation unit 111.

[0100] Furthermore, in a case where a plurality of attributes for each landmark are detected by the landmark detection unit 101, the collation condition determination unit 104B may employ a method for selecting an attribute of a dominant landmark. The dominant landmark is, for example, a landmark that becomes foreground when a plurality of landmarks appear in an image area. In this case, the feature collation unit 105 collates the feature of the landmark according to the attribute of the dominant landmark that is foreground with respect to the dominant landmark and estimates the position of the landmark. Here, the collation condition determination unit 104B included in the external world recognition device 100B, which is capable of selecting a dominant landmark from a plurality of landmarks detected from the images 40a and 41a, will be described.

[0101] Fig. 9 is a block diagram illustrating an internal configuration example of the external world recognition device 100B according to the third embodiment. In the external world recognition device 100B according to the third embodiment, in a case where landmarks with a plurality of attributes are mixed in the image area designated by the landmark detection unit 101, a process of selecting a specific landmark as a template matching target is performed by the feature collation unit 105.

[0102] The external world recognition device 100B includes the landmark detection units 101 and 102, a camera parameter 103, a collation condition determination unit 104B, a feature collation unit 105, and a position estimation unit 106. The collation condition determination unit 104B included in the external world recognition device 100B includes a transformation selection unit 115 in addition to the imaging plane estimation unit 111 and the geometric transformation estimation unit 112.

[0103] The transformation selection unit (transformation selection unit 115) selects a landmark within a short distance from a plurality of landmarks detected from the first image (image 40a) and the second image (image 41a), and selects an image transformation parameter (geometric transformation parameter) estimated by the image transformation estimation unit (geometric transformation estimation unit 112) for the selected landmark. For example, the transformation selection unit 115 selects a geometric transformation parameter estimated for a landmark with a dominant attribute from the geometric transformation parameters for a plurality of landmarks estimated by the geometric transformation estimation unit 112. Then, the geometric transformation parameter selected by the transformation selection unit 115 is output to the feature collation unit 105.

[0104] The feature collation unit (feature collation unit 105) performs an image transformation of the landmark detected from the first image (image 40a) using the image transformation parameter (geometric transformation parameter) selected by the transformation selection unit (transformation selection unit 115). Then, the feature collation unit 105 collates the feature of the landmark after the geometric transformation with the feature of the landmark detected by the landmark detection unit 102 and outputs a collation result to the position estimation unit 106.

[0105] Here is a specific example of the processing carried out by the Fig. 9 illustrated collation condition determination unit 104B is performed, with reference to Fig. 10. Here, as an example, an example will be described in which the collation condition determination unit 104B selects an attribute of a landmark near, that is, in front of the own vehicle 1, using a plane equation.

[0106] Fig. 10 is a diagram illustrating an example of areas 56 and 57 where the vehicle 2 and the background are mixed. Assume that the vehicle 2 is traveling while facing the host vehicle 1.

[0107] In the area 56, a part of the left side of the vehicle 2 and the road surface are blended as the background. Further, in the area 57, a part of the lower side of the vehicle 2 and the road surface are blended as the background. Normally, in a case where the road surface is included on the top of the image, the three-dimensional object is located on the front side of the road surface. In a case where the frame to be subjected to template matching by the feature matching unit 105 is the area 56, the shape of the vehicle 2 against the background is preferably adopted as a matching target.

[0108] On the other hand, in a case where the image includes the three-dimensional object and the road surface, the road surface on the lower side of the image is closer than the three-dimensional object. In a case where the frame to be subjected to template matching by the feature collation unit 105 is the area 57, the shape of the road surface in front of the own vehicle 1 is preferably adopted as a matching target because the connection between the lower part of the vehicle 2 and the road surface is closer to the own vehicle.

[0109] As another example, from the perspective of vehicle control, a three-dimensional object (e.g., the vehicle 2) that has a great influence on driving in a case where the host vehicle 1 comes into contact can be prioritized and can be easily determined as a matching target from attribute information for each vehicle. In this case, the transformation selection unit (transformation selection unit 115) preferentially selects a landmark having an attribute that greatly influences the driving of the host vehicle. Consequently, since the distance to the landmark having an attribute that greatly influences the driving of the host vehicle is preferentially estimated by the position estimation unit 106, the host vehicle can perform control such as avoiding the landmark for which the distance has been estimated.

[0110] In the external world recognition device 100B according to the third embodiment described above, in a case where a landmark having a plurality of attributes is detected in an image region in the image 40a, it is possible to select a geometric transformation parameter for the one landmark, perform the geometric transformation of the landmark, and collate the landmark with the landmark in the corresponding image region of the image 41a. Therefore, in the feature collation unit 105, the landmarks to be collated in the images 40a and 41a are likely to coincide, and in the position estimation unit 106, the position estimation of the collated landmark can also be accurately performed. [Fourth Embodiment]

[0111] Next, a configuration example and a processing example of an external world recognition device 100C according to a fourth embodiment of the present invention will be described with reference to Fig. 11 and Fig. 12 described.

[0112] The road surface model (plane equation) estimated by the imaging plane estimation unit 111 is expected to deviate from the actual plane model because the landmark is farther away from the host vehicle 1. That is, the plane equation itself estimated by the imaging plane estimation unit 111 based on the detection result of the landmark detection unit 101 may contain an error. Therefore, it is assumed that the error included in the plane equation may affect the accuracy of feature collation by the feature collation unit 105B. Here, a configuration example of an external world recognition device 100C in which the error included in the plane equation does not affect the accuracy of feature collation by the feature collation unit 105B will be described.

[0113] Fig. 11 is a block diagram illustrating an internal configuration example of the external world recognition device 100C according to the fourth embodiment. In the external world recognition device 100C according to the fourth embodiment, in a case where the plane equation includes an error, a process of collating the landmark features is performed.

[0114] The external world detection device 100C has the same configuration as that shown in Fig. 3, but differs in that the collation condition determining unit 104 is replaced by a collation condition determining unit 104C and an error estimating unit 107 connected to a geometric transformation estimating unit 112A of the collation condition determining unit 104C is provided.

[0115] The error estimation unit (error estimation unit 107) estimates an error of the imaging plane estimated by the imaging plane estimation unit (imaging plane estimation unit 111). For example, the error estimation unit 107 estimates an error of the plane equation estimated by the imaging plane estimation unit 111 and outputs an estimation result to the geometric transformation estimation unit 112A. The error of the plane equation is, for example, a value determined in an experiment or a design stage conducted in advance. As an estimation result of the error by the error estimation unit 107, a method of increasing the output of the error as the landmark is farther away is conceivable. Therefore, the error estimation unit 107 estimates a large error in an object far from the host vehicle 1 and estimates a small error in an object near the host vehicle 1.

[0116] As one way to use the error output from the error estimation unit 107, for example, the geometric transformation estimation unit 112A is caused not to output the geometric transformation parameter to a plane equation in which the error estimation result by the error estimation unit 107 is equal to or greater than a certain value. As a result, the feature collation unit 105B does not need to use the geometric transformation parameter estimated by the geometric transformation estimation unit 112A. The reason why such processing is performed is that the geometric transformation of the imaging plane by the feature collation unit 105B is expected to adversely affect the feature collation processing.In a case where the feature collation unit 105B does not use the geometric transformation parameter, the geometric transformation of the imaging plane is not performed, and the landmarks detected from the images 40a and 41a are collated as they are.

[0117] The image transformation estimation unit (geometric transformation estimation unit 112A) determines the error range of the imaging plane to be subjected to image transformation based on the error of the imaging plane. Then, the geometric transformation estimation unit 112A uses the estimation result of the plane equation error input from the error estimation unit 107 to determine whether the geometric transformation parameter for the imaging plane estimated by the imaging plane estimation unit 111 can be output. Since the error range of the imaging plane is determined in this way, the geometric transformation parameter estimated for the imaging plane with a large error is not output.

[0118] In a case where the geometric transformation parameter is not output from the geometric transformation estimation unit 112A, the feature collation unit 105B collates the areas where the landmarks of the images 40a and 41a are detected without performing the geometric transformation, and outputs the collation result to the position estimation unit 106.

[0119] In the external world recognition device 100C according to the fourth embodiment described above, the geometric transformation estimation unit 112A determines whether to output the geometric transformation parameter using the estimation result of the plane equation error. In a case where the geometric transformation parameter is not output, the feature collation unit 105B collates the area where the landmark is detected without performing the geometric transformation. Therefore, the accuracy of collating the landmark feature is improved compared to a case where the area geometrically transformed from the imaging plane using the plane equation with a large error is collated. [Modification of the fourth embodiment]

[0120] Here, another method for using the error estimation result output by the error estimation unit 107 will be described.

[0121] Fig. 12 is a block diagram illustrating an internal configuration example of the geometric transformation estimation unit 112A and the feature collation unit 105B according to a modification of the external world recognition device 100C. Here, configuration examples of the geometric transformation estimation unit 112A and the feature collation unit 105B are mainly shown in the Fig. 11, the external world detection device 100C is described.

[0122] It is empirically assumed that the parameter of the plane equation estimated from the imaging plane by the imaging plane estimation unit 111 is shifted by an error (±εα, ±εβ, ±εγ, ±εδ) with respect to the original parameter (α, β, γ, δ). In this case, the geometric transformation estimation unit 112A duplicates the estimation result of the imaging plane by N samples for each imaging plane within the range of the error width. There are various methods for duplication, and Fig. 12 illustrates a configuration example and a processing example of the geometric transformation estimation unit 112A capable of executing one of the methods.

[0123] One in Fig. The image transformation estimation unit (geometric transformation estimation unit 112A) illustrated in FIG. 12 generates a plurality of imaging planes based on the error of the imaging plane estimated by the error estimation unit (error estimation unit 107) and estimates a plurality of image transformation parameters (geometric transformation parameters) for each of the plurality of imaging planes. The geometric transformation estimation unit 112A includes an imaging plane N generation unit 112A-1 and a geometric transformation N estimation unit 112A-2.

[0124] The imaging plane N generation unit 112A-1 generates a plurality of imaging planes of N patterns by, for example, randomly assigning parameters so that they fall within the above error range and adding the parameters to the parameters of the plane equation estimated by the imaging plane estimation unit 111.

[0125] The geometric transformation N estimation unit 112A-2 estimates a plurality of geometric transformation parameters for each of the generated N-pattern imaging planes. Therefore, the geometric transformation N estimation unit 112A-2 can estimate the geometric N-pattern transformation parameters. The geometric transformation N estimation unit 112A-2 estimated by the geometric transformation N estimation unit 112A-2 is output to the feature collation unit 105B.

[0126] The feature collation unit (feature collation unit 105B) performs collation processing between a feature of a landmark detected from the first image (image 40a) subjected to image transformation using a plurality of image transformation parameters (geometric transformation parameters) and a feature of a landmark detected from the second image (image 41a) a plurality of times, and outputs a result of the collation processing with a high score to the position estimation unit (position estimation unit 106). The feature collation unit 105B includes a feature-N collation unit 105B-1 and a feature collation result selection unit 105B-2.

[0127] The feature N collation unit 105B-1 receives the estimation result of the N-geometric pattern transformation parameters from the N-geometric transformation estimation unit 112A-2 of the geometric transformation estimation unit 112A. The feature N collation unit 105B-1 performs N different collation processing on the feature of the landmark based on the input estimation result of the N-geometric pattern transformation parameters. The results of the N collation processing are output to the feature collation result selection unit 105B-2.

[0128] The feature collation result selecting unit 105B-2 selects the result of the collation processing having the highest evaluation at the time of collation from among the results of the N collation processing input from the feature-N collation unit 105B-1, and outputs the selected result of the collation processing to the position estimating unit 106.

[0129] The position estimation unit 106 estimates the position of the landmark based on the result of the collation processing input from the feature collation unit 105B.

[0130] The external world recognition device 100C according to the modification of the fourth embodiment described above includes the geometric transformation estimation unit 112A and the feature collation unit 105B, so that the estimation result of the N-pattern geometric transformation parameters can be obtained from the estimation result for the N-pattern imaging planes. Then, after the collation processing is performed N times according to the estimation result of the N-pattern geometric transformation parameters, the result of the collation processing with the highest evaluation at the time of collation is selected. Therefore, even in a case where the parameter of the plane equation estimated by the imaging plane estimation unit 111 includes an error with respect to the original parameter, the position estimation unit 106 can accurately estimate the position of the landmark appearing in the images 40a and 41a. [Fifth Embodiment]

[0131] Next, a configuration example and a processing example of an external world recognition device 100D according to a fifth embodiment of the present invention will be described with reference to Fig. 13 described.

[0132] Fig. 13 is a block diagram illustrating an internal configuration example of the external world recognition device 100D.

[0133] The external world recognition device 100D includes a parallax calculation unit 108 and a parallax validity verification unit 109 in addition to the landmark detection units 101 and 102, the camera parameter 103, the collation condition determination unit 104, and the position estimation unit 106.

[0134] The parallax calculation unit (parallax calculation unit 108) calculates the parallax from a first image (image 40a) input from the imaging unit 40 and a second image (image 41a) input from the imaging unit 41, and collates the positions of the landmarks appearing in the first image (image 40a) and the second image (image 41a). That is, the parallax calculation unit 108 calculates the parallax between the images 40a and 41a without using the plane equation estimated by the imaging plane estimation unit 111 based on the landmark detection result by the landmark detection unit 101. As the method for calculating the parallax by the parallax calculation unit 108, a known method for measuring parallax can be used.The parallax calculation unit 108 calculates the parallax and outputs a result of collating the positions corresponding to the same landmark shown in Images 40a and 41a to the parallax validity verification unit 109 as a second collation result.

[0135] The parallax validity verification unit 109 is a modification of the feature collation unit 105 according to the first embodiment. The feature collation unit (parallax validity verification unit 109) compares a first collation result obtained by collating the feature of the landmark detected from the first image (image 40a) subjected to image transformation using the image transformation parameter (geometric transformation parameter) and the feature of the landmark detected from the second image (image 41a) with a second collation result obtained by collating based on parallax, and outputs the first collation result or the second collation result to the position estimation unit (position estimation unit 106) based on the validity of the comparison result.That is, the parallax validity verification unit 109 collates the landmark of the image region, which is geometrically transformed based on the geometric transformation parameter estimated by the geometric transformation estimation unit 112 using the plane equation estimated by the imaging plane estimation unit 111, with the landmark of the image region of the image 41a and obtains the first collation result. Then, the parallax validity verification unit 109 compares the first collation result with the second collation result and verifies the validity of the second collation result through the parallax calculation unit 108. For example, in template matching, a method of determining the validity of the collation result by the parallax validity verification unit 109 based on the degree of matching is adopted.

[0136] Note that in the range where the plane equation is estimated by the imaging plane estimation unit 111, the collation accuracy of the landmark geometrically transformed based on the geometric transformation parameter estimated by the geometric transformation estimation unit 112 is assumed to be high. However, in the method where the parallax calculation unit 108 obtains the corresponding position of the landmark from the parallax calculated from images 40a and 41a without the plane equation estimation by the imaging plane estimation unit 111, no conditional branching occurs due to the plane equation. Therefore, when the parallax calculation unit 108 is converted to hardware, there is an advantage in that the parallax calculation processing can be easily accelerated.

[0137] Therefore, the parallax validity verification unit 109 restricts the first collation result obtained by the plane equation and the geometric transformation parameter, and uses the restricted first collation result for the verification processing of whether the second collation result obtained by the parallax calculation unit 108 is correct. The reason why the processing is performed in this way is that the processing load of the parallax validity verification unit 109 increases when the parallax validity verification unit 109 obtains the first collation result obtained by the plane equation and the geometric transformation parameter for all landmarks commonly included in the images 40a and 41a.

[0138] The parallax validity verification unit 109 compares the restricted first collation result with the second collation result, and when the validity of the first collation result using the geometric transformation is high, accepts the first collation result and outputs the first collation result to the position estimation unit 106. For example, when a landmark is detected at a position close to the host vehicle 1, the parallax increases, and thus the validity of the second collation result obtained by collation using the parallax decreases. On the other hand, when the landmark is located at a position close to the host vehicle 1, the imaging plane is accurately estimated, so the validity of the first collation result is increased by the geometric transformation.Therefore, similarly to the external world recognition apparatus 100 according to the first embodiment described above, the position estimation unit 106 estimates the position of the landmark using the first collation result.

[0139] Note that the parallax validity verification unit 109 compares the first collation result with the second collation result, and when the validity of the second collation result is low, calculates the parallax of the images 40a and 41a and outputs the collated second collation result to the position estimation unit 106. For example, when the landmark is a landmark detected at a position far from the host vehicle 1, the parallax becomes small, and thus the validity of the second collation result obtained by collation using parallax becomes high. On the other hand, since the landmark is far from the host vehicle 1, the imaging plane is estimated inaccurately, so the validity of the first collation result is reduced by geometric transformation.Therefore, the position estimation unit 106 estimates the position of the landmark using the second collation result. Note that in a case where the validity of the second collation result is low, there is a landmark that has a large difference between the first collation result and the second collation result. Therefore, the parallax validity verification unit 109 may re-compare the first collation result with the second collation result for the image region where the landmark appears.

[0140] In the external world recognition device 100D according to the fifth embodiment described above, the validity of the collation result is verified by comparing the first collation result obtained by collating the image region of the image 40a, which is geometrically transformed based on the geometric transformation parameter output from the collation condition determination unit 104, and the image region of the image 41a, with the second collation result of the images 40a and 41a using the parallax calculated by the parallax calculation unit 108. Then, the parallax validity verification unit 109 determines the result to be output to the position estimation unit 106 as either the first collation result or the second collation result according to the validity of the collation result.In a case where the validity of the comparison between the first collation result and the second collation result is low, the position estimation unit 106 can estimate the position of the landmark using the second collation result. Therefore, either the first collation result or the second collation result is output to the position estimation unit 106 depending on whether the landmark is close to or far from the host vehicle 1. Therefore, the position estimation unit 106 can estimate the position of the landmark at various positions at high speed. [Modifications]

[0141] Note that in each of the above-described embodiments, the present invention has been described as an external world recognition device mounted on the vehicle 1. However, the external world recognition device can be used, for example, in a self-driving robot that includes a plurality of cameras or in an infrastructure monitoring system that monitors premises through videos captured by a plurality of cameras. In the infrastructure monitoring system, for example, a configuration is assumed in which a plurality of cameras are installed at one location.

[0142] Furthermore, in addition to the embodiments described above, it is also possible to configure an external world recognition device with a function of directly calculating a distance from a machine learning result. In this case, if the external world recognition device can calculate the distance to the landmark through machine learning, the distance can be calculated when the landmark is detected from the image, and the computation load of the external world recognition device can be reduced.

[0143] In addition to the embodiments described above, the external world recognition device may perform parallax adjustment, which is used in a conventional stereo camera. In this case, the external world recognition device may refer to the result estimated to have the best collation result from the results of collating the plurality of images, and may further estimate the positional relationship for each landmark using map information prepared in advance.

[0144] It should be noted that the present invention is not limited to the embodiments described above, and it is understood that various other applications and modifications can be made without departing from the gist of the present invention described in the claims.

[0145] For example, the embodiments described above describe the configurations of the device and the system in detail, and specifically to describe the present invention in an easy-to-understand manner, and are not necessarily limited to those having all of the described configurations. Furthermore, part of the configuration of the embodiment described here may be replaced with the configuration of another embodiment, and further, the configuration of another embodiment may be added to the configuration of a particular embodiment. Furthermore, some of the configurations of each embodiment may be omitted, replaced with other configurations, and added to other configurations.

[0146] Furthermore, only control lines and information lines considered necessary for explanation are illustrated, but not all control lines and information lines for a manufacturing process are illustrated. In practice, almost all configurations can be considered interconnected. List of reference symbols 1, 2 vehicle 40, 41 imaging unit 40a, 41a Image 100 Device for detecting an external world 101, 102 Landmark detection unit 103 Camera parameters 104 Collation condition determination unit 105 Feature collation unit 106 Position estimation unit 111 Imaging plane estimation unit 112 geometric transformation estimation unit 113 Imaging plane estimation unit 114 geometric transformation estimation unit QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2020-12735 A

[0008] JP 2019-16308 A

[0008]

Claims

[1] Device for detecting an external world, comprising: a first landmark detection unit that detects a landmark based on a first image acquired by at least one first imaging unit among a plurality of imaging units in which at least a part of an imaging field of view with respect to an external world overlaps; a second landmark detection unit that detects a landmark based on a second image acquired by a second imaging unit among the plurality of imaging units; an imaging plane estimation unit that estimates an imaging plane in the first image based on a detection result of the landmark by the first landmark detection unit; an image transformation estimation unit that estimates an image transformation parameter corresponding to an imaging plane in the second image based on information about the imaging plane estimated by the imaging plane estimation unit and imaging parameters of the plurality of imaging units; a feature collation unit that collates a feature of the landmark detected from the first image subjected to image transformation using an image transformation parameter by the image transformation estimation unit with a feature of the landmark detected from the second image; and a position estimation unit that estimates a three-dimensional position of the landmark based on a collation result of the feature collation unit. [2] The external world recognition apparatus according to claim 1, wherein the imaging plane estimation unit estimates the imaging plane based on a size of the landmark and a distance to the landmark. [3] The external world recognition apparatus according to claim 2, wherein the imaging plane estimation unit estimates the imaging plane in an image area in which the landmark is imaged. [4] An external world recognition device according to claim 3, wherein a plurality of imaging plane estimation units and a plurality of image transformation estimation units are provided for the landmark detected from the first image and the landmark detected from the second image, respectively, and the feature collation unit acquires a collation result between a result of image transformation of the first image using the image transformation parameter estimated by one of the image transformation estimation units and the second image, acquires a collation result between a result of image transformation of the second image using the image transformation parameter estimated by another of the image transformation estimation units and the first image, compares the respective collation results, and, in a case where the landmark detected from the first image and the landmark detected from the second image are collated at a same three-dimensional position, imparts high reliability to the collation result. [5] An external world recognition device according to claim 3, comprising: a transformation selection unit that selects the landmark at a short distance from a host vehicle from a plurality of landmarks detected from the first image and the second image and selects the image transformation parameter estimated by the image transformation estimation unit for the selected landmark, wherein the feature collation unit performs an image transformation of the landmark detected from the first image using the image transformation parameter selected by the transformation selection unit. [6] The external world recognition device according to claim 5, wherein the transformation selecting unit preferentially selects the landmark having an attribute that greatly influences the driving of the own vehicle. [7] An external world recognition device according to claim 3, comprising: an error estimation unit that estimates an error of the imaging plane estimated by the imaging plane estimation unit, wherein the image transformation estimation unit determines an error range of the imaging plane to be subjected to image transformation based on an error of the imaging plane. [8] An external world recognition device according to claim 7, wherein the image transformation estimation unit generates a plurality of imaging planes based on an error of the imaging plane estimated by the error estimation unit and estimates a plurality of image transformation parameters for each of the plurality of imaging planes, and the feature collation unit performs collation processing between a feature detected from the first image subjected to image transformation using the plurality of image transformation parameters and a feature detected from the second image a plurality of times, and outputs a result of the collation processing with high score to the position estimation unit. [9] An external world recognition device according to claim 3, comprising: a parallax calculation unit that calculates a parallax from the first image and the second image and collates a position of a landmark appearing in the first image and the second image, wherein the feature collation unit compares a first collation result obtained by collation between a feature of the landmark detected from the first image subjected to image transformation using the image transformation parameter and a feature of the landmark detected from the second image with a second collation result obtained by collation on a parallax basis, and outputs the first collation result or the second collation result to the position estimation unit on a validity basis of a comparison result. [10] A method for recognizing an external world, comprising: a process of detecting, by a first landmark detection unit, a landmark based on a first image acquired by at least a first imaging unit among a plurality of imaging units in which at least a part of an imaging field of view with respect to an external world overlaps; a process of detecting, by a second landmark detecting unit, a landmark based on a second image acquired by a second imaging unit among the plurality of imaging units; a process of estimating, by an imaging plane estimation unit, an imaging plane in the first image on the basis of a detection result of the landmark by the first landmark detection unit; a process of estimating, by an image transformation estimation unit, an image transformation parameter in accordance with an imaging plane in the second image based on information of the imaging plane estimated by the imaging plane estimation unit and imaging parameters of the plurality of imaging units; a process of collating a feature of the landmark detected from the first image subjected to image transformation using an image transformation parameter by the image transformation estimation unit with a feature of the landmark detected from the second image; and a process of estimating a three-dimensional position of the landmark based on a result of collation processing.

Images