Environment recognition device

Abstract

Provided is an environment recognition device that can determine with high accuracy the validity of parallaxes in various cases in which parallaxes are difficult to calculate. The environment recognition device comprises: an image acquisition unit 100 that acquires a first image and a second image; a feature amount calculation unit 101 that determines a feature amount from each of the first image and the second image; a cost computation unit 102 that determines a matching cost between the first image and the second image in a prescribed search range on the basis of the feature amount; a parallax estimation unit 103 that estimates parallaxes of a target pixel on the basis of the matching cost; a cost pattern storage unit 104 in which a cost pattern indicating a pattern of the matching cost is stored; a score estimation unit 105 that calculates a score expressing the uncertainty of the parallaxes from the matching cost and the cost pattern in the search range; and a parallax invalidation unit 106 that uses the score to determine a parallax to be invalidated among the parallaxes estimated by the parallax estimation unit 103.

Description

environment recognition device

[0001] The present invention relates to an environment recognition device.

[0002] Three-dimensional measurement of the driving environment is important for realizing preventive safety features and autonomous driving. Stereo cameras are effective for three-dimensional measurement, including moving objects such as other vehicles and pedestrians. Stereo cameras can determine depth from the camera by performing stereo matching. Stereo matching is a procedure for identifying areas in parallelized left and right images where the same area is captured. In stereo matching, the amount of pixel positional deviation between the left and right images is called disparity, and depth can be calculated using the disparity and internal camera parameters. While stereo matching can calculate disparity with high accuracy in areas with distinctive textures, it is known to be difficult to accurately calculate disparity in areas with repetitive patterns where similar texture patterns are repeated or road surfaces with uncharacteristic textures. Therefore, it is important to determine the validity of the estimated disparity. Patent Document 1 discloses a method for detecting a minimum value in the matching cost calculated by stereo matching, counting minimum values ​​similar to that minimum value, and determining the validity of the disparity.

[0003] JP 2017-54481 A

[0004] By using the method disclosed in Patent Document 1, it is possible to detect repetitive patterns and invalidate parallax. However, in real images, in addition to repetitive patterns, there are also cases where the texture is not distinctive, resulting in a low overall matching cost, or where it is difficult to calculate parallax due to the influence of disturbances (raindrops) or occlusions. In such a variety of cases, the method described in Patent Document 1 cannot accurately determine the effectiveness of parallax.

[0005] In view of the above problems, the present invention aims to provide an environment recognition device that can determine the validity of disparity with high accuracy in various cases where disparity calculation is difficult.

[0006] In order to achieve the above-mentioned object, the environment recognition device of the present invention includes an image acquisition unit that acquires a first image and a second image, a feature calculation unit that calculates feature amounts from each of the first image and the second image, a cost calculation unit that calculates a matching cost between the first image and the second image in a predetermined search range based on the feature amounts, a disparity estimation unit that estimates a disparity of a target pixel based on the matching cost, a cost pattern storage unit that stores a cost pattern indicating a pattern of the matching cost, a reliability estimation unit that calculates a reliability of the disparity in the search range based on the matching cost and the cost pattern, and a disparity invalidation unit that determines a disparity to be invalidated from among the disparities estimated by the disparity estimation unit based on the reliability.

[0007] According to the present invention, it is possible to determine the validity of parallax with high accuracy in various cases where parallax calculation is difficult.

[0008] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0009] 1 is a functional block diagram showing the configuration of an environment recognition device according to a first embodiment of the present invention. FIG. 2 is a processing flow diagram in the environment recognition device according to the first embodiment. FIG. 3 is a diagram showing a method for calculating a matching cost. FIG. 4 is a diagram showing the calculated matching cost. FIG. 5 is a diagram showing data stored in a cost pattern storage unit. FIG. 6 is a diagram explaining a method for determining a matching cost (cost pattern) to be stored in a cost pattern storage unit. FIG. 7 is a diagram showing the relationship between the matching cost and the calculated score. FIG. 8 is a functional block diagram showing the configuration of an environment recognition device according to a second embodiment of the present invention. FIG. 9 is a diagram showing the relationship between a convolutional neural network and the functional blocks of the embodiments. FIG. 10 is a diagram explaining a procedure for calculating a score related to the uncertainty of disparity. FIG. 11 is a diagram explaining a learning method for a convolutional neural network. FIG. 12 is a diagram explaining a method for calculating a score using a plurality of cost patterns. FIG. 13 is a functional block diagram showing the configuration of an environment recognition device according to a third embodiment of the present invention. FIG. 14 is a diagram explaining disparity correction processing. FIG. 15 is a functional block diagram showing the configuration of an environment recognition device according to a fourth embodiment of the present invention.

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0011] [First Embodiment] FIG. 1 is a functional block diagram showing the configuration of an environment recognition device 1 according to the first embodiment. In the configuration of the first embodiment, two cameras are connected to the environment recognition device 1. In the following description, the environment recognition device 1 is connected to a stereo camera (a configuration in which two cameras are arranged horizontally, one on the left and one on the right) and is mounted on a vehicle. Note that the cameras connected to the environment recognition device 1 are not limited to a stereo camera, but may be a multi-camera consisting of three or more cameras installed on a vehicle. When a multi-camera is connected to the environment recognition device 1, the processing described in the embodiments of the present invention can be applied to images captured by any two camera pairs having overlapping imaging areas in which at least a portion of the fields of view overlap.

[0012] The environment recognition device 1 of this embodiment is composed of a camera, a computer, a memory, a storage device, etc., and the computer operates as various functional units by executing control programs stored in the memory, etc.

[0013] As shown in FIG. 1, the environment recognition device 1 has, as functional units realized by the operation of a camera and a computer, an image acquisition unit 100, a feature calculation unit 101, a cost calculation unit 102, a disparity estimation unit 103, a cost pattern storage unit 104, a score estimation unit 105, and a disparity nullification unit 106.

[0014] The image acquisition unit 100 acquires two images taken at the same time by a left camera and a right camera installed on the left and right sides.

[0015] The feature calculation unit 101 calculates feature amounts for each of the left camera image (hereinafter also referred to as the left image) and the right camera image (hereinafter also referred to as the right image) acquired by the image acquisition unit 100. Here, the feature amounts can be well-known SURF features or BRIEF features. In addition to these, the pixel values ​​of the target image can be used as feature amounts directly, or the pixel values ​​of an 8 x 8 range surrounding the target pixel can be used as feature amounts.

[0016] The cost calculation unit 102 calculates a matching cost for each pixel using the feature calculated by the feature calculation unit 101. The matching cost can be calculated by using the right image as a reference and scanning a predetermined search range of the left image. The matching cost can be calculated as the absolute value of the difference in feature between the target pixel in the right image and the scanned pixel in the left image.

[0017] The disparity estimation unit 103 estimates the disparity at each pixel from the matching cost calculated by the cost calculation unit 102. The disparity that achieves the minimum matching cost is set as the estimated value.

[0018] The cost pattern storage unit 104 stores cost patterns used by the score estimation unit 105. Like the matching costs calculated by the cost calculation unit 102, the cost pattern is information having a size equivalent to a predetermined search range and expresses the matching cost for each search position. This cost pattern is calculated using a data set prepared in advance. Specifically, this cost pattern is calculated by calculating the disparity for the data set using the disparity estimation unit 103 and comparing the calculated disparity with a correct value (the true value of the disparity). If the comparison shows that the two values ​​deviate by a certain value or more, the matching cost calculated by the cost calculation unit 102 is stored as a cost pattern. That is, the matching cost of an area where the deviation between the disparity calculated by the disparity estimation unit 103 and the true value data of disparity prepared in advance is a certain value or more is stored as the cost pattern.

[0019] The score estimation unit 105 calculates a score related to the uncertainty of the disparity estimated by the disparity estimation unit 103 using the matching cost calculated by the cost calculation unit 102 and the cost pattern stored in the cost pattern storage unit 104. The score calculation utilizes the matching cost calculated by the cost calculation unit 102 and the cost pattern stored in the cost pattern storage unit 104. As will be described later, the score can be calculated by numerically expressing the matching cost and the cost pattern as vectors and calculating the dot product of these vectors. In addition to the dot product, the similarity between two vectors can also be measured using metrics such as cosine similarity and the difference between vectors. The higher the similarity between the two vectors (i.e., the matching cost and the cost pattern), the higher the score calculated by the score estimation unit 105. The lower the similarity between the two vectors (i.e., the matching cost and the cost pattern), the lower the score calculated by the score estimation unit 105. That is, the larger the score calculated by the score estimation unit 105, the more similar the two vectors are and the lower the reliability of the disparity estimated by the disparity estimation unit 103. Conversely, the smaller the score calculated by the score estimation unit 105, the more dissimilar the two vectors are and the higher the reliability of the disparity estimated by the disparity estimation unit 103. In this way, the score estimation unit 105 can use the similarity between the matching cost calculated by the cost calculation unit 102 and the cost pattern stored in the cost pattern storage unit 104 to find the reliability of the disparity estimated by the disparity estimation unit 103 by calculating a score related to the uncertainty of the disparity.

[0020] The parallax invalidation unit 106 compares the score value output by the score estimation unit 105 with a predetermined threshold, and if the score is equal to or greater than the threshold, determines that the parallax estimated by the parallax estimation unit 103 is invalid.

[0021] An example of the operation of the environment recognition device 1 of this embodiment will be described in detail below with reference to the flowchart in Fig. 2. In the following example of the operation, a method of calculating the parallax for the right image of the stereo camera will be mentioned.

[0022] The environment recognition device 1 sequentially executes an image acquisition process P101, a feature amount calculation process P102, a cost calculation process P103, a parallax estimation process P104, a score estimation process P105, and a parallax nullification process P106.

[0023] In the image acquisition process P101, right and left images of the stereo camera are acquired at the same time.

[0024] In feature calculation processing P102, feature values ​​are calculated from the right and left images captured by the stereo camera. The feature values ​​are obtained by arranging pixel values ​​contained in an 8x8 area around the target pixel.

[0025] In the cost calculation process P103, the feature values ​​calculated in the feature value calculation process P102 are used to calculate a matching cost for each pixel in the right image. The matching cost calculation method is shown in FIG. 3. FIG. 3 shows a method for calculating the matching cost by scanning the left image 301 with respect to a target pixel 302 (position (x, y)) in the right image 300. To calculate the matching cost, D pixels are scanned in the right direction from position (x, y) in the left image 301 (scanning from pixel 303 to pixel 304). Specifically, the absolute value of the difference in feature values ​​is calculated at each position in the left image. By repeating this calculation at each position, the final matching cost 401 shown in FIG. 4 can be calculated.

[0026] In the disparity estimation process P104, the disparity of the target pixel is determined from the matching cost calculated in the cost calculation process P103. When the matching cost 401 shown in FIG. 4 has been calculated, d' that achieves the minimum value of the matching cost 401 is determined as the disparity of the target pixel.

[0027] In the score estimation process P105, the matching cost calculated by the cost calculation unit 102 and the cost pattern stored in the cost pattern storage unit 104 are used to calculate a score that represents the uncertainty of the parallax.

[0028] FIG. 5 shows the cost patterns stored in the cost pattern storage unit 104. The cost pattern storage unit 104 stores multiple cost patterns, such as cost pattern 501 and cost pattern 502. These cost patterns are stored as vector data. Specifically, cost pattern 501 is expressed as a vector of size 501v (search range + 1), and each element stores the matching cost at that position. FIG. 6 shows a method for acquiring these cost patterns. A pre-prepared dataset is used to acquire the cost patterns. An image 600 is an image in the dataset, and feature calculation processing P102, cost calculation processing P103, and disparity estimation processing P104 are performed on this image. Then, a matching cost 602 is calculated for a target pixel 601, and a disparity d' is estimated. This estimated disparity d' is compared with the true disparity value d~ (603), and if the absolute value of the difference is equal to or greater than a predetermined value (e.g., 3 pixels), the matching cost 602 is stored in the cost pattern storage unit 104. That is, in a data set prepared in advance, the matching cost of an area where the deviation between the parallax calculated by the parallax estimation process P104 and the true value data of the parallax prepared in advance is a predetermined value or more, and the matching cost of an area where the parallax estimation process P104 was unable to correctly estimate the parallax is stored as a cost pattern.

[0029] In the score estimation process P105, the inner product (score) of each of the previously acquired cost patterns and the matching cost (401 in FIG. 4) calculated by the cost calculation unit 102 is calculated. Here, the matching cost calculated by the cost calculation unit 102 is also expressed as a vector, similar to the cost pattern. Furthermore, as shown in FIG. 5 , when multiple cost patterns are stored in the cost pattern storage unit 104, the inner product (score) is calculated for each cost pattern (501v and 502v in FIG. 5 ), and the maximum value is used as the final inner product value (score). By performing this calculation in this manner, as shown in the upper part of FIG. 7 , if the matching cost (701) calculated by the cost calculation unit 102 has an outer shape that allows accurate calculation of disparity, it is not similar to any of the cost patterns stored in the cost pattern storage unit 104, and therefore the score (701s) is small. On the other hand, as shown in the lower part of FIG. 7 , in the case of a matching cost (702) for which accurate calculation of disparity is difficult, it is similar to any of the cost patterns stored in the cost pattern storage unit 104, and therefore the score (702s) is large. The matching cost (702) for which it is difficult to calculate the disparity correctly is not limited to the cost pattern of the outline shown in the lower part of Figure 7, but also includes a multi-peak matching cost due to a repeating pattern, a matching cost with a low overall value due to few undulations in the road surface area, etc.

[0030] In the parallax invalidation process P106, pixels to be invalidated are determined based on the score calculated in the score estimation process P105. Specifically, a predetermined threshold (e.g., 30) is used, and if the score is equal to or greater than the threshold, the parallax is determined to be invalid.

[0031] As described above, the environment recognition device 1 of this embodiment calculates the uncertainty of the parallax (conversely, the reliability of the parallax) using the matching cost and multiple pre-stored cost patterns to determine the validity of the parallax. This allows the validity of the matching cost to be verified against multiple conditions, rather than being determined based on a single condition, thereby enabling the validity of the parallax to be determined with high accuracy even in a wider variety of scenes. Furthermore, by comparing the cost pattern using the outline information of the entire matching cost, the validity of the parallax can be determined based not only on the peak information of the matching cost but also on global cost information. This allows the validity of the parallax to be determined with higher accuracy even in cases where the validity cannot be determined based on local determination alone, such as a repetitive pattern, because it can be determined that the pattern is a repetitive pattern from the outline of the entire matching cost.

[0032] Furthermore, the environment recognition device 1 of this embodiment stores matching costs for which it is difficult to actually calculate disparity as cost patterns. By using a large amount of actual data, it is possible to acquire various cost patterns for which it is difficult to calculate disparity. This makes it possible to determine the effectiveness of disparity with high accuracy in a wider variety of scenes. Furthermore, the above-described cost pattern acquisition method makes it possible to acquire a large number of cost patterns simply by collecting actual data. This eliminates the need to manually design an algorithm for calculating disparity uncertainty, thereby reducing the man-hours required for design.

[0033] Furthermore, in the environment recognition device 1 of this embodiment, the matching cost and the cost pattern are each numerically represented as vectors, and the dot product of these two vectors is calculated to calculate the score. Because calculating the dot product of vectors can be performed very efficiently, the processing time required to determine whether disparity is invalid can be reduced. Furthermore, instead of the simple dot product calculation described above, cosine similarity can also be used. That is, the matching cost and the cost pattern are each normalized to a unit vector, and the dot product of the unit vectorized matching cost and cost pattern is calculated. This allows the similarity between the two vectors to be determined in a normalized state, thereby enabling a more accurate determination of the validity of disparity.

[0034] Furthermore, in the parallax nullification process P106 of the environment recognition device 1 described above, a fixed threshold is always applied to the score representing the uncertainty of the parallax to determine the validity of the parallax. However, the threshold can also be adaptively set according to the value of the parallax d' calculated in the parallax estimation process P104. Specifically, the threshold used in the parallax nullification process P106 is set as C*d', where C is a predetermined constant (e.g., 15). By setting the threshold according to the estimated value of the parallax d' in this way, it is possible to set a low threshold for distant locations (small d') and a high threshold for nearby locations (large d'). In this way, the parallax nullification process P106 uses the parallax estimated in the parallax estimation process P104 and nullifies parallax more aggressively the smaller the parallax value (the farther away). This allows for a stricter determination of the validity of the parallax at distant locations where the influence of parallax error is greater, and only highly accurate parallax can be determined to be valid.

[0035] 8 is a functional block diagram showing the configuration of an environment recognition apparatus 2 according to the second embodiment. In the configuration of the second embodiment, the feature amount calculation unit 101, the cost calculation unit 102, the disparity estimation unit 103, the cost pattern storage unit 104, and the score estimation unit 105 are executed as the respective functions of a convolutional neural network N800. The following describes the feature amount calculation unit 101, the cost calculation unit 102, the disparity estimation unit 103, the cost pattern storage unit 104, and the score estimation unit 105, which have different configurations from those of the first embodiment.

[0036] The feature calculation unit 101 extracts features from each image by performing two-dimensional convolution on the left and right images acquired by the image acquisition unit 100. The kernel used in the two-dimensional convolution is determined by learning, which will be described later.

[0037] The cost calculation unit 102 calculates the matching cost using the feature amounts calculated by the feature amount calculation unit 101. The matching cost is calculated by repeatedly shifting the feature amounts of the left image relative to the feature amounts of the right image and combining them in the channel direction, as in the well-known GC-Net. Then, the matching cost is calculated by performing a three-dimensional convolution process. The kernel used in the three-dimensional convolution is determined by learning, which will be described later.

[0038] The disparity estimation unit 103 estimates the disparity for each target pixel using the matching cost calculated by the cost calculation unit 102. The disparity is estimated by applying a known soft argmin process to the matching cost.

[0039] The cost pattern storage unit 104 stores cost patterns used by the score estimation unit 105, expressed as convolution kernels. These kernels are determined by learning, which will be described later.

[0040] The score estimation unit 105 calculates the disparity score for the target pixel by performing a convolution operation using the convolution kernel stored in the cost pattern storage unit 104 on the matching cost output by the cost calculation unit 102. The calculated score represents the uncertainty of the disparity.

[0041] The processing order of the environment recognition device 2 shown in Fig. 8 is the same as that of the first embodiment, and therefore, hereinafter, an example of the operation of the environment recognition device 2 of this embodiment will be described in detail with reference to the flowchart in Fig. 2 and the configuration of the convolutional neural network N800 in Fig. 9. Furthermore, of the specific processing contents of the second embodiment, the processes that differ from those of the first embodiment are the feature amount calculation process P102, the cost calculation process P103, the disparity estimation process P104, and the score estimation process P105, and therefore the following description will focus on these processes.

[0042] As shown in FIG. 9, by executing a convolutional neural network N800, a feature amount calculation process P102, a cost calculation process P103, a disparity estimation process P104, and a score estimation process P105 are executed.

[0043] In the feature calculation process P102, two-dimensional convolution is performed on each of the right camera image 300 and the left camera image 301. As a result, feature amounts (maps) 300f and 301f are calculated from each of the right camera image 300 and the left camera image 301. The feature calculation process P102 is composed of a three-stage convolutional neural network ((H, W, C), (H / 2, W / 2, 2C), and (H / 4, W / 4, 4C)), with each stage consisting of two convolutional layers. Here, H, W, and C represent height, width, and channel (C=16), respectively. Each convolutional layer (except for the layers corresponding to 300f and 301f) consists of convolution, batch normalization, and a rectified linear unit. In the layers corresponding to 300f and 301f, only convolution is performed.

[0044] In the cost calculation process P103, first, 300v is generated from the features 300f and 301f calculated in the feature calculation process P102. Similar to the known GC-Net, 300v can be generated by repeating the process of combining 300f and 301f shifted to the left in the channel direction a number of times equal to the disparity search range. Then, 3D convolution is repeatedly performed on the generated 300v to generate a matching cost 300c. The cost calculation process P103 is a four-stage encoder-decoder type convolutional neural network ((D / 4, H / 4, W / 4, 4C), (D / 8, H / 8, W / 8, 8C), (D / 16, H / 16, W / 16, 16C), (D / 32, H / 32, W / 32, 32C)). Here, D indicates the disparity search range (D = 192). Each stage consists of two convolutional layers. Each convolutional layer, except for the layer corresponding to 300c, is composed of batch normalization and a rectified linear unit. 300c performs only convolution. Furthermore, skip connections (addition of feature maps) are performed between each encoder and decoder.

[0045] In the disparity estimation process P104, a known soft argmin process is performed on the matching cost 300c to generate a disparity image 300d. In the soft argmin process, the sign of the matching cost of the target pixel is inverted and a softmax function is applied. Then, the output result of the softmax function is applied to each disparity index, and the disparity is calculated by obtaining the sum.

[0046] In the score estimation process P105, a convolution is performed on the matching cost 300c to output a score 300s that represents the uncertainty of the disparity. The score estimation process P105 will be described with reference to FIG. 10 . A convolution operation is performed on the matching cost 300c using a kernel 1000 to generate the score 300s. Here, the kernel 1000 is a vector representation of the matching cost as described in the first embodiment, and indicates the cost pattern stored in the cost pattern storage unit 104. Here, the convolution operation is equivalent to calculating the correlation between the vectors of the input and the kernel, so the score estimation process P105 calculates the correlation between the matching cost and the cost pattern, i.e., the similarity.

[0047] The above-described feature calculation process P102, cost calculation process P103, and score estimation process P105 utilize convolution operations. Below, we will describe a method for learning the kernel used in the convolution operations. FIG. 11 illustrates this learning method. In the environment recognition device 2 described in embodiment 2, two values, a disparity (image) 300d and a score 300s (disparity uncertainty), must be output from the convolutional neural network N800. Therefore, loss functions are defined for the disparity and the score, respectively, and the kernel is learned by minimizing the loss functions. In FIG. 11, 300d is the disparity image estimated in the disparity estimation process P104, and 300s is the output of the score estimation process P105. FIG. 11 also illustrates a method for defining losses for spatially identical regions (i.e., regions corresponding to the same position on the image) 300dt and 300st. First, we will describe the definition of the loss for region 300dt. d~)。 In other words, the kernel value is learned so that the output of 300dt, d', approaches the true value d~. Next, the definition of the loss related to the score will be described. If the output of region 300st is score, the output of 300dt is d', and the true value of the disparity of 300dt is d~, cost2(abs(score-abs(d'-d~)))) is minimized as the loss function. In other words, the kernel is learned so that the output of 300st (score) approaches the difference between the estimated disparity d' and the true value d~. In this way, if there is a large discrepancy between the disparity d' and the true value d~, the output of 300st (score) will also take a large value, making it possible to express the uncertainty of the estimated disparity. Furthermore, in the loss of cost2, abs(d'-d~) is treated as a constant. That is, the gradient of cost2 due to error backpropagation is prevented from propagating to the parallax image 300d. In learning, the kernel is estimated by simultaneously minimizing cost1 and cost2. This minimization makes it possible to determine, through learning, not only the kernels used in the feature calculation process P102 and the cost calculation process P103, but also the kernel used to output the score 300s, i.e., the cost pattern 1000 (FIG. 10) stored in the cost pattern storage unit 104.

[0048] As described above, the environment recognition device 2 of this embodiment calculates disparity and disparity uncertainty using a convolutional neural network. The kernel of the convolutional neural network is determined by defining loss functions for scores that represent disparity and disparity uncertainty and minimizing these loss functions. In other words, the kernel is estimated through learning. Here, because the cost pattern used in calculating disparity uncertainty is expressed as a kernel, the cost pattern can be automatically determined simply by collecting learning data, eliminating the need to manually design the cost pattern. Furthermore, using a large amount of data as learning data makes it possible to determine cost patterns that are useful for determining the effectiveness of disparity in a wider variety of scenes, leading to improved estimation accuracy.

[0049] In the environment recognition device 2 of the second embodiment described above, as shown in FIG. 10 , only one cost pattern (1000) is stored in the cost pattern storage unit 104. On the other hand, as shown in FIG. 12 , multiple cost patterns can be used to estimate the score 300s, which represents the uncertainty of the disparity. FIG. 12 illustrates a case in which four cost patterns, 1000, 1001, 1002, and 1003, are stored in the cost pattern storage unit 104. To estimate the score 300s, a convolution operation is performed using each cost pattern as a kernel. The resulting outputs are 1000f, 1001f, 1002f, and 1003f, respectively. These four outputs are combined in the channel direction, and then two-dimensional convolution is performed (conv in FIG. 12 ) to output the score 300s. The kernel (including the cost pattern) shown in FIG. 12 can be estimated by minimizing cost2 shown in FIG. 11 .

[0050] In this way, by representing multiple cost patterns as kernels of a convolutional neural network and estimating them through training, we can evaluate the uncertainty of disparity using matching costs under various conditions. In other words, we can improve the accuracy of determining the validity of disparity in a variety of scenes.

[0051] 13 is a functional block diagram showing the configuration of an environment recognition device 3 according to a third embodiment. The configuration of the third embodiment is configured by adding a disparity correction unit 107 to the configuration of the second embodiment. The disparity correction unit 107 is executed by a convolutional neural network N801. The disparity correction unit 107 will be described below.

[0052] The parallax correction unit 107 corrects the parallax estimated by the parallax estimation unit 103 using the right camera image acquired by the image acquisition unit 100, the parallax estimated by the parallax estimation unit 103, the score estimated by the score estimation unit 105, and the invalid determination result output by the parallax nullifying unit 106, and outputs a more accurate parallax. FIG. 14 shows the processing executed by the parallax correction unit 107. The right camera image 300 acquired by the image acquisition unit 100, the parallax 300d estimated by the parallax estimation unit 103, the score 300s estimated by the score estimation unit 105, and the invalid determination result 300i output by the parallax nullifying unit 106 are input to the parallax correction unit 107. As shown in the lower part of FIG. 14, the invalid determination result 300i is assumed to have a value of 1 for parts determined to be invalid and a value of 0 for parts determined to be valid. Furthermore, these input data are each upsampled (nearest neighbor interpolated) so that they have the same spatial resolution as the width and height of the right camera image 300, and these four inputs are combined in the channel direction to produce 300''. The disparity correction unit 107 outputs corrected disparity 300d'' by running a convolutional neural network N801 on 300''. The convolutional neural network N801 is a two-stage encoder-decoder ((H, W, C), (H / 2, W / 2, 2C)), and like N800, each stage consists of two convolutional layers. Furthermore, each convolutional layer, except for the final layer, consists of convolution, batch normalization, and a rectified linear unit. The final layer only performs convolution.

[0053] In training the convolutional neural network N801, the loss function cost1 related to disparity is minimized, as explained in FIG. 11 . That is, the convolutional neural network N801 is trained by minimizing abs(300d''-300d~) using the corrected disparity 300d'' and the true value 300d~. In the environment recognition device 3 of the third embodiment shown in FIG. 13, there are two convolutional neural networks, N800 and N801, and in training, N800 is trained first, followed by N801.

[0054] In the environment recognition device 3 of the above-described third embodiment, the disparity estimated by the disparity estimation unit 103 is corrected by the disparity correction unit 107, and a highly accurate disparity is finally output. The disparity correction unit 107 uses the image acquired by the image acquisition unit 100 and the disparity estimated by the disparity estimation unit 103, as well as the score estimated by the score estimation unit 105 and the output of the disparity nullification unit 106. The outputs of the score estimation unit 105 and the disparity nullification unit 106 include information on areas with high disparity uncertainty and areas that should be nullified. By transmitting this information to the disparity correction unit 107, information on areas that would normally be nullified or have low disparity accuracy can be transmitted to the convolutional neural network (N801), making it possible to actively correct the disparity in such areas. This allows for even higher accuracy of the disparity.

[0055] 15 is a functional block diagram showing the configuration of an environment recognition device 4 according to a fourth embodiment. In the configuration of the fourth embodiment, a communication unit 108 is added to the configuration of the first embodiment. The communication unit 108 and the score estimation unit 105 will be described below.

[0056] The communication unit 108 cooperates with an external server (not shown), receives cost pattern information transmitted from the external server, and adds the received cost pattern information to the cost pattern storage unit 104. The external server transmits cost patterns determined to be effective in score estimation to the communication unit 108. The communication unit 108 may be connected to the external server wirelessly or via a wired connection.

[0057] The score estimation unit 105 uses the newly added cost pattern in addition to the already stored cost patterns to calculate the dot product with the matching cost output by the cost calculation unit 102, thereby calculating the uncertainty of the disparity.

[0058] In the environment recognition device 4 of the above-described fourth embodiment, a cost pattern transmitted from an external server is newly added to the cost pattern storage unit 104, and the validity of parallax is determined. The external server uses the latest data set to calculate a valid cost pattern and transmits it to the environment recognition device 4. This configuration makes it possible to add cost patterns to be used for determining the validity of parallax, and to accurately determine the validity of parallax in a wider variety of scenes.

[0059] The environment recognition device 4 of the fourth embodiment described above is configured to add a cost pattern. However, this is not limited to adding a cost pattern; it can also replace a cost pattern. That is, a cost pattern received from an external server can be replaced with a cost pattern already stored in the cost pattern storage unit 104. When replacing, a cost pattern that is used less frequently is deleted and replaced with a cost pattern received from the external server. Specifically, the score estimation unit 105 calculates the inner product of each cost pattern and the matching cost output by the cost calculation unit 102. The largest inner product value (i.e., the highest correlation value) is selected as the final score, and the number of uses of the cost pattern that achieved this largest inner product value is incremented by 1. This process is performed for each frame, and the least frequently used cost pattern after a predetermined number of frames (e.g., 100 frames) is selected as the target for deletion. In this way, deleting an infrequently used cost pattern instead of adding a cost pattern not only reduces memory consumption in the cost pattern storage unit 104 but also suppresses an increase in the execution time of the score estimation unit 105.

[0060] Furthermore, while the above-described fourth embodiment has been described in terms of adding the communication unit 108 to the first embodiment, the communication unit 108 can also be added to the configuration described in the second embodiment. That is, the communication unit 108 receives a cost pattern, in other words, a convolution kernel to be used by the score estimation unit 105. The processing performed by the score estimation unit 105 in this case will be described with reference to FIG. 10 . When the communication unit 108 receives a convolution kernel, the cost pattern storage unit 104 stores not only the illustrated kernel 1000 but also an additional kernel 1001 (not shown in FIG. 10 ). By performing a convolution operation using these two kernels 1000 and 1001, two scores 300s_1000 and 300s_1001 (not shown in FIG. 10 ) are generated. The maximum value for each element of these two scores is then calculated, and the final score is output. As described above, receiving a new cost pattern from an external server allows the effectiveness of disparity to be accurately determined even for scenes that were initially incompatible. Furthermore, by changing only the cost pattern (i.e., the kernel used by the score estimation unit 105) in the cost pattern storage unit 104 of the convolutional neural network N800 shown in FIG. 8, it is possible to expand the scenes in which the validity of disparity can be correctly determined without affecting the output result of the disparity estimation unit 103.

[0061] [Summary] As described above, the environment recognition device of this embodiment includes: an image acquisition unit 100 that acquires a first image and a second image; a feature calculation unit 101 that calculates feature amounts from each of the first image and the second image; a cost calculation unit 102 that calculates a matching cost (dissimilarity or dissimilarity) between the first image and the second image in a predetermined search range based on the feature amounts; a disparity estimation unit 103 that estimates a disparity of a target pixel based on the matching cost; a cost pattern storage unit 104 that stores a cost pattern indicating a pattern of the matching cost; a reliability estimation unit that calculates a reliability of the disparity in the search range based on the matching cost and (the similarity of) the cost pattern (a score estimation unit 105 that calculates a score that expresses the uncertainty of the disparity in the search range from the matching cost and the cost pattern); and a disparity nullification unit 106 that determines a disparity to be nullified from the disparities estimated by the disparity estimation unit 103 based on the reliability (the score).

[0062] The cost pattern is a matching cost for an area where the deviation between the disparity calculated by the disparity estimation unit 103 and the prepared true value data of the disparity is equal to or greater than a certain value.

[0063] The feature calculation unit 101, the cost calculation unit 102, the disparity estimation unit 103, the cost pattern storage unit 104, and the reliability estimation unit (the score estimation unit 105) are a single convolutional neural network N800, and the cost pattern stored in the cost pattern storage unit 104 is expressed as a convolution kernel. The convolution kernel is determined by learning to use images and true value data of disparity that are prepared in advance, so that the reliability estimation unit (the score estimation unit 105) predicts the amount of error between the disparity estimated by the disparity estimation unit 103 and the true value data of the disparity (embodiment 2).

[0064] The environment recognition device further includes a disparity correction unit 107 that corrects disparity using a convolutional neural network N801 that receives as input the image acquired by the image acquisition unit 100, the disparity estimated by the disparity estimation unit 103, the reliability estimated by the reliability estimation unit (the score estimated by the score estimation unit 105), and the determination result of the disparity nullification unit 106 (Embodiment 3).

[0065] The reliability estimation unit (score estimation unit 105) numerically expresses the matching cost and the cost pattern as vectors, respectively, and determines the score as the inner product value of these two vectors.

[0066] The parallax nullifying unit 106 uses the parallax estimated by the parallax estimating unit 103, and nullifies the parallax more actively as the parallax value becomes smaller (the farther away).

[0067] The environment recognition device further includes a communication unit 108 capable of communicating with an external server, and the communication unit 108 receives a cost pattern transmitted from the external server and adds the received cost pattern to the cost pattern storage unit 104 or replaces the cost pattern stored in the cost pattern storage unit 104 (embodiment 4).

[0068] According to this embodiment, the validity of the parallax is determined using a cost pattern that is acquired or calculated in advance, and therefore the validity of the parallax can be determined with high accuracy in various cases where it is difficult to calculate the parallax.

[0069] Although the present invention has been described above with reference to Embodiments 1 to 4, the present invention is not limited to the above embodiments. In particular, the network structures used in the feature calculation unit and cost calculation unit described in the present invention have been described using examples, but it is easy to imagine changes to the number of stages, the number of convolutional layers between stages, and the like. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0070] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely realized by hardware, for example, by designing them as integrated circuits. Furthermore, the above-described configurations, functions, etc. may be realized by software, in which a processor interprets and executes a program that realizes each function. Information such as the program, tape, and file that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (solid state drive), or a recording medium such as an IC card, SD card, or DVD.

[0071] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0072] REFERENCE SIGNS LIST 1 Environment recognition device 100 Image acquisition unit 101 Feature amount calculation unit 102 Cost calculation unit 103 Disparity estimation unit 104 Cost pattern storage unit 105 Score estimation unit (reliability estimation unit) 106 Disparity nullification unit

Claims

1. An environment recognition device comprising: an image acquisition unit that acquires a first image and a second image; a feature calculation unit that calculates features from each of the first image and the second image; a cost calculation unit that calculates a matching cost between the first image and the second image in a predetermined search range based on the feature; a disparity estimation unit that estimates a disparity of a target pixel based on the matching cost; a cost pattern storage unit that stores a cost pattern indicating a pattern of the matching cost; a reliability estimation unit that calculates a reliability of the disparity in the search range based on the matching cost and the cost pattern; and a disparity invalidation unit that determines, based on the reliability, disparities to be invalidated among the disparities estimated by the disparity estimation unit.

2. An environment recognition device according to claim 1, wherein the reliability estimation unit determines the reliability of the disparity in the search range based on the similarity between the matching cost and the cost pattern.

3. An environmental recognition device as described in claim 1, wherein the reliability estimation unit calculates a score representing the uncertainty of disparity from the matching cost and the cost pattern in the search range, and the disparity invalidation unit determines, based on the score, disparities to be invalidated among the disparities estimated by the disparity estimation unit.

4. An environment recognition device according to claim 1, wherein the cost pattern is a matching cost of an area where the deviation between the disparity calculated by the disparity estimation unit and pre-prepared true disparity data is equal to or greater than a certain value.

5. An environmental recognition device as described in claim 1, wherein the feature calculation unit, the cost calculation unit, the disparity estimation unit, the cost pattern storage unit, and the reliability estimation unit are a single convolutional neural network, the cost pattern stored in the cost pattern storage unit is expressed as a convolution kernel, and the convolution kernel is determined by using previously prepared images and true value data of disparity and learning such that the reliability estimation unit predicts the amount of error between the disparity estimated by the disparity estimation unit and the true value data of disparity.

6. An environmental recognition device according to claim 5, further comprising a disparity correction unit that corrects disparity using a convolutional neural network that receives as input the image acquired by the image acquisition unit, the disparity estimated by the disparity estimation unit, the reliability estimated by the reliability estimation unit, and the judgment result of the disparity nullification unit.

7. An environment recognition device according to claim 3, wherein the reliability estimation unit numerically represents the matching cost and the cost pattern as vectors, and the score is the inner product value of the two vectors.

8. An environment recognition device according to claim 1, wherein the disparity nullifying unit utilizes the disparity estimated by the disparity estimating unit, and nullifies the disparity more actively as the disparity value becomes smaller.

9. An environmental recognition device as described in claim 1, further comprising a communication unit capable of communicating with an external server, the communication unit receiving a cost pattern transmitted from the external server and adding the cost pattern to the cost pattern storage unit or replacing the cost pattern with a cost pattern stored in the cost pattern storage unit.

Images