Environment recognition device

Abstract

The invention provides an environment recognition device which can determine the effectiveness of parallax with high precision under the condition of diversity of difficulty in parallax calculation. Comprises: an image acquisition unit (100) that acquires a first image and a second image; a feature amount calculation unit (101) that obtains feature amounts from the first image and the second image, respectively; a cost calculation unit (102) that finds a matching cost between the first image and the second image within a predetermined search range on the basis of the feature amount; a parallax estimation unit (103) that estimates the parallax of the target pixel on the basis of the matching cost; a cost pattern storage unit (104) in which a cost pattern indicating a pattern of the matching cost is stored; a score estimation unit (105) that calculates, from the matching cost and the cost pattern within the search range, a score exhibiting parallax uncertainty; and a parallax invalidation unit (106) that determines, using the score, a parallax that should be invalidated among the parallax estimated by the parallax estimation unit (103).

Description

TECHNICAL FIELD

[0001] The present application relates to an environment recognition device. BACKGROUND

[0002] In implementing a preventive safety function or automated driving, three-dimensional measurement in a travel environment is very important. In three-dimensional measurement including mobile bodies such as other vehicles and pedestrians, a stereo camera is effective. In a stereo camera, by performing stereo matching, it is possible to determine a depth from the camera. Stereo matching refers to a step of determining a region that has photographed the same region in a left image and a right image that have been parallelized. In stereo matching, an amount of shift of pixel positions of the left image and the right image is a disparity, and using the disparity and an internal parameter of the camera, it is possible to calculate a depth. It is known that by stereo matching, it is possible to calculate a disparity with high accuracy in a region having a texture feature, but in a repetitive pattern in which a similar texture pattern is continuous or a road surface having no texture feature, it is difficult to accurately calculate a disparity, and thus it is important to determine the validity of a disparity that is estimated. Patent Literature 1 discloses a method of detecting a local minimum in a matching cost calculated by stereo matching, counting local minima that take values similar to the local minimum, and determining the validity of a disparity.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Laid-Open No. 2017-54481 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] By using the method disclosed in Patent Literature 1, it is possible to detect a repetitive pattern and invalidate a disparity. On the other hand, in an actual image, in addition to a repetitive pattern, there are cases where a matching cost as a whole becomes low due to a lack of a texture feature, or a calculation of a disparity becomes difficult due to an influence of external disturbance (raindrops) or occlusion (shading). For such diversified cases, the technology described in Patent Literature 1 cannot accurately determine the validity of a disparity.

[0008] In view of the above problems, an object of the present application is to provide an environment recognition device that can accurately determine the validity of a disparity in diversified cases where a calculation of a disparity becomes difficult.

[0009] TECHNICAL SOLUTION TO THE PROBLEMS

[0010] To achieve the above object, the environment recognition device according to the present application includes: an image acquisition unit that acquires a first image and a second image; a feature amount calculation unit that calculates a feature amount from the first image and the second image, respectively; a cost calculation unit that calculates a matching cost between the first image and the second image within a predetermined search range, based on the feature amount; a disparity estimation unit that estimates a disparity of an object pixel based on the matching cost; a cost pattern storage unit that stores a cost pattern indicating a pattern of the matching cost; a reliability estimation unit that calculates a reliability of the disparity of the search range based on the matching cost and the cost pattern; and a disparity invalidation unit that invalidates a disparity estimated by the disparity estimation unit based on the reliability.

[0011] Effects of the Invention

[0012] According to the present application, the validity of the disparity can be determined with high accuracy in diversified situations in which disparity calculation becomes difficult.

[0013] The technical problems, structures, and effects other than the above will be further clarified by the following embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a functional block diagram showing the structure of the environment recognition device according to Embodiment 1 of the present application.

[0015] Figure 2 is a processing flowchart of the environment recognition device according to Embodiment 1.

[0016] Figure 3 is a diagram showing a method of calculating a matching cost.

[0017] Figure 4 is a diagram showing a calculated matching cost.

[0018] Figure 5 is a diagram showing data stored in the cost pattern storage unit.

[0019] Figure 6 is a diagram showing a method of determining a matching cost (cost pattern) stored in the cost pattern storage unit.

[0020] Figure 7 is a diagram showing the correlation between a matching cost and a calculated score.

[0021] Figure 8 is a functional block diagram showing the structure of the environment recognition device according to Embodiment 2 of the present application.

[0022] Figure 9 is a diagram showing the correlation between a convolutional neural network and a functional block diagram according to the embodiment.

[0023] Figure 10 This is a diagram illustrating the steps involved in calculating a score related to the uncertainty of disparity.

[0024] Figure 11 This is a diagram illustrating the learning method of convolutional neural networks.

[0025] Figure 12 This is a diagram illustrating a fractional calculation method that utilizes multiple cost models.

[0026] Figure 13 This is a functional block diagram illustrating the structure of the environmental identification device according to Embodiment 3 of the present invention.

[0027] Figure 14 This is a diagram illustrating parallax correction processing.

[0028] Figure 15 This is a functional block diagram illustrating the structure of the environmental identification device according to Embodiment 4 of the present invention. Detailed Implementation

[0029] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0030] [Implementation Method 1]

[0031] Figure 1 This is a functional block diagram illustrating the structure of the environment recognition device 1 in Embodiment 1. In the structure of Embodiment 1, two cameras are connected to the environment recognition device 1. Furthermore, the environment recognition device 1 is configured to be connected to a stereo camera (a structure with two cameras side-by-side in the horizontal direction) and mounted on a vehicle. Moreover, the cameras connected to the environment recognition device 1 are not limited to stereo cameras; they can also be multiple cameras consisting of three or more cameras mounted on the vehicle. When multiple cameras are connected to the environment recognition device 1, the processing described in this embodiment can be applied to images captured by any two cameras within a repeatedly captured area that has at least a portion of its field of view.

[0032] The environmental recognition device 1 of this embodiment consists of a camera, a computer, a memory, and a storage device. The computer executes control programs stored in the memory and the like to perform various functional actions.

[0033] like Figure 1 As shown, the environmental recognition device 1, as a functional unit implemented by the action of a camera or computer, includes an image acquisition unit 100, a feature calculation unit 101, a cost calculation unit 102, a disparity estimation unit 103, a cost pattern storage unit 104, a score estimation unit 105, and a disparity invalidation unit 106.

[0034] The image acquisition unit 100 acquires two images captured by the left camera and the right camera disposed left and right at the same time.

[0035] The feature amount calculation unit 101 calculates a feature amount for the left camera image (hereinafter also referred to as a left image) and the right camera image (hereinafter also referred to as a right image) acquired by the image acquisition unit 100, respectively. Here, the feature amount can use a known SURF feature amount or a BRIEF feature amount. In addition, it is also possible to directly use a pixel value of an object image as a feature amount, or to use pixel values of a range of 8 x 8 around an object pixel as a feature amount.

[0036] The cost calculation unit 102 calculates a matching cost at each pixel using the feature amount calculated by the feature amount calculation unit 101. The matching cost can be calculated by scanning a prescribed search range for the left image with the right image as a reference. The matching cost can be calculated as an absolute value of a difference in feature amount between an object pixel of the right image and a scanned pixel of the left image.

[0037] The disparity estimation unit 103 estimates a disparity of each pixel from the matching cost calculated by the cost calculation unit 102. As the estimation of the disparity, a disparity that achieves a minimum value of the matching cost is taken as an estimated value.

[0038] The cost pattern storage unit 104 stores a cost pattern used by the score estimation unit 105. The cost pattern, like the matching cost calculated by the cost calculation unit 102, is information having a size of a prescribed search range, and represents a matching cost for each search position. This cost pattern is calculated using a data set prepared in advance. Specifically, the cost pattern is calculated by calculating a disparity using the disparity estimation unit 103 for the data set, and comparing the calculated disparity with a correct value (a true value of the disparity). As a result of the comparison, in a case where the values differ by a certain amount or more, the matching cost calculated by the cost calculation unit 102 is stored as the cost pattern. That is, the matching cost of a region where the disparity calculated by the disparity estimation unit 103 and the true value data of the disparity prepared in advance deviate by a certain value or more, and the matching cost of a position where the disparity cannot be correctly calculated by the disparity estimation unit 103 are stored as the cost pattern.

[0039] The score estimation section 105 calculates a score related to the uncertainty of the disparity estimated by the disparity estimation section 103 using the matching cost calculated by the cost calculation section 102 and the cost pattern stored in the cost pattern storage section 104. In the calculation of the score, the matching cost calculated by the cost calculation section 102 and the cost pattern stored in the cost pattern storage section 104 are used. As described later, the score can numerically express the matching cost and the cost pattern as vectors respectively, and is calculated by obtaining the inner product of the vectors. In addition, the similarity of the two vectors can be measured using a cosine similarity, a difference between the vectors, or the like, in addition to the inner product. The higher the similarity of the two vectors (i.e., the matching cost and the cost pattern), the larger the score calculated by the score estimation section 105. The lower the similarity of the two vectors (i.e., the matching cost and the cost pattern), the smaller the score calculated by the score estimation section 105. That is, the larger the score calculated by the score estimation section 105, the more similar the two vectors, and the lower the reliability of the disparity estimated by the disparity estimation section 103. The smaller the score calculated by the score estimation section 105, the less similar the two vectors, and the higher the reliability of the disparity estimated by the disparity estimation section 103. Thus, the score estimation section 105 can calculate the reliability of the disparity estimated by the disparity estimation section 103 by calculating the score related to the uncertainty of the disparity using the similarity of the matching cost calculated by the cost calculation section 102 and the cost pattern stored in the cost pattern storage section 104.

[0040] The disparity invalidation section 106 compares the value of the score output from the score estimation section 105 with a predetermined threshold value, and determines that the disparity estimated by the disparity estimation section 103 is invalid in the case where the score is equal to or higher than the threshold value.

[0041] Next, the operation example of the environment recognition device 1 of the present embodiment will be described in detail with reference to the flowchart of Figure 2 In the operation example below, the method of calculating the disparity with respect to the right image of the stereo camera will be described.

[0042] The environment recognition device 1 sequentially performs the image acquisition process P101, the feature amount calculation process P102, the cost calculation process P103, the disparity estimation process P104, the score estimation process P105, and the disparity invalidation process P106.

[0043] In the image acquisition process P101, the right and left images of the stereo camera at the same time are acquired.

[0044] In the feature amount calculation process P102, the feature amount is calculated from the right and left images of the stereo camera. As the feature amount, the amount obtained by arranging the pixel values of the region of 8 x 8 pixels around the target pixel is used as the feature amount.

[0045] In the cost operation processing P103, the matching cost is calculated for each pixel of the right image using the feature amount calculated by the feature amount operation processing P102. Figure 3 A method of calculating the matching cost is shown. Figure 3 A method of calculating the matching cost by scanning the left image 301 is shown for the object pixel 302 (position (x, y)) of the right image 300. In the calculation of the matching cost, from the position (x, y) of the left image 301, D pixels are scanned in the right direction (from the pixel 303 to the pixel 304). Specifically, at each position of the left image, the absolute value of the difference of the feature amount is calculated. By repeating this calculation at each position, the matching cost 401 shown can be calculated. Figure 4 The final matching cost 401 shown.

[0046] In the disparity estimation processing P104, the disparity of the object pixel is decided from the matching cost calculated by the cost operation processing P103. In the case where the matching cost 401 shown is calculated, Figure 4 In the case where the matching cost 401 shown is calculated,

[0047] In the score estimation processing P105, the score expressing the uncertainty of the disparity is calculated using the matching cost calculated by the cost operation section 102 and the cost pattern stored in the cost pattern storage section 104.

[0048] Figure 5 A cost pattern stored in the cost pattern storage section 104 is shown. A plurality of cost patterns, such as the cost pattern 501 or the cost pattern 502, are stored in the cost pattern storage section 104. These cost patterns are stored as vector data. Specifically, the cost pattern 501 is expressed as a vector of the size of the search range + 1 as 501v, and each element stores the matching cost at that position. Figure 6 A method of acquiring these cost patterns is shown. The acquisition of the cost pattern uses a data set prepared in advance. The image 600 is an image in the data set, and the feature amount operation processing P102, the cost operation processing P103, and the disparity estimation processing P104 are performed on this image. Then, it is set that the matching cost 602 is calculated for the object pixel 601, and the disparity d' is estimated. The estimated disparity d' is compared with the true value d~ of the disparity (603), and when the absolute value of the difference thereof is above a prescribed value (for example, 3 pixels), the matching cost 602 is stored in the cost pattern storage section 104. That is, in the data set prepared in advance, the matching cost of the region where the deviation between the disparity calculated by the disparity estimation processing P104 and the true value data of the disparity prepared in advance reaches a prescribed value or more, and the matching cost of the region where the disparity cannot be correctly estimated by the disparity estimation processing P104 are stored as the cost pattern.

[0049] In the score estimation process P105, the inner product (score) between each of the cost patterns acquired in advance and the matching cost (401) calculated by the cost calculation section 102 is calculated. Here, the matching cost calculated by the cost calculation section 102 is also expressed as a vector like the cost patterns. In addition, as shown in the upper part of FIG. 4, in the case where a plurality of cost patterns are stored in the cost pattern storage section 104, the inner product (score) is calculated for each of the cost patterns (501v and 502v) and the maximum value thereof is taken as the final inner product value (score). By so doing, as shown in the upper part of FIG. 4, in the case where the matching cost (701) calculated by the cost calculation section 102 is an outline for which the disparity can be correctly calculated, it is not similar to any of the cost patterns stored in the cost pattern storage section 104, and thus the value of the score (701s) becomes small. On the other hand, as shown in the lower part of FIG. 4, in the case where the matching cost (702) is one for which it is difficult to correctly calculate the disparity, it is similar to any of the cost patterns stored in the cost pattern storage section 104, and thus the score (702s) becomes large. The matching cost (702) for which it is difficult to correctly calculate the disparity is not limited to the outline of the cost pattern shown in the lower part of FIG. 4, but also includes a multi-peak matching cost caused by a repetitive pattern, or a matching cost having a low overall value in a road surface region where the fluctuation is small, and the like. Figure 4 Figure 5 Figure 5 Figure 7 Figure 7 Figure 7

[0050] In the disparity invalidation process P106, the score calculated in the score estimation process P105 is used to determine the pixel to be invalidated. Specifically, using a prescribed threshold value (for example, a value of 30), in the case where the score is equal to or higher than the threshold value, the disparity is determined to be invalid.

[0051] As described above, the environment recognition device 1 of the present embodiment calculates the uncertainty of the disparity (in other words, the reliability of the disparity) using the matching cost and the plurality of cost patterns stored in advance, and determines the validity of the disparity. Thus, rather than determining the appropriateness of the matching cost based on a single condition, the appropriateness of the matching cost can be verified against a plurality of conditions, and thus the validity of the disparity can be determined with high accuracy in a wider variety of scenes. Furthermore, by comparing the outline information of the matching cost as a whole with the cost patterns, the validity of the disparity can be determined based not only on the peak information of the matching cost but also on the global cost information. Thus, in the case where the validity cannot be determined by the local determination by the repetitive pattern or the like, the validity can be determined from the outline of the matching cost as a whole whether it is a repetitive pattern or the like, and thus the validity of the disparity can be determined with higher accuracy.

[0052] ​​​​​​In addition, the environment recognition device 1 of the present embodiment stores, as the cost pattern, a matching cost that is actually difficult to calculate the disparity. By using a large amount of actual data, a variety of cost patterns that cause difficulty in calculating the disparity can be acquired. Thus, the validity of the disparity can be determined with high accuracy in a wider variety of scenes. Furthermore, in the method of acquiring the cost pattern described above, a large number of cost patterns can be acquired by simply collecting actual data. Thus, it is not necessary to manually design an algorithm that calculates the uncertainty of the disparity, and the amount of work required for the design can be reduced.

[0053] In addition, the environment recognition device 1 of the present embodiment, when calculating the score, numerically represents the matching cost and the cost pattern as vectors, respectively, and calculates the inner product of these two vectors. The inner product calculation of the vectors with each other can be calculated very efficiently, and thus the processing time required to determine the invalidation of the disparity can be shortened. Furthermore, as described above, the cosine similarity can be used instead of the simple inner product calculation. That is, the matching cost and the cost pattern are each normalized to a unit vector, and the inner product is calculated using the unit vectorized matching cost and the cost pattern. Thus, the similarity of the two vectors can be determined in the normalized state, and thus the validity of the disparity can be determined more accurately.

[0054] In addition, in the disparity invalidation processing P106 of the environment recognition device 1 described above, although a certain threshold value is always applied to the score that represents the uncertainty of the disparity to determine the validity of the disparity, the threshold value can be adaptively set according to the value of the disparity d' calculated by the disparity estimation processing P104. Specifically, the threshold value used in the disparity invalidation processing P106 is set to C d'. Here, C is a prescribed constant (for example, a value of 15). In this way, the threshold value is set according to the value of the estimated disparity d', and thus the threshold value can be set lower in the case of a distant place (d' is small) and set higher in the case of a close place (d' is large). In this way, the disparity invalidation processing P106 uses the disparity estimated in the disparity estimation processing P104, and the smaller the value of the disparity (the farther the distance), the more actively the disparity is invalidated, and thus the validity of the disparity can be more strictly determined in a distant place where the influence of the disparity error is large, and only a disparity with high accuracy can be determined to be valid.

[0055] [Embodiment 2]

[0056] Figure 8is a functional block diagram showing the structure of the environment recognition device 2 of Embodiment 2. In the structure of Embodiment 2, the feature amount operation section 101, the cost operation section 102, the disparity estimation section 103, the cost pattern storage section 104, and the score estimation section 105 are executed as respective functions of the convolutional neural network N800. Hereinafter, the feature amount operation section 101, the cost operation section 102, the disparity estimation section 103, the cost pattern storage section 104, and the score estimation section 105 that are different from those of Embodiment 1 are described.

[0057] In the feature amount operation section 101, the feature amount is extracted from each of the left and right images acquired by the image acquisition section 100 by performing two-dimensional convolution on the left and right images. The kernel used in the two-dimensional convolution is determined by learning described later.

[0058] In the cost operation section 102, the matching cost is calculated using the feature amount calculated by the feature amount operation section 101. The matching cost is calculated by repeatedly performing the process of shifting the feature amount of the left image and combining it in the channel direction with respect to the feature amount of the right image as in the known GC-Net. Thereafter, the matching cost is calculated by performing three-dimensional convolution processing. The kernel used in the three-dimensional convolution is determined by learning described later.

[0059] In the disparity estimation section 103, the disparity is estimated for each of the subject pixels using the matching cost calculated by the cost operation section 102. The disparity is estimated by applying the known soft argmin process to the matching cost.

[0060] In the cost pattern storage section 104, the cost pattern used by the score estimation section 105 is expressed as a convolution kernel and stored. The kernel is determined by learning described later.

[0061] In the score estimation section 105, the score of the disparity at the subject pixel is calculated with respect to the matching cost output from the cost operation section 102 by convolution operation using the convolution kernel stored in the cost pattern storage section 104. The calculated score represents the uncertainty of the disparity.

[0062] Figure 8 The processing sequence of the environment recognition device 2 shown in FIG. 10 is the same as that of Embodiment 1, and thus the operation example of the environment recognition device 2 of this embodiment is described in detail with reference to the flowchart of FIG. 9 and the structure of the convolutional neural network N800 of FIG. 8. Figure 2 Figure 9 The processing sequence of the environment recognition device 2 shown in FIG. 10 is the same as that of Embodiment 1, and thus the operation example of the environment recognition device 2 of this embodiment is described in detail with reference to the flowchart of FIG. 9 and the structure of the convolutional neural network N800 of FIG. 8.

[0063] As described above, the environment recognition device 2 of this embodiment can estimate the disparity of the subject pixel with higher accuracy by using the convolutional neural network N800 in which the feature amount operation section 101, the cost operation section 102, the disparity estimation section 103, the cost pattern storage section 104, and the score estimation section 105 are executed as respective functions. Figure 9 ​As shown, by executing the convolutional neural network N800, the feature amount operation processing P102, the cost operation processing P103, the disparity estimation processing P104, and the score estimation processing P105 are executed.

[0064] In the feature amount operation processing P102, two-dimensional convolution is executed on the right camera image 300 and the left camera image 301, respectively. Thereby, the feature amounts (maps) 300f and 301f are calculated from the right camera image 300 and the left camera image 301, respectively. The feature amount operation processing P102 is constituted by a three-stage convolutional neural network ((H, W, C), (H / 2, W / 2, 2C), (H / 4, W / 4, 4C)) constituted by two convolutional layers in each stage. Here, H, W, C represent height, width, and channel (C=16), respectively. Each convolutional layer (except for the layers corresponding to 300f and 301f) is constituted by convolution, batch normalization, and a rectified linear unit (ReLU). In the layers corresponding to 300f and 301f, only convolution is executed.

[0065] In the cost operation processing P103, first, 300v is generated from the feature amounts 300f and 301f calculated by the feature amount operation processing P102. 300v can be generated by repeating the processing of combining 300f and 301f shifted to the left direction in the channel direction the number of times of the search range of disparity, similarly to the known GC-Net. Thereafter, by repeatedly executing three-dimensional convolution on the generated 300v, the matching cost 300c is generated. The cost operation processing P103 is a four-stage convolutional neural network of an encoder-decoder form ((D / 4, H / 4, W / 4, 4C), (D / 8, H / 8, W / 8, 8C), (D / 16, H / 16, W / 16, 16C), (D / 32, H / 32, W / 32, 32C)). Here, D represents the search range of disparity (D=192). Each stage is constituted by two convolutional layers. Each convolutional layer is constituted by batch normalization and a rectified linear unit, except for the layers corresponding to 300c. 300c only executes convolution. In addition, between each encoder and decoder, skip connection (addition of maps to each other) is executed.

[0066] In the disparity estimation processing P104, by executing the known soft argmin processing on the matching cost 300c, the disparity image 300d is generated. In the soft argmin processing, the matching cost of the target pixel is sign-reversed, and a softmax function is applied. Thereafter, the index of each disparity is applied to the output result of the softmax function, and the disparity is calculated by taking the sum.

[0067] In the score estimation process P105, the score 300s that shows uncertainty of the disparity is output by performing convolution on the matching cost 300c. The score 300s is used in the disparity estimation process P106. Figure 10 The score estimation process P105 is described. The score 300s is generated by performing convolution operation on the matching cost 300c using the kernel 1000. Here, the kernel 1000 vectorizes the matching cost as described in Embodiment 1, and indicates the cost pattern stored in the cost pattern storage 104. Here, the convolution operation is equivalent to calculating correlation of the input and the vector of the kernel with each other, and thus the correlation, i.e., the similarity, of the matching cost and the cost pattern is calculated in the score estimation process P105.

[0068] In the above feature quantity operation process P102, the cost operation process P103, and the score estimation process P105, the convolution operation is used. Next, a learning method of the kernel used in the convolution operation is described. Figure 11 The learning method is described. In the environment recognition device 2 described in Embodiment 2, it is necessary to output both the disparity (image) 300d and the score 300s (uncertainty of the disparity) from the convolution neural network N800. Therefore, with respect to the disparity and the score, the loss function is defined respectively, and the kernel is learned by minimizing the loss function. Figure 11 The 300d is the disparity image estimated by the disparity estimation process P104, and the 300s is the output of the score estimation process P105. In addition, Figure 11The definition of the loss for the spatially identical regions (i.e., corresponding to the same position on the image) 300dt and 300st is illustrated. First, the loss definition related to the region 300dt is described. In the case where the output of 300dt is d' and the true value of the disparity of 300dt is d~, the cost1(abs(d' - d~)) is used as the loss function. That is, the value of the kernel is learned so that the output d' of 300dt approaches the true value d~. Next, the loss definition related to the score is described. In the case where the output of the region 300st is score, the output of 300dt is d', and the true value of the disparity of 300dt is d~, the cost2(abs(score - abs(d' - d~))) is minimized as the loss function. That is, the kernel is learned so that the output (score) of 300st approaches the difference between the estimated disparity d' and the true value d~. Thus, in the case where the disparity d' and the true value d~ deviate greatly, the output (score) of 300st also takes a large value, and the uncertainty of the estimated disparity can be expressed. In addition, in the loss of cost2, abs(d' - d~) is treated as a constant. That is, the gradient of cost2 based on error backpropagation is not propagated to the disparity image 300d. In learning, the kernel is estimated by simultaneously minimizing cost1 and cost2. Through this minimization, the kernel used for the output of the score 300s, i.e., the cost pattern 1000 stored in the cost pattern storage section 104 Figure 10 ) can also be determined through learning.

[0069] As described above, the environment recognition device 2 of the present embodiment calculates the disparity and the uncertainty of the disparity using the convolutional neural network. In addition, the kernel of the convolutional neural network is determined by defining the loss function for the score expressing the disparity and the uncertainty of the disparity and minimizing these loss functions. That is, the kernel is estimated through learning. Here, since the cost pattern used in the calculation of the uncertainty of the disparity is expressed as a kernel, the cost pattern can be automatically determined by simply collecting data for learning, without manually designing the cost pattern. In addition, by using a large amount of data as the data for learning, a cost pattern that is beneficial for determining the validity of the disparity in a more diversified scene can be determined, and thus the estimation accuracy can be improved.

[0070] In the environment recognition device 2 of the above-described embodiment 2, as shown in FIG. 10, there is only one cost pattern (1000) stored in the cost pattern storage section 104. On the other hand, as shown in FIG. 11, it is also possible to estimate the score 300s expressing the uncertainty of the disparity using a plurality of cost patterns. Figure 10 Figure 12 Figure 12 ​​The case where four cost patterns 1000, 1001, 1002, 1003 are stored in the cost pattern storage section 104 is shown. At the time of estimation of the score 300s, convolution operations are performed using each of the cost patterns as a kernel. The outputs after the operations are 1000f, 1001f, 1002f, 1003f, respectively. These four outputs are combined in the channel direction, and then the score 300s is output by performing two-dimensional convolution (conv) in the following equation. Figure 12 Figure 12 The kernel (including the cost pattern) shown in the equation can be estimated by minimizing Figure 11

[0071] Thus, by expressing a plurality of cost patterns as kernels of a convolutional neural network and estimating them through learning, it is possible to evaluate the uncertainty of disparity using matching costs of a variety of conditions. That is, it is possible to improve the accuracy of disparity validity determination in a diversified scene.

[0072] [Embodiment 3]

[0073] Figure 13 is a functional block diagram showing the structure of the environment recognition device 3 of Embodiment 3. The structure of Embodiment 3 is a structure in which the disparity correction section 107 is added to the structure of Embodiment 2. In addition, the disparity correction section 107 is executed by the convolutional neural network N801. Hereinafter, the disparity correction section 107 will be described.

[0074] In the disparity correction section 107, the disparity estimated by the disparity estimation section 103 is corrected using the right camera image acquired by the image acquisition section 100, the disparity estimated by the disparity estimation section 103, the score estimated by the score estimation section 105, and the invalidity determination result output by the disparity invalidation section 106, and a disparity of higher accuracy is output. Figure 14 The process executed by the disparity correction section 107 is shown. The right camera image 300 acquired by the image acquisition section 100, the disparity 300d estimated by the disparity estimation section 103, the score 300s estimated by the score estimation section 105, and the invalidity determination result 300i output by the disparity invalidation section 106 are input to the disparity correction section 107. As shown in the equation, the disparity 300d is corrected using the right camera image 300, the score 300s, and the invalidity determination result 300i. Figure 14 ​​The invalidity determination result 300i is set to a value of 1 for a position determined to be invalid and a value of 0 for a position determined to be valid, as shown in the lower part of FIG. 30. Further, these input data are each up-sampled (nearest neighbor interpolation) to have the same spatial resolution as the width and height of the right camera image 300, and the data obtained by combining these four inputs in the channel direction is 300''. In the disparity correction section 107, the corrected disparity 300d'' is output by executing the convolutional neural network N801 on 300''. The convolutional neural network N801 is a two-stage encoder-decoder ((H, W, C), (H / 2, W / 2, 2C)), and like N800, each stage is composed of two convolutional layers. Further, each convolutional layer is composed of convolution, batch normalization, and a rectified linear unit except for the final layer. In the final layer, only convolution is performed.

[0075] In the learning of the convolutional neural network N801, like Figure 11 Similarly to the above, the loss function cost1 related to disparity is minimized. That is, the convolutional neural network N801 is learned by minimizing abs(300d'' - 300d~) using the corrected disparity 300d'' and the true value 300d~. In Figure 13 In the environment recognition device 3 of Embodiment 3 shown in FIG. 31, there are two convolutional neural networks, N800 and N801, but in learning, N800 is learned first, and then N801 is learned.

[0076] The environment recognition device 3 of Embodiment 3 described above corrects the disparity estimated by the disparity estimation section 103 by the disparity correction section 107, and finally outputs a high-precision disparity. In the disparity correction section 107, in addition to the image acquired by the image acquisition section 100 and the disparity estimated by the disparity estimation section 103, the score estimated by the score estimation section 105 and the output of the disparity invalidation section 106 are also used. The output of the score estimation section 105 and the disparity invalidation section 106 includes positions where the uncertainty of the disparity is high and information related to positions that should be invalidated. By passing this information to the disparity correction section 107, it is possible to pass information of positions that were originally invalidated or where the precision of the disparity is low to the convolutional neural network (N801), and thus it is possible to actively correct the disparity for these positions. As a result, it is possible to further improve the precision of the disparity.

[0077] [Embodiment 4]

[0078] Figure 15 is a functional block diagram showing the structure of the environment recognition device 4 of Embodiment 4. The structure of Embodiment 4 is a structure in which the communication section 108 is added to the structure of Embodiment 1. Hereinafter, the communication section 108 and the score estimation section 105 will be described.

[0079] The communication section 108 cooperates with an external server not shown, receives cost pattern information transmitted from the external server, and appends it to the cost pattern storage section 104. The external server transmits a cost pattern judged to be effective in the score estimation to the communication section 108. The communication section 108 can be connected to the external server in a wireless connection manner, or wired connection can be made.

[0080] The score estimation section 105 calculates the uncertainty of the disparity by calculating the inner product of the matching cost output from the cost operation section 102, using not only the stored cost pattern but also the newly appended cost pattern.

[0081] The environment recognition device 4 of the above-described embodiment 4 newly appends a cost pattern transmitted from the external server to the cost pattern storage section 104, and judges the effectiveness of the disparity. The external server calculates an effective cost pattern using the latest data set, and transmits it to the environment recognition device 4. With this structure, it is possible to append a cost pattern for judging the effectiveness of the disparity, and thus it is possible to accurately judge the effectiveness of the disparity in a more diversified scene.

[0082] The environment recognition device 4 of the above-described embodiment 4 explains the structure of appending a cost pattern, but is not limited to appending a cost pattern, and can also perform replacement of a cost pattern. That is, it is possible to replace a cost pattern received from the external server with a cost pattern already stored in the cost pattern storage section 104. At the time of replacement, a cost pattern with a low frequency of use is deleted, and is replaced with a cost pattern received from the external server. Specifically, in the score estimation section 105, the inner product of the matching cost output from the cost operation section 102 and each cost pattern is calculated, and the largest inner product value (i.e., the one with a high correlation value) is selected as the final score, but the number of times of use of the cost pattern reaching the largest inner product value is incremented by one. This is performed per frame, and the cost pattern with the lowest number of times of use at the time when a prescribed number of frames (e.g., 100 frames) have elapsed is selected as a deletion target. In this way, without increasing the cost pattern, by deleting a cost pattern with a low frequency of use, it is possible to not only suppress the memory consumption amount of the cost pattern storage section 104, but also suppress the increase in the execution time of the score estimation section 105.

[0083] In addition, the above-described embodiment 4 has been explained in a form in which the communication section 108 is added to the embodiment 1, but the communication section 108 can also be added to the structure explained in the embodiment 2. That is, the communication section 108 receives a cost pattern, in other words, a kernel of convolution used in the score estimation section 105. The processing contents in the score estimation section 105 in this case are explained with reference to Figure 10 In the case where a kernel of convolution is received in the communication section 108, not only the illustrated kernel 1000 but also an added kernel 1001 is stored in the cost pattern storage section 104 (Figure 10 The two scores 300s_1000 and 300s_1001 are generated by performing a convolution operation using the two kernels 1000 and 1001 (not illustrated in FIG. 8). Thereafter, the maximum value of each element in the two scores is taken, and the final score is output. As described above, by receiving a new cost pattern from an external server, it is possible to correctly determine the validity of the disparity even for a scene that could not be dealt with initially. Furthermore, by changing only the cost pattern of the cost pattern storage section 104 in the convolution neural network N800 illustrated in FIG. 8, that is, the kernel used in the score estimation section 105, it is possible to expand the scene in which the validity of the disparity can be correctly determined without affecting the output result of the disparity estimation section 103. Figure 10 Figure 8 The cost pattern of the cost pattern storage section 104 in the convolution neural network N800 illustrated in FIG. 8, that is, the kernel used in the score estimation section 105, is changed, so it is possible to expand the scene in which the validity of the disparity can be correctly determined without affecting the output result of the disparity estimation section 103.

[0084] [SUMMARY]

[0085] As described above, the environment recognition device of the present embodiment includes: an image acquisition section 100 that acquires a first image and a second image; a feature amount operation section 101 that respectively calculates a feature amount from the first image and the second image; a cost operation section 102 that calculates a matching cost (a degree of difference or a degree of dissimilarity) between the first image and the second image within a predetermined search range on the basis of the feature amount; a disparity estimation section 103 that estimates a disparity of an object pixel on the basis of the matching cost; a cost pattern storage section 104 in which a cost pattern representing a pattern of the matching cost is stored; a reliability estimation section (a score estimation section 105 that calculates a score that represents uncertainty of the disparity from the matching cost and the cost pattern within the search range) that calculates a reliability of the disparity within the search range on the basis of the matching cost and the cost pattern (a degree of similarity); and a disparity invalidation section 106 that determines a disparity that should be invalidated from the disparity estimated by the disparity estimation section 103 on the basis of the reliability (the score).

[0086] The cost pattern is a matching cost of a region in which the disparity calculated by the disparity estimation section 103 and true value data of the disparity prepared in advance deviate by a certain value or more.

[0087] The feature amount operation section 101, the cost operation section 102, the disparity estimation section 103, the cost pattern storage section 104, and the reliability estimation section (the score estimation section 105) are a single convolution neural network N800, and the cost pattern stored in the cost pattern storage section 104 is expressed as a convolution kernel that is decided by learning using an image and true value data of the disparity so that the reliability estimation section (the score estimation section 105) predicts an error amount between the disparity estimated by the disparity estimation section 103 and the true value data of the disparity (Embodiment 2). ​

[0088] The environment recognition device further includes a disparity correction section 107 that corrects disparity by a convolutional neural network N801 that inputs an image acquired by the image acquisition section 100, disparity estimated by the disparity estimation section 103, reliability estimated by the reliability estimation section (score estimated by the score estimation section 105), and a determination result of the disparity invalidation section 106 (Embodiment 3).

[0089] The reliability estimation section (score estimation section 105) numerically expresses the matching cost and the cost pattern as vectors, respectively, and takes an inner product value of the two vectors as the score.

[0090] The disparity invalidation section 106 uses disparity estimated by the disparity estimation section 103, and the smaller (the farther) the value is, the more actively the disparity is invalidated.

[0091] The environment recognition device further includes a communication section 108 that communicates with an external server, and the communication section 108 receives a cost pattern transmitted from the external server and newly adds it to the cost pattern storage section 104 or performs replacement of a cost pattern stored in the cost pattern storage section 104 (Embodiment 4).

[0092] According to the present embodiment, since the validity of disparity is determined using a cost pattern acquired or calculated in advance, the validity of disparity can be determined with high accuracy in diversified scenes in which it is difficult to calculate disparity.

[0093] The present application has been described above with reference to Embodiments 1 to 4, but the present application is not limited to the above-described embodiments. In particular, although a network structure used in the feature quantity operation section and the cost operation section described in the present application has been exemplified, it is easy to imagine changing the number of stages, the number of convolution layers between stages, and the like. As for the structure or details of the present application, various changes that can be understood by those skilled in the art can be added within the scope of the present application.

[0094] Further, each of the above-described structures, functions, processing sections, processing units, and the like can be designed, for example, by an integrated circuit, or the like, to be implemented in hardware. In addition, each of the above-described structures, functions, and the like can be implemented in software by a processor interpreting and executing a program that implements each function. Information of the program, tables, files, and the like that implement each function can be stored in a memory or a hard disk, a recording device such as an SSD (solid state drive), or an IC card, an SD card, a DVD, or the like.

[0095] In addition, control lines and information lines necessary for the explanation are shown, but the number of control lines and information lines necessary for the product is not limited to the number shown. In fact, it can be considered that almost all the structures are connected to each other.

[0096] Label Explanation

[0097] 1 Environment recognition device

[0098] 100 Image acquisition section

[0099] 101 Feature amount operation section

[0100] 102 Cost operation section

[0101] 103 Parallax estimation section

[0102] 104 Cost pattern storage section

[0103] 105 Score estimation section (reliability estimation section)

[0104] 106 Parallax invalidation section

Claims

1. An environmental identification device, characterized in that, include: An image acquisition unit acquires a first image and a second image; The feature calculation unit calculates feature quantities from the first image and the second image respectively; The cost calculation unit calculates the matching cost between the first image and the second image within a specified search range based on the feature quantity. A disparity estimation unit that estimates the disparity of an object pixel based on the matching cost; A cost pattern storage unit stores cost patterns representing patterns of the matching cost; A reliability estimation unit, which calculates the reliability of the disparity within the search range based on the matching cost and the cost pattern; as well as The parallax invalidation unit determines, based on the reliability, the parallaxes that should be invalidated from the parallaxes estimated by the parallax estimation unit.

2. The environmental identification device as described in claim 1, characterized in that, The reliability estimation unit calculates the reliability of the disparity within the search range based on the similarity between the matching cost and the cost pattern.

3. The environmental identification device as described in claim 1, characterized in that, The reliability estimation unit calculates a score representing the uncertainty of disparity from the matching cost and the cost pattern within the search range, and the disparity invalidation unit determines, based on the score, the disparities that should be invalidated from the disparities estimated by the disparity estimation unit.

4. The environmental identification device as described in claim 1, characterized in that, The cost pattern is the matching cost for regions where the deviation between the disparity calculated by the disparity estimation unit and the pre-prepared true disparity data reaches a certain value or more.

5. The environmental identification device as described in claim 1, characterized in that, The feature calculation unit, the cost calculation unit, the disparity estimation unit, the cost pattern storage unit, and the reliability estimation unit are a single convolutional neural network. The cost patterns stored in the cost pattern storage unit are represented as convolutional kernels. The convolutional kernels are determined by learning using pre-prepared image and disparity ground truth data so that the reliability estimation unit predicts the amount of error between the disparity estimated by the disparity estimation unit and the disparity ground truth data.

6. The environmental identification device as described in claim 5, characterized in that, It also includes a disparity correction unit, which corrects disparity by using a convolutional neural network as input to the image acquired by the image acquisition unit, the disparity estimated by the disparity estimation unit, the reliability estimated by the reliability estimation unit, and the determination result of the disparity invalidation unit.

7. The environmental identification device as described in claim 3, characterized in that, The reliability estimation unit represents the matching cost and the cost pattern as vectors and uses the inner product of these two vectors as the score.

8. The environmental identification device as described in claim 1, characterized in that, The parallax invalidation unit uses the parallax estimated by the parallax estimation unit, and the smaller the value of the parallax, the more actively it invalidates the parallax.

9. The environmental identification device as described in claim 1, characterized in that, It also includes a communications unit, which can communicate with external servers. The communication unit receives a cost pattern sent from the external server and adds it to the cost pattern storage unit, or performs a replacement with the cost pattern stored in the cost pattern storage unit.

Images