Environment recognition device and environment recognition method

Abstract

To provide an environment recognition device which calculates a correspondence for each pixel between images with a smaller computation amount.SOLUTION: An environment recognition device comprises: a feature extraction unit 20 which obtains a first feature map and a second feature map obtained by acquiring a feature amount from a first image and a second image; a cost calculation unit 30 which obtains matching cost expressing a probability about a correspondence of each partial region by performing convolutional arithmetic operation to the first feature map and the second feature map; and a matching unit 50 which determines a correspondence between the first image and the second image by analyzing the obtained matching cost. The cost calculation unit executes the convolutional arithmetic operation using a first parameter and a second parameter for cost calculation to the first feature map and the second feature map, obtains the first convolutional arithmetic operation result and the second convolutional arithmetic operation result, and obtains the matching cost by integrating the first convolutional arithmetic operation result and the second convolutional arithmetic operation result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an environment recognition device and an environment recognition method. [Background technology]

[0002] There is a known technology that calculates the distance to an object captured in an image and the flow of the object by calculating the pixel-by-pixel correspondence between multiple images. This technology uses a camera mounted on a vehicle to calculate the distance and flow information, making it possible to analyze the behavior of other vehicles around the vehicle, and can be used for preventive safety functions.

[0003] Patent Document 1 discloses an example of a method for calculating the correspondence between left and right images using a stereo camera. That is, Patent Document 1 describes a technology based on deep learning. Specifically, the technology described in Patent Document 1 generates a 3D matching cost feature by linking left and right feature maps shifted by a disparity value and further linking the linked feature maps over the disparity search range. Furthermore, this 3D matching cost feature is further linked with a grouped cross-correlation feature. Then, a 3D convolution process is performed multiple times to calculate the disparity for each pixel. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2021-526683 Summary of the Invention [Problem to be solved by the invention]

[0005] The 3D convolution processing for 3D matching cost features as described in Patent Document 1 involves convolution processing for width, height, and parallax direction, which results in a high processing load. In particular, the first convolution processing for 3D matching cost features involves large sizes for width, height, and parallax, making it difficult to apply to devices with limited computational resources, and there has been a demand for reducing the amount of computation.

[0006] In view of the above, it is an object of the present invention to provide an environment recognition device and an environment recognition method that are capable of calculating pixel-by-pixel correspondences between a plurality of images with a smaller amount of calculation. [Means for solving the problem]

[0007] In order to solve the above problems, for example, the configurations described in the claims are adopted. The present application includes a plurality of means for solving the above problems, and one example thereof is an environment recognition device comprising: an image acquisition unit that acquires a first image and a second image; a feature extraction unit that obtains a first feature map by obtaining feature amounts for a plurality of partial regions of a first image, and a second feature map by obtaining feature amounts for a plurality of partial regions of a second image; a cost calculation unit that calculates a matching cost that represents the likelihood of a correspondence relationship between each subregion by performing a convolution operation on the first feature map and the second feature map; The image processing system includes a matching unit that determines the correspondence between the first image and the second image by analyzing the matching cost calculated by the cost calculation unit. The cost calculation unit performs a convolution operation using the first parameter and the second parameter for cost calculation on the first feature map and the second feature map, respectively, to obtain a first convolution operation result and a second convolution operation result, and then integrates the first convolution operation result and the second convolution operation result to obtain a matching cost. The first parameter for cost calculation and the second parameter for cost calculation are convolution kernels determined by learning to minimize a loss function based on the correspondence. [Effects of the Invention]

[0008] According to the present invention, it is possible to calculate the correspondence between pixels between a plurality of images with a smaller amount of calculation. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a configuration diagram of an example of an environment recognition device according to an embodiment of the present invention; [Figure 2] 1 is a diagram illustrating an example of the overall configuration and hardware configuration mounted on a vehicle according to an embodiment of the present invention; [Figure 3] 10 is a flowchart illustrating an example of environment recognition processing according to an embodiment of the present invention. [Figure 4] FIG. 10 is a diagram illustrating a feature extraction process according to an embodiment of the present invention. [Figure 5] 10 is a flowchart illustrating an example of a cost calculation process according to an embodiment of the present invention. [Figure 6] FIG. 10 is a diagram illustrating a cost calculation process according to an embodiment of the present invention. [Figure 7] FIG. 10 is a diagram illustrating a cost correction process according to an embodiment of the present invention. [Figure 8] FIG. 10 is a diagram showing a calculation method during learning in the cost calculation process according to an embodiment of the present invention. [Figure 9] FIG. 10 is a diagram illustrating a method of integrating kernels in the cost calculation process according to an embodiment of the present invention. [Figure 10] 10A to 10C are diagrams illustrating a method for generating a kernel for a right feature map in the cost calculation process according to an embodiment of the present invention. [Figure 11] 10A to 10C are diagrams illustrating a method for generating a kernel for a left feature map in the cost calculation process according to an embodiment of the present invention. [Figure 12] 10 is a flowchart showing a case where processing is performed for every certain number of rows in the cost calculation process according to an embodiment of the present invention. [Figure 13]10 is a flowchart illustrating a flow calculation process according to an embodiment of the present invention. [Figure 14] 10 is a flowchart of a cost calculation process according to an embodiment of the present invention. [Figure 15] FIG. 10 is a diagram illustrating a cost calculation process according to an embodiment of the present invention. [Figure 16] FIG. 10 is a diagram showing a calculation method during learning in the cost calculation process according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] An embodiment of the present invention will now be described with reference to the accompanying drawings. FIG. 1 is a functional block diagram showing the configuration of the environment recognition device 1. The environment recognition device 1 of this embodiment is composed of a camera, a computer, a memory, a storage device, etc., as described below, and operates as various functional units by the computer executing control programs stored in the memory, etc.

[0011] [Configuration of the environment recognition device] As shown in FIG. 1, the environment recognition device 1 has an image acquisition unit 10, a feature extraction unit 20, a cost calculation unit 30, a cost aggregation unit 40, and a matching unit 50 as functional units realized by the operation of a camera and a computer. The image acquisition unit 10 has a first image storage unit 11 and a second image storage unit 12, and performs image acquisition processing to acquire multiple images captured by a camera. In the case of a stereo camera, the first image storage unit 11 and the second image storage unit 12 acquire two left and right images captured at the same time. In the case of a monocular camera, the first image storage unit 11 and the second image storage unit 12 acquire multiple images captured at different times.

[0012] The feature extraction unit 20 has a convolution operation unit 21 and a feature extraction parameter storage unit 22. The convolution operation unit 21 performs a convolution operation on a plurality of images acquired by the image acquisition unit 10. The feature extraction parameter storage unit 22 stores parameters of a kernel used for the convolution. The kernel used for the convolution is determined by learning so as to minimize a predetermined loss function.

[0013] The cost calculation unit 30 performs a cost calculation process to calculate costs relating to correspondence between a plurality of images acquired by the image acquisition unit 10. Specifically, the cost calculation unit 30 includes a first image convolution calculation unit 31 and a second image convolution calculation unit 32. The first image convolution calculation unit 31 performs a convolution calculation using first parameters stored in a first parameter storage unit 33, and the second image convolution calculation unit 32 performs a convolution calculation using second parameters stored in a second parameter storage unit 34.

[0014] The results of the convolution operations in the image convolution operation units 31 and 32 are integrated in the integration unit 35. In the case of a stereo camera, this cost calculation unit 30 calculates the cost related to the parallax between the left and right images, and in the case of a monocular camera, it calculates the cost related to the flow between images captured at different times. In the cost calculation process, the image convolution operation units 31 and 32 perform convolution operations independently for each feature map calculated in the feature extraction unit 20, and the results are integrated in the integration unit 35 to calculate a cost map. The kernel used in the convolution operation is determined so as to minimize a predetermined loss function.

[0015] The cost aggregator 40 includes a convolution calculator 41 and a feature extraction parameter storage 42 . The convolution calculation unit 41 corrects the cost map by performing a convolution calculation on the cost map calculated by the cost calculation unit 30. The feature extraction parameter storage unit 42 stores parameters of the convolution kernel executed by the convolution calculation unit 41. The kernel used here is a 3D convolution kernel, determined to minimize a predetermined loss function. The matching unit 50 determines correspondences between multiple images acquired by the image acquisition unit 10 by having the cost analysis unit 51 analyze the cost map calculated and corrected by the cost aggregation unit 40. That is, the matching unit 50 determines the parallax between images in the case of a stereo camera, and determines the flow in the case of a monocular camera. The cost analysis unit 51 can perform analysis processing using SoftArgMin processing or a kernel.

[0016] The feature extraction unit 20, cost calculation unit 30, cost aggregation unit 40, and matching unit 50 use kernels determined by learning to minimize a predetermined loss function. These kernels are determined by minimizing a loss function defined by comparing the matching result determined by the matching unit 50 with ground truth data. In the case of a stereo camera, loss due to disparity between images is minimized, and in the case of a monocular camera, loss due to flow between images is minimized.

[0017] [Configuration of a vehicle equipped with an environmental recognition device] FIG. 2 shows the configuration of a vehicle 5 equipped with a computer 4 including an environment recognition device 1 and a vehicle control device 2 according to this embodiment. In order to monitor the road ahead, on-board cameras 3L and 3R are installed on a vehicle (automobile) 5. Here, the two on-board cameras 3L and 3R are stereo cameras that capture left and right images, but there are also cases where a single camera is connected to the environment recognition device 1.

[0018] The images captured by the on-board cameras 3L, 3R are supplied to the environment recognition device 1 in the computer 4. The environment recognition device 1 processes the captured images and performs environment recognition processing. When the environment recognition device 1 recognizes the environment, it may use not only the images captured by the on-board cameras 3L, 3R but also other detection mechanisms such as radar (not shown). Information about the environment recognized by the environment recognition device 1 is sent to a vehicle control device 2 also configured in the computer 4, which controls the driving of the vehicle 5. However, the environment recognition device 1 and the vehicle control device 2 may each be configured as separate computers.

[0019] As shown in FIG. 2, the computer 4 incorporating the environment recognition device 1 and the vehicle control device 2 includes a CPU (central processing unit) 4a, a work memory 4b, a storage unit 4c, and an interface (I / F) 4d, which are interconnected to enable data transfer between them. The CPU 4a executes the program stored in the storage unit 4c in the work memory 4b, thereby configuring each processing unit that performs environment recognition processing and the like in the work memory 4b. The storage unit 4c stores programs for performing environment recognition processing and vehicle control, as well as images, control data, and the like. The interface 4d performs input processing of image data from the on-board cameras 3L and 3R and detection data from various sensors, and outputs control data for the vehicle.

[0020] [Example 1 of processing performed by the environment recognition device: Example of calculating disparity using a stereo camera] Next, as an example 1 of an operation performed by the environment recognition device 1, an operation of calculating parallax using the vehicle-mounted cameras 3L and 3R, which are stereo cameras, will be described. Hereinafter, a process of calculating the parallax for the right image out of the left and right images acquired by the stereo camera will be described.

[0021] FIG. 3 is a flowchart showing the process of calculating the parallax performed by the environment recognition device 1. First, the environment recognition device 1 sequentially executes an image acquisition process (P101), a feature extraction process (P102), a matching cost calculation process (P103), a cost correction process (P104), and a parallax calculation process (P105).

[0022] In the image acquisition process (P101), the image acquisition unit 10 acquires data of left and right images from the stereo camera. In the feature extraction process (P102), the feature extraction unit 20 performs a convolution operation on the left and right images acquired in the image acquisition process (P101) to generate a feature map.

[0023] FIG. 4 shows the details of the feature extraction process (P102) performed by the feature extraction unit 20. Here, the feature extraction unit 20 calculates a left feature map F100 and a right feature map F101 for a left image I100 and a right image I101, respectively, using a trained convolutional neural network N100. The convolutional neural network N100 is configured with a repeated structure of convolution operations, normalization operations, and activation operations. Batch normalization is used for the normalization operation, and a rectified linear unit is used for activation. However, only the final layer does not use batch normalization or a rectified linear unit.

[0024] Here, the feature extraction unit 20 performs convolution operations on the left image I100 and the right image I101 using the same convolutional neural network N100. This allows the feature extraction unit 20 to extract the same feature amounts for areas that appear the same. In addition, the convolutional neural network N100 uses parameters estimated by learning, which will be described later. Returning to the explanation of FIG. 3, in the matching cost calculation process (P103), the cost calculation unit 30 generates a cost for each disparity using the left feature map F100 and right feature map F101 calculated in the feature extraction process (P102).

[0025] FIG. 5 is a flowchart showing the details of the matching cost calculation process (P103) performed by the cost calculation unit 30. In the matching cost calculation process (P103), the cost calculation unit 30 performs left feature map convolution process (P201), right feature map convolution process (P202), and cost generation corresponding to each disparity (P203). In generating a cost corresponding to each disparity (P203), the cost calculation unit 30 performs a shift process on the left feature map (P204), and then performs an element addition process on the right feature map and the left feature map (P205) to generate a cost corresponding to each disparity. Here, the shifting process of the left feature map (P204) and the element addition process of the right feature map and the left feature map (P205) are repeatedly executed within the search range of the cost generation (P203) corresponding to the disparity. Thereafter, the cost calculation unit 30 performs normalization using Batch Normalization and activation using a Rectified Linear Unit on the generated costs.

[0026] FIG. 6 shows an example of the processing status of the matching cost calculation process (P103) by the cost calculation unit 30 described in the flowchart of FIG. In the left feature map convolution process (P201), the cost calculation unit 30 executes a convolution operation LC using a convolution parameter KL on the left feature map F100 to obtain a left feature map F110. Furthermore, in the right feature map convolution process (P202), the cost calculation unit 30 executes a convolution operation RC using the convolution parameter KR on the right feature map F101 to obtain a right feature map F111.

[0027] In generating costs corresponding to each disparity (P203), the following process is executed a number of times according to the disparity range to be calculated (for example, 64 pix). Here, the cost generation process corresponding to a disparity of 4 pix will be described. In the left feature map shift process (P204), the cost calculation unit 30 shifts the left feature map F110 by 4 pix to the left relative to the right feature map F111. Then, in the element addition process (P205), the cost calculation unit 30 performs element addition on the right feature map F111 and the shifted left feature map F110 to generate a cost map C104 corresponding to a disparity of 4 pix.

[0028] Then, the cost calculation unit 30 executes the shift process (P204) and element addition process (P205) of this left feature map the number of times (D+1 times) for the calculated disparity range, and combines the results in the depth direction of the feature map to generate a cost map C200. Here, a cost corresponding to each disparity is stored at each position in the cost map C200.

[0029] In the above explanation, for convenience of explanation, description of the channels of the feature maps was omitted. However, the cost calculation unit 30 can use multiple kernels as the convolution parameters KL and KR. As a result, the sizes of the left feature map F110 and the right feature map F111 are (HL, WL, WL) and (HR, WR, CR), respectively. Here, HL, WL, and WL respectively represent the height, width, and channel of the left feature map F110. HR, WR, and CR respectively represent the height, width, and channel of the right feature map F111.

[0030] In the element addition process, the cost calculation unit 30 calculates a cost map C200 of size (D+1, HR, WR, CR) by shifting the left feature map F110 D+1 times corresponding to the disparity range and adding elements between the left feature map F110 and the right feature map F111. Then, a cost corresponding to the disparity is stored for each channel at each position in the cost map C200 calculated in this way.

[0031] Furthermore, by inputting the left feature map F100 itself, which has a size such that WL = WR + D as the width of the left feature map F110, the cost calculation unit 30 can arrange the shifted left feature map F110 so that it overlaps the right feature map F111 in the horizontal direction. Here, setting the width of the left feature map F110 to WL = WR + D means that the left image I100 shown in Fig. 4 is input to the convolutional neural network N100 at a size larger than the right image I101. Note that the convolution parameters KL and KR are parameters estimated by learning, which will be described later.

[0032] In the cost correction process (P104), the cost aggregator 40 performs a convolution operation using the convolutional neural network N200 on the cost map C200 calculated in the matching cost calculation process (P103), thereby calculating a corrected cost map C201. The size of the cost map C201 is (D+1, HR, WR, 1), and the cost for each disparity is stored at each position. The corrected cost map C201 stores costs corrected in consideration of the costs of surrounding pixels. In the convolutional neural network N200 shown in FIG. 7, the convolution operation unit 41 of the cost aggregator 40 uses a three-dimensional convolution kernel to perform convolution operations on the width, height, and depth directions of the cost map C200. The convolutional neural network N200 consists of a repeated structure of convolution, normalization, and activation operations.

[0033] Batch normalization is used as the normalization operation, and a rectified linear unit is used as the activation operation. However, batch normalization and a rectified linear unit are not used in the final layer. The convolutional neural network N200 is an encoder-decoder type network that downsamples the input cost map and then upsamples it. The kernels used in the neural network N200 are parameters estimated through learning, which will be described later.

[0034] In the disparity calculation process (P105), the cost analysis unit 51 of the matching unit 50 calculates the disparity at each pixel of the right image by analyzing the cost stored in the cost map C201 calculated in the cost correction process (P104). The cost analysis unit 51 executes the well-known SoftArgMin process as a cost analysis method. That is, the cost analysis unit 51 performs SoftMax processing on the depth direction D for each position. At this time, the cost analysis unit 51 calculates the likelihood for each disparity by inverting the sign of the cost. Thereafter, the cost analysis unit 51 calculates the disparity for each pixel by calculating the sum of the disparity value corresponding to the depth index in the cost map C201 and the likelihood calculated by the SoftMax processing.

[0035] As described above, in the feature extraction process (P102), matching cost calculation process (P103), and cost correction process (P104) shown in Fig. 3, convolution operations are performed using kernels obtained by learning. Batch normalization parameters are also used. Below, we explain the learning method used to estimate the convolution kernel and the parameters of Batch Normalization. In this learning method, a loss function is defined using the final disparity estimated in the disparity calculation process (P105) in Figure 3 and the ground truth data, and the kernel is estimated by minimizing this loss function. The absolute value of the difference between the estimated disparity and the ground truth data is used as the loss function. The cost analysis unit 51 of the matching unit 50 performs the above-described calculation on pixels that include correct value data in the estimated parallax image, and uses the average value as the loss function. The feature extraction process (P102), matching cost calculation process (P103), cost correction process (P104), and disparity calculation process (P105) are all composed of only differentiable operations, and the parameters of the feature extraction process (P102), matching cost calculation process (P103), and cost correction process (P104) are determined by defining a loss function for the disparity calculation process (P105) and minimizing it.

[0036] As described above, the environment recognition device 1 of this embodiment performs a convolution operation independently on the right image and the left image once in the matching cost calculation process (P103), and then integrates the results to calculate the matching cost. Conventionally, the matching cost was calculated by connecting the right image and the left image in the parallax direction and performing a 3D convolution operation once, but this required convolution operations the same number of times as the number of parallax search ranges. In contrast, by applying this embodiment, the number of convolution operations can be reduced, making it possible to reduce the amount of calculation.

[0037] Furthermore, in the matching cost calculation process (P103), the environment recognition device 1 of this embodiment shifts the left image from the convolution operation results of the right image and the left image, and calculates the matching cost by performing element addition processing on the shifted result. In this way, the environment recognition device 1 of this embodiment can generate the matching cost more quickly by generating the cost by shifting the image and simple addition processing without using complex calculations. Furthermore, the environment recognition device 1 of this embodiment is configured to correct the cost calculated in the matching cost calculation process (P103) in a cost correction process (P104). As a result, even if the cost calculated in the matching cost calculation process (P103) contains an error, the environment recognition device 1 of this embodiment can correct it by using surrounding costs, and can calculate disparity with higher accuracy.

[0038] The above-described operational example 1 is configured to use the kernel determined during learning in the matching cost calculation process (P103) during inference. On the other hand, it is also possible to configure the system so that the kernels determined by learning are integrated and used in the matching cost calculation process (P103) based on the processing described in Figure 8 onwards. In other words, it is also possible to configure the system so that the calculation method is different between learning and inference. Furthermore, by generating the kernel to be used during inference according to the procedure described in Figure 8 onwards, it is possible to obtain the same output results as with the configuration during learning.

[0039] FIG. 8 is a diagram illustrating the configuration and procedure of the cost calculation unit 30 during learning. In the matching cost calculation process (P103), processing is performed on the left feature map F100 and the right feature map F101. As shown in Fig. 8, the cost calculation unit 30 shifts the left feature map F100 by the number of times corresponding to the disparity search range (D+1 pix) relative to the right feature map F101, and generates left-right pairs corresponding to each disparity. The feature map combined in the channel direction without shifting is F0 (no shift), the feature map combined after a 1-pix shift is F1 (1-pix shift), and the feature map combined after a D-pix shift is FD (D-pix shift). During learning, the cost calculation unit 30 further combines these D+1 pairs in the depth direction and generates a matching cost by performing a convolution operation using the 3D convolution kernel K100.

[0040] Here, the 3D convolution kernel K100 is a kernel that has dimensions not only in width and height but also in the depth direction. In the following explanation, the size in the depth direction will be assumed to be "3". During learning, the cost calculation unit 30 generates a matching cost according to the procedure described above. The generated matching cost is used to execute the cost aggregator 40 and the matching unit 50, thereby finally calculating the disparity. Then, the cost analysis unit 51 defines a loss function based on the disparity estimated in the disparity calculation process (P105), and determines the value of the 3D convolution kernel K100 in the matching cost calculation process (P103) by minimizing this function. The configuration used during inference is that shown in Fig. 6. That is, the cost calculation unit 30 determines the convolution parameters KL and KR shown in Fig. 6 using the value of the three-dimensional convolution kernel K100 calculated during learning.

[0041] 9, 10, and 11 are diagrams showing a method for determining the convolution parameters KL and KR using the three-dimensional convolution kernel K100 obtained during learning. As shown in Figure 9, the depth size of the 3D convolution kernel K100 is "3", so it is a convolution kernel for three left-right pairs. The 3D convolution kernel K100 can be considered as an independent convolution kernel for each feature map (F0R, F1R, F2R, F0L, F1L, F2L). F0R, F1R, and F2R indicate the right feature maps of the three left-right pairs, and F0L, F1L, and F2L indicate the left feature maps of the three left-right pairs. Here, F0R, F1R, and F2R are the same feature maps, whereas F0L, F1L, and F2L are maps where F1L is shifted by 1 pix from F0L, and F2L is shifted by 2 pix from F0L.

[0042] Of the three-dimensional convolution kernel K100, the convolution kernels for F0R, F1R, and F2R are designated as K0R, K1R, and K2R. Similarly, of the three-dimensional convolution kernel K100, the convolution kernels for F0L, F1L, and F2L are designated as K0L, K1L, and K2L. The cost calculation unit 30 combines the kernels K0R, K1R, and K2R to generate a two-dimensional convolution kernel K101 for the right feature map. The cost calculation unit 30 also combines the kernels K0L, K1L, and K2L to generate a two-dimensional convolution kernel K102 for the left feature map. The generated two-dimensional convolution kernel K101 for the right feature map and two-dimensional convolution kernel K102 for the left feature map correspond to the convolution parameters KR and KL, respectively, used during inference shown in FIG. 6.

[0043] FIG. 10 shows how to generate a two-dimensional convolution kernel K101 for the right feature map. The cost calculation unit 30 generates a two-dimensional convolution kernel K101 for the right feature map by performing element addition on the convolution kernels K0R, K1R, and K2R.

[0044] FIG. 11 shows how to generate a two-dimensional convolution kernel K102 for the left feature map. In the process of generating the two-dimensional convolution kernel K102 for the left feature map, the cost calculation unit 30 shifts the convolution kernel K0L by one pixel to the left with respect to the convolution kernel K1L and performs element addition. That is, the cost calculation unit 30 adds each element in the second column of the convolution kernel K0L to each element in the first column of the convolution kernel K1L. Next, the cost calculation unit 30 shifts the convolution kernel K2L by one pixel to the right and performs element addition on the convolution kernel K1L that has been added to the convolution kernel K0L. That is, the cost calculation unit 30 adds each element in the first column of the convolution kernel K2L to each element in the second column of the convolution kernel K1L. Note that in the element addition process for the first column of the convolution kernel K0L and the third column of the convolution kernel K2L, elements are added to the 0th and 4th columns of the convolution kernel K1L, which do not exist. In this case, the cost calculation unit 30 performs element addition processing, assuming that all elements in the 0th and 4th columns of the convolution kernel K1L are 0. The two-dimensional convolution kernel K102 for the left feature map generated by the above-described procedure is a kernel that has a larger size in the horizontal direction than the sizes of the convolution kernels K0R, K1R, and K2R.

[0045] These generated two-dimensional convolution kernels K101 for the right feature map and K102 for the left feature map are used as the convolution parameters KR and KL, respectively, shown in Figure 6. This makes it possible to obtain the same output as the configuration during training using a different calculation method.

[0046] The environment recognition device 1 of this embodiment integrates the 3D convolution kernels obtained during learning and uses them as 2D convolution kernels during inference. In this way, the environment recognition device 1 of this embodiment can execute a model learned using the 3D convolution kernels during inference without the need for re-learning and with a smaller amount of calculation. Furthermore, when comparing the number of parameters of the 3D convolution kernels used during learning with the number of parameters used during inference, the number of parameters used during inference is smaller. This makes it possible to perform disparity calculation with less memory usage.

[0047] In the operation example 1 described above, the matching cost calculation process (P103) by the cost calculation unit 30 is performed to generate a cost map for the entire image at once. However, the process can also be performed for each partial region, as shown in the flowchart of FIG. 12, which will be described next.

[0048] That is, as shown in Fig. 12, in the matching cost calculation process (P301), the same processes as those of P201 to P205 shown in the flowchart of Fig. 5 are executed for a certain number of consecutive rows. The individual processes (P201, P202, P203, P204, P205) executed for a certain number of rows are as described in Fig. 5. The cost calculation unit 30 generates the final cost map by repeating the processes of P201 to P205 for every certain number of rows.

[0049] In this way, the cost calculation unit 30 generates a cost map for each set of consecutive rows and divides the area to be processed, thereby reducing the amount of memory used. Furthermore, in an execution environment that allows distributed inference, the cost calculation unit 30 performs inference for each set of consecutive rows and integrates the results, thereby reducing the inference time.

[0050] In the feature extraction process (P102) described so far (Operation Example 1), it is assumed that the size of the input image and the size of the output feature map are equal. On the other hand, the feature extraction unit 20 can also reduce the size of the output feature map compared to the size of the input image. The feature extraction unit 20 can output a feature map smaller in size than the input image by performing a convolution operation with a stride of 2 or more using the convolution operation unit 21 in the convolutional neural network (N100). For example, the feature map output by the convolution operation unit 21 performing convolution with a stride of 2 twice will be 1 / 4 the size of the input image. That is, 1 pix in the output feature map corresponds to 4 pix in the input image. In this case, if the disparity search range for the input image is 64 pix, the cost calculation unit 30 executes the cost calculation process 16 pix times, which corresponds to 1 / 4 of that range, in the matching cost calculation process (P103). In this way, the feature extraction process of this embodiment can be modified to output feature maps of various sizes.

[0051] Furthermore, in the cost correction process (P104) explained so far (Operation Example 1), the process was explained assuming that a cost map of the same size as the input cost map is output, but it is also possible to change the process so that a cost map that is larger or smaller than the input cost map is output. Specifically, when a cost map larger than the input is to be output, the convolution operation unit 41 in the convolutional neural network (N200) can output a large-sized feature map by setting a stride larger than the number of strides used for encoding at the time of decoding, whereas when a cost map smaller than the input is to be output, the convolution operation unit 41 in the convolutional neural network (N200) can output a small-sized feature map by setting a stride smaller than the number of strides used for encoding at the time of decoding.

[0052] Furthermore, we have explained a method in which the 3D convolution kernels determined during learning are integrated and used during inference using the processes shown in Figures 8 to 11. However, taking into account the influence of padding in the horizontal and parallax directions on the convolution calculation, the following modifications may be made. In the following explanation, we assume that the size of the 3D convolution kernel is 3 pixels in each of the width, height, and depth directions. That is, when generating a pair of left and right feature maps during learning, the cost calculation unit 30 trims the left and right ends of the right feature map (F101) by 1 pixel. Then, the cost calculation unit 30 sets the size of the right feature map (F101) to WR-2. Thereafter, the cost calculation unit 30 generates a pair of the left feature map (F100) and the right feature map (F101). As a result, the size of the left feature map is WL=WR+D, so D+3 pairs are generated. Then, the cost calculation unit 30 performs 3D convolution processing on the D+3 pairs. Then, the cost calculation unit 30 trims the left and right ends of the output cost map by 1 pixel in the width direction, and trims one minimum and one maximum feature map in the depth direction.

[0053] The size of the output feature map is thus (D+1, WR-4, H). During inference, the cost calculation unit 30 trims the left and right ends of the input right feature map (F101) by 1 pixel each, setting the size of the right feature map (F101) to WR-2. Thereafter, the cost calculation unit 30 performs convolution operations on the right feature map (F101) and the left feature map (F100) using the integrated kernel. The cost calculation unit 30 then further trims each of the left and right ends of the right feature map (F111) by 1 pix, setting the size of the right feature map (F111) to WR-4. Thereafter, the cost calculation unit 30 further trims each of the left and right ends of the left feature map (F110) by 2 pix, setting the size of the left feature map (F110) to WL=WR+D-4. The cost calculation unit 30 then performs element addition processing to generate a cost map of (D+1, WR-4, H). In this way, the cost calculation unit 30 can exclude cost maps affected by padding during learning during inference, and can match the calculation results during learning and inference, even for image edges and maximum and minimum disparity values.

[0054] [Example 2 of processing performed by the environment recognition device: Example of flow calculation using a monocular camera] Next, as a second example of an operation performed by the environment recognition device 1, an operation of calculating a flow (optical flow) that represents the amount of movement of pixels of the same subject from images captured at multiple times by a monocular camera will be described. In the following, we will explain the processing operation for calculating the flow from time T to time T-1 using two images: an image captured by a monocular camera at time T-1 and an image captured immediately thereafter at time T. In addition, the maximum and minimum values ​​of the flow are ±3 pixels in the vertical and horizontal directions, and we will explain the process of calculating the flow within a 7 × 7 pixel area at the target pixel at time T.

[0055] FIG. 13 shows the process of flow calculation in the second operational example in this embodiment. In the flow calculation, an image acquisition process (P401), a feature extraction process (P402), a matching cost calculation process (P403), a cost correction process (P404), and an optical flow calculation process (P405) are executed in this order. First, in the image acquisition process (P401), the image acquisition unit 10 acquires two images captured at time T-1 and time T. In the feature extraction process (P402), the feature extraction unit 20 generates a feature map from each image captured at time T-1 and time T acquired in the image acquisition process (P401) using a convolutional neural network.

[0056] The difference from the feature extraction process in disparity calculation shown in Fig. 4 is that the feature extraction unit 20 calculates feature maps for two images taken at time T-1 and time T, rather than for left and right images. In addition, the kernel used in the convolution process of the convolutional neural network is estimated by learning, which will be described later. In the matching cost calculation process (P403), the cost calculation unit 30 calculates the cost for each flow using the feature map F201 at time T-1 and the feature map F200 at time T, both of which were acquired in the feature extraction process (P402).

[0057] FIG. 14 is a flowchart showing the matching cost calculation process (P403) performed by the cost calculation unit 30. In the matching cost calculation process (P403), the cost calculation unit 30 performs a feature map convolution process at time T-1 (P201') and a feature map convolution process at time T (P202'). Then, the cost calculation unit 30 executes cost calculations corresponding to the candidate flows as many times as the number of candidate flows (7 × 7) (P203). In the feature map convolution process at time T-1 (P201') and the feature map convolution process at time T (P202'), the images in the left image feature map convolution process (P201) and the right image feature map convolution process (P202) described in FIG. 5 are replaced with the images at time T-1 and time T, respectively.

[0058] In cost calculation corresponding to the candidate flow (P203), the cost calculation unit 30 shifts the feature map at time T-1 by the amount of the candidate flow (P204). Then, the cost calculation unit 30 calculates the cost for the flow by performing element addition processing on the shifted feature map at time T-1 and the feature map at time T (P205). The cost calculation unit 30 repeatedly performs cost calculation corresponding to this candidate flow (P203) within the search range.

[0059] FIG. 15 shows the procedure of the matching cost calculation process (P403) of FIG. 13 performed by the cost calculation unit 30. The cost calculation unit 30 performs a convolution operation C(T-1) using a convolution parameter K(T-1) on the time T-1 feature map F201 to calculate a time T-1 feature map F211. The cost calculation unit 30 also performs a convolution operation C(T) using a convolution parameter K(T) on the feature map F200 at time T to calculate a feature map F210 at time T. After that, the cost calculation unit 30 calculates costs for the candidate flows (-3, -3) to (3, 3). Then, in generating a cost for the candidate flow (-3, -3), the cost calculation unit 30 shifts the time T-1 feature map F211 left by 3 pix and upward by 3 pix with respect to the time T feature map F210, and performs element addition processing.

[0060] The cost calculation unit 30 executes the above process in order within the range of (-3,-3) to (3,3), calculates the cost corresponding to each candidate flow, and generates the cost map C200 by linking the respective costs in the depth direction. The cost corresponding to the flow candidate is stored at each position in the cost map (C200).

[0061] For the sake of convenience, the explanation so far has not included description of feature map channels. However, each feature map has a size in the channel direction. Furthermore, as explained in the disparity calculation of Operation Example 1, the cost map C200 stores costs for each channel. Furthermore, in the element addition process of the feature map F210 at time T and the feature map F211 at time T-1, the cost calculation unit 30 shifts the feature map F211 at time T-1. As a result, the feature map F210 at time T and the feature map F211 at time T-1 do not overlap, and the element addition process may not be possible. Therefore, to prevent the cost calculation unit 30 from being unable to perform the element addition process, the feature extraction unit 20 specifies a wider range for the image at time T-1 in the feature extraction process (P402) than for the image at time T.

[0062] In the cost correction process (P404) by the cost aggregation unit 40, the convolution operation unit 41 performs convolution processing on the cost map C200 calculated in the matching cost calculation process (P403) as shown in Figure 7, thereby generating a cost map that takes into account the surrounding area. In the optical flow calculation process (P405), a feature map with a size of (H, W, 2) is generated by a convolution operation performed by the cost analysis unit 51 on the cost map corrected in the cost correction process (P404). H and W indicate the height and width of the feature map, respectively, and the flow coordinates (horizontal flow, vertical flow) are stored for each position.

[0063] As described above, in the feature extraction process (P402), matching cost calculation process (P403), cost correction process (P404), and optical flow calculation process (P405) shown in FIG. 13, convolution operations are performed using kernels determined by learning.

[0064] Next, the learning method in this case will be described. In this learning method, a loss function is defined for the flow calculated in the optical flow calculation process (P405), and the defined loss function is minimized. The loss function calculates the absolute value of the difference between the horizontal flow and vertical flow of the correct answer data for each horizontal flow and vertical flow. Therefore, the sum of the absolute values ​​of the difference between the horizontal flow and vertical flow becomes the error for each pixel. This error is calculated only for pixels that store correct answer data, and the average value is used as the loss function.

[0065] As described above, the environment recognition device 1 of this embodiment can calculate the cost for a flow by performing a convolution operation only once on the image at time T and the image at time T-1 in calculating the matching cost in flow calculation. Therefore, the amount of calculation can be reduced compared to when performing a 3D convolution operation on each flow candidate.

[0066] In the operational example 2 explained up to this point, the kernel determined during learning is used in the matching cost calculation process (P403) during inference. On the other hand, based on the process described below, it is also possible to use an integrated kernel determined by learning in the matching cost calculation process (P403). In other words, it is possible to configure the calculation method to be different between learning and inference. Furthermore, by generating a kernel to be used during inference using the process described below, it is possible to obtain the same output results as with the configuration during learning.

[0067] FIG. 16 shows the matching cost calculation process (P403) during learning. For each candidate flow, the cost calculation unit 30 creates a pair of feature maps, one for a feature map F200 at time T and the other for a feature map F201 at time T-1. Specifically, for the candidate flow (-3, -3), the cost calculation unit 30 shifts the feature map F201 at time T-1 by 3 pix to the left and 3 pix upward relative to the feature map F200 at time T, creating a feature map pair F(-3, -3). The cost calculation unit 30 generates F(-3,-3) to F(3,3) by executing this procedure a number of times corresponding to each candidate flow. Then, the cost calculation unit 30 concatenates the generated feature map pairs F(-3,-3) to F(3,3) in the depth direction and performs a convolution operation using the 3D convolution kernel (K100). The 3D convolution kernel (K100) used in this case has a size of (49,3,3). In other words, the 3D convolution kernel (K100) has a size of 3x3 in the spatial direction and a size in the depth direction that allows convolution with all feature map pairs F(-3,-3) to F(3,3).

[0068] During inference, the cost calculation unit 30 integrates and uses the three-dimensional convolution kernel (K100) used during learning. The convolution parameter K(T-1) and the convolution parameter K(T) shown in FIG. 15 are the integrated kernels used during inference. The integration method is basically a procedure similar to that of Operation Example 1. Specifically, the cost calculation unit 30 generates the convolution parameter K(T) by performing element addition of each kernel of the three-dimensional convolution kernel (K100) for the feature map F200 at time T. Furthermore, for the convolution parameter K(T-1), the cost calculation unit 30 uses each kernel of the three-dimensional convolution kernel (K100) for the feature map F201 at time T-1.

[0069] Here, the cost calculation unit 30 performs element addition of the kernels for F(-3,-3) to F(3,3) to the kernel for the feature map pair F(0,0), but shifts F(-3,-3) to F(3,3) before performing element addition. When performing element addition of F(-3,-3), the cost calculation unit 30 shifts the kernel by 3 pix to the left and 3 pix upward before performing element addition. By performing the above process for all kernels, the cost calculation unit 30 generates the convolution parameter K(T-1). The difference from the operation example 1 is that the kernel is shifted not only in the left-right direction but also in the up-down direction.

[0070] In the environment recognition device 1 of this embodiment, the 3D convolution kernels calculated during learning are integrated and used as 2D convolution kernels during inference. In this way, a model learned using a 3D convolution kernel can be executed with a smaller amount of calculation without re-learning during inference. Furthermore, when the number of parameters of the 3D convolution kernel used during learning is compared with the number of parameters used during inference, the number of parameters used during inference is smaller. This makes it possible to perform disparity calculation with a smaller amount of memory usage.

[0071] [Variations] Although the present invention has been described above with reference to operation examples 1 and 2 according to the embodiments, the present invention is not limited to the above-described embodiments. For example, the above-described embodiment is configured to include the cost aggregator 40. However, the disparity or flow may be calculated without including the cost aggregator 40.

[0072] For example, in the first operational example, the parallax calculation corresponding to the right image of the left and right images of the stereo camera is executed, but the parallax calculation corresponding to the left image can also be used. In addition, in Operation Example 2, calculation of the flow corresponding to the image of the monocular camera at time T-1 and the image at time T is executed, but it is also possible to calculate the flow of an image of another time T-2 before time T-1 and an image at time T. In other words, Operation Example 2 can be applied to calculation of the flow between any frames. Furthermore, in the feature extraction section (P102), matching cost calculation process (P103), and cost correction process (P104), the network structure and kernel size described in the figure are merely specific examples, and may be other than the above-mentioned structure or size.

[0073] Furthermore, the embodiments described so far have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. 2, the application example of the present invention is also an example of environment recognition from images captured by a camera mounted on a vehicle and vehicle control, but the present invention may also be applied to cases where environment recognition is performed from cameras mounted on various moving bodies such as other vehicles. The use of the environment recognition results is also not limited to vehicle control. 1 and 2, only the control lines and information lines considered necessary for explanation are shown, and not all control lines and information lines in the product are necessarily shown. In reality, it can be assumed that almost all components are interconnected. Furthermore, the flow of operational processing in the flowcharts shown in FIG. 3 and the like is also an example, and as long as the processing results are the same, the order of some of the processing may be changed or multiple processes may be executed simultaneously.

[0074] Furthermore, the processing performed within the environment recognition device of the present invention is performed by executing a program (software), and for example, an existing computer (information processing device) may be made to function as the environment recognition device of the present invention by installing a program that executes the processing described in the embodiment. In this case, the program information can be stored in various recording media such as memory, a hard disk, an SSD (Solid State Drive), an IC card, an SD card, an optical disk, etc. In addition, in the configuration shown in Figure 2, the environment recognition device is an example configured as a computer executed under the control of a CPU, but some or all of the functions performed by the environment recognition device may be realized by dedicated hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). [Explanation of symbols]

[0075] 1...environment recognition device, 2...vehicle control device, 3L...on-board camera, 4...computer, 4a...CPU, 4b...work memory, 4c...storage unit, 4d...interface, 5...vehicle, 10...image acquisition unit, 11...image storage unit, 12...image storage unit, 20...feature extraction unit, 21...convolution operation unit, 22...feature extraction parameter storage unit, 30...cost calculation unit, 31...image convolution operation unit, 32...image convolution operation unit, 33...parameter storage unit, 34...parameter storage unit, 35...integration unit, 40...cost aggregation unit, 41...convolution operation unit, 42...feature extraction parameter storage unit, 50...matching unit, 51...cost analysis unit

Claims

1. an image acquisition unit that acquires a first image and a second image; a feature extraction unit that obtains a first feature map obtained by obtaining feature amounts for a plurality of partial regions of the first image, and a second feature map obtained by obtaining feature amounts for a plurality of partial regions of the second image; a cost calculation unit that calculates a matching cost that represents the likelihood of a correspondence relationship between each partial region by performing a convolution operation on the first feature map and the second feature map; a matching unit that determines a correspondence between a first image and a second image by analyzing the matching cost calculated by the cost calculation unit, the cost calculation unit performs a convolution operation using a first parameter and a second parameter for cost calculation on the first feature map and the second feature map, respectively, to obtain a first convolution operation result and a second convolution operation result, and calculates the matching cost by integrating the first convolution operation result and the second convolution operation result; the first parameter for cost calculation and the second parameter for cost calculation are convolution kernels determined by learning that minimize a loss function based on the correspondence relationship; Environment recognition device.

2. the first image and the second image are images acquired at the same time by a first camera and a second camera, respectively; The correspondence determined by the matching unit is a parallax The environment recognition device according to claim 1 .

3. When integrating the first convolution operation result and the second convolution operation result, the cost calculation unit shifts either the first convolution operation result or the second convolution operation result in a left-right direction and performs element addition. The environment recognition device according to claim 2 .

4. the first image and the second image are images acquired by a single camera at different times, The correspondence determined by the matching unit is a flow that expresses the amount of movement. The environment recognition device according to claim 1 .

5. When integrating the first convolution operation result and the second convolution operation result, the cost calculation unit shifts one of the first convolution operation result and the second convolution operation result in an up / down or left / right direction and performs element addition. The environment recognition device according to claim 4 .

6. one of the first parameter for cost calculation and the second parameter for cost calculation is obtained by element-adding a part of the convolution parameters estimated during the learning without shifting them in a spatial direction, The other of the first parameter for cost calculation and the second parameter for cost calculation is obtained by shifting a part of the estimated convolution parameter in the spatial direction and adding elements during the learning. The environment recognition device according to claim 1 .

7. The cost calculation unit calculates the matching cost for each of a certain number of rows. The environment recognition device according to claim 1 .

8. a cost aggregator that corrects the matching cost; The matching unit determines a correspondence relationship for the matching cost corrected by the cost aggregator. The environment recognition device according to any one of claims 1 to 7.

9. an image acquisition process for acquiring a first image and a second image; a feature extraction process for obtaining a first feature map obtained by obtaining feature amounts for a plurality of partial regions of the first image, and a second feature map obtained by obtaining feature amounts for a plurality of partial regions of the second image; a cost calculation process for calculating a matching cost that expresses the likelihood of a correspondence relationship between each partial region by performing a convolution operation on the first feature map and the second feature map; a matching process for determining a correspondence between the first image and the second image by analyzing the matching cost calculated by the cost calculation process, the cost calculation process performs a convolution operation using a first parameter and a second parameter for cost calculation on the first feature map and the second feature map, respectively, to obtain a first convolution operation result and a second convolution operation result, and calculates the matching cost by integrating the first convolution operation result and the second convolution operation result; the first parameter for cost calculation and the second parameter for cost calculation are convolution kernels determined by learning that minimize a loss function based on the correspondence relationship; Environmental recognition method.

Images