Environmental recognition device and environmental recognition method

Abstract

To provide an environment recognition device and an environment recognition method which allow a depth of a visual field non-overlapping region to be estimated with high accuracy.SOLUTION: The environment recognition device comprises: an image acquisition unit which acquires an image captured by a camera; a first depth calculation unit which calculates a first depth in a first region being a region partially overlapping or adjacent to a field of view of the camera; and a second depth calculation unit which calculates a second depth in a second region being a region not included in the first region in the field of view of the camera by using the first depth in the first region and the image captured by the camera.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an environmental recognition device and an environmental recognition method that perform environmental recognition using information from a camera.

Background Art

[0002] In the realization of preventive safety functions and autonomous driving, three-dimensional sensing is important, and by using LiDAR or a stereo camera, three-dimensional high-precision measurement becomes possible. However, in the case of a stereo camera, there are regions where the fields of view overlap and regions where they do not overlap (monocular vision), and generally, the depth accuracy in the non-overlapping field of view (monocular vision) region is lower than the depth accuracy in the overlapping field of view region.

[0003] Regarding this point, Patent Document 1 discloses a depth estimation method using a stereo camera. Here, when an object is imaged across a region where the fields of view overlap and a region where they do not overlap (monocular vision), it is proposed to use the depth measured in the overlapping field of view region as the distance to the object.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the case of Patent Document 1, it is possible to improve the distance accuracy of an object imaged across two regions. On the other hand, it cannot be applied to an object imaged only in the non-overlapping field of view region (i.e., an object that is not imaged across). Also, the distance in the overlapping field of view region is directly used as the distance to the object, and if the distance in the overlapping field of view region is incorrect, there is a problem that the estimation accuracy decreases.

[0006] Based on the above, the present invention aims to provide an environmental recognition device and an environmental recognition method that can accurately estimate the depth of non-overlapping areas of the field of view. [Means for solving the problem]

[0007] Based on the above, the present invention is characterized by comprising "an image acquisition unit that acquires an image captured by a camera; a first depth calculation unit that calculates a first depth in a first region which is a region that partially overlaps with or is adjacent to the camera's field of view; and a second depth calculation unit that uses the first depth in the first region and the image captured by the camera to calculate a second depth in a second region which is a region of the camera's field of view that is not included in the first region."

[0008] Furthermore, the present invention is described as "an environmental recognition method characterized by obtaining two-dimensional and three-dimensional information about the environment, determining the first depth and first depth feature quantities of the first region in the environment from the three-dimensional information, determining the two-dimensional information feature quantities for the second region other than the first region in the environment, determining the correlation between the two-dimensional information feature quantities and the first depth feature quantities, and calculating the second depth in the second region using the two-dimensional information feature quantities modified according to the correlation."

[0009] Furthermore, the present invention is characterized by comprising "an input unit for obtaining two-dimensional and three-dimensional information about the environment; a first depth calculation unit for determining the first depth of a first region in the environment from the three-dimensional information; and a second depth calculation unit for determining the feature quantities of the first depth, determining the feature quantities of the two-dimensional information for a second region other than the first region in the environment, determining the degree of correlation between the feature quantities of the two-dimensional information and the feature quantities of the first depth, and calculating the second depth in the second region using the feature quantities of the two-dimensional information modified according to the degree of correlation." [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an environmental recognition device that can accurately estimate the depth of non-overlapping areas of the field of view. [Brief explanation of the drawing]

[0011] [Figure 1] A diagram showing a schematic configuration example of an environmental recognition device according to an embodiment of the present invention. [Figure 2] A diagram showing an example of a camera image and an overlaid image. [Figure 3] A diagram showing an example of the processing flow of an environmental recognition device according to an embodiment of the present invention. [Figure 4] This diagram illustrates the concept of depth feature extraction processing in the first feature calculation unit. [Figure 5] This diagram illustrates the concept of image feature extraction processing in the second feature calculation unit. [Figure 6] A diagram showing specific examples of the relevance calculation process and feature integration process related to Example 1. [Figure 7] A diagram illustrating the concept of calculating depth in non-overlapping fields of view. [Figure 8] This figure shows an example of the relevance calculation process and feature integration process according to Embodiment 2 of the present invention. [Figure 9] This figure shows an example of the relevance calculation process and feature integration process according to Embodiment 3 of the present invention. [Figure 10] This figure shows a schematic configuration example of an environmental recognition device according to Embodiment 4 of the present invention. [Figure 11] A figure showing an example of a camera and superimposed image according to Embodiment 4 of the present invention. [Modes for carrying out the invention]

[0012] The embodiments of the present invention will be described below with reference to the drawings. In this invention, we deal with overlapping regions where 3D information can be obtained and non-overlapping regions where 2D information can be obtained. As a means of obtaining 3D information, there are cases where multiple monocular cameras or a stereo camera (composed of multiple monocular cameras) is used, and cases where a combination of LiDAR and a monocular camera is used. Therefore, embodiments 1, 2, and 3 will describe the former embodiments, and embodiment 4 will describe the combination of LiDAR and a monocular camera. [Examples]

[0013] FIG. 1 is a diagram showing a schematic configuration example of an environmental recognition device according to an embodiment of the present invention. The environmental recognition device 1 is mounted on, for example, a vehicle, obtains image information D1 and D2 from cameras CS (CS1, CS2) on the vehicle, and finally measures depth D3 (D3a, D3b) from the images. The measured depth D3 (D3a, D3b) is given to the vehicle control device 7 and is used to obtain the distance from the vehicle to the target object and perform vehicle control.

[0014] In this case, the camera CS for obtaining image information is a plurality of monocular cameras or a stereo camera (composed of a plurality of monocular cameras), and the superimposed image D of the image information D1 and D2 obtained from the camera CS is as illustrated in FIG. 2.

[0015] FIG. 2 is a diagram showing an example of a camera and a superimposed image. In the superimposed image D of FIG. 2, the image regions R1 and R2 are regions on the image D1 captured by the right front camera CS1 mounted on the vehicle, and the image regions R2 and R3 are regions on the image D2 captured by the left front camera CS2 mounted on the vehicle.

[0016] This superimposed image D is formed by combining the monocular images D1 and D2 grasped by the monocular left and right cameras C1 and C2. The left and right image regions R1 and R3 are non-overlapping regions of the two images, and the central image region R2 is the overlapping region of the two images. As a result, in the central overlapping region R2, a stereo region is formed and three-dimensional information can be obtained, so that precise depth measurement is possible as is well known. On the other hand, since the left and right non-overlapping regions R1 and R3 are two-dimensional information, precise depth measurement is difficult.

[0017] In order to eliminate the non-overlapping regions R1 and R3, methods such as using a wide-angle camera or arranging a large number of cameras around the vehicle can be considered. However, since these methods are inevitably expensive, in the present invention shown in FIG. 1, depth measurement is made possible by image information processing for the left and right non-overlapping regions R1 and R3 of FIG. 2.

[0018] In the environmental recognition device 1 shown in Figure 1, the image acquisition unit 2 first obtains image information D1 and D2 from the cameras CS (CS1, CS2) on the vehicle. The image information D1 and D2 are provided to the first depth calculation unit 3A and the second depth calculation unit 3B. The first depth calculation unit 3A calculates the depth D3a in the overlapping region R2 in Figure 2, and the second depth calculation unit 3B calculates the depth D3b in the non-overlapping regions R1 and R3 in Figure 2.

[0019] In the processing of the first depth calculation unit 3A, the depth D3a is determined for the stereo region R2 by 3D information processing. The depth D3a can be determined by performing a well-known process here. For example, the depth D3a can be calculated by using known stereo matching (searching the left image based on the right image of the left and right cameras and determining the most similar position). Alternatively, the depth D3a can be calculated from the two left and right images using a known deep learning model.

[0020] In the processing of the second depth calculation unit 3B, the depth D3b for the left and right non-overlapping regions R1 and R3 in Figure 2 is calculated as follows. In this process, first, the first feature calculation unit 4a calculates the feature Pa of the depth D3a of the overlapping region R2 obtained in the first depth calculation unit 3A, and then the second feature calculation unit 4b calculates the feature Pb of the images of the non-overlapping regions R1 and R3.

[0021] Specifically, for example, the first feature calculation unit 4a performs a convolution operation on the depth image of the non-overlapping region R2 calculated by the first depth calculation unit 3A to calculate the first feature Pa. The kernel value used for convolution in this process is determined by the learning process described later. The kernel size and nonlinear function for convolution can be any desired values. The same applies to subsequent convolutions.

[0022] The second feature calculation unit 4b performs a convolution operation on the non-overlapping regions R1 and R3 of the images to calculate the second feature Pb. The kernel value used for convolution is determined during the learning process described later.

[0023] In the relevance calculation unit 5, the relevance Q of the first feature Pa and the second feature Pb is first calculated. The relevance Q is calculated as the dot product of the first feature Pa and the second feature Pb. Next, the relevance calculation unit 5 weights the first feature Pa based on the calculated relevance Q and adds it to the second feature Pb to obtain feature P. The relevance Q may be obtained by directly using the first feature Pa and the second feature Pb, or it may be calculated using feature P obtained by convolving the first feature Pa and the second feature Pb, respectively. The kernel value used for convolution is determined in the learning process described later.

[0024] The depth calculation unit 6 takes the feature quantities P updated by the relevance calculation unit 5 as input and performs a convolution operation to estimate the final depth D3b of the non-overlapping regions R1 and R3. The kernel values ​​used for convolution are determined during the learning process described later.

[0025] Regarding learning: The kernel used for convolution is determined during learning. For learning, ground truth depth data collected in advance by LiDAR is used. The values ​​of the kernel used by the first feature calculation unit 4a, the second feature calculation unit 4b, the relevance calculation unit 5, and the depth calculation unit 6 are updated so that the absolute value between the depth calculated by the depth calculation unit 6 and the ground truth depth collected by LiDAR is minimized.

[0026] Figure 3 shows an example of the processing flow of an environmental recognition device according to an embodiment of the present invention. However, it is assumed that the kernel used for convolution in the subsequent processing has been trained in advance and determined so that the difference between the estimated result and the correct depth is minimized.

[0027] In this process, first, in processing step S100, the image acquisition unit 2 acquires two images of information D1 and D2. Next, in processing step S101, the first depth calculation unit 3A performs stereo matching using, for example, the two images as a depth calculation process for the overlapping field of view region R2. This estimates the depth D3a in the overlapping field of view region R2.

[0028] In processing step S102, the first feature calculation unit 4a performs depth feature extraction processing on the overlapping field of view region R2. Figure 4 shows the concept of depth feature extraction processing, where the first feature Pa is calculated by applying a convolutional neural network NNWA to the depth image (first depth image) in the overlapping field of view region R2 as input.

[0029] In processing step S103, the second feature calculation unit 4b performs feature extraction processing on non-overlapping regions R1 and R3. Figure 5 shows the concept of image feature extraction processing, where images in non-overlapping regions R1 and R3 (non-overlapping region images) are taken as input, and the convolutional neural network NNWB is applied to calculate the second feature Pb.

[0030] The second feature vector Pb is calculated for the non-overlapping regions R1 and R3, respectively. Note that the convolutional neural network NNWA used in depth feature extraction and the convolutional neural network NNWB used in image feature extraction have different configurations.

[0031] In processing step S104, the correlation calculation unit 5 calculates the correlation Q between the first feature Pa and the second feature Pb. The correlation Q is calculated as the dot product of the first feature Pa and the second feature Pb.

[0032] In processing step S105, the relevance calculation unit 5 weights the first feature Pa based on the calculated relevance Q and adds it to the second feature Pb to obtain feature P. The relevance Q may be obtained by directly using the first feature Pa and the second feature Pb, or it may be calculated using features obtained by convolving the first feature Pa and the second feature Pb, respectively.

[0033] In processing step S106, the depth calculation unit 6 performs depth calculation processing for the non-overlapping field regions R1 and R3. Here, for example, the updated feature quantity P from the relevance calculation unit 5 is used as input, and a convolution operation is performed to estimate the final depth D3b of the non-overlapping field regions R1 and R3.

[0034] Figure 6 shows specific examples of the relevance calculation process (processing step S104) and feature integration process (processing step S105) in the flow of Figure 3. In Figure 6, the second feature Pb obtained from the R1 image among the non-overlapping regions is shown in the upper right, and the first feature Pa obtained from the depth of the overlapping region R2 is shown in the upper left. The following explanation will focus on the process for R1, but the same process can be performed on R3.

[0035] In Figure 6, the information obtained from the first feature Pa, which is derived from the depth of the overlapping region R2, is convolved using different kernels for each element (f1...fn) from the top left to the bottom right, resulting in the values ​​(v1...vn) and keys (k1...kn).

[0036] Figure 6 describes a method for updating the features of a target pixel Pix in the non-overlapping region R1 image. A convolution operation is performed once on the feature Pb of the target pixel Pix to obtain query q2. Using the relationship between query q2 and the key (k1···kn), the correlation between query q2 and the first feature Pa is finally obtained as f2. A similar process is repeated with different target pixels Pix, and finally the features are updated for the entire second feature Pb, and the feature P is obtained.

[0037] The specific process is explained below using mathematical formulas. Here, since the pattern shown by the first feature Pa contains depth information, the procedure for calculating the correlation between the second feature Pb and a certain target pixel Pix of the first feature Pa is shown, and the feature of the target pixel is updated.

[0038] Therefore, first, a 1x1 convolution is performed on the second feature Pb to calculate the query q2. On the other hand, a 1x1 convolution is also performed on all of the first feature Pa's features (f1...fn, total number n) to calculate the value v1 and key k1. Here, the convolution kernels used for v1 and k1 are different. Performing this operation calculates v1...vn and k1...kn. Next, the inner product of each ki (i=1...n) with q2 is calculated. At this time, using a predetermined constant C, ki' (i=1...n) is calculated according to equation (1). Here, the * operator is the inner product. This ki' represents the degree of association Q between the first feature Pa and the second feature Pb. [Mathematics 1] ki' = ki * q² / C (1) Next, we perform normalization by calculating ai(i=1…n) according to equation (2) so that the sum of the correlations Q is 1. Here, exp is exponential. [Math 2] ai = exp(ki') / Σ j exp(kj') (2) Next, we use ai and vi to calculate si as shown in equation (3) below. Since ai is a scalar and vi is a vector, si is also a vector. [Math 3] si = ai * vi (3) Then, r1 is calculated according to equation (4). [Math 4] r1 = Σ i si (4) Finally, update q2 according to equation (5). [Number 5] f2 = q2 + r1 (5) In other words, the above process calculates the correlation (normalized a1...an) between the target pixel Pix of the second feature Pb and the first feature Pa, weights the first features (v1...vn) based on this correlation, and adds them to the second feature q2. Furthermore, although the above calculation was performed only for a specific target pixel Pix, the correlation calculation process and the feature integration process perform the same calculation for all pixels of the second feature. In this case, v1...vn and k1...kn, which are calculated from the first features (f1...fn), are not changed. That is, when performing the above calculation on different second features, the value of q2 will change, but vi and ki will use the same values.

[0039] Referring to Figure 7, the depth calculation process for the non-overlapping field of view regions (processing step S106) in Figure 3 is described as follows: Here, the features updated in the relevance calculation process and the feature integration process are used as input, and the convolutional neural network NNWC is executed to calculate the depth D3b in the non-overlapping field of view regions R1 and R3.

[0040] In the present invention described above, it is possible to reflect the depth information of 3D information in the 2D image information of the non-overlapping fields R1 and R3. As a specific example, let us assume that the captured environmental information consists of clouds in the sky, trees, and the ground. In this case, the information of the clouds in the sky, trees, and the ground each has its own directionality and is reflected in the first feature quantity Pa and the second feature quantity Pb ​​as vectors of different magnitudes and directions.

[0041] In this case, the images from cameras C1 and C2 capture the clouds in the sky, trees, and the ground, and the key (k1···kn) of the first feature Pa, which is aggregated to depth, also contains an information sequence that includes the depth of the clouds in the sky, trees, and ground. In contrast, let's assume that the target pixel Pix of the second feature Pb, which is 2D image information, is the region of the clouds in the sky.

[0042] However, when both the serial information and the target pixel Pix are parts of the sky clouds, the directions of their respective vector information will be the same. Conversely, when the serial information is trees and the ground, the directions of their respective vector information will be different. As a result of the dot product of the vectors, the former value will be larger and the latter value will be smaller. Consequently, the relevance of this target pixel Pix strongly reflects the depth of the sky information, and therefore is ultimately captured as a feature.

[0043] In this embodiment, depth information D3a of the overlapping field of view region R2 is input, and the depth D3b of the non-overlapping field of view regions R1 and R3 is estimated. By utilizing not only the image of the overlapping field of view region R2 but also the depth information D3a of the overlapping field of view region R2, it is possible to estimate the depth with high accuracy. Furthermore, instead of directly using the depth D3a of the overlapping field of view region R2 as the depth D3b of the non-overlapping field of view regions R1 and R3, it is used to update the feature quantities of the overlapping field of view region R2. This reduces the impact of errors even if errors occur in the depth information D3a of the overlapping field of view region R2.

[0044] In this embodiment, the degree of relevance is calculated and feature data is integrated. When estimating the depth of the road surface in non-overlapping field-of-view regions R1 and R3, information on the depth of the sky in overlapping field-of-view regions is not useful. By calculating the degree of relevance as in the present invention and integrating features based on that degree of relevance, the use of unnecessary features can be reduced, and further accuracy can be achieved.

[0045] In this example, the degree of relevance was calculated using the dot product. Since the dot product can be implemented using the sum-of-products operation on vectors, it can be processed quickly. This allows for the calculation of relevance with less computation. [Examples]

[0046] In Example 1, the relevance calculation process and feature integration process calculated the relevance Q for all regions of the first feature Pa. In contrast, in Example 2, the relevance is calculated only for the first feature Pa that is on the same line as the target pixel Pix of the second feature Pb.

[0047] Figure 8 shows an example of the relevance calculation process and feature integration process according to Example 2. Here, the second feature Pb and the first feature Pa are compared, for example, when the target pixel Pix on the second feature Pb is located on the first line, and the element information of the first feature Pa compared with it is only compared with the element information sequence on the same first line. The figure shows the case where there are 8 pieces of element information on the same line.

[0048] As shown in Example 2, the computational complexity can be reduced by limiting the number of subjects when calculating the degree of relevance. This makes it possible to calculate the degree of relevance with even less computation. [Examples]

[0049] In Examples 1 and 2, the relevance was calculated for images taken at the same time, and the features were integrated. Alternatively, previously acquired information can be used as the depth calculated in the overlapping field of view region.

[0050] Figure 9 shows the calculation of relevance using depth information of overlapping field-of-view regions acquired in the past, and the method of integrating features. The current time is t, and here the case where depth information from the previous frame t-1 is used is illustrated. From right to left, the current second feature, the current first feature, and the past first feature are shown. Here, the past first feature is calculated by aligning past depth information with the current time using the vehicle's speed and yaw rate and past depth information, and then using the depth feature extraction process shown in Figure 4. The main difference from Examples 1 and 2 is that the relevance of the current second feature q2 is calculated not only for the current first feature but also for the past first feature. Furthermore, when normalizing the relevance according to equation (2), the normalization process is performed so that the sum of the relevances for the present and past becomes 1.

[0051] As shown in Example 3, by utilizing past depth information, a wider range of depth information can be used, enabling highly accurate depth estimation. [Examples]

[0052] In Examples 1, 2, and 3, it was assumed that the camera CS for obtaining image information consisted of multiple monocular cameras or a stereo camera (composed of multiple monocular cameras).

[0053] Alternatively, a combination of LiDAR and a monocular camera can be used. For example, as shown in Figure 10, the system can be changed to a monocular camera C1 and LiDAR. The first depth calculation unit 3A processes the LiDAR point cloud information D4 to obtain the depth D3a, and the first feature calculation unit 4a uses this to obtain the feature quantity Pa of the depth D3a of the overlapping region R2 obtained by the first depth calculation unit 3A.

[0054] In this case, the relationship between the LiDAR point cloud information and the imaging area of ​​the monocular camera C1 is as shown in Figure 11. A portion of the imaging area D4 of the monocular camera C1 is covered by the LiDAR point cloud information.

[0055] In the embodiments of the present invention described above, the relevance calculation and the integration of features from overlapping and non-overlapping fields were performed only once. However, these processes can also be performed multiple times. That is, after performing the depth feature extraction process, image feature extraction process, relevance calculation process, and feature integration process shown in Figure 3, the depth feature extraction process, image feature extraction process, relevance calculation process, and feature integration process can be performed again. In the second depth feature extraction process, the output of the first depth feature extraction process becomes the input, and in the second image feature extraction process, the output of the first feature integration process becomes the input. [Explanation of symbols]

[0056] 1:Environment recognition device 2: Image acquisition unit 3A: First depth calculation section 3B: Second depth calculation section 4a: First Feature Calculation Unit 4b: Second feature calculation unit 5: Relevance calculation unit 6: Depth calculation part 7: Vehicle Control Unit

Claims

1. An image acquisition unit that acquires images captured by the camera, A first depth calculation unit calculates a first depth in a first region which is a region that partially overlaps with or is adjacent to the field of view of the camera, The system includes a second depth calculation unit that uses the first depth in the first region and the image captured by the camera to calculate a second depth in a second region, which is a region of the camera's field of view not included in the first region. The environmental recognition device is characterized in that the second depth calculation unit calculates the degree of correlation between the first region and the second region, and uses the information of the first depth based on the degree of correlation.

2. An image acquisition unit that acquires an image 1 captured by camera 1, A first depth calculation unit calculates a first depth in a first region which is a region that partially overlaps with or is adjacent to the field of view of the camera, The system includes a second depth calculation unit that uses the first depth in the first region and the image captured by the camera to calculate a second depth in a second region, which is a region of the camera's field of view not included in the first region. The environmental recognition device is characterized in that the first depth calculation unit calculates the first depth for the first region within the image using information from LiDAR.

3. An environmental recognition device according to claim 1, The image acquisition unit acquires multiple images captured by multiple cameras, The first depth calculation unit defines the region where the fields of view of the multiple cameras overlap as the first region, and calculates the first depth from the multiple images. The environmental recognition device is characterized in that the second depth calculation unit calculates the second depth in the second region captured by only one of the plurality of cameras.

4. An environmental recognition device according to claim 1, The aforementioned relevance is calculated by the dot product of the first feature calculated by the convolution operation on the first depth and the second feature calculated by the convolution operation on the portion of the image included in the second region. An environmental recognition device characterized in that the first feature quantity is weighted by the degree of relevance and added to the second feature quantity.

5. An environmental recognition device according to claim 1, The environmental recognition device is characterized in that the degree of relevance is calculated between identical lines on the image in the first region and the second region.

6. An environmental recognition device according to claim 1, The second depth calculation unit uses past values ​​of the first depth calculated by the first depth calculation unit, The environmental recognition device is characterized in that the degree of relevance is calculated using the current and past values ​​of the first depth.

7. An environmental recognition method characterized by obtaining two-dimensional and three-dimensional information about the environment, determining the first depth and feature quantities of the first depth of the first region in the environment from the three-dimensional information, determining the feature quantities of the two-dimensional information for the second region other than the first region in the environment, determining the correlation between the feature quantities of the two-dimensional information and the feature quantities of the first depth, and calculating the second depth of the second region using the feature quantities of the two-dimensional information modified according to the correlation.

8. An input unit that obtains two-dimensional and three-dimensional information about the environment, A first depth calculation unit that determines the first depth of the first region in the environment from three-dimensional information, An environmental recognition device comprising a second depth calculation unit that determines the feature quantities of the first depth, determines the feature quantities of the two-dimensional information for a second region other than the first region in the environment, determines the degree of correlation between the feature quantities of the two-dimensional information and the feature quantities of the first depth, and calculates the second depth in the second region using the feature quantities of the two-dimensional information modified according to the degree of correlation.

Images