Environment recognition device

Abstract

Disclosed is an environment recognition device which, when performing three-dimensional recognition from a two-dimensional image, eliminates the influence of distortion generated when converting two-dimensional image information into three-dimensional information and improves the accuracy of the three-dimensional recognition. The environment recognition device comprises: an image acquisition unit 100 that acquires images captured by a plurality of cameras C1-Cn mounted on a vehicle; a depth-related information acquisition unit 110 that obtains depth-related information including the depth of each pixel of the plurality of images; a feature amount extraction unit 121 that has a learning model trained to extract a feature amount in a single aggregated coordinate space by using the plurality of images and the depth-related information as inputs; and an external environment recognition unit 122 that recognizes the environment outside the vehicle on the basis of the feature amount.

Description

Environmental recognition device

[0001] The present invention relates to an environmental recognition device, and particularly to an environmental recognition device mounted on a moving body such as a vehicle to recognize the surrounding environment.

[0002] In recent years, in moving bodies such as vehicles, in order to realize automatic driving and support for a driver's driving operation, a multi-camera system has been developed that monitors the surroundings of the moving body using a plurality of cameras installed on the moving body. In the multi-camera system, image recognition processing such as object detection and segmentation is performed in each of the plurality of cameras, and the recognition results from these plurality of cameras are finally integrated by an integration processing unit and output as a recognition result.

[0003] Recently, for the purpose of improving the efficiency of recognition processing, an environmental recognition device using a deep learning model that performs recognition processing by integrating a plurality of acquired images without performing image recognition in individual cameras has been developed. In such an environmental recognition device, since integration processing is performed with one deep learning model, there are merits such as reducing the development man-hours and the calculation load. Also, with this method, the final recognition result can be output without an integration processing unit.

[0004] On the other hand, as a technology for recognizing the environment around a vehicle using a learned model, for example, in Patent Document 1, a technology is disclosed in which an image captured by a camera is converted into a bird's-eye view image, and the bird's-eye view image is input into a learned model to detect a three-dimensional object and a drivable area.

[0005] Japanese Patent Application Laid-Open No. 2023-117203

[0006] However, in the technology described in Patent Document 1, when converting two-dimensional image information into three-dimensional bird's-eye view information, simple assumptions such as the road surface being flat are used, so distortions such as a three-dimensional object being stretched radially may occur, and there is a risk that the three-dimensional recognition accuracy by the model will decrease.

[0007] An object of the present invention is to provide an environmental recognition device that can accurately perform three-dimensional conversion of an image captured by a camera and perform highly accurate image recognition in view of such problems.

[0008] In one preferred embodiment, the environmental recognition device according to the present invention comprises: an image acquisition unit that acquires multiple images captured by multiple cameras mounted on a vehicle; a depth-related information acquisition unit that uses the multiple images to obtain depth-related information, including the depth to the subject, for each pixel corresponding to a subject captured in the multiple images; a feature extraction unit having a learning model that has learned to acquire feature quantities in a single aggregated coordinate space using image information and depth-related information obtained from the multiple images as input; and an external environment recognition unit that recognizes the external environment of the vehicle based on the feature quantities.

[0009] According to the present invention, the recognition accuracy of three-dimensional recognition tasks, such as object detection using multiple images, can be improved. Other novel features of the present invention and the technical problems solved thereby will become apparent from the description and drawings herein.

[0010] This is a schematic block diagram showing the functional configuration of one embodiment of an environmental recognition device. This is a schematic overhead view showing the imaging area captured by the camera. This is a flowchart showing an example of processing performed by the environmental recognition device. This is a schematic diagram showing an example of a bird's-eye view space. This is a schematic diagram for explaining the depth of the image. This is a schematic diagram for explaining the stereo region and the monocular region. This is a schematic block diagram showing an example of the hardware configuration of the environmental recognition device.

[0011] Hereinafter, representative embodiments of the present invention will be described with reference to the drawings. Note that the embodiments and drawings described below are illustrative examples for explaining the present invention, and have been omitted or simplified as appropriate for clarity of explanation. Furthermore, please note that the position, size, shape, and extent of each component shown in the drawings may not necessarily accurately represent them, in order to facilitate understanding of the invention.

[0012] Figure 1 is a schematic block showing the functional configuration of one embodiment of an environmental recognition device to which the present invention is applied. In the following explanation, the application to a vehicle with preventive safety functions and autonomous driving functions will be used as an example, but it may also be applied to devices other than vehicles, such as surveillance cameras.

[0013] The environmental recognition device 10 is mounted on a vehicle, such as an automobile, and connected to a plurality of cameras C1, C2, ... Cn provided on the vehicle. The environmental recognition device 10 has an image acquisition unit 100, a depth-related information acquisition unit 110, and an integrated recognition unit 120, and processes images captured by the plurality of cameras C1, C2, ... Cn (hereinafter referred to as camera images) to recognize the environment around the vehicle.

[0014] The image acquisition unit 100 uses cameras C1, C2, ... Cn to capture images of the area around the vehicle and acquires the captured camera images.

[0015] The depth-related information acquisition unit 110 uses the camera images acquired by the image acquisition unit 100 to acquire depth-related information for each pixel of a subject captured by multiple camera images taken by cameras C1, C2, ... Cn. When acquiring depth-related information, camera-related information such as intrinsic and extrinsic parameters of each camera C1, C2, ... Cn may be used. The acquisition of depth-related information may be performed by geometric calculation, or a learning model trained by machine learning or deep learning may be used. In this embodiment, the depth-related information may be information indicating the depth itself.

[0016] The integrated recognition unit 120 includes a feature extraction unit 121 and an external environment recognition unit 122, and recognizes the surrounding environment using multiple image information acquired by the image acquisition unit 100 and depth-related information acquired by the depth-related information acquisition unit 110.

[0017] The feature extraction unit 121 extracts features in a coordinate space integrated into one around the vehicle 20, using camera images captured by multiple cameras C1, C2, ... Cn acquired by the image acquisition unit 100 and depth-related information acquired by the depth-related information acquisition unit 110. The camera images used by the feature extraction unit 121 may have undergone preprocessing such as normalization in the image acquisition unit 100 beforehand, or they may have been converted into feature maps using a deep learning model such as a CNN (convolutional neural network).

[0018] The integrated coordinate space may be a bird's-eye view space or a three-dimensional space. Furthermore, the features may be defined in formats such as grid maps, voxels, or point clouds, or they may be intermediate outputs of a deep learning model. Feature extraction is performed using deep learning models such as CNNs or Transformers.

[0019] The external environment recognition unit 122 recognizes the external environment using the features extracted by the feature extraction unit 121. The recognition process performed here may be, for example, 3D object detection, segmentation, or lane detection. These recognition processes may be performed individually, or multiple recognition processes may be performed simultaneously. Furthermore, deep learning models or machine learning models may be used for the recognition processes.

[0020] Physically, the environmental recognition device 10 is configured as shown in Figure 7, comprising a computing device 11, a memory 12, an interface 13 for exchanging data with a camera and other control devices, and a bus 14 for interconnecting these components. The functions of each component described above are realized when the computing device 11 executes a control program stored in the memory 12. The computing device 11 can be a processing unit such as a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit). Some or all of the functions of these components may be realized by hardware such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0021] Figure 2 is a schematic overhead view showing the imaging area captured by the camera mounted on the vehicle in this embodiment.

[0022] In this embodiment, the vehicle 20 is equipped with a total of six cameras: three cameras C1, C2, and C3 at the front of the vehicle body, and three cameras C4, C5, and C6 at the rear. The areas captured by each camera C1 to C6 are the areas indicated by H1 to H6, respectively. Cameras C1 and C2, C1 and C3, C2 and C4, C3 and C5, C4 and C6, and C5 and C6 each form a pair and have common imaging areas that overlap with each other.

[0023] Figure 3 is a flowchart showing an example of a process performed by the environmental recognition device 10.

[0024] The environmental recognition device 10 first acquires camera images captured by cameras C1 to C6 in the image acquisition unit 100 (step S100).

[0025] The camera images acquired by the image acquisition unit 100 are passed to the depth-related information acquisition unit 110. The depth-related information acquisition unit 110 uses, for example, a trained deep learning model that takes one camera image as input and estimates the depth corresponding to each pixel, to infer depth-related information for each pixel of each camera image, taking each of the camera images acquired in step S100 as input. If there are overlapping regions with images captured by other cameras in a pair, it is sufficient to perform depth-related information inference for at least one camera image for the overlapping regions (step S110). Furthermore, it is not necessarily required to perform depth-related information inference for all regions captured by cameras C1 to C6; depth-related information may be inferred for at least two or more images.

[0026] The depth-related information for each pixel acquired by the depth-related information acquisition unit 110 is passed to the feature extraction unit 121 along with the camera image. The feature extraction unit 121 uses the depth-related information and the camera parameters of each camera C1 to C6 to convert the coordinate position of each pixel in the camera image from a camera coordinate system based on each camera to a unified vehicle coordinate system based on the vehicle 20, and acquires it as a 3D point cloud in the vehicle coordinate system. Furthermore, all the obtained point clouds are aggregated to form a point cloud in a single coordinate space around the vehicle. The information in the point cloud obtained here includes both the position coordinates in the vehicle coordinate system and the RGB values ​​as image information obtained from the camera image (step S120).

[0027] The feature extraction unit 121 then extracts features of the bird's-eye view based on the acquired 3D point cloud. First, the feature extraction unit 121 defines a grid space centered on the vehicle, maps all the points acquired in step S120 to the grid space, and obtains the bird's-eye view space.

[0028] Figure 4 is a schematic diagram showing an example of the acquired bird's-eye view space. The bird's-eye view space is the space viewed from above the vehicle 210, within a grid space 200 defined around the vehicle 210. In the bird's-eye view space, point clouds obtained corresponding to moving objects such as vehicles 220, pedestrians 230, and bicycles 240, as well as stationary objects such as curbs 250 and trees 260, which are imaged by cameras C1 to C6, are mapped onto this grid space 200 as black dots.

[0029] Next, the feature extraction unit 121 inputs the point clouds present in each grid of the acquired bird's-eye view space into a fully connected layer and obtains feature vectors corresponding to each grid. Then, a CNN convolution process is performed on the feature vectors of the entire grid obtained, and a feature map is acquired.

[0030] The feature map obtained here corresponds positionally to the defined grid space, and features are stored corresponding to each grid in the grid space. Hereafter, this feature map will be referred to as grid features (step S130).

[0031] Finally, in the external environment recognition unit 122, the acquired grid features are input to a trained deep learning model consisting of fully connected layers, and a 3D bounding box of the objects around the vehicle is output (step S140).

[0032] In this embodiment, depth-related information is estimated for each pixel from multiple camera images, and features in an integrated coordinate space are obtained using the image information and the estimated depth-related information to perform external environment recognition. This allows image information to be accurately deployed in a three-dimensional coordinate space, improving the accuracy of three-dimensional recognition. In particular, by obtaining accurate depth-related information in advance, it becomes possible to recognize the position coordinates of surrounding objects with small errors.

[0033] In the embodiment described above, depth-related information is acquired for each image, but it is also possible to acquire depth-related information by using two images obtained from a pair of cameras as input and performing stereo matching. The process when acquiring depth-related information using stereo matching is as follows.

[0034] In step S100, the environmental recognition device 10 performs parallelization processing on the camera images acquired by the image acquisition unit 100. In the parallelization processing, the images are transformed so that the projection planes of the paired cameras are placed on the same plane and the epipolar lines are made parallel. By parallelizing the images in this way before performing stereo matching, stereo matching can be performed efficiently with a small amount of computation.

[0035] In this embodiment, each of the six cameras C1 to C6 is paired with two adjacent cameras, and parallelization processing is performed between each adjacent camera. As a result, two parallelized images (parallelized images) are created from one camera image, resulting in a total of six pairs, or twelve parallelized images. In subsequent processing, the created parallelized images are processed as camera images.

[0036] In step S110, the depth-related information acquisition unit 110 obtains stereo disparity information using a deep learning model trained to output a stereo disparity image including stereo disparity, taking parallelized paired images as input. Then, the depth of each pixel is obtained as depth-related information from the obtained disparity information by geometric calculation. Stereo disparity information may also be obtained using rule-based methods such as template matching. By using such a method, the computational load when acquiring depth-related information can be reduced compared to when using a deep learning model.

[0037] Subsequently, the environmental recognition device 10 performs the processes described in steps S120 to S140 above to detect objects.

[0038] In this way, by using depth information obtained through stereo matching as depth-related information, it becomes possible to use accurate depth for object recognition, thereby improving the accuracy of the final 3D recognition.

[0039] When using stereo matching, instead of obtaining information indicating the depth itself as described above, it is also possible to obtain stereo parallax or cost volume as depth-related information.

[0040] The cost volume is obtained by calculating a value (matching cost) indicating the degree of agreement or mismatch between multiple disparity candidates at each pixel position of two paired images, and storing it in a three-dimensional array of image width, image height, and disparity. The cost volume is typically obtained as intermediate information when calculating stereo disparity.

[0041] When the cost volume is used as depth-related information, the depth-related information acquisition unit 110 acquires the stereo disparity in the same way as when determining the stereo disparity, and outputs the cost volume obtained as intermediate information. By converting each of the disparity candidates in the cost volume into depth, as shown in Figure 5, each pixel 310 in the image 300 will have the same number of depth candidates 320 as the number of disparity candidates.

[0042] The feature extraction unit 121 expands all the depth candidates of each pixel into three-dimensional point cloud information. The point cloud obtained thereby holds, in addition to the position coordinates of each point, the RGB value of the pixel and the matching cost as information. The feature extraction unit 121 integrates the point clouds corresponding to all the images into one point cloud around the vehicle. Thereafter, the processes of steps S130 and S140 are performed.

[0043] In this way, by using the cost volume as depth-related information, the deep learning model can reflect a plurality of depth candidates and their probabilities for the depth of a pixel, and the accuracy of three-dimensional recognition can be improved.

[0044] In the embodiments described so far, as depth-related information, depth information directly or indirectly indicating depth is used as depth-related information. The depth-related information may include, as information related to depth, the reliability indicating the probability of the depth at that pixel together with the depth for each pixel. The reliability of the depth can be given, for example, as information represented by a value from 0 to 1, where the larger the value, the higher the probability, and the smaller the value, the lower the probability. The reliability of the depth can be obtained, for example, by using a deep learning model trained to obtain the reliability of the depth with the original camera image and depth information such as the depth and stereo disparity described above as inputs.

[0045] The depth-related information acquisition unit 110 creates depth-related information including the acquired depth information and the reliability of the depth, and passes it to the feature extraction unit 121. The feature extraction unit 121 excludes regions (pixels) where the reliability of the depth is less than or equal to a preset threshold from each of the camera images, obtains a three-dimensional point cloud from the remaining pixels based on the depth information, and integrates the obtained three-dimensional point clouds to create one point cloud around the vehicle. In the integrated recognition unit 120, the processes of steps S130 and S140 are executed using the three-dimensional point cloud obtained in this way to detect an object.

[0046] Generally, when depth information is inaccurate, distortion occurs when converting image information from 2D to 3D, which causes a decrease in recognition accuracy. As described above, by including the depth reliability in the depth-related information, only the regions with high depth reliability can be converted into 3D point clouds for use in processing, and the accuracy of 3D recognition can be improved.

[0047] When stereo matching is used to obtain depth-related information, there may be regions that appear only in one of the paired camera images due to geometric constraints of the camera or occlusion of the object. For example, consider the case of imaging a vehicle from behind with two cameras placed at different positions. Here, the left image is the reference image 400, and the right image is the reference image 410. The rear surface of the vehicle appears in region 401 in the reference image 400 and in region 411 in the reference image 410, respectively. In this specification, the imaging region that appears in both the reference image 400 and the reference image 410 that form a pair is called the stereo region. On the other hand, the left side of the vehicle appears slightly in region 402 of the reference image 400, but does not appear in the reference image 410. In this specification, the imaging region that appears in the reference image 400 but does not appear in the reference image 410 is called the monocular region.

[0048] When stereo matching is used, usually, the estimation accuracy of the depth in the monocular region is lower than that in the stereo region. When stereo matching is used to obtain depth-related information, by including information indicating whether a pixel in the reference image is in the stereo region or the monocular region for each pixel, it is possible to exclude the monocular region with low depth estimation accuracy when obtaining the 3D point cloud and obtain the 3D point cloud, thereby improving the accuracy of 3D recognition. For example, using the above-described depth reliability information, the reliability can be set to 1 for the stereo region and 0 for the monocular region, and the same processing can be performed as in the case of using the depth reliability.

[0049] To determine whether an area in an image is stereo or monocular, the cost volume obtained during stereo disparity acquisition can be used. Typically, in monocular areas, no corresponding points are found on the reference image, resulting in high matching costs for multiple disparity candidates. Therefore, it is possible to identify areas with high overall matching costs as monocular areas and identify the remaining areas as stereo areas.

[0050] In this way, the accuracy of 3D recognition can also be improved by distinguishing between stereoscopic and monocular regions and excluding the monocular region from the acquisition of 3D point cloud data.

[0051] The embodiments described above perform object recognition using grid features obtained from a 3D point cloud as input. However, grid features can also be used in other processes such as predicting the behavior of surrounding objects, planning the vehicle's actions, predicting risks, and vehicle control.

[0052] For example, when performing behavior prediction using grid features obtained through the above-described process, the environmental recognition device 10 can be equipped with a new processing unit, a behavior prediction unit having a deep learning model trained to estimate the future movement path of an object by inputting time-series object information and grid features, and can perform the following processing. That is, the environmental recognition device 10 performs the processing shown in steps S100 to S140 in Figure 3 on several sets of camera images taken in a time series to obtain time-series object information and grid features for the number of sets of input camera images. Then, the time-series object information and grid features obtained using the behavior prediction unit are input to a deep learning model, and the deep learning model can estimate the future movement path of the imaged object and perform behavior prediction.

[0053] By using grid features to predict the behavior of an object, factors other than the object being predicted, such as road conditions and the movement of other surrounding objects, can be taken into consideration, thereby improving the accuracy of the prediction.

[0054] In addition to behavior prediction, other processes such as behavior planning, risk prediction, and vehicle control can also be performed using a processing unit equipped with a deep learning model trained to obtain the desired output from grid features or time-series grid features as input. This allows for similar processing to be performed to plan the vehicle's behavior, predict risks, and output commands for vehicle control. Grid features represent the situation around the vehicle in the form of features, and by using these as input to a deep learning model for various estimations, improved estimation accuracy can be obtained. Furthermore, by using grid features in common with other processes such as external environment recognition and behavior prediction, computational processing can be made more efficient compared to extracting features separately.

[0055] In the embodiments described above, an environmental recognition device mounted on a vehicle was used as an example. However, some or all of the functions of the above-mentioned parts may be located outside the vehicle, such as on the cloud. For example, image information from a camera mounted on the vehicle, or information at an intermediate stage of processing, may be transmitted to an external device via wireless communication for processing, and the processing results, or intermediate processing results, may be sent to the vehicle's device for use.

[0056] Although the present invention has been described above using representative embodiments as examples, the present invention is not limited thereto and can be implemented in various ways without departing from the spirit of the invention as described in the claims. Furthermore, the embodiments described above are explained in detail for the purpose of clearly illustrating the present invention and are not necessarily limited to those having all the configurations described.

[0057] 10: Environmental recognition device, 20: Vehicle, 100: Image acquisition unit, 110: Depth-related information acquisition unit, 120: Integrated recognition unit, 121: Feature extraction unit, 122: External environment recognition unit, C1-Cn: Camera, H1-H6: Shooting area

Claims

1. An environmental recognition device comprising: an image acquisition unit that acquires multiple images captured by multiple cameras mounted on a vehicle; a depth-related information acquisition unit that uses the multiple images to obtain depth-related information, including the depth to the subject, for each pixel corresponding to a subject captured in the multiple images; a feature extraction unit having a learning model that has learned to acquire feature quantities in a single aggregated coordinate space using the image information obtained from the multiple images and the depth-related information as input; and an external environment recognition unit that recognizes the external environment of the vehicle based on the feature quantities.

2. The environment recognition device according to claim 1, wherein the plurality of images include pairs of images capturing overlapping regions, and the depth-related information acquisition unit acquires stereo parallax obtained based on the pairs of images as depth-related information.

3. The environmental recognition device according to claim 1, wherein the plurality of images include pairs of images capturing overlapping regions, and the depth-related information acquisition unit acquires cost volume obtained based on the pairs of images as depth-related information.

4. The environment recognition device according to any one of claims 1 to 3, wherein the depth-related information acquisition unit estimates the reliability of the depth, and the depth-related information includes information relating to the reliability.

5. The environmental recognition device according to claim 2 or 3, wherein the depth-related information acquisition unit identifies whether the pixel is included in a monocular region or a stereo region, estimates the confidence level of the depth based on the result of the identification, and the depth-related information includes information relating to the confidence level.

6. The environmental recognition device according to any one of claims 1 to 3, wherein the coordinate space is a three-dimensional space centered on the vehicle, the feature extraction unit converts the coordinate position of the pixels into a coordinate position in the coordinate space using the depth-related information and the camera parameters of the plurality of cameras to create point cloud information in the coordinate space, and the point cloud information is used as input to the learning model.

7. The environmental recognition device according to claim 6, wherein the point cloud information includes the coordinate position in the coordinate space and the image information obtained from the image.

8. The environmental recognition device according to any one of claims 1 to 3, further comprising a processing unit that performs at least one of the following processes: behavior prediction, risk prediction, action planning, and vehicle control, using the features extracted by the feature extraction unit.

Images