Environmental awareness device

Abstract

When performing 3D recognition from 2D images, the distortion that occurs when converting 2D image information to 3D information is eliminated, thereby improving the accuracy of 3D recognition. [Solution] An environmental recognition device comprising: an image acquisition unit 100 that acquires images captured by multiple cameras C1 to Cn mounted on a vehicle; a depth-related information acquisition unit 110 that obtains depth-related information including the depth of each pixel in the multiple images; a feature extraction unit 121 having a learning model that has been trained to extract feature quantities in a single aggregated coordinate space using multiple images and depth-related information as input; and an external environment recognition unit 122 that recognizes the external environment of the vehicle based on the feature quantities.

Description

Technical Field

[0001] The present invention relates to an environment recognition device, and particularly to an environment recognition device mounted on a moving body such as a vehicle to recognize the surrounding environment.

Background Art

[0002] In recent years, in moving bodies such as vehicles, in order to realize automatic driving and support for a driver's driving operation, a multi-camera system has been developed that monitors the surroundings of the moving body using a plurality of cameras installed on the moving body. In the multi-camera system, image recognition processing such as object detection and segmentation is performed in each of the plurality of cameras, and the recognition results from these plurality of cameras are finally integrated by an integration processing unit and output as a recognition result.

[0003] Recently, for the purpose of improving the efficiency of recognition processing, an environment recognition device using a deep learning model that performs recognition processing by integrating a plurality of acquired images without performing image recognition in individual cameras has been developed. In such an environment recognition device, since integration processing is performed by one deep learning model, there are advantages such as reducing the development man-hours and the computational load. Also, in this method, the final recognition result can be output without an integration processing unit.

[0004] On the other hand, as a technology for recognizing the environment around a vehicle using a learned model, for example, Patent Document 1 discloses a technology for converting an image captured by a camera into a bird's-eye view image and inputting the bird's-eye view image into a learned model to detect a three-dimensional object and a drivable area.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, the technology described in Patent Document 1 uses simple assumptions, such as the road surface being flat, when converting 2D image information into 3D bird's-eye view information. As a result, distortions such as radial stretching of 3D objects may occur, potentially reducing the accuracy of 3D recognition by the model.

[0007] In view of these problems, the object of the present invention is to provide an environmental recognition device that can accurately convert images captured by a camera into three dimensions and perform highly accurate image recognition. [Means for solving the problem]

[0008] In one preferred embodiment, the environmental recognition device according to the present invention comprises: an image acquisition unit that acquires multiple images captured by multiple cameras mounted on a vehicle; a depth-related information acquisition unit that uses the multiple images to obtain depth-related information, including the depth to the subject, for each pixel corresponding to a subject captured in the multiple images; a feature extraction unit having a learning model that has learned to acquire feature quantities in a single aggregated coordinate space using image information and depth-related information obtained from the multiple images as input; and an external environment recognition unit that recognizes the external environment of the vehicle based on the feature quantities. [Effects of the Invention]

[0009] According to the present invention, the recognition accuracy of three-dimensional recognition tasks, such as object detection using multiple images, can be improved. Other novel features of the present invention and the technical problems they solve will become apparent from the description and drawings herein. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic block diagram showing the functional configuration of one embodiment of an environmental recognition device. [Figure 2] This is a schematic overview showing the imaging area captured by the camera. [Figure 3]This flowchart shows an example of a process performed by an environmental recognition device. [Figure 4] This is a schematic diagram illustrating an example of a bird's-eye view space. [Figure 5] This is a schematic diagram illustrating the possible depths of an image. [Figure 6] This is a schematic diagram illustrating the stereoscopic and monocular regions. [Figure 7] This is a schematic block diagram showing an example of the hardware configuration of an environmental recognition device. [Modes for carrying out the invention]

[0011] Hereinafter, representative embodiments of the present invention will be described with reference to the drawings. Note that the embodiments and drawings described below are illustrative examples for explaining the present invention, and have been omitted or simplified as appropriate for clarity of explanation. Furthermore, please note that the position, size, shape, and extent of each component shown in the drawings may not necessarily accurately represent them, in order to facilitate understanding of the invention.

[0012] Figure 1 is a schematic block showing the functional configuration of one embodiment of an environmental recognition device to which the present invention is applied. In the following explanation, the application to a vehicle with preventive safety functions and autonomous driving functions will be used as an example, but it may also be applied to devices other than vehicles, such as surveillance cameras.

[0013] The environmental recognition device 10 is mounted on a vehicle, such as an automobile, and connected to a plurality of cameras C1, C2, ... Cn provided on the vehicle. The environmental recognition device 10 has an image acquisition unit 100, a depth-related information acquisition unit 110, and an integrated recognition unit 120, and processes images captured by the plurality of cameras C1, C2, ... Cn (hereinafter referred to as camera images) to recognize the environment around the vehicle.

[0014] The image acquisition unit 100 uses cameras C1, C2, ...Cn to capture images of the area around the vehicle and acquires the captured camera images.

[0015] The depth-related information acquisition unit 110 acquires depth-related information related to the depth to the subject for each pixel of the subject shown in a plurality of camera images captured by the cameras C1, C2, ··· Cn using the camera images acquired by the image acquisition unit 100. When acquiring the depth-related information, camera-related information such as the internal parameters and external parameters of each camera C1, C2, ··· Cn may be used. The acquisition of the depth-related information may be performed by geometric calculation, or a learning model learned by machine learning or deep learning may be used. In the present embodiment, the depth-related information may be information indicating the depth itself.

[0016] The integrated recognition unit 120 includes a feature extraction unit 121 and an external environment recognition unit 122, and recognizes the surrounding environment using the plurality of image information acquired by the image acquisition unit 100 and the depth-related information acquired by the depth-related information acquisition unit 110.

[0017] The feature extraction unit 121 extracts features in a coordinate space integrated into one of the surroundings of the vehicle 20 using the camera images captured by the plurality of cameras C1, C2, ··· Cn acquired by the image acquisition unit 100 and the depth-related information acquired by the depth-related information acquisition unit 110. The camera images used by the feature extraction unit 121 may be pre-processed such as normalization in the image acquisition unit 100 in advance, or may be converted into feature maps using a deep learning model such as a CNN (Convolutional Neural Network).

[0018] The integrated coordinate space may be a bird's-eye view space or a three-dimensional space. Further, the features may be defined in a format such as a grid map format, a voxel format, a point cloud format, etc., or may be something like an intermediate output of a deep learning model. The extraction of the features is performed using a deep learning model such as a CNN or a Transformer.

[0019] The external environment recognition unit 122 recognizes the external environment using the feature amounts extracted by the feature amount extraction unit 121. The recognition process executed here may be, for example, 3D object detection, segmentation, or lane detection. Also, these recognition processes may be executed individually, or a plurality of recognition processes may be executed simultaneously. Further, a deep learning model or a machine learning model may be used for the recognition process.

[0020] Physically, the environment recognition device 10 includes, as shown in FIG. 7, an arithmetic device 11, a memory 12, an interface 13 for exchanging data with a camera or other control devices, a bus 14 for interconnecting these, etc. The functions of the above-described respective units are realized by the arithmetic device 11 executing a control program stored in the memory 12. For the arithmetic device 11, a processing device such as a so-called CPU (Central Processing Unit) and / or GPU (Graphics Processing Unit) can be used. Note that some or all of the functions of these respective units may be realized by hardware using an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

[0021] FIG. 2 is a plan view schematically showing an imaging region imaged by a camera mounted on a vehicle in the present embodiment.

[0022] In the present embodiment, for example, a total of six cameras, i.e., three cameras C1, C2, C3 at the front of the vehicle body and three cameras C4, C5, C6 at the rear, are mounted on the vehicle 20. The regions imaged by the respective cameras C1 to C6 are regions indicated by H1 to H6, respectively. Cameras C1 and C2, C1 and C3, C2 and C4, C3 and C5, C4 and C6, and C5 and C6 are paired with each other and have a common imaging region that overlaps.

[0023] Figure 3 is a flowchart showing an example of a process performed by the environmental recognition device 10.

[0024] The environmental recognition device 10 first acquires camera images captured by cameras C1 to C6 in the image acquisition unit 100 (step S100).

[0025] The camera images acquired by the image acquisition unit 100 are passed to the depth-related information acquisition unit 110. The depth-related information acquisition unit 110 uses, for example, a trained deep learning model that takes one camera image as input and estimates the depth corresponding to each pixel, to infer depth-related information for each pixel of each camera image acquired in step S100. If there are overlapping regions with images captured by other cameras in a pair, it is sufficient to perform depth-related information inference for at least one camera image for the overlapping regions (step S110). Furthermore, it is not necessarily required to perform depth-related information inference for all regions captured by cameras C1 to C6; depth-related information may be inferred for at least two or more images.

[0026] The depth-related information for each pixel acquired by the depth-related information acquisition unit 110 is passed to the feature extraction unit 121 along with the camera image. The feature extraction unit 121 uses the depth-related information and the camera parameters of each camera C1 to C6 to convert the coordinate position of each pixel in the camera image from a camera coordinate system based on each camera to a unified vehicle coordinate system based on the vehicle 20, and acquires it as a 3D point cloud in the vehicle coordinate system. Furthermore, all the obtained point clouds are aggregated to form a point cloud in a single coordinate space around the vehicle. The information in the point cloud obtained here includes both the position coordinates in the vehicle coordinate system and the RGB values ​​as image information obtained from the camera image (step S120).

[0027] The feature extraction unit 121 then extracts features of the bird's-eye view based on the acquired 3D point cloud. First, the feature extraction unit 121 defines a grid space centered on the vehicle, maps all the points acquired in step S120 to the grid space, and obtains the bird's-eye view space.

[0028] Figure 4 is a schematic diagram showing an example of the acquired bird's-eye view space. The bird's-eye view space is the space viewed from above the vehicle 210, within a grid space 200 defined around the vehicle 210. In the bird's-eye view space, point clouds obtained corresponding to moving objects such as vehicles 220, pedestrians 230, and bicycles 240, as well as stationary objects such as curbs 250 and trees 260, which are imaged by cameras C1 to C6, are mapped onto this grid space 200 as black dots.

[0029] Next, the feature extraction unit 121 inputs the point clouds present in each grid of the acquired bird's-eye view space into a fully connected layer and obtains feature vectors corresponding to each grid. Then, a CNN convolution process is performed on the obtained feature vectors for the entire grid to obtain a feature map.

[0030] The feature map obtained here corresponds positionally to the defined grid space, and features are stored corresponding to each grid in the grid space. Hereafter, this feature map will be called grid features (step S130).

[0031] Finally, in the external environment recognition unit 122, the acquired grid features are input to a trained deep learning model consisting of fully connected layers, and a 3D bounding box of the objects around the vehicle is output (step S140).

[0032] In this embodiment, depth-related information is estimated for each pixel from multiple camera images, and features in an integrated coordinate space are obtained using the image information and the estimated depth-related information to perform external environment recognition. This allows image information to be accurately deployed in a 3D coordinate space, improving the accuracy of 3D recognition. In particular, by obtaining accurate depth-related information in advance, it becomes possible to recognize the position coordinates of surrounding objects with small errors.

[0033] In the embodiment described above, depth-related information is acquired for each image, but it is also possible to acquire depth-related information by using two images obtained from a pair of cameras as input and performing stereo matching. The process when acquiring depth-related information using stereo matching is as follows.

[0034] In step S100, the environmental recognition device 10 performs parallelization processing on the camera images acquired by the image acquisition unit 100. In the parallelization processing, the images are transformed so that the projection planes of the paired cameras lie on the same plane and the epipolar lines are parallel. By parallelizing the images in this way before performing stereo matching, stereo matching can be performed efficiently with a small amount of computation.

[0035] In this embodiment, each of the six cameras C1 to C6 is paired with two adjacent cameras, and parallelization processing is performed between each adjacent camera. As a result, two parallelized images (parallelized images) are created from one camera image, resulting in a total of six pairs, or twelve parallelized images. In subsequent processing, the created parallelized images are processed as camera images.

[0036] In step S110, the depth-related information acquisition unit 110 obtains stereo disparity information using a deep learning model trained to output a stereo disparity image including stereo disparity, taking parallelized paired images as input. Then, the depth of each pixel is obtained as depth-related information by geometric calculation from the obtained disparity information. Stereo disparity information may also be obtained using rule-based methods such as template matching. By using such a method, the computational load when acquiring depth-related information can be reduced compared to when using a deep learning model.

[0037] Subsequently, the environmental recognition device 10 performs the processes described in steps S120 to S140 above to detect objects.

[0038] In this way, by using depth information obtained through stereo matching as depth-related information, it becomes possible to use accurate depth for object recognition, thereby improving the accuracy of the final 3D recognition.

[0039] When using stereo matching, instead of obtaining information indicating the depth itself as described above, it is also possible to obtain stereo parallax or cost volume as depth-related information.

[0040] The cost volume is obtained by calculating a value (matching cost) indicating the degree of agreement or mismatch between multiple disparity candidates at each pixel position of two paired images, and storing it in a three-dimensional array of image width, image height, and disparity. The cost volume is typically obtained as intermediate information when calculating stereo disparity.

[0041] When the cost volume is used as depth-related information, the depth-related information acquisition unit 110 acquires the stereo disparity in the same way as when determining the stereo disparity, and outputs the cost volume obtained as intermediate information. By converting each of the disparity candidates in the cost volume into depth, as shown in Figure 5, each pixel 310 in the image 300 will have the same number of depth candidates 320 as the number of disparity candidates.

[0042] The feature extraction unit 121 expands all depth candidates for each pixel into 3D point cloud information. The resulting point cloud stores the position coordinates of each point, as well as the RGB values ​​and matching cost of the pixels. The feature extraction unit 121 integrates the point clouds corresponding to all images into a single point cloud representing the area around the vehicle. Subsequently, steps S130 and S140 are performed.

[0043] In this way, by using cost volume as depth-related information, deep learning models can reflect multiple depth candidates and their likelihood in the pixel depth, thereby improving the accuracy of 3D recognition.

[0044] In the embodiments described so far, depth-related information is used, either directly or indirectly, to indicate depth. The depth-related information may include, as information related to depth, the depth of each pixel, along with a confidence level indicating the likelihood of the depth at that pixel. The confidence level of depth can be expressed as a value between 0 and 1, for example, with a higher value indicating higher likelihood and a lower value indicating lower likelihood. The confidence level of depth can be obtained, for example, using a deep learning model that has been trained to obtain the confidence level of depth by taking the original camera image and depth information such as the depth and stereo disparity mentioned above as input.

[0045] The depth-related information acquisition unit 110 creates depth-related information including the acquired depth information and depth confidence, and passes it to the feature extraction unit 121. The feature extraction unit 121 removes areas (pixels) from each camera image where the depth confidence is below a preset threshold, obtains a 3D point cloud from the depth information for the remaining pixels, and integrates the acquired 3D point clouds to create a single point cloud representing the area around the vehicle. The integrated recognition unit 120 uses the 3D point cloud obtained in this way to perform the processing in steps S130 and S140 and detect objects.

[0046] Generally, inaccurate depth information causes distortion when converting image information from 2D to 3D, leading to a decrease in recognition accuracy. As mentioned above, by including depth confidence in depth-related information, only regions with high depth confidence can be converted into 3D point clouds and used for processing, thereby improving the accuracy of 3D recognition.

[0047] When using stereo matching to acquire depth-related information, geometric constraints of the cameras or object occlusion may result in areas being captured in only one of the paired camera images. For example, consider the case where a vehicle is imaged from the rear by two cameras positioned at different locations, as shown in Figure 6. Here, the image on the left is the reference image 400, and the image on the right is the reference image 410. The rear of the vehicle is captured in area 401 in the reference image 400 and in area 411 in the reference image 410. In this specification, the captured area that is captured in both the paired reference image 400 and reference image 410 is called the stereo region. On the other hand, area 402 in the reference image 400 shows a small portion of the left side of the vehicle, but the left side is not captured in the reference image 410. In this specification, the captured area that is captured in the reference image 400 but not in the reference image 410 is called the monocular region.

[0048] When using stereo matching, the depth estimation accuracy is typically lower in monocular regions compared to stereo regions. When using stereo matching to acquire depth-related information, including information indicating whether each pixel in the reference image is in a stereo or monocular region allows for the exclusion of monocular regions with low depth estimation accuracy during 3D point cloud acquisition, thereby improving the accuracy of 3D recognition. For example, by using the depth confidence information mentioned above and setting the confidence level to 1 for stereo regions and 0 for monocular regions, processing can be performed in the same way as when using depth confidence.

[0049] To determine whether an area in an image is stereo or monocular, the cost volume obtained during stereo disparity acquisition can be used. Typically, in monocular areas, no corresponding points are found on the reference image, resulting in high matching costs for multiple disparity candidates. Therefore, it is possible to identify areas with high overall matching costs as monocular areas and identify the remaining areas as stereo areas.

[0050] In this way, the accuracy of 3D recognition can also be improved by distinguishing between stereoscopic and monocular regions and excluding the monocular region from the acquisition of 3D point cloud data.

[0051] The embodiments described above perform object recognition processing using grid features obtained from a 3D point cloud as input. However, grid features can also be used in other processes such as predicting the behavior of surrounding objects, planning the vehicle's actions, predicting risks, and vehicle control.

[0052] For example, when performing behavior prediction using grid features obtained through the above-described process, the environment recognition device 10 can be equipped with a new processing unit, a behavior prediction unit having a deep learning model trained to estimate the future movement path of an object by inputting time-series object information and grid features, and can perform the following processing. That is, the environment recognition device 10 performs the processing shown in steps S100 to S140 in Figure 3 on several sets of camera images taken in a time series to obtain time-series object information and grid features for the number of sets of input camera images. Then, the time-series object information and grid features obtained using the behavior prediction unit are input to a deep learning model, and the deep learning model can estimate the future movement path of the imaged object and perform behavior prediction.

[0053] By using grid features to predict the behavior of an object, factors other than the object being predicted, such as road conditions and the movement of other surrounding objects, can be taken into consideration, thereby improving the accuracy of the prediction.

[0054] In addition to behavior prediction, other processes such as behavior planning, risk prediction, and vehicle control can also be performed using a processing unit equipped with a deep learning model trained to obtain the desired output from grid features or time-series grid features as input. This allows for similar processing to be performed to plan the vehicle's behavior, predict risks, and output commands for vehicle control. Grid features represent the situation around the vehicle in the form of features, and by using these as input to a deep learning model for various estimations, improved estimation accuracy can be obtained. Furthermore, by using grid features in common with other processes such as external environment recognition and behavior prediction, computational processing can be made more efficient compared to extracting features separately.

[0055] In the embodiments described above, an environmental recognition device mounted on a vehicle was used as an example. However, some or all of the functions of the above-mentioned parts may be located outside the vehicle, such as on the cloud. For example, image information from a camera mounted on the vehicle, or information at an intermediate stage of processing, may be transmitted to an external device via wireless communication for processing, and the processing results, or intermediate processing results, may be sent to the vehicle's device for use.

[0056] Although the present invention has been described above using representative embodiments as examples, the present invention is not limited thereto and can be implemented in various ways without departing from the spirit of the invention as described in the claims. Furthermore, the embodiments described above are explained in detail for the purpose of clearly illustrating the present invention and are not necessarily limited to those having all the configurations described. [Explanation of Symbols]

[0057] 10: Environmental recognition device, 20: Vehicle, 100: Image acquisition unit, 110: Depth-related information acquisition unit, 120: Integrated recognition unit, 121: Feature extraction unit, 122: External environment recognition unit, C1~Cn: Camera, H1~H6: Shooting area

Claims

1. An image acquisition unit that acquires multiple images captured by multiple cameras mounted on the vehicle, A depth-related information acquisition unit obtains depth-related information, including the depth to the subject, for each pixel corresponding to a subject captured in the plurality of images, using the plurality of images. A feature extraction unit having a learning model that has been trained to acquire feature quantities in a single aggregated coordinate space using image information obtained from the plurality of images and the depth-related information as input, An external environment recognition unit recognizes the external environment of the vehicle based on the aforementioned feature quantities, An environmental recognition device equipped with the following features.

2. The environmental recognition device according to claim 1, wherein the plurality of images include pairs of images capturing overlapping regions, and the depth-related information acquisition unit acquires stereo parallax obtained based on the pairs of images as depth-related information.

3. The environmental recognition device according to claim 1, wherein the plurality of images include pairs of images capturing overlapping regions, and the depth-related information acquisition unit acquires cost volume obtained based on the pairs of images as depth-related information.

4. The environment recognition device according to any one of claims 1 to 3, wherein the depth-related information acquisition unit estimates the reliability of the depth, and the depth-related information includes information relating to the reliability.

5. The environment recognition device according to claim 2 or 3, wherein the depth-related information acquisition unit identifies whether the pixel is included in a monocular region or a stereo region, estimates the confidence level of the depth based on the result of the identification, and the depth-related information includes information relating to the confidence level.

6. The environmental recognition device according to any one of claims 1 to 3, wherein the coordinate space is a three-dimensional space centered on the vehicle, the feature extraction unit converts the coordinate position of the pixels into a coordinate position in the coordinate space using the depth-related information and the camera parameters of the plurality of cameras to create point cloud information in the coordinate space, and the point cloud information is used as input to the learning model.

7. The environmental recognition device according to claim 6, wherein the point cloud information includes the coordinate position in the coordinate space and the image information obtained from the image.

8. The environmental recognition device according to any one of claims 1 to 3, further comprising a processing unit that performs at least one of the following processes: behavior prediction, risk prediction, action planning, and vehicle control, using the features extracted by the feature extraction unit.

Images