Estimation device and estimation method

Abstract

To obtain a more reliable estimation result for estimation of three-dimensional spatial information.SOLUTION: An estimation device 100 includes a coordinate calculator 210 for calculating, on the basis of consecutive two-dimensional images captured by the same camera in a time series, three-dimensional coordinate of objects in the two-dimensional images by using Visual Odometry as a self-location estimation method, and calculating three-dimensional coordinates of objects around a vehicle 500 on the basis of each of two-dimensional images of the outside of the vehicle captured by a plurality of cameras 300A to 300F provided on the vehicle, and a feature value acquisition unit 220 for acquiring BEV feature values on the basis of the three-dimensional coordinates and the two-dimensional images of the outside of the vehicle by using an algorithm for BEV estimation for acquiring BEV feature values which are feature values in a BEV space, and a bird's-eye view generation unit 230 for generating, on the basis of the BEV feature values, a bird's-eye view which is an image showing an aspect that the vehicle is viewed from above.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an estimation device and an estimation method. [Background technology]

[0002] In the technology described in Patent Document 1, three-dimensional spatial information is estimated using a machine learning model based on two-dimensional images captured by multiple cameras, and a bird's-eye view is generated based on the estimated three-dimensional spatial information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2021 / 0398338 Summary of the Invention [Problem to be solved by the invention]

[0004] When using a machine learning model to estimate 3D spatial information based on 2D images, the accuracy of the estimation result is affected by the training data and the method of training the machine learning model. For example, the machine learning model may not be able to output a correct estimation result for an untrained event. When using a machine learning model, the estimation result obtained is not necessarily reliable.

[0005] Therefore, there is a need for technology that can obtain more reliable estimation results when estimating 3D spatial information based on 2D images. [Means for solving the problem]

[0006] According to one embodiment of the present disclosure, an estimation device (100) is provided. The estimation device includes a coordinate calculation unit (210) that calculates three-dimensional coordinates of an object in a two-dimensional image by using Visual Odometry as a self-location estimation method based on consecutive two-dimensional images captured in time series by the same camera, the coordinate calculation unit (210) calculating the three-dimensional coordinates of an object present around the vehicle based on each of the two-dimensional images of the outside of the vehicle captured by a plurality of cameras (300A to 300F) provided on the vehicle (500), a feature amount acquisition unit (220) that acquires a BEV feature amount based on the three-dimensional coordinates and the two-dimensional image of the outside of the vehicle by an algorithm for BEV estimation that acquires a BEV feature amount that is a feature amount in a BEV (Bird's eye view) space, and a bird's eye view generation unit (230) that generates a bird's eye view that is an image showing the vehicle viewed from above based on the BEV feature amount.

[0007] According to the estimation device of the above embodiment, visual odmetry is used to calculate the 3D coordinates of objects around the vehicle by a geometric method based on 2D images and the internal and external parameters of the camera, so that it is possible to obtain more reliable estimation results compared to a method of estimating the 3D coordinates of objects around the vehicle using only a machine learning model. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a schematic configuration of a driving assistance system. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of camera arrangement. [Diagram 3] FIG. 1 is an explanatory diagram of coordinate transformation. [Figure 4] 4 is a flowchart showing a method for generating a bird's-eye view in the first embodiment. [Diagram 5] 10 is a flowchart showing a method for generating a bird's-eye view in the second embodiment. [Figure 6] 13 is a flowchart showing a method for generating a bird's-eye view in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A. First embodiment: A driving assistance system 1 shown in Fig. 1 estimates the surrounding situation of a vehicle 50 using a plurality of two-dimensional images acquired in time series by an image sensor provided in the vehicle 50, and generates a bird's-eye image of the vehicle 50 viewed from above. The bird's-eye image is an image of the vehicle 50 viewed from above. For example, the driving assistance system 1 uses a plurality of two-dimensional images acquired in time series by an image sensor provided in the vehicle 50 to display an image of the vehicle 50 viewed from above on an in-vehicle monitor of the vehicle 50. The vehicle 50 is capable of traveling using an ADAS (Advanced Driving Assistant System).

[0010] The driving assistance system 1 includes an estimation device 100 and a camera group 300. The estimation device 100 and the camera group 300 are mounted on a vehicle 50.

[0011] As shown in FIG. 2, the camera group 300 includes six cameras 300A to 300F provided on the vehicle 50. The cameras 300A to 300F are monocular cameras capable of capturing color images. The camera 300A is provided on the upper part of the windshield of the vehicle 50. The camera 300A captures an image of a predetermined range in front of the vehicle 50. The camera 300B is provided on the lower part of the door mirror on the right side of the vehicle 50. Hereinafter, the right side of the vehicle 50 refers to the right side as seen by a person sitting in the vehicle 50 facing the front of the vehicle 50. The camera 300B captures an image of a predetermined range diagonally to the right front of the vehicle 50.

[0012] Camera 300C is provided at the bottom of the left door mirror. The left side of vehicle 50 refers to the left side as seen by a person sitting in vehicle 50 facing the front of vehicle 50. Camera 300C captures an image of a predetermined range diagonally forward to the left of vehicle 50. Camera 300D is provided on the right door pillar. Camera 300D captures an image of a predetermined range on the right side of vehicle 50. Camera 300E is provided on the left door pillar. Camera 300E captures an image of a predetermined range on the left side of vehicle 50. Camera 300F is provided in the center of the back door. Camera 300F captures an image of a predetermined range behind vehicle 50.

[0013] The cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the acquired images to the estimation device 100. For ease of understanding of the technology, in the embodiment, it is assumed that the cameras 300A to 300F capture images at the same timing.

[0014] The estimation device 100 is a computer including a memory 110, an input / output interface 120, and a CPU (Central Processing Unit) 150. The memory 110 and the input / output interface 120 are connected to the CPU 150 via a bus 190. For example, the functions of the estimation device 100 are realized by a driving control ECU (Electronic Control Unit) that controls driving of the vehicle 50.

[0015] The memory 110 stores various programs and various data used for various processes executed by the estimation device 100. The memory 110 stores data representing a machine learning model used for various processes executed by the estimation device 100.

[0016] The input / output interface 120 is connected to the cameras 300A to 300F via Ethernet (registered trademark).

[0017] CPU 150 is a processor that realizes various functions by executing programs stored in memory 110. For example, CPU 150 stores two-dimensional images received from cameras 300A to 300F in memory 110. In the embodiment, CPU 150 executes the programs stored in memory 110 to function as coordinate calculation unit 210, feature amount acquisition unit 220, and bird's-eye view generation unit 230.

[0018] The coordinate calculation unit 210 calculates the three-dimensional coordinates of an object in a two-dimensional image by using Visual Odometry as a self-position estimation method based on two-dimensional images consecutive in time series. In this embodiment, the coordinate calculation unit 210 calculates the three-dimensional coordinates of an object existing around the vehicle 50 by using Visual Odometry based on two-dimensional images of the outside of the vehicle captured by the cameras 300A to 300F. Details of the processing by the coordinate calculation unit 210 will be described later.

[0019] The feature amount acquisition unit 220 acquires BEV feature amounts, which are feature amounts in a BEV (Bird's eye view) space, based on the three-dimensional coordinates of objects present around the vehicle 50 and the two-dimensional image used to calculate the three-dimensional coordinates. The BEV feature amount includes coordinate values ​​of a point cloud in a three-dimensional space and information on the color of the point cloud, similar to point cloud data detected by Lidar. The details of the processing by the feature amount acquisition unit 220 will be described later.

[0020] The bird's-eye view generating unit 230 generates a bird's-eye view based on the BEV feature amount. The bird's-eye view is an image showing the vehicle 50 viewed from above. An existing method can be used as a method for generating a bird's-eye view based on the BEV feature amount. The bird's-eye view generating unit 230 can detect objects around the vehicle 50 based on the BEV feature amount and generate a bird's-eye view showing the objects around the vehicle 50. Note that the generated bird's-eye view does not need to include all objects around the vehicle 50. For example, the bird's-eye view generating unit 230 may detect lanes around the vehicle 50 based on the BEV feature amount and generate a bird's-eye view including the vehicle 50 and the lanes.

[0021] Next, we will explain Visual Odometry (hereinafter, VO). In VO, feature points are detected in multiple consecutive 2D images in time series, and the 3D coordinates of the feature points in the world coordinate system are estimated using the coordinates of the feature points in the 2D images. A feature point is a point on a 2D image that can be reliably detected.

[0022] A corner detection method is used to detect feature points. For example, an intersection of two edges is detected as a corner. Alternatively, a point where two prominent edges with different directions exist in a certain local neighborhood is detected as a corner. For example, Harris corner detection, SHIFT (scale-invariant feature transform), SURF (Speeded up robust features), and ORB (Oriented FAST and Rotated BRIEF) can be used as the corner detection method. In addition, many corner detection methods can detect not only corner points but also feature points. The number of feature points detected from one two-dimensional image is, for example, 100.

[0023] For example, feature points are detected from three consecutive two-dimensional images captured in time series by the same camera. The times at which the three two-dimensional images are captured are assumed to be times t0, t1, and t2. Time t0 is the earliest time, and time t2 is the latest time. Feature points in the two-dimensional images captured at times t0, t1, and t2 are detected, and the coordinates and brightness values ​​of each detected feature point are recorded. Examples of the detected feature points include corners of buildings, outlines of traffic signs, color boundaries of traffic signs, and corners of curbs.

[0024] As shown in FIG. 3, the coordinates of the detected feature points are obtained by projecting the three-dimensional position of an object, etc. in the world coordinate system onto the image coordinate system. The relationship between the coordinates of the feature points in the two-dimensional image and the three-dimensional position of the feature points in the world coordinate system can be expressed as in the following formula (1). In formula (1), f x , f y represents the focal length. x , cy represents the position of the intersection of the optical axis and the projection plane. 11 ~r 33 represents the rotation matrix. t1 to t3 represent translation vectors. s represents the scale factor. The scale factor s is estimated using the results of measuring the position and orientation of the camera in advance using an IMU (Inertial Measurement Unit) or the like.

[0025]

number

[0026] Equation (1) can be expressed as equation (2).

[0027]

number

[0028] The matrix K in equation (2) is called the internal parameters of the camera. The matrix [RT] is called the external parameters of the camera. The external parameters of the camera are determined by the position T and orientation R of the camera coordinate system. As shown in equation (2), the position u in the 2D image I =[uv 1] t is the position T and orientation R in the camera coordinate system, and the position X in the world coordinate system. w =[X W Y W Z W ] t It is expressed as a function of

[0029] When tracking the positions of feature points corresponding to the same object, etc. detected in multiple consecutive 2D images in time series, the position of the feature point of interest changes due to the movement of the vehicle 50, i.e., the movement of the camera group.

[0030] In VO, the position and orientation of the camera and the 3D coordinates of the feature points in the world coordinate system at time t1 are estimated by bundle adjustment based on the positions of the feature points detected in each 2D image. Specifically, the position and orientation of the camera and the 3D coordinates of the feature points in the world coordinate system at time t1 are calculated using the coordinates of corresponding feature points detected in multiple 2D images and the assumed camera position T and orientation R. Then, the calculated 3D coordinates are reprojected onto the image coordinate system, and the distance (reprojection error) between the projected point and the feature point detected from the 2D image is estimated.

[0031] The position of a feature point in a two-dimensional image can be found by a projection function as shown in equation (3). Note that the internal parameters are assumed to be fixed, and are not taken into consideration in equation (3). The projection function is a function that maps the coordinate system of the object space (world coordinate system) to the coordinate system of the image space (image coordinate system).

[0032]

number

[0033] Position of feature points detected from a 2D image, u I and the position of the feature point obtained by the projection function, u P The error between is defined as the following equation (4). In equation (4), u I represents a set of coordinates of feature points detected from a 2D image.

[0034]

number

[0035] The error function C expressed in equation (4) camThe three-dimensional coordinates of the feature points in the world coordinate system and the position and orientation of the camera at time t1 are estimated by an optimization process that minimizes. The estimated three-dimensional coordinates of the feature points in the world coordinate system are the three-dimensional coordinates of the feature points in the two-dimensional image acquired at time t1 in the world coordinate system. The estimated position and orientation of the camera are expressed by a rotation matrix and a translation vector. Minimizing the error expressed in equation (4) means minimizing the error between the actual coordinates of the detected feature points in the image and the coordinates of the feature points calculated by the projection function. In this way, in VO, the three-dimensional coordinates of objects, etc. in the world coordinate system are estimated.

[0036] Next, an algorithm for estimating the BEV in this embodiment will be described. In this embodiment, a bird's-eye image is generated by combining the LSS (Lift, Splat, Shoot) algorithm (Jonah Philion, Sanja Fidler, "Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D", in European Conference on Computer Vision. Springer, 2020, pp. 194-210) with the estimation result of the three-dimensional coordinates of the objects around the vehicle 50 by VO.

[0037] In the LSS algorithm, first, two-dimensional images are acquired by the cameras 300A to 300F, respectively. A feature vector of each of the two-dimensional images is extracted using a machine learning model such as CNN. The depth of each pixel is estimated based on each of the two-dimensional images acquired by each camera. Since the estimated depth is ambiguous, a ray (a ray from the camera to a point in three-dimensional space) corresponding to each pixel is assumed, and a discrete depth probability distribution is predicted for each pixel. The cross product of the predicted discrete depth probability distribution and the feature vector is calculated, and a series of points are obtained as information representing the pixel depth. As a result, a frustum-shaped point cloud is generated for each camera. BEV features are generated from the frustum point cloud using the internal parameters and external parameters of the camera.

[0038] Fig. 4 shows a bird's-eye view estimation method executed by the estimation device 100 in this embodiment. For example, when the vehicle 50 starts traveling, the process shown in Fig. 4 starts. The cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the captured two-dimensional images to the estimation device 100. The captured two-dimensional images are color images. The memory 110 sequentially stores the two-dimensional images captured by the cameras 300A to 300F.

[0039] In step S101, a two-dimensional image is read from memory 110. In step S102, VO is executed. In this embodiment, three-dimensional coordinates of an object or the like in the world coordinate system are calculated by VO based on each of the two-dimensional images acquired by cameras 300A to 300F. The processes of steps S101 and S102 are executed by coordinate calculation unit 210.

[0040] In step S103, a depth map is generated using the three-dimensional coordinates in the world coordinate system calculated in step S102 and the two-dimensional image (the two-dimensional image acquired at time t1) used in the calculation of the three-dimensional coordinates. The two-dimensional image (the two-dimensional image acquired at time t1) used in the calculation of the three-dimensional coordinates is also called a corresponding image. In this embodiment, the coordinates are estimated by a feature based method. Therefore, the coordinates in the world coordinate system are estimated only for the feature points detected from the corresponding image. Therefore, the coordinates in the world coordinate system are not estimated for pixels other than the feature points. Therefore, the depths of the pixels other than the feature points can be obtained by, for example, linear interpolation using the depths of nearby pixels. The process of step S103 is executed by the feature amount acquisition unit 220.

[0041] In step S104, image features (feature vectors) of the two-dimensional images acquired by the camera group 300 are extracted using a machine learning model such as CNN. To extract image features of the two-dimensional images acquired by the camera 300A, a machine learning model is used that is trained to extract image features of the two-dimensional images inputted from the two-dimensional images acquired by the camera 300A as inputs. This machine learning model is generated by machine learning using two-dimensional images captured by a camera attached at a position similar to that of the camera 300A as training data. Here, 1×1 convolution is performed to extract image features. Therefore, the size of the two-dimensional image is not changed. Here, the corresponding image (two-dimensional image acquired at time t1) used to calculate the three-dimensional coordinates is inputted. For the cameras 300B to 300F, image features of the two-dimensional images are extracted using the corresponding machine learning models. The process of step S104 is executed by the feature acquisition unit 220.

[0042] In step S105, the cross product of the image feature extracted in step S104 and the depth map generated from the corresponding two-dimensional image is calculated as the cross product feature. The process of step S105 is executed by the feature acquisition unit 220.

[0043] In step S106, a point cloud of the view frustum is generated based on the cross product feature obtained in step S105, similarly to the LSS algorithm. Specifically, a light ray (a light ray from the camera to a point in three-dimensional space) corresponding to each pixel of each two-dimensional image is assumed, and a discrete depth probability distribution is predicted for each pixel. The cross product of the predicted discrete depth probability distribution and a vector representing the cross product feature is obtained as the depth. Using the estimated depth, the coordinates of each pixel of the two-dimensional image, and the internal and external parameters of the camera, a point cloud of the view frustum is generated for the two-dimensional image acquired by each camera. The process of step S106 is executed by the feature acquisition unit 220.

[0044] In step S107, BEV features are generated from the point cloud of the view frustum generated in step S106. The process of step S107 is executed by feature acquisition unit 220. In step S108, a bird's-eye image is generated from the BEV features. The generated bird's-eye image is displayed, for example, on a monitor provided in vehicle 50. The process of step S108 is executed by bird's-eye view generation unit 230. The process shown in FIG. 4 is executed repeatedly at predetermined time intervals, for example, while vehicle 50 is traveling.

[0045] As described above, in this embodiment, a depth map generated based on three-dimensional coordinate information acquired using a VO is used to generate a bird's-eye view. Using the VO, three-dimensional coordinates of objects around the vehicle are calculated by a geometric method based on a two-dimensional image, internal parameters and external parameters of the camera. Therefore, a more reliable result can be obtained for estimating the three-dimensional coordinates of objects around the vehicle. Therefore, the reliability of the accuracy of the estimated bird's-eye view can be increased compared to a mode in which a depth map is generated based on three-dimensional coordinate information acquired using a machine learning model and the generated depth map is used to estimate the bird's-eye view.

[0046] B. Second embodiment: The driving assistance system 1 according to the second embodiment will be described below. The description will focus on configurations different from the first embodiment, and descriptions of configurations similar to the first embodiment will be omitted.

[0047] In this embodiment, a bird's-eye image is generated by combining the algorithm disclosed in U.S. Patent Application Publication No. 2023 / 0053785 with the results of VO estimation of the three-dimensional coordinates of objects around the vehicle 50.

[0048] In the algorithm disclosed in the above document, first, two-dimensional images are respectively acquired by a plurality of cameras. Image features of each of the two-dimensional images are extracted using a backbone network. The backbone network is, for example, a CNN. The extracted image features are input to a transformer engine. The transformer engine is a machine learning model that utilizes an attention mechanism. The transformer engine is a machine learning model that is trained to receive image features extracted from a plurality of two-dimensional images as input, fuse the image features, and project the image features into a BEV space.

[0049] Fig. 5 shows a bird's-eye view estimation method executed by the estimation device 100 in this embodiment. For example, when the vehicle 50 starts traveling, the process shown in Fig. 5 starts. While the following process is being executed, the cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A to 300F are sequentially stored in the memory 110.

[0050] In step S201, a two-dimensional image is read from memory 110. In step S202, VO is executed. In this embodiment, three-dimensional coordinates of an object or the like in the world coordinate system are calculated by VO based on each of the two-dimensional images captured by cameras 300A to 300F. The processes of steps S201 and S202 are executed by coordinate calculation unit 210.

[0051] In step S203, a depth map is generated using the three-dimensional coordinates in the world coordinate system calculated in step S202 and the two-dimensional image (the two-dimensional image acquired at time t1) used in calculating the three-dimensional coordinates. As in the first embodiment, the depths of pixels other than the feature points are obtained by, for example, linear interpolation using the depths of neighboring pixels. The process of step S203 is executed by the feature amount acquisition unit 220.

[0052] In step S204, the depth map acquired in step S203 is fused with a corresponding two-dimensional image. The corresponding two-dimensional image is a two-dimensional image used to calculate the three-dimensional coordinates used to generate the depth map. For example, the depth map is fused with the corresponding two-dimensional image by RGB-D fusion of the ASIF-Net algorithm (Chongyi Li, et al., “ASIF-Net: Attention Steered Interweave Fusion Network for RGB-D Salient Object Detection,” in IEEE Transactions on Cybernetics, vol. 51, no. 1, pp. 88-100, Jan. 2021, [online], [Retrieved September 4, 2023], Internet URL: https: / / doi.org / 10.1109 / TCYB.2020.2969255). As a result, RGB-D data is generated. The process of step S204 is executed by the feature acquisition unit 220.

[0053] In step S205, RGB-D features, which are features of the RGB-D data acquired in step S204, are extracted using a backbone network. This backbone network is a trained machine learning model that has been trained to receive RGB-D data as input and output RGB-D data features, which are features of the RGB-D data. The RGB-D data features also include features related to depth. The process of step S205 is executed by the feature acquisition unit 220.

[0054] In step S206, a BEV feature in the BEV space is extracted using a transformer engine based on the RGB-D data feature acquired in step S205. The process of step S206 is executed by the feature acquisition unit 220.

[0055] In step S207, a bird's-eye view is generated based on the BEV feature amount acquired in step S206. The generated bird's-eye view image is displayed, for example, on a monitor provided in the vehicle 50. The process of step S207 is executed by the bird's-eye view generating unit 230. The process shown in Fig. 5 is executed repeatedly at predetermined time intervals, for example, while the vehicle 50 is traveling.

[0056] As described above, in this embodiment, a depth map generated based on three-dimensional coordinate information acquired using Visual Odometry is used to generate a bird's-eye view. Using VO, three-dimensional coordinates of objects around the vehicle are calculated by a geometric method based on two-dimensional images, internal parameters and external parameters of the camera. Therefore, it is possible to obtain a result with higher reliability for estimating the three-dimensional coordinates of objects around the vehicle. Therefore, it is possible to increase the reliability of the accuracy of the estimated bird's-eye view compared to a mode in which a depth map is generated based on three-dimensional coordinate information acquired using a machine learning model and the generated depth map is used to estimate the bird's-eye view.

[0057] C. Third embodiment: The driving assistance system 1 according to the third embodiment will be described below. The description will focus on configurations different from the first embodiment, and descriptions of configurations similar to those of the first embodiment will be omitted.

[0058] In this embodiment, a bird's-eye image is generated by combining the BEVFusion algorithm (Zhijian Liu, et al., “BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation”, CoRR, 2022) with the VO estimation results of the three-dimensional coordinates of objects around the vehicle 50.

[0059] In the BEVFusion algorithm, first, 2D images are acquired by multiple cameras. Image features of each 2D image are extracted using an encoder model for the cameras. A bird's-eye view representation of the image (bev representation) is generated using the image features extracted from the multiple 2D images. Lidar features are extracted from a point cloud detected by Lidar using an encoder model for Lidar. A bird's-eye view representation of Lidar is generated using the extracted Lidar features. The bird's-eye view representation of the image and the bird's-eye view representation of Lidar are combined. A unified bird's-eye view representation is generated from the combined bird's-eye view representation using an encoder for the BEV. Here, both geometric and semantic information are maintained in the unified bird's-eye view representation (unified BEV representation).

[0060] Fig. 6 shows a bird's-eye view estimation method executed by the estimation device 100 in this embodiment. For example, when the vehicle 50 starts traveling, the process shown in Fig. 6 starts. The cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A to 300F are sequentially stored in the memory 110.

[0061] In step S301, a two-dimensional image is read from memory 110. In step S302, image features (feature vectors) of each of the two-dimensional images acquired by cameras 300A-300F are extracted using a machine learning model such as an encoder model for the camera. This machine learning model is a trained machine learning model that has been trained to receive a two-dimensional image as input and output image features that are features of the two-dimensional image. A corresponding image (two-dimensional image acquired at time t1) used to calculate the three-dimensional coordinates is input to this machine learning model. The processes of steps S301 and S302 are executed by feature acquisition unit 220.

[0062] In step S303, data of a bird's-eye view representation is generated based on the image features acquired in step S302. The generated data of the bird's-eye view representation represents a first BEV feature in the BEV space. Here, as in the LSS algorithm, a ray (a ray from the camera to a point in the three-dimensional space) corresponding to each pixel of each two-dimensional image is assumed, and a discrete depth probability distribution is predicted for each pixel. The cross product of the predicted discrete depth probability distribution and the feature vector is obtained as the depth. Using the estimated depth, the coordinates of each pixel of the two-dimensional image, and the internal and external parameters of the camera, a frustum-shaped point cloud is generated for the two-dimensional image acquired by each camera. The first BEV feature is generated from the frustum point cloud. The process of step S303 is executed by the feature acquisition unit 220.

[0063] In step S304, VO is executed. In this embodiment, the three-dimensional coordinates of an object or the like in the world coordinate system are calculated by VO based on each of the two-dimensional images acquired by the cameras 300A to 300F. Point cloud data is generated using the estimated three-dimensional coordinates. The process of step S304 is executed by the coordinate calculation unit 210.

[0064] In step S305, a three-dimensional feature amount, which is a feature amount of the point cloud data generated based on the three-dimensional coordinates acquired in step S304, is extracted using a machine learning model such as an encoder model for Lidar. This machine learning model is a trained machine learning model that has been trained to input point cloud data and output three-dimensional feature amounts. This machine learning model is also called a second machine learning model. The process of step S305 is executed by the feature amount acquisition unit 220.

[0065] In step S306, data of a bird's-eye view representation (BEV representation) is generated using a machine learning model based on the three-dimensional feature acquired in step S305. The data of the bird's-eye view representation generated here represents a second BEV feature, which is a feature in the BEV space. This machine learning model is a trained machine learning model that receives the three-dimensional feature as input and is trained to output the second BEV feature. The process of step S306 is executed by the feature acquisition unit 220.

[0066] In step S307, the first BEV feature acquired in step S303 and the second BEV feature acquired in step S306 are fused to generate a fused feature. The process of step S307 is executed by the feature acquisition unit 220.

[0067] In step S308, features in the integrated bird's-eye view representation are extracted based on the fusion features using a machine learning model such as an encoder for a BEV. The process of step S308 is executed by the feature acquisition unit 220. In step S309, a bird's-eye image is generated from the fusion features. The generated bird's-eye image is displayed, for example, on a monitor provided in the vehicle 50. The process of step S309 is executed by the bird's-eye view generation unit 230. The process shown in FIG. 6 is executed repeatedly at predetermined time intervals, for example, while the vehicle 50 is traveling.

[0068] As described above, in this embodiment, the second BEV feature used to estimate the bird's-eye view is generated based on three-dimensional coordinate information acquired using the VO. Using the VO, the three-dimensional coordinates of the objects around the vehicle are calculated by a geometric method based on the two-dimensional image, the internal parameters and the external parameters of the camera. Therefore, it is possible to obtain a result with higher reliability for the estimation of the three-dimensional coordinates of the objects around the vehicle. Therefore, it is possible to increase the reliability of the accuracy of the estimated bird's-eye view compared to the aspect in which the second BEV feature is generated based on the three-dimensional coordinate information acquired using a machine learning model and the bird's-eye view is estimated using the generated second BEV feature.

[0069] In addition, in this embodiment, the feature points in the world coordinate system estimated by VO are used instead of a point cloud detected by Lidar, so there is no need to use an expensive Lidar.

[0070] D. Other embodiments: D1. Alternative embodiment 1: In the first embodiment, an example in which VO is used as the self-location estimation method has been described, but Visual Inertial Odometry (hereinafter, VIO) may also be used as the self-location estimation method. In VIO, a camera and an inertial sensor are used. Also, in the configurations according to the second and third embodiments, VIO can be used as the self-location estimation method. According to this embodiment, in order to estimate the three-dimensional coordinates of objects existing around the vehicle, measurement values ​​by an inertial measurement unit are used in addition to two-dimensional images, so that a more reliable result can be obtained for the estimation of the bird's-eye view.

[0071] D2. Alternative embodiment 2: In addition, in any of the first to third embodiments, the VIO is used as the self-position estimation method, and further, the detection value by the wheel speed sensor can be used. For example, an IMU equipped with a correction function using the wheel speed can be used. That is, in this embodiment, the measurement value by the inertial measurement unit and the detection value by the wheel speed sensor are used. In this case, the vehicle 50 is assumed to be equipped with a wheel speed sensor. In order to estimate the three-dimensional coordinates of the object existing around the vehicle, in addition to the two-dimensional image, the measurement value by the inertial measurement unit and the detection value by the wheel speed sensor are used, so that a more reliable result can be obtained for the estimation of the bird's-eye view.

[0072] D3. Alternative embodiment 3: In the first embodiment, an example has been described in which the coordinates of objects around the vehicle 500 are estimated by a feature-based method. Alternatively, the coordinates of objects around the vehicle 500 may be estimated by a direct method. For example, the depth of all pixels of an image can be restored by DTAM (Richard A. Newcombe, et al., “DTAM: Dense Tracking and Mapping in Real-Time”, [online], [Retrieved September 4, 2023], Internet URL: https: / / doi.org / 10.1109 / iccv.2011.6126513).

[0073] D4. Alternative embodiment 4: In the first embodiment, an example has been described in which the coordinates of objects around the vehicle 50 are estimated by a feature-based method, and the depth of pixels other than the feature points is interpolated by linear interpolation. Alternatively, the depth may be interpolated by a depth interpolation method using a machine learning model (Alex Wong, et al., “Unsupervised Depth Completion from Visual Inertial Odometry”. IEEE Robotics and Automation Letters, 5(2), 1899-1906. [online], [searched September 4, 2023], Internet URL: https: / / doi.org / 10.1109 / lra.2020.2969938).

[0074] D5. Alternative embodiment 5: In the first embodiment, an example in which the processes are executed sequentially as shown in FIG. 4 has been described. However, the processes of steps S102 and S103 shown in FIG. 4 and the process of step S104 may be executed in parallel.

[0075] D6. Alternative embodiment 6: In the third embodiment, an example in which the processes are executed sequentially as shown in FIG. 6 has been described. However, the processes of steps S302 and S303 and the processes of steps S304 to S306 shown in FIG. 6 may be executed in parallel.

[0076] The estimation apparatus and the method described in the present disclosure may be realized by a special-purpose computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, the estimation apparatus and the method described in the present disclosure may be realized by a special-purpose computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the estimation apparatus and the method described in the present disclosure may be realized by one or more special-purpose computers configured by a combination of a processor and a memory programmed to execute one or more functions and a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored in a computer-readable non-transitory tangible recording medium as instructions executed by a computer.

[0077] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention column can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0078] 1 ... driving assistance system, 50 ... vehicle, 100 ... estimation device, 110 ... memory, 120 ... input / output interface, 150 ... CPU, 190 ... bus, 210 ... coordinate calculation unit, 220 ... feature amount acquisition unit, 230 ... bird's-eye view generation unit, 300 ... camera group, 300A to 300F ... cameras

Claims

1. An estimation device (100), A coordinate calculation unit (210) that calculates three-dimensional coordinates of an object in a time-series consecutive two-dimensional image captured by the same camera using visual odometry as a self-position estimation method, the coordinate calculation unit (210) calculating the three-dimensional coordinates of an object present around the vehicle (500) based on each of the two-dimensional images in which the outside of the vehicle is captured by a plurality of cameras (300A to 300F) provided on the vehicle; a feature amount acquisition unit (220) that acquires a BEV feature amount, which is a feature amount in a BEV (Bird's eye view) space, based on the three-dimensional coordinates and the two-dimensional image in which the outside of the vehicle is captured, by an algorithm for BEV estimation; a bird's-eye view generating unit (230) that generates a bird's-eye view, which is an image showing a state in which the vehicle is viewed from above, based on the BEV feature amount; Equipped with Estimation device.

2. The estimation device according to claim 1 , The feature acquisition unit is using the three-dimensional coordinates to generate a depth map for each of the two-dimensional images used to calculate the three-dimensional coordinates; a plurality of machine learning models each trained to receive the two-dimensional image captured by one of the plurality of cameras as an input and output image features that are features of the two-dimensional image captured by the one camera, the two-dimensional image captured by the corresponding camera is input to each of the plurality of machine learning models, and the image features output by each of the plurality of machine learning models are obtained; generating a frustum-shaped point cloud of the two-dimensional images captured by each of the plurality of cameras using the image feature amounts of the two-dimensional images and the depth map corresponding to the two-dimensional images; generating the BEV feature based on the point cloud of the viewing frustum; Estimation device.

3. The estimation device according to claim 2, Visual Inertial Odometry is used as the self-location estimation method. Estimation device.

4. The estimation device according to claim 3, The self-position estimation method further includes using a detection value obtained by a wheel speed sensor. Estimation device.

5. The estimation device according to claim 1 , The feature acquisition unit is using the three-dimensional coordinates to generate a depth map for each of the two-dimensional images used to calculate the three-dimensional coordinates; fusing the two-dimensional image and the depth map corresponding to the two-dimensional image to generate RGB-D data; a plurality of machine learning models each trained to receive the RGB-D data generated based on the two-dimensional image captured by one of the plurality of cameras as an input and output RGB-D data feature amounts that are feature amounts of the RGB-D data captured by the one camera, the RGB-D data generated based on the two-dimensional image captured by a corresponding camera is input to the plurality of machine learning models, and the RGB-D data feature amounts output by each of the plurality of machine learning models are acquired; generating the BEV feature based on the RGB-D data feature; Estimation device.

6. The estimation device according to claim 5, Visual Inertial Odometry is used as the self-location estimation method. Estimation device.

7. The estimation device according to claim 6, The self-position estimation method further includes using a detection value obtained by a wheel speed sensor. Estimation device.

8. The estimation device according to claim 1 , The feature acquisition unit is a plurality of machine learning models each trained to receive the two-dimensional image captured by one of the plurality of cameras as an input and output image features that are features of the two-dimensional image captured by the one camera, the two-dimensional image captured by the corresponding camera is input to each of the plurality of machine learning models, and the image features output by each of the plurality of machine learning models are obtained; generating a first BEV feature, which is a feature in the BEV space, based on the image feature; inputting the three-dimensional coordinates into a second machine learning model that has been trained to receive the three-dimensional coordinates as an input and output three-dimensional feature values ​​that are feature values ​​of the three-dimensional coordinates, and acquiring the three-dimensional feature values ​​output by the second machine learning model; generating a second BEV feature, which is a feature in the BEV space, based on the three-dimensional feature; 、 The bird's-eye view generating unit generates the bird's-eye view based on a fusion feature obtained by fusing the first BEV feature and the second BEV feature. Estimation device.

9. The estimation device according to claim 8, Visual Inertial Odometry is used as the self-location estimation method. Estimation device.

10. The estimation device according to claim 9, The self-position estimation method further includes using a detection value obtained by a wheel speed sensor. Estimation device.

11. 1. A method of estimation comprising: A step of calculating three-dimensional coordinates of an object in a time-series consecutive two-dimensional image captured by a same camera using Visual Odometry as a self-location estimation method, the step of calculating the three-dimensional coordinates of an object present around the vehicle based on each of the two-dimensional images in which the outside of the vehicle is captured by a plurality of cameras provided on the vehicle; acquiring BEV features, which are features in a BEV (Bird's eye view) space, based on the three-dimensional coordinates and the two-dimensional image in which the outside of the vehicle is captured, by an algorithm for BEV estimation; generating a bird's-eye view based on the BEV feature amount, the bird's-eye view being an image showing a state of the vehicle as seen from above; An estimation method including:

Images