Estimation device and estimation method

Abstract

To provide technique for reducing the influence of synchronization errors in adjusting BEV features.SOLUTION: An estimation device 100 includes: a position calculation unit 210 configured to calculate three-dimensional coordinates of an object in a two-dimensional image, using Visual Odometry as a self-position estimation method, on the basis of time-series sequential two-dimensional images captured by the same camera 300A, the position calculation unit calculating a position and an orientation of a vehicle, on the basis of two-dimensional images of the outside of the vehicle captured by at least one of a plurality of cameras provided on the vehicle; a feature acquisition unit 220 configured to acquire a BEV (Bird-eye-view) feature on the basis of the two-dimensional images of the outside of the vehicle, using a BEV estimation algorithm for acquiring BEV feature which is a feature in a BEV space; and a correction unit 225 configured to correct the BEV feature acquired by the feature acquisition unit, using information representing the position and the orientation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an estimation device and an estimation method. [Background technology]

[0002] In the technology described in Patent Document 1, three-dimensional spatial information (BEV feature amount) is estimated using a machine learning model based on two-dimensional images acquired by multiple cameras mounted on a vehicle, and a bird's-eye view is generated based on the estimated three-dimensional spatial information. In the technology described in Patent Document 1, the three-dimensional spatial information (BEV feature amount) is adjusted using vehicle motion information. The vehicle motion information used includes acceleration, speed, yaw rate, etc. detected by various sensors provided on the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2023 / 0053785 Summary of the Invention [Problem to be solved by the invention]

[0004] These various sensors transmit their detection values ​​to the ECU that generates the bird's-eye image via the CAN (Controller Area Network) bus. Multiple ECUs are connected to the CAN bus, and when the communication load of the CAN bus increases, communication delays occur.

[0005] Here, the multiple cameras capturing 2D images are individually connected to the ECU that generates the bird's-eye images via Ethernet. Therefore, there is almost no time lag between the timing when the ECU that generates the bird's-eye images receives the images captured by the cameras and the timing when the cameras transmit the images.

[0006] On the other hand, if a communication delay occurs in the CAN bus, the time lag between the timing when the ECU that generates the bird's-eye image receives the detection values ​​of various sensors and the timing when the various sensors transmit the detection values ​​becomes large. It is desirable that the timing when the ECU that generates the bird's-eye image acquires the 2D image of the camera and the timing when it acquires the detection values ​​of the sensors are the same. However, due to the influence of the communication delay of the CAN bus, a synchronization error occurs when adjusting the BEV feature values. Therefore, a technology that can reduce the influence of the synchronization error when adjusting the BEV feature values ​​is required. [Means for solving the problem]

[0007] According to one embodiment of the present disclosure, an estimation device (100) is provided. The estimation device includes a position calculation unit (210) that calculates a position and orientation of a vehicle using Visual Odometry as a self-location estimation method based on consecutive two-dimensional images captured in time series by a single camera (300A), the position calculation unit (210) calculating the position and orientation of the vehicle based on the two-dimensional image of the outside of the vehicle captured by at least one of a plurality of cameras provided on the vehicle, a feature amount acquisition unit (220) that acquires BEV feature amounts based on the two-dimensional image of the outside of the vehicle using an algorithm for BEV estimation that acquires BEV feature amounts that are feature amounts in a BEV (Bird's eye view) space, and a correction unit (225) that corrects the BEV feature amounts acquired by the feature amount acquisition unit using information representing the position and orientation.

[0008] According to the estimation device of the above embodiment, the vehicle position and orientation are estimated by visual odometry using images acquired by a camera that is less affected by communication delays compared to sensors connected to an ECU via a CAN bus, and the BEV feature values ​​are corrected using information representing the estimated position and orientation. This reduces the effect of synchronization errors compared to an embodiment in which the BEV feature values ​​are corrected using detection values ​​by a sensor connected to an ECU via a CAN bus. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a schematic configuration of a driving assistance system. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of camera arrangement. [Diagram 3] FIG. 1 is an explanatory diagram of coordinate transformation. [Figure 4] 4 is a flowchart showing a method for generating a bird's-eye view in the first embodiment. [Diagram 5] 10 is a flowchart showing a method for generating a bird's-eye view in the second embodiment. [Figure 6] 13 is a flowchart showing a method for generating a bird's-eye view in the third embodiment. [Figure 7] 13 is a flowchart showing a method for generating a bird's-eye view in the fourth embodiment. [Figure 8] 13 is a flowchart showing a method for generating a bird's-eye view in the fifth embodiment. [Figure 9] 23 is a flowchart showing a method for generating a bird's-eye view in the seventh embodiment. [Figure 10] 10 is a flowchart of a calculation process of the position and orientation in FIG. 9. [Figure 11] 23 is a flowchart showing a method for generating a bird's-eye view in the eighth embodiment. [Figure 12] 12 is a flowchart of a calculation process of the position and orientation in FIG. 11. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] A. First embodiment: A driving assistance system 1 shown in Fig. 1 estimates the surrounding situation of a vehicle 50 using a plurality of two-dimensional images acquired in time series by an image sensor provided in the vehicle 50, and generates a bird's-eye image of the vehicle 50 viewed from above. The bird's-eye image is an image of the vehicle 50 viewed from above. For example, the driving assistance system 1 uses a plurality of two-dimensional images acquired in time series by an image sensor provided in the vehicle 50 to display an image of the vehicle 50 viewed from above on an in-vehicle monitor of the vehicle 50. The vehicle 50 is capable of traveling using an ADAS (Advanced Driving Assistant System).

[0011] The driving assistance system 1 includes an estimation device 100 and a camera group 300. The estimation device 100 and the camera group 300 are mounted on a vehicle 50.

[0012] As shown in FIG. 2, the camera group 300 includes six cameras 300A to 300F provided on the vehicle 50. The cameras 300A to 300F are monocular cameras capable of capturing color images. The camera 300A is provided on the upper part of the windshield of the vehicle 50. The camera 300A captures an image of a predetermined range in front of the vehicle 50. The camera 300B is provided on the lower part of the door mirror on the right side of the vehicle 50. Hereinafter, the right side of the vehicle 50 refers to the right side as seen by a person sitting in the vehicle 50 facing the front of the vehicle 50. The camera 300B captures an image of a predetermined range diagonally to the right front of the vehicle 50.

[0013] Camera 300C is provided at the bottom of the left door mirror. The left side of vehicle 50 refers to the left side as seen by a person sitting in vehicle 50 facing the front of vehicle 50. Camera 300C captures an image of a predetermined range diagonally forward to the left of vehicle 50. Camera 300D is provided on the right door pillar. Camera 300D captures an image of a predetermined range on the right side of vehicle 50. Camera 300E is provided on the left door pillar. Camera 300E captures an image of a predetermined range on the left side of vehicle 50. Camera 300F is provided in the center of the back door. Camera 300F captures an image of a predetermined range behind vehicle 50.

[0014] The cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the acquired images to the estimation device 100. For ease of understanding of the technology, in the embodiment, it is assumed that the cameras 300A to 300F capture images at the same timing.

[0015] The estimation device 100 is a computer including a memory 110, an input / output interface 120, and a CPU (Central Processing Unit) 150. The memory 110 and the input / output interface 120 are connected to the CPU 150 via a bus 190. For example, the functions of the estimation device 100 are realized by a driving control ECU (Electronic Control Unit) that controls driving of the vehicle 50.

[0016] The memory 110 stores various programs and various data used for various processes executed by the estimation device 100. The memory 110 stores data representing a machine learning model used for various processes executed by the estimation device 100.

[0017] The input / output interface 120 is connected to the cameras 300A to 300F via Ethernet (registered trademark).

[0018] CPU 150 is a processor that realizes various functions by executing programs stored in memory 110. For example, CPU 150 stores two-dimensional images received from cameras 300A to 300F in memory 110. In the embodiment, CPU 150 executes the programs stored in memory 110 to function as a position calculation unit 210, a feature acquisition unit 220, a correction unit 225, and a bird's-eye view generation unit 230.

[0019] The position calculation unit 210 calculates the position and attitude of the vehicle 50 and the three-dimensional coordinates of objects in the two-dimensional images using Visual Odometry as a self-position estimation method based on consecutive two-dimensional images in a time series. In this embodiment, the position calculation unit 210 calculates the position and attitude of the vehicle 50 and the three-dimensional coordinates of objects existing around the vehicle 50 using Visual Odometry based on two-dimensional images captured by at least one of the cameras 300A to 300F. Details of the processing by the position calculation unit 210 will be described later.

[0020] The feature amount acquiring unit 220 acquires BEV feature amounts, which are feature amounts in a BEV (Bird's eye view) space, based on two-dimensional images of the outside of the vehicle 50 captured by the cameras 300A to 300F. The BEV feature amount includes coordinate values ​​of a point cloud in a three-dimensional space and information on the color of the point cloud, similar to point cloud data detected by Lidar. The details of the processing of the feature amount acquiring unit 220 will be described later.

[0021] The correction unit 225 corrects the BEV feature amounts generated by the feature amount acquisition unit 220 in accordance with the position and attitude of the vehicle 50. The process of the correction unit 225 will be described in detail later.

[0022] The bird's-eye view generating unit 230 generates a bird's-eye view based on the BEV feature amount. The bird's-eye view is an image showing the vehicle 50 viewed from above. An existing method can be used as a method for generating a bird's-eye view based on the BEV feature amount. The bird's-eye view generating unit 230 can detect objects around the vehicle 50 based on the BEV feature amount and generate a bird's-eye view showing the objects around the vehicle 50. Note that the generated bird's-eye view does not need to include all objects around the vehicle 50. For example, the bird's-eye view generating unit 230 may detect lanes around the vehicle 50 based on the BEV feature amount and generate a bird's-eye view including the vehicle 50 and the lanes.

[0023] Next, Visual Odometry (hereinafter, VO) will be described. In VO, feature points are detected in multiple two-dimensional images that are consecutive in time series, and the position and attitude of the vehicle 50 are estimated using the coordinates of the feature points in the two-dimensional images. A feature point is a point on an image that can be reliably detected.

[0024] A corner detection method is used to detect feature points. For example, an intersection of two edges is detected as a corner. Alternatively, a point where two prominent edges with different directions exist in a certain local neighborhood is detected as a corner. For example, Harris corner detection, SHIFT (scale-invariant feature transform), SURF (Speeded up robust features), and ORB (Oriented FAST and Rotated BRIEF) can be used as the corner detection method. In addition, many corner detection methods can detect not only corner points but also feature points. The number of feature points detected from one two-dimensional image is, for example, 100.

[0025] For example, feature points are detected from three consecutive two-dimensional images captured in time series by the same camera. The times at which the three two-dimensional images are captured are assumed to be times t0, t1, and t2. Time t0 is the earliest time, and time t2 is the latest time. Feature points in the two-dimensional images captured at times t0, t1, and t2 are detected, and the coordinates and brightness values ​​of each detected feature point are recorded. Examples of the detected feature points include corners of buildings, outlines of traffic signs, color boundaries of traffic signs, and corners of curbs.

[0026] As shown in FIG. 3, the coordinates of the detected feature points are obtained by projecting the three-dimensional position of an object, etc. in the world coordinate system onto the image coordinate system. The relationship between the coordinates of the feature points in the two-dimensional image and the three-dimensional position of the feature points in the world coordinate system can be expressed as in the following formula (1). In formula (1), f x , f y represents the focal length. x , c yrepresents the position of the intersection of the optical axis and the projection plane. 11 ~r 33 represents the rotation matrix. t1 to t3 represent translation vectors. s represents the scale factor. The scale factor s is estimated using the results of measuring the position and orientation of the camera in advance using an IMU (Inertial Measurement Unit) or the like.

[0027]

number

[0028] Equation (1) can be expressed as equation (2).

[0029]

number

[0030] The matrix K in equation (2) is called the internal parameters of the camera. The matrix [RT] is called the external parameters of the camera. The external parameters of the camera are determined by the position T and orientation R of the camera coordinate system. As shown in equation (2), the position u in the 2D image I =[uv 1] t is the position T and orientation R in the camera coordinate system, and the position X in the world coordinate system. w =[X W Y W Z W ] t It is expressed as a function of

[0031] When tracking the positions of feature points corresponding to the same object, etc. detected in multiple consecutive 2D images in time series, the position of the feature point of interest changes due to the movement of the vehicle 50, i.e., the movement of the camera group.

[0032] In VO, the position and orientation of the camera and the 3D coordinates of the feature points in the world coordinate system at time t1 are estimated by bundle adjustment based on the positions of the feature points detected in each 2D image. Specifically, the position and orientation of the camera and the 3D coordinates of the feature points in the world coordinate system at time t1 are calculated using the coordinates of corresponding feature points detected in multiple 2D images and the assumed camera position T and orientation R. Then, the calculated 3D coordinates are reprojected onto the image coordinate system, and the distance (reprojection error) between the projected point and the feature point detected from the 2D image is estimated.

[0033] The position u of a feature point in a two-dimensional image can be found by a projection function as shown in equation (3). Note that the internal parameters are assumed to be fixed, and are not considered in equation (3). The projection function is a function that maps the coordinate system of the object space (world coordinate system) to the coordinate system of the image space (image coordinate system).

[0034]

number

[0035] Position of feature points detected from a 2D image, u I and the position of the feature point obtained by the projection function, u P The error between and is defined as the following equation (4). In equation (4), u I represents a set of coordinates of feature points detected from a 2D image.

[0036]

number

[0037] The error function C expressed in equation (4) camThe position and orientation of the camera and the three-dimensional coordinates of the feature points in the world coordinate system at time t1 are estimated by an optimization process that minimizes. The estimated position and orientation of the camera are expressed by a rotation matrix and a translation vector. Minimizing the error expressed in equation (4) means minimizing the error between the actual coordinates of the detected feature points in the image and the coordinates of the feature points calculated by the projection function. In this way, in VO, the position T and orientation R of the camera in the world coordinate system are estimated.

[0038] Next, an algorithm for estimating the BEV in this embodiment will be described.

[0039] In this embodiment, a bird's-eye image is generated by combining the algorithm disclosed in U.S. Patent Application Publication No. 2023 / 0053785 with the estimation results of the position and attitude of the vehicle 50 by VO. In the algorithm disclosed in the above document, a machine learning model having a Transformer architecture is used to convert image features corresponding to each of the multiple cameras from the image space to the BEV space, thereby acquiring BEV features.

[0040] In the algorithm disclosed in the above document, first, two-dimensional images are respectively acquired by a plurality of cameras. Image features of each of the two-dimensional images are extracted using a backbone network. The backbone network is, for example, a CNN. The extracted image features are input to a Transformer engine, which is a machine learning model having a Transformer architecture. The Transformer engine is a machine learning model that utilizes an Attention mechanism. The Transformer engine is a machine learning model trained to input image features extracted from a plurality of two-dimensional images, fuse the image features, and project the image features into a BEV space.

[0041] Fig. 4 shows a bird's-eye view estimation method executed by the estimation device 100 in this embodiment. For example, when the autonomous driving function is enabled, the process shown in Fig. 4 is started. While the following process is being executed, the cameras 300A-300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A-300F are sequentially stored in the memory 110.

[0042] In step S101, the feature acquisition unit 220 extracts image features of the two-dimensional image. Specifically, the image features (feature vectors) of the two-dimensional image acquired by the camera 300A are extracted using a machine learning model such as CNN. The extracted image features of the two-dimensional image are stored in the memory 110. A machine learning model that has been trained to output the image features of the two-dimensional image when the two-dimensional image is input is used to extract the image features of the two-dimensional image acquired by the camera 300A. This machine learning model is generated by machine learning using two-dimensional images captured by a camera attached at a position similar to that of the camera 300A as training data. In addition, the two-dimensional images acquired by the cameras 300B to 300F are used as input, and the image features of the two-dimensional images are extracted using the machine learning models corresponding to the respective cameras. The extracted image features of the two-dimensional images are stored in the memory 110.

[0043] In step S101, for example, image feature amounts of one set of two-dimensional images may be extracted. One set of two-dimensional images is six two-dimensional images captured at the same time by the cameras 300A to 300F.

[0044] Alternatively, in step S101, the image feature extraction process may be performed on two-dimensional images included in a predetermined number of sets among the two-dimensional images stored in memory 110 and not subjected to the image feature extraction process. The predetermined number is, for example, "2". In this case, when the process of step S101 is performed once, for example, image feature amounts of each two-dimensional image captured by cameras 300A to 300F at time tc1 and image feature amounts of each two-dimensional image captured by cameras 300A to 300F at time (tc1-1) are extracted. Time tc1 is the time when the immediately preceding two-dimensional image was captured. tc1-1 is the time when the two-dimensional image was captured prior to time tc1. No two-dimensional image is captured between time tc1-1 and time tc1.

[0045] In step S103, the feature amount acquiring unit 220 generates BEV features in the BEV space using a transformer engine based on the image feature amounts extracted in step S101. The generated BEV features are stored in the memory 110.

[0046] For example, if image features of a set of two-dimensional images are extracted in step S101, BEV features are generated based on the image features of the set of two-dimensional images in step S103.

[0047] In step S105, VO is executed by position calculation unit 210. In this embodiment, the position and orientation of camera 300A in the world coordinate system are calculated by VO based on a two-dimensional image acquired by camera 300A. The calculated position and orientation of camera 300A are treated as the position and orientation of vehicle 50. Note that the processes of steps S101 and S103 and the process of step S105 are executed in parallel.

[0048] In step S105, the position and attitude of the vehicle 50 at two or more consecutive times may be calculated. In this case, when the process of step S105 is executed once, the position and attitude of the vehicle 50 at the time (tc1-1) and the time tc1 are calculated, for example.

[0049] In step S107, correction unit 225 determines whether or not correction processing can be started. Correction processing can be started when all of the following conditions (i) to (iv) are satisfied. Hereinafter, the current frame refers to the two-dimensional image acquired immediately before, and the previous frame refers to the two-dimensional image acquired before acquiring the current frame. No other frames have been acquired between the acquisition of the current frame and the acquisition of the previous frame. (i) The BEV feature is generated based on the image feature of the current frame acquired by each of the cameras 300A to 300F. (ii) The BEV feature amount is generated based on the image feature amount of the previous frame captured by each of the cameras 300A to 300F. (iii) The position and orientation of the vehicle 50 have been calculated based on the current frame. (iv) The position and orientation of the vehicle 50 have been calculated based on the previous frame.

[0050] If it is determined that the correction process can be started (step S107; YES), step S109 is executed. If the correction process cannot be started (step S107; NO), steps S101 and S105 are executed again.

[0051] In step S109, the correction unit 225 acquires motion information of the vehicle 50. The motion information of the vehicle 50 is the amount of change between the position and attitude of the vehicle 50 at time tc1 and the position and attitude of the vehicle 50 at time (tc1-1).

[0052] In step S111, the correction unit 225 corrects the BEV feature amount. Specifically, first, the BEV feature amount based on the previous frame is aligned using the motion information of the vehicle 50. Furthermore, the aligned BEV feature amount and the BEV feature amount based on the current frame are combined to obtain a corrected BEV feature amount. The corrected BEV feature amount is stored in the memory 110. For example, when a vehicle detected in the previous frame is hidden by another vehicle or the like and not detected in the current frame, the corrected BEV feature amount is used in a process of estimating the position of the vehicle in the current frame.

[0053] In step S113, a bird's-eye view is generated based on, for example, the BEV feature amount based on the current frame by the bird's-eye view generating unit 230. The generated bird's-eye view image is displayed on, for example, a monitor provided in the vehicle 50.

[0054] In step S115, it is determined whether or not to continue the process according to a predetermined condition. For example, it is determined that the process is to be continued while the autonomous driving function is enabled (step S115; YES). In this case, the processes of steps S101 and S105 are executed again. Also, for example, when a state is reached in which an abnormality in a sensor is detected, it is determined that the process is not to be continued (step S115; NO). In this case, the process shown in FIG. 4 is ended. The process of step S115 is executed, for example, by the bird's-eye view generating unit 230.

[0055] As described above, in this embodiment, the vehicle position and orientation are estimated using visual odometry using images acquired by a camera that is less affected by communication delays compared to sensors connected to an ECU via a CAN bus, and the BEV feature values ​​are corrected using information representing the estimated position and orientation. This reduces the effects of synchronization errors compared to an embodiment in which the BEV feature values ​​are corrected using values ​​detected by a sensor connected to an ECU via a CAN bus.

[0056] B. Second embodiment: The driving assistance system 1 according to the second embodiment will be described below. The description will focus on configurations different from the first embodiment, and descriptions of configurations similar to the first embodiment will be omitted.

[0057] In this embodiment, a bird's-eye image is generated by combining the estimation results of the position and orientation of the vehicle 50 by VO with a machine learning model having an Attention mechanism disclosed in the UniFusion algorithm (Zequn Qin, et al., “UniFusion: Unified Multi-view Fusion Transformer for Spatial-Temporal Representation in Bird's-Eye-View”, arXiv, [online], [searched September 4, 2023], Internet URL: https: / / arxiv.org / abs / 2207.08536). In the UniFusion algorithm, a virtual image generated using the position and orientation information of the vehicle 50 based on the image features corresponding to each of the multiple cameras acquired in the past, and the image features corresponding to each of the multiple cameras acquired this time, is converted from the image space to the BEV space using a machine learning model, and the BEV features are acquired. In this embodiment, a virtual image is generated using the estimated results of the position and attitude of the vehicle 50 estimated by VO, thereby eliminating the effects of synchronization errors between the various sensors obtained via the camera and CAN bus mentioned above.

[0058] 5 shows a bird's-eye view estimation method executed by the estimation device 100 in this embodiment. While the following process is being executed, the cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A to 300F are stored in the memory 110 one by one.

[0059] In step S201, the feature acquisition unit 220 extracts image features of the two-dimensional image. Specifically, the image features (feature vectors) of the two-dimensional image acquired by the camera 300A are extracted using a machine learning model such as CNN. The extracted image features of the two-dimensional image are stored in the memory 110. To extract the image features of the two-dimensional image acquired by the camera 300A, a machine learning model that is trained to output the image features of the two-dimensional image when the two-dimensional image is input is used. This machine learning model is generated by machine learning using two-dimensional images captured by a camera attached at a position similar to that of the camera 300A as training data. In addition, the two-dimensional images acquired by the cameras 300B to 300F are input, and the image features of the two-dimensional images are extracted using the machine learning models corresponding to the respective cameras. The extracted image features of the two-dimensional images are stored in the memory 110.

[0060] In step S201, image feature amounts may be extracted for one set of two-dimensional images, which are six two-dimensional images captured at the same time by the cameras 300A to 300F.

[0061] Alternatively, in step S201, image feature extraction processing may be performed on two-dimensional images included in a predetermined number of sets that are stored in memory 110 and have not been subjected to image feature extraction processing.

[0062] In step S203, VO is executed by position calculation unit 210. In this embodiment, the position and orientation of camera 300A in the world coordinate system are calculated by VO based on a two-dimensional image acquired by camera 300A. Note that the process of step S201 and the process of step S203 are executed in parallel.

[0063] In step S205, it is determined whether or not the generation process of the BEV feature can be started. The case where the generation process of the BEV feature can be started refers to a case where all of the conditions (i) to (vi) are satisfied. Time tc2 is the time when the immediately preceding two-dimensional image was acquired. Time (tc2-1) is the time when the two-dimensional image was acquired prior to time tc2. No two-dimensional image is acquired between time (tc2-1) and time tc2. Time (tc2-2) is the time when the two-dimensional image was acquired prior to time tc2-1. No two-dimensional image is acquired between time (tc2-2) and time (tc2-1). (i) Image feature amounts of the two-dimensional images captured by the cameras 300A to 300F at time tc2 are extracted. (ii) Image feature amounts are extracted from the two-dimensional images captured by the cameras 300A to 300F at time (tc2-1). (iii) Image feature amounts are extracted from the two-dimensional images captured by the cameras 300A to 300F at time (tc2-2). (iv) The position and attitude of the vehicle 50 at time tc2 are calculated. (v) The position and attitude of the vehicle 50 at time (tc2-1) are calculated. (vi) The position and attitude of the vehicle 50 at time (tc2-2) are calculated.

[0064] If it is determined that the generation process of the BEV feature can be started (step S205; YES), step S207 is executed. If the correction process cannot be started (step S205; NO), steps S201 and S203 are executed again.

[0065] In step S207, feature acquisition unit 220 generates BEV features in the BEV space using the attention network in the above-mentioned UniFusion algorithm based on the image features of each two-dimensional image acquired by cameras 300A to 300F and the position and orientation information of vehicle 50. The generated BEV features are stored in memory 110. In the UniFusion algorithm, the BEV features are generated based on information obtained by converting the image features of the two-dimensional images acquired by each camera at time (tc2-2) and time (tc2-1) into a virtual image at time tc2 using the vehicle position and orientation information, and the image features acquired at time tc2.

[0066] In step S209, a bird's-eye view is generated based on the BEV feature amount by the bird's-eye view generating unit 230. The generated bird's-eye view image is displayed on a monitor provided in the vehicle 50, for example.

[0067] In step S211, it is determined whether or not to continue the process according to a predetermined condition. For example, it is determined that the process is to be continued while the autonomous driving function is enabled (step S211; YES). In this case, the processes of steps S201 and S203 are executed again. Also, for example, when a state is reached in which an abnormality in a sensor is detected, it is determined that the process is not to be continued (step S211; NO). In this case, the process shown in FIG. 5 is ended. The process of step S211 is executed, for example, by the bird's-eye view generating unit 230.

[0068] As described above, in this embodiment, the vehicle position and orientation are estimated using visual odometry using images acquired by a camera that is less affected by communication delays compared to sensors connected to an ECU via a CAN bus, and the BEVt feature values ​​are corrected using information representing the estimated position and orientation. This can reduce the effects of synchronization errors compared to an embodiment in which the BEV feature values ​​are corrected using values ​​detected by a sensor connected to an ECU via a CAN bus.

[0069] C. Third embodiment: The driving assistance system 1 according to the third embodiment will be described below. The description will focus on configurations different from the first embodiment, and descriptions of configurations similar to those of the first embodiment will be omitted.

[0070] In this embodiment, a bird's-eye view image is generated by combining the LSS (Lift, Splat, Shoot) algorithm (Jonah Philion, Sanja Fidler, “Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D”, in European Conference on Computer Vision. Springer, 2020, pp. 194-210) with the estimation results of the position and attitude of the vehicle 50 using VO.

[0071] In the LSS algorithm, first, two-dimensional images are acquired by the cameras 300A to 300F, respectively. A feature vector of each of the two-dimensional images is extracted using a machine learning model such as CNN. The depth of each pixel is estimated for each of the two-dimensional images acquired by each camera. Since the estimated depth is ambiguous, a light ray (a light ray from the camera to a point in three-dimensional space) corresponding to each pixel is assumed, and a discrete depth probability distribution is predicted for each pixel. The cross product of the predicted discrete depth probability distribution and the feature vector is calculated, and a series of points are obtained as information representing the pixel depth. As a result, a frustum-shaped point cloud is generated for each camera. BEV features are generated from the frustum point cloud using the internal parameters and external parameters of the camera.

[0072] Fig. 6 shows a bird's-eye view estimation method executed by the estimation device 100 in this embodiment. For example, when the autonomous driving function is enabled, the process shown in Fig. 6 is started. While the following process is being executed, the cameras 300A-300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A-300F are sequentially stored in the memory 110.

[0073] In step S301, the feature acquisition unit 220 extracts image features of the two-dimensional image. Specifically, the image features (feature vector) of the two-dimensional image acquired by the camera 300A are extracted using a machine learning model such as CNN. The extracted image features of the two-dimensional image are stored in the memory 110. A machine learning model that is trained to output the image features of the two-dimensional image when the two-dimensional image is input is used to extract the image features of the two-dimensional image acquired by the camera 300A. This machine learning model is generated by machine learning using two-dimensional images captured by a camera attached at a position similar to that of the camera 300A as training data. In addition, the two-dimensional images acquired by the cameras 300B to 300F are input, and the image features of the two-dimensional images are extracted using the machine learning models corresponding to each camera. The extracted image features of the two-dimensional images are stored in the memory 110. In step S301, 1×1 convolution is performed to extract the image features (feature vector). Therefore, the size of the two-dimensional image is not changed.

[0074] In step S301, image feature amounts may be extracted for one set of two-dimensional images, which are six two-dimensional images captured at the same time by the cameras 300A to 300F.

[0075] Alternatively, in step S301, image feature extraction processing may be performed on 2D images included in a predetermined number of sets that are stored in memory 110 and have not been subjected to image feature extraction processing.

[0076] In step S303, a depth map is generated. Specifically, first, VO is executed by the position calculation unit 210. Here, the three-dimensional coordinates of objects around the vehicle 50 in the world coordinate system are calculated by VO based on the two-dimensional images acquired by the cameras 300A to 300F.

[0077] Thereafter, the feature amount acquiring unit 220 generates a depth map using the three-dimensional coordinates of the feature points in the two-dimensional image in the world coordinate system and the two-dimensional image used to calculate the three-dimensional coordinates. Here, depth maps are generated for the two-dimensional images acquired by the cameras 300A to 300F, respectively. In this embodiment, the coordinates are estimated by a feature based method. Therefore, the coordinates in the world coordinate system are estimated only for the feature points detected from the corresponding image. Therefore, the coordinates in the world coordinate system are not estimated for pixels other than the feature points. Therefore, the depths of the pixels other than the feature points can be obtained by, for example, linear interpolation using the depths of nearby pixels.

[0078] In step S303, a depth map of one set of two-dimensional images may be generated. One set of two-dimensional images is six two-dimensional images captured at the same time by the cameras 300A to 300F. Alternatively, depth maps of two or more sets of two-dimensional images may be generated.

[0079] In step S305, VO is executed by position calculation unit 210. In this embodiment, the position and orientation of camera 300A in the world coordinate system are calculated by VO based on a two-dimensional image acquired by camera 300A. Note that the processes of steps S301 and S303 and the process of step S305 are executed in parallel.

[0080] In step S307, the feature amount acquiring unit 220 obtains the cross product of the image feature amount (feature amount vector) extracted from the two-dimensional image in step S301 and the depth map generated in step S303 as a cross product feature amount.

[0081] In step S309, similar to the LSS algorithm, the feature acquisition unit 220 generates a point cloud of the view frustum based on the cross product feature acquired in step S307. Specifically, assuming a light ray (a light ray from the camera to a point in three-dimensional space) corresponding to each pixel of each two-dimensional image, a discrete depth probability distribution is predicted for each pixel. The cross product of the predicted discrete depth probability distribution and a vector representing the cross product feature is obtained as the depth. Using the estimated depth, the coordinates of each pixel of the two-dimensional image, and the internal and external parameters of the camera, a point cloud of the view frustum is generated for the two-dimensional image acquired by each camera.

[0082] In step S311, similarly to the LSS algorithm, the feature amount acquisition unit 220 generates BEV feature amounts from the point group of the view frustum generated in step S309. The generated BEV feature amounts are stored in the memory 110.

[0083] In step S313, it is determined whether or not the correction process can be started by the correction unit 225. The conditions for determining whether or not the correction process can be started are the same as those in the first embodiment.

[0084] If it is determined that the correction process can be started (step S313; YES), step S315 is executed. If the correction process cannot be started (step S313; NO), steps S301 and S305 are executed again.

[0085] In step S315, the correction unit 225 acquires motion information of the vehicle 50, similarly to the first embodiment (see step S109 in FIG. 4).

[0086] In step S317, the correction unit 225 corrects the BEV feature amount, similarly to the first embodiment (see step S111 in FIG. 4).

[0087] In step S319, a bird's-eye view is generated based on the BEV feature amount by the bird's-eye view generating unit 230. The generated bird's-eye view image is displayed on a monitor provided in the vehicle 50, for example.

[0088] In step S321, it is determined whether or not to continue the process according to a predetermined condition. For example, it is determined that the process is to be continued while the autonomous driving function is enabled (step S321; YES). In this case, the processes of steps S301 and S305 are executed again. Also, for example, when a state is reached in which an abnormality in a sensor is detected, it is determined that the process is not to be continued (step S321; NO). In this case, the process shown in FIG. 6 is ended. The process of step S321 is executed, for example, by the bird's-eye view generating unit 230.

[0089] As described above, in this embodiment, the vehicle position and orientation are estimated using visual odometry using images acquired by a camera that is less affected by communication delays compared to sensors connected to an ECU via a CAN bus, and the BEV feature values ​​are corrected using information representing the estimated position and orientation. This reduces the effects of synchronization errors compared to an embodiment in which the BEV feature values ​​are corrected using values ​​detected by a sensor connected to an ECU via a CAN bus.

[0090] D. Fourth embodiment: The driving assistance system 1 according to the fourth embodiment will be described below. The description will focus on configurations different from the first embodiment, and descriptions of configurations similar to the first embodiment will be omitted.

[0091] In the fourth embodiment, as in the first embodiment, a bird's-eye image is generated by combining the algorithm disclosed in U.S. Patent Application Publication No. 2023 / 0053785 with the estimation results of the three-dimensional coordinates of objects around the vehicle 50 by VO.

[0092] 7 shows a bird's-eye view estimation method executed by the estimation device 100 in this embodiment. While the following process is being executed, the cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A to 300F are stored in the memory 110 one by one.

[0093] In step S401, similarly to the first embodiment (see step S101 in FIG. 4), the feature acquisition unit 220 extracts image features of the two-dimensional image, and the extracted image features of the two-dimensional image are stored in the memory 110.

[0094] In step S401, image features of one set of two-dimensional images may be extracted. Alternatively, image feature extraction may be performed on two-dimensional images included in a predetermined number of sets among the two-dimensional images stored in memory 110 and not subjected to image feature extraction processing.

[0095] In step S403, the feature acquisition unit 220 generates BEV features in the BEV space using a transformer engine based on the image features of each of the two-dimensional images acquired by the cameras 300A to 300F. The generated BEV features are stored in the memory 110.

[0096] In step S405, VO is executed by position calculation section 210. In this embodiment, the positions and orientations of cameras 300A to 300F in the world coordinate system are calculated by VO for each of the two-dimensional images captured by cameras 300A to 300F.

[0097] In step S405, the position and attitude of the vehicle 50 at two or more consecutive times may be calculated.

[0098] In step S407, the average of the positions and orientations of cameras 300A to 300F is obtained. In this embodiment, the obtained average of the positions and orientations is treated as the position and orientation of vehicle 50. In the first embodiment, the position and orientation of camera 300A was treated as the position and orientation of vehicle 50, but in this embodiment, the average of the positions and orientations of cameras 300A to 300F is treated as the position and orientation of vehicle 50. Note that the processes of steps S401 and S403 and the processes of steps S405 and S407 are executed in parallel.

[0099] In step S407, the average of the positions and orientations of the cameras 300A to 300F at two or more consecutive times may be calculated.

[0100] In step S409, it is determined whether or not the correction process can be started by the correction unit 225. The conditions for determining whether or not the correction process can be started are the same as those in the first embodiment.

[0101] In step S411, the correction unit 225 acquires motion information of the vehicle 50, similarly to the first embodiment (see step S109 in FIG. 4).

[0102] In step S413, the correction unit 225 corrects the BEV feature amount, similarly to the first embodiment (see step S111 in FIG. 4).

[0103] In step S415, a bird's-eye view is generated based on the BEV feature amount by the bird's-eye view generating unit 230. The generated bird's-eye view image is displayed on a monitor provided in the vehicle 50, for example.

[0104] In step S417, it is determined whether or not to continue the process according to a predetermined condition. For example, it is determined that the process is to be continued while the autonomous driving function is enabled (step S417; YES). In this case, the processes of steps S401 and S405 are executed again. Also, for example, when the vehicle 50 stops traveling, it is determined that the process is not to be continued (step S417; NO). In this case, the process shown in FIG. 7 is ended. The process of step S417 is executed, for example, by the bird's-eye view generating unit 230.

[0105] In this embodiment, the position and orientation of each camera are obtained based on two-dimensional images acquired by the multiple cameras, and the average of the positions and orientations of the multiple cameras is regarded as the position and orientation of the vehicle 50. For example, there are cases where a sufficient number of feature points outside the vehicle 50 cannot be detected using only a camera such as camera 300A that captures an image in front of the vehicle 50. In such cases, the accuracy of estimating the position and orientation of the vehicle 50 can be improved by obtaining the average of the positions and orientations of the multiple cameras.

[0106] E. Fifth embodiment: In this embodiment, the position calculation unit 210 calculates the position and attitude of the vehicle 50 using Visual Inertial Odometry (VIO) as a self-position estimation method based on two-dimensional images that are continuous in time series. In this case, the vehicle 50 is assumed to be equipped with an IMU.

[0107] In VIO, the position and attitude of the vehicle 50, the three-dimensional coordinates of objects around the vehicle 50, and the synchronization error between the camera and the IMU are calculated using the image acquired by the camera and the detection value of the IMU installed in the vehicle 50. As a VIO method, for example, a known technique disclosed in the literature (Tong Qin, and Shaojie Shen, “Online Temporal Calibration for Monocular Visual-Inertial Systems”, IEEE International Workshop on Intelligent Robots and Systems (IROS), [online], [searched September 4, 2023], Internet URL: https: / / doi.org / 10.1109 / iros.2018.8593603) can be used.

[0108] Fig. 8 shows a bird's-eye view estimation method executed by the estimation device 100 in this embodiment. In Fig. 8, the same processes as those in the first embodiment (see Fig. 4) are denoted by the same reference numerals. For example, when the autonomous driving function is enabled, the process shown in Fig. 8 is started. While the following process is being executed, the cameras 300A-300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A-300F are stored sequentially in the memory 110.

[0109] In step S101, similarly to the first embodiment, image features (feature vectors) of a two-dimensional image captured by camera 300A are extracted by feature acquisition unit 220. The extracted image features of the two-dimensional image are stored in memory 110.

[0110] In step S103, similarly to the first embodiment, the feature amount acquiring unit 220 generates a BEV feature amount based on the image feature amount of each of the two-dimensional images acquired by the cameras 300A to 300F. The generated BEV feature amount is stored in the memory 110.

[0111] In step S106, VIO is executed by position calculation unit 210. In this embodiment, the position and orientation of camera 300A in the world coordinate system are calculated by VIO for the two-dimensional image acquired by camera 300A. The calculated position and orientation of camera 300A are treated as the position and orientation of vehicle 50. Note that the processes of steps S101 and S103 and the process of step S106 are executed in parallel.

[0112] In step S106, the position and attitude of the vehicle 50 at two or more consecutive times may be calculated.

[0113] The processes from step S107 onwards are similar to those in the first embodiment, and therefore the description thereof will be omitted.

[0114] In this embodiment, in order to estimate the position and attitude of the vehicle 50, in addition to two-dimensional images, measurement values ​​from an inertial measurement unit are used, so that more reliable results can be obtained for estimating the position and attitude of the vehicle 50.

[0115] F. Sixth embodiment: In any of the first to fourth embodiments, the VIO can be used as a self-location estimation method. Also, in any of the first to fourth embodiments, the VIO with a wheel speed sensor added can be used as a self-location estimation method. In general, immediately after the vehicle starts moving from a stopped state, the VIO cannot distinguish between vehicle acceleration and acceleration changes due to attitude changes, resulting in a large estimation error in the vehicle's position and attitude. By adding a wheel speed sensor, the estimation error in the vehicle's position and attitude of the VIO can be reduced. In this case, the vehicle 50 is assumed to be equipped with a wheel speed sensor. In order to estimate the position and attitude of the vehicle 50, a measurement value by an inertial measurement unit and a detection value by a wheel speed sensor are used in addition to a two-dimensional image, so that a more reliable result can be obtained for the estimation of the position and attitude of the vehicle 50.

[0116] G. Seventh embodiment: The driving assistance system 1 according to the seventh embodiment will be described below. The description will focus on configurations different from the first embodiment, and descriptions of configurations similar to the first embodiment will be omitted.

[0117] Fig. 9 shows a bird's-eye view estimation method executed by the estimation device 100 in this embodiment. In Fig. 9, the same processes as those in the first embodiment (see Fig. 4) are denoted by the same reference numerals. While the following processes are being executed, the cameras 300A-300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A-300F are successively stored in the memory 110.

[0118] The processes in steps S101 and S103 are similar to those in the first embodiment, and therefore the description thereof will be omitted.

[0119] In step S102, a position and orientation calculation process is executed. Note that the processes in steps S101 and S103 and the process in step S102 are executed in parallel.

[0120] Fig. 10 shows a flowchart of the position and orientation calculation process in Fig. 9. In step S1021, VIO is executed by the position calculation unit 210. In this embodiment, the measurement values ​​by the inertial measurement unit and the detection values ​​by the wheel speed sensors are used.

[0121] In this embodiment, the VIO calculates the position and orientation of the camera 300A in the world coordinate system for the two-dimensional image captured by the camera 300A. The calculated position and orientation of the camera 300A are treated as the position and orientation of the vehicle 50. By executing the VIO, the position T and orientation R of the vehicle 50 in the world coordinate system and the three-dimensional coordinates X of a set of feature points are calculated. W and is calculated.

[0122] In step S1023, the bias error and the synchronization error are estimated by the position calculation unit 210. Specifically, the position T and the attitude R of the vehicle 50 and the three-dimensional coordinates X W Based on the above, the bias error of the IMU is estimated. A known estimation method is used to estimate the bias error. Furthermore, the position T and the orientation R of the vehicle 50 and the three-dimensional coordinates X W Based on the above, a synchronization error between the camera and the IMU is estimated. A known estimation method is used to estimate the synchronization error.

[0123] In step S1025, the synchronization error is corrected by the position calculation unit 210. Specifically, using the synchronization error estimated in step S1023, a detection value synchronized with the two-dimensional image acquired by the camera 300A is selected from the time-series detection values ​​of the IMU.

[0124] In step S1027, the position calculation unit 210 corrects the detection value of the IMU selected in step S1025 using the bias error estimated in step S1023.

[0125] In step S1029, the position calculation unit 210 calculates the position and orientation of the vehicle 50 based on the detection value of the IMU whose bias error has been corrected in step S1027. The above is the calculation process of the position and orientation. In this embodiment, the position and orientation of the vehicle 50 are calculated based on the detection value of the IMU. Thereafter, the processes from step S107 onwards shown in FIG. 9 are executed.

[0126] 9, in step S107, it is determined whether or not the correction process can be started by the correction unit 225. The correction process can be started when all of the following conditions (i) to (iv) are satisfied. (i) The BEV feature is generated based on the image feature of the current frame acquired by each of the cameras 300A to 300F. (ii) The BEV feature amount is generated based on the image feature amount of the previous frame captured by each of the cameras 300A to 300F. (iii) The position and orientation of the vehicle 50 at the time the current frame is captured are calculated. (iv) The position and attitude of the vehicle 50 at the time the previous frame was acquired have been calculated.

[0127] If it is determined that the correction process can be started (step S107; YES), step S109 is executed. If the correction process cannot be started (step S107; NO), steps S101 and S105 are executed again.

[0128] In step S109, the correction unit 225 acquires motion information of the vehicle 50. The motion information of the vehicle 50 is the amount of change between the position and attitude of the vehicle 50 at the acquisition time of the current frame and the position and attitude of the vehicle 50 at the acquisition time of the previous frame. As described above, the position and attitude of the vehicle 50 are calculated based on the measurement values ​​of the IMU.

[0129] The processes from step S111 onwards are similar to those in the first embodiment, and therefore the description thereof will be omitted.

[0130] In the VIO, the estimation result of the position and attitude of the vehicle 50 based on the two-dimensional image acquired by the camera and the estimation result of the position and attitude of the vehicle 50 based on the detection value by the IMU are integrated. Since the angular velocity information output by the IMU contains a bias error, if the period during which the position and attitude are estimated is long, the error of the estimation result due to the bias error of the IMU becomes large. When the two-dimensional image acquired by the camera is used to estimate the position and attitude, the estimation result contains an error due to the influence of blur contained in the two-dimensional image. Therefore, if the period during which the position and attitude are estimated is short, the accuracy of the estimation result is low. On the other hand, since there is no bias error in the two-dimensional image acquired by the camera, if the period during which the position and attitude are estimated is long, the accuracy of the estimation result is high.

[0131] In consideration of the above characteristics, in this embodiment, the bias error of the detection value of the IMU is estimated using the estimation result of the VIO. After correcting the bias error of the detection value of the IMU, the position and attitude of the vehicle 50 are calculated based on the detection value of the IMU. In this way, the problems caused by the characteristics of each sensor described above are solved.

[0132] H. Eighth embodiment: The driving assistance system 1 according to the eighth embodiment will be described below. The description will focus on configurations different from the first embodiment, and descriptions of configurations similar to the first embodiment will be omitted.

[0133] Fig. 11 shows a bird's-eye view estimation method executed by the estimation device 100 in this embodiment. In Fig. 11, the same processes as those in the first embodiment (see Fig. 4) are denoted by the same reference numerals. While the following processes are being executed, the cameras 300A-300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A-300F are stored sequentially in the memory 110.

[0134] The processes in steps S101 and S103 are similar to those in the first embodiment, and therefore the description thereof will be omitted.

[0135] In step S104, a process of calculating the position and orientation is executed. Note that the processes in steps S101 and S103 and the process in step S104 are executed in parallel.

[0136] Fig. 12 shows a flowchart of the calculation process of the position and orientation in Fig. 11. In step S1041, the position calculation unit 210 detects a moving object included in the two-dimensional image acquired by the camera 300A and subjected to the process of extracting image features in step S101. For example, the moving object is detected using a machine learning model such as Pseudo-LiDAR. Alternatively, the moving object may be detected using a known detection method of a moving object using epipolar geometry.

[0137] In step S1043, mask processing is performed on the target two-dimensional image by the position calculation unit 210. The target two-dimensional image is the two-dimensional image that has been subjected to the processing for extracting image feature amounts in step S101.

[0138] In step S1045, the position calculation unit 210 executes the VIO. Here, the VIO is executed based on the two-dimensional image masked in step S1043. By executing the VIO, the position T and the orientation R of the vehicle 50 in the world coordinate system and the three-dimensional coordinates X W The above is the calculation process of the position and orientation. After that, the processes from step S107 onwards shown in FIG. 11 are executed.

[0139] The process from step S107 onward shown in FIG. 11 is similar to that in the first embodiment, and therefore a description thereof will be omitted.

[0140] When estimating the position and attitude of the vehicle 50 by the VIO, it is desirable that objects around the vehicle 50 are stationary. When other vehicles, pedestrians, etc. around the vehicle 50 are moving, the error included in the estimation result by the VIO tends to be large. For this reason, in this embodiment, a moving object is detected in advance in a two-dimensional image, and the VIO is executed based on the two-dimensional image in which the detected moving object is masked. Therefore, the error included in the estimation result by the VIO can be reduced.

[0141] I. Other embodiments: I1. Alternative embodiment 1: In the first embodiment, an example has been described in which the coordinates of objects around the vehicle 50 are estimated by a feature-based method. Alternatively, the coordinates of objects around the vehicle 50 may be estimated by a direct method. For example, the depth of all pixels in an image can be restored by DTAM (Richard A. Newcombe, et al., “DTAM: Dense Tracking and Mapping in Real-Time”, IEEE Xplore, [online], [Retrieved September 4, 2023], Internet URL: https: / / doi.org / 10.1109 / ICCV.2011.6126513).

[0142] I2. Alternative embodiment 2: In the first embodiment, an example has been described in which the coordinates of objects around the vehicle 50 are estimated by a feature-based method, and the depth of pixels other than the feature points is interpolated by linear interpolation. Alternatively, the depth may be interpolated by a depth interpolation method using a machine learning model (Alex Wong, et al., “Unsupervised Depth Completion from Visual Inertial Odometry” IEEE Xplore, [online], [searched September 4, 2023], Internet URL: https: / / doi.org / 10.1109 / lra.2020.2969938).

[0143] The estimation apparatus and the method described in the present disclosure may be realized by a special-purpose computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, the estimation apparatus and the method described in the present disclosure may be realized by a special-purpose computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the estimation apparatus and the method described in the present disclosure may be realized by one or more special-purpose computers configured by a combination of a processor and a memory programmed to execute one or more functions and a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored in a computer-readable non-transitory tangible recording medium as instructions executed by a computer.

[0144] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention column can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0145] 1 ... driving assistance system, 50 ... vehicle, 100 ... estimation device, 110 ... memory, 120 ... input / output interface, 150 ... CPU, 190 ... bus, 210 ... position calculation unit, 220 ... feature amount acquisition unit, 225 ... correction unit, 230 ... bird's-eye view generation unit, 300 ... camera group, 300A to 300F ... cameras

Claims

1. An estimation device (100), a position calculation unit (210) that calculates the position and attitude of a vehicle using visual odometry as a self-position estimation method based on consecutive two-dimensional images captured in time series by a single camera (300A), the position calculation unit calculating the position and attitude of the vehicle based on the two-dimensional images of the outside of the vehicle captured by at least one of a plurality of cameras provided on the vehicle; a feature acquisition unit (220) that acquires BEV features, which are features in a BEV (Bird's eye view) space, based on the two-dimensional image of the outside of the vehicle, using an algorithm for BEV estimation to acquire the BEV features; a correction unit (225) that corrects the BEV feature amount acquired by the feature amount acquisition unit using information representing the position and orientation; An estimation device comprising:

2. The estimation device according to claim 1 , In the algorithm for BEV estimation, The two-dimensional images captured by the corresponding cameras are input to a plurality of machine learning models that have been trained to input the two-dimensional images captured by corresponding cameras among a plurality of cameras provided on a vehicle, and output image feature amounts that are feature amounts of the two-dimensional images captured by the corresponding cameras, thereby acquiring the image feature amounts of the two-dimensional images; A view frustum feature for each of the plurality of cameras is generated based on the image feature corresponding to each of the plurality of cameras; A feature of a view frustum for each of the plurality of cameras is converted into the BEV feature. Estimation device.

3. The estimation device according to claim 2, The position calculation unit uses Visual Inertial Odometry as the self-position estimation method. Estimation device.

4. The estimation device according to claim 2, The position calculation unit further uses detection values ​​of wheel speed sensors in calculating the position and attitude of the vehicle. Estimation device.

5. The estimation device according to claim 1 , In the algorithm for BEV estimation, The two-dimensional images captured by the corresponding cameras are input to a plurality of machine learning models that have been trained to input the two-dimensional images captured by corresponding cameras among a plurality of cameras provided on a vehicle, and output image feature amounts that are feature amounts of the two-dimensional images captured by the corresponding cameras, thereby acquiring the image feature amounts of the two-dimensional images; Based on the image feature amounts corresponding to each of the multiple cameras previously acquired and the image feature amounts corresponding to each of the multiple cameras currently acquired, the image feature amounts corresponding to each of the multiple cameras are converted from an image space to the BEV space using a machine learning model having an attention mechanism, thereby acquiring the BEV feature amounts. Estimation device.

6. The estimation device according to claim 5, The position calculation unit uses Visual Inertial Odometry as the self-position estimation method. Estimation device.

7. The estimation device according to claim 5, The position calculation unit further uses detection values ​​of wheel speed sensors in calculating the position and attitude of the vehicle. Estimation device.

8. A method for estimating a bird's-eye view, comprising the steps of: A step of calculating a position and an attitude of the vehicle by using Visual Odometry as a self-location estimation method based on consecutive two-dimensional images captured in time series by a same camera, the step of calculating a position and an attitude of the vehicle based on the two-dimensional images of the outside of the vehicle captured by at least one of a plurality of cameras provided on the vehicle; acquiring BEV features, which are features in a BEV (Bird's eye view) space, based on the two-dimensional image of the outside of the vehicle, using an algorithm for BEV estimation for acquiring the BEV features, which are features in a BEV space; correcting the acquired BEV feature amount using information representing the position and orientation; An estimation method including:

Images