Estimation device and estimation method

Abstract

The invention provides an estimation device and an estimation method. This estimation device is provided with: a coordinate calculation unit for calculating three-dimensional coordinates of an object in a two-dimensional image, using a visual odometer as a self-position estimation method, on the basis of the two-dimensional image which is captured by the same camera and is continuous in time series; a coordinate calculation unit that calculates three-dimensional coordinates of an object present around the vehicle on the basis of two-dimensional images of the outside of the vehicle captured by a plurality of cameras provided on the vehicle; a feature amount acquisition unit that acquires a BEV feature amount, which is a bird's-eye view, on the basis of the three-dimensional coordinates and a two-dimensional image obtained by imaging the outside of the vehicle, by means of a BEV estimation algorithm for acquiring the BEV feature amount, which is a feature amount in a BEV space; and a bird's-eye view generation unit that generates, on the basis of the BEV feature amount, a bird's-eye view that is an image indicating a situation in which the vehicle is viewed from above. As a result, it is possible to obtain a more reliable estimation result for the estimation of three-dimensional space information.

Description

Technical Field

[0001] The present application relates to an inference device and an inference method. Background Art

[0002] In the technology described in Patent Document 1, three-dimensional space information is estimated using a machine learning model based on two-dimensional images acquired by a plurality of cameras, and a bird's-eye view is generated based on the estimated three-dimensional space information.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: U.S. Patent Application Publication No. 2021 / 0398338 Summary of the invention

[0006] When using a machine learning model to infer three-dimensional spatial information based on a two-dimensional image, the accuracy of the inference result is affected by the teaching data used in the training and the method used to train the machine learning model. For example, there are cases where the machine learning model cannot output the correct inference result for an untrained phenomenon. When a machine learning model is used, it cannot be said that the inference result obtained is necessarily reliable.

[0007] Therefore, when inferring three-dimensional spatial information based on two-dimensional images, a technology that can obtain more reliable inference results is required.

[0008] According to one aspect of the present application, an inference device is provided. The inference device comprises: a coordinate calculation unit, which calculates the three-dimensional coordinates of an object in the two-dimensional image based on the two-dimensional images taken by the same camera in a time series, using a visual odometer as a method of estimating the own position, and the coordinate calculation unit calculates the three-dimensional coordinates of the object existing around the vehicle based on each of the two-dimensional images taken by multiple cameras installed on the vehicle to take the exterior of the vehicle; a feature quantity acquisition unit, which acquires the BEV feature quantity based on the three-dimensional coordinates and the two-dimensional image taken to take the exterior of the vehicle by using an algorithm for BEV inference for acquiring the feature quantity in the BEV space, namely, the BEV feature quantity, and the BEV is a bird's-eye view; and a bird's-eye view generation unit, which generates an image, namely, a bird's-eye view, showing the situation of the vehicle viewed from above, based on the BEV feature quantity.

[0009] According to the inference device of the above-mentioned method, the visual odometry is used to calculate the three-dimensional coordinates of objects around the vehicle through a geometric method based on two-dimensional images, internal parameters of the camera and external parameters. Therefore, compared with the method of inferring the three-dimensional coordinates of objects around the vehicle using only a machine learning model, a more reliable inference result can be obtained.

[0010] According to another embodiment of the present application, an inference method is provided, which includes: a step of calculating the three-dimensional coordinates of objects in the two-dimensional images taken in a time series by the same camera, using a visual odometer as a method for inferring the vehicle's own position, in which the three-dimensional coordinates of objects existing around the vehicle are calculated based on each of the two-dimensional images obtained by taking the outside of the vehicle by multiple cameras installed on the vehicle; a step of obtaining the BEV feature quantity based on the three-dimensional coordinates and the two-dimensional image obtained by taking the outside of the vehicle by using an algorithm for BEV inference for obtaining a feature quantity in the BEV space, namely, a BEV feature quantity, wherein the BEV is a bird's-eye view; and a step of generating an image representing the situation of observing the vehicle from above, namely, a bird's-eye view, based on the BEV feature quantity. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a block diagram showing a schematic configuration of a driving assistance system.

[0012] Figure 2 This is an explanatory diagram showing an example of camera configuration.

[0013] Figure 3 This is a diagram to illustrate coordinate transformation.

[0014] Figure 4 This is a flowchart showing the method for generating a bird's-eye view in the first embodiment.

[0015] Figure 5 This is a flowchart showing a method for generating a bird's-eye view in the second embodiment.

[0016] Figure 6 This is a flowchart showing a method for generating a bird's-eye view in the third embodiment. DETAILED DESCRIPTION

[0017] A. First Implementation Method:

[0018] Figure 1 The driving assistance system 1 shown uses a plurality of two-dimensional images acquired in a time series by an image sensor provided on the vehicle 50 to estimate the conditions around the vehicle 50 and generate a bird's-eye view image of the vehicle 50 viewed from above. The bird's-eye view image is an image of the vehicle 50 viewed from above. For example, the driving assistance system 1 uses a plurality of two-dimensional images acquired in a time series by an image sensor provided on the vehicle 50 to display an image of the vehicle 50 viewed from above on a vehicle monitor of the vehicle 50. The vehicle 50 can travel using an ADAS (Advanced Driving Assistant System).

[0019] The driving support system 1 includes an estimation device 100 and a camera group 300 . The estimation device 100 and the camera group 300 are mounted on a vehicle 50 .

[0020] like Figure 2 As shown, the camera group 300 includes six cameras 300A to 300F installed on the vehicle 50. Cameras 300A to 300F are monocular cameras capable of capturing color images. Camera 300A is disposed on the upper portion of the front windshield of the vehicle 50. Camera 300A captures a predetermined range in front of the vehicle 50. Camera 300B is disposed on the lower portion of the rearview mirror on the right side of the vehicle 50. Hereinafter, the right side of the vehicle 50 refers to the right side when viewed from a person sitting in the vehicle 50 facing the front of the vehicle 50. Camera 300B captures a predetermined range in the right oblique front of the vehicle 50.

[0021] The camera 300C is disposed at the lower part of the left rearview mirror. The left side of the vehicle 50 refers to the left side when viewed from a person sitting in the vehicle 50 facing the front of the vehicle 50. The camera 300C photographs a predetermined range in the left oblique front of the vehicle 50. The camera 300D is disposed at the right door pillar. The camera 300D photographs a predetermined range on the right side of the vehicle 50. The camera 300E is disposed at the left door pillar. The camera 300E photographs a predetermined range on the left side of the vehicle 50. The camera 300F is disposed at the center of the rear door. The camera 300F photographs a predetermined range behind the vehicle 50.

[0022] Cameras 300A to 300F capture images of a predetermined range outside vehicle 50 at a predetermined frame rate, and output the acquired images to estimation device 100. In order to facilitate technical understanding, in the embodiment, cameras 300A to 300F capture images at the same timing.

[0023] The inference device 100 is a computer including a memory 110, an input / output interface 120, and a CPU (Central Processing Unit) 150. The memory 110 and the input / output interface 120 are connected to the CPU 150 via a bus 190. For example, the functions of the inference device 100 are implemented by a driving control ECU (Electronic Control Unit) that is responsible for driving control of the vehicle 50.

[0024] Various programs and various data used in various processes executed by the inference device 100 are stored in the memory 110. Data representing a machine learning model used in various processes executed by the inference device 100 is stored in the memory 110.

[0025] The cameras 300A to 300F are connected to the input / output interface 120 via Ethernet (registered trademark).

[0026] The CPU 150 is a processor that implements various functions by executing programs stored in the memory 110. For example, the CPU 150 stores two-dimensional images received from the cameras 300A to 300F in the memory 110. In the embodiment, the CPU 150 functions as the coordinate calculation unit 210, the feature quantity acquisition unit 220, and the bird's-eye view generation unit 230 by executing the programs stored in the memory 110.

[0027] The coordinate calculation unit 210 calculates the three-dimensional coordinates of the object in the two-dimensional image based on the two-dimensional image continuous in time series, using the visual odometry as a method of estimating the own position. In the present embodiment, the coordinate calculation unit 210 calculates the three-dimensional coordinates of the object existing around the vehicle 50 based on the two-dimensional image obtained by taking the outside of the vehicle by the cameras 300A to 300F, using the visual odometry. The details of the processing of the coordinate calculation unit 210 will be described later.

[0028] The feature acquisition unit 220 acquires a feature in a BEV (Bird's eye view) space, i.e., a BEV feature, based on the three-dimensional coordinates of objects existing around the vehicle 50 and the two-dimensional image used in the calculation of the three-dimensional coordinates. The BEV feature includes information on the coordinate values ​​of the point cloud in the three-dimensional space and the color of the point cloud, similar to point cloud data detected by a laser radar (Lidar). The details of the processing of the feature acquisition unit 220 will be described later.

[0029] The bird's-eye view generation unit 230 generates a bird's-eye view based on the BEV feature quantity. The bird's-eye view is an image showing the situation of observing the vehicle 50 from above. As a method for generating a bird's-eye view based on the BEV feature quantity, an existing method can be used. The bird's-eye view generation unit 230 can detect objects around the vehicle 50 based on the BEV feature quantity and generate a bird's-eye view showing objects around the vehicle 50. In addition, it is not necessary to include all objects around the vehicle 50 in the generated bird's-eye view. For example, the bird's-eye view generation unit 230 can also detect lanes around the vehicle 50 based on the BEV feature quantity and generate a bird's-eye view including the vehicle 50 and the lanes.

[0030] Next, the visual odometry (hereinafter referred to as VO) is explained. In VO, feature points are detected in multiple two-dimensional images in a time series, and the coordinates of the feature points in the two-dimensional images are used to infer the three-dimensional coordinates of the feature points in the world coordinate system. Feature points refer to points on a two-dimensional image that can be reliably detected.

[0031] Feature points are detected using a corner detection method. For example, the intersection of two edges is detected as a corner point. Alternatively, a point where two significant edges with different directions exist in a certain local vicinity is detected as a corner point. As a method for corner detection, for example, Harris corner detection, SHIFT (scale-invariant feature transform), SURF (Speeded up robust features), and ORB (Oriented FAST and Rotated BRIEF) can be used. In addition, most methods of corner detection can detect not only corner points but also feature points. The number of feature points detected from a two-dimensional image is, for example, 100.

[0032] For example, feature points are detected from three consecutive two-dimensional images taken by the same camera in a time series. The time when the three two-dimensional images are obtained is set as time t0, t1, and t2. Time t0 is the earliest time, and time t2 is the latest time. Feature points of the two-dimensional images obtained at time t0, t1, and t2 are detected respectively, and the coordinates and brightness values ​​of each detected feature point are recorded. The detected feature points are, for example, the corners of buildings, the outlines of traffic signs, the boundaries of the colors of traffic signs, and the corners of curbstones.

[0033] like Figure 3 As shown in , the coordinates of the detected feature points are the coordinates obtained by projecting the three-dimensional position of the object in the world coordinate system onto the image coordinate system. The relationship between the coordinates of the feature points in the two-dimensional image and the three-dimensional position of the feature points in the world coordinate system can be expressed as in the following formula (1). In formula (1), f x 、f y Indicates focal length. c x 、c y Indicates the position of the intersection of the optical axis and the projection surface. 11 ~r 33 represents the rotation matrix. t1 to t3 represent translation vectors. s represents the scale factor. The scale factor s is estimated using the results of measuring the position and posture of the camera in advance using an IMU (Inertial Measurement Unit) or the like.

[0034]

Number 1

[0035] [Math.1]

[0036]

[0037] Formula (1) can be expressed in the form of formula (2).

[0038]

Number 2

[0039] [Math.2]

[0040] su I =K[RT]X W …(2)

[0041] The matrix K in equation (2) is called the internal parameters of the camera. The matrix [RT] is called the external parameters of the camera. The external parameters of the camera are determined by the position T and posture R of the camera coordinate system. As shown in equation (2), the position u in the two-dimensional image I =[uv 1] t The position T and posture R of the camera coordinate system and the position X of the world coordinate system W =[X W Y W Z W ] t The function representation of .

[0042] When the positions of feature points corresponding to the same object or the like detected in a plurality of two-dimensional images are continuously tracked in a time series, the positions of the feature points of interest change due to the movement of the vehicle 50 , that is, the movement of the camera group.

[0043] In VO, the position and posture of the camera at time t1 and the three-dimensional coordinates of the feature points in the world coordinate system are estimated based on the positions of the feature points detected in each two-dimensional image through bundle adjustment. Specifically, the position and posture of the camera at time t1 and the three-dimensional coordinates of the feature points in the world coordinate system are calculated using the coordinates of the corresponding feature points detected in multiple two-dimensional images and the assumed position T and posture R of the camera. In addition, the calculated three-dimensional coordinates are reprojected to the image coordinate system, and the distance between the projected point and the feature point detected from the two-dimensional image (reprojection error) is estimated.

[0044] The position of the feature point in the two-dimensional image can be obtained by the projection function as shown in formula (3). In addition, the internal parameters are set to be fixed and are not considered in formula (3). The projection function is a function that maps the coordinate system of the object space (world coordinate system) to the coordinate system of the image space (image coordinate system).

[0045]

Number 3

[0046] [Math.3]

[0047] u P =proj(R, T, X W )…(3)

[0048] The position u of the feature point detected from the two-dimensional image is defined as follows: I , and the position u of the feature point obtained by the projection function P In formula (4), u I Represents a set of coordinates of feature points detected from a two-dimensional image.

[0049]

Number 4

[0050] [Math.4]

[0051] C cam =|u I -proj(R,T,X W )|…(4)

[0052] By using the error function C represented by equation (4) cam The three-dimensional coordinates of the feature points at time t1 in the world coordinate system, as well as the position and posture of the camera are inferred by minimizing the optimization process. The inferred three-dimensional coordinates of the feature points in the world coordinate system are the three-dimensional coordinates of the feature points in the two-dimensional image obtained at time t1 in the world coordinate system. The inferred position and posture of the camera are represented by a rotation matrix and a translation vector. Minimizing the error represented by equation (4) means minimizing the error between the actual coordinates of the detected feature points in the image and the coordinates of the feature points obtained by the projection function. In this way, in VO, the three-dimensional coordinates of objects in the world coordinate system are inferred.

[0053] Next, the algorithm for BEV inference in this embodiment is described. In the embodiment, a bird's-eye view image is generated by combining the inference results of the three-dimensional coordinates of objects around the vehicle 50 based on VO in the LSS (Lift, Splat, Shoot) algorithm (Jonah Philion, Sanja Fidler, "Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D", in European Conference on Computer Vision. Springer, 2020, pp. 194-210).

[0054] In the LSS algorithm, first, two-dimensional images are obtained by cameras 300A~300F respectively. The feature vectors of multiple two-dimensional images are extracted respectively using machine learning models such as CNN. The depth of each pixel is inferred based on the two-dimensional images obtained by each camera. Since the inferred depth is blurred, it is assumed that the light corresponding to each pixel (the light from the camera to a certain point in the three-dimensional space) is assumed, and the probability distribution of the discrete depth is predicted for each pixel. The cross product of the predicted probability distribution of the discrete depth and the feature vector is calculated, and a series of points are calculated as information representing the depth of the pixel. As a result, a frustum-shaped point cloud is generated for each camera. The BEV feature is generated from the point cloud of the frustum using the internal and external parameters of the camera.

[0055] exist Figure 4 The method for estimating a bird's-eye view executed by the estimation device 100 in this embodiment is shown in FIG. Figure 4 The cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate and output the acquired two-dimensional images to the estimation device 100. The acquired two-dimensional images are color images. The two-dimensional images captured by the cameras 300A to 300F are sequentially stored in the memory 110.

[0056] In step S101, a two-dimensional image is read from the memory 110. In step S102, VO is executed. In this embodiment, based on the two-dimensional images acquired by the cameras 300A to 300F, the three-dimensional coordinates of the object in the world coordinate system are calculated by VO. The processing of steps S101 and S102 is executed by the coordinate calculation unit 210.

[0057] In step S103, a depth map is generated using the three-dimensional coordinates in the world coordinate system calculated in step S102 and the two-dimensional image used for the calculation of the three-dimensional coordinates (the two-dimensional image obtained at time t1). The two-dimensional image used for the calculation of the three-dimensional coordinates (the two-dimensional image obtained at time t1) is called the corresponding image. In the present embodiment, the coordinates are inferred by a feature point-based method. Therefore, the coordinates of the world coordinate system are inferred only for the feature points detected from the corresponding image. Therefore, for pixels other than the feature points, the coordinates of the world coordinate system are not inferred. Therefore, the depth of pixels other than the feature points can be obtained, for example, by linear interpolation using the depths of nearby pixels. The processing of step S103 is performed by the feature quantity acquisition unit 220.

[0058] In step S104, a machine learning model such as CNN is used to extract the image feature quantity (feature quantity vector) of the two-dimensional image obtained by the camera group 300. For extracting the image feature quantity of the two-dimensional image obtained by camera 300A, a machine learning model that has been learned in the following manner is used: the two-dimensional image obtained by camera 300A is used as input, and the image feature quantity of the input two-dimensional image is extracted. The machine learning model is generated by machine learning using a two-dimensional image captured by a camera installed at the same position as camera 300A as learning data. Here, a 1×1 convolution is performed to extract the image feature quantity. Therefore, the size of the two-dimensional image is not changed. Here, the corresponding image (two-dimensional image obtained at time t1) used for the calculation of the three-dimensional coordinates is input. For cameras 300B to 300F, the corresponding machine learning models are also used to extract the image feature quantity of the two-dimensional image. The processing of step S104 is performed by the feature quantity acquisition unit 220.

[0059] In step S105 , the outer product of the image feature extracted in step S104 and the depth map generated from the corresponding two-dimensional image is obtained as the outer product feature. The process of step S105 is executed by the feature acquisition unit 220 .

[0060] In step S106, similarly to the LSS algorithm, a point cloud of the viewing cone is generated based on the outer product feature quantity calculated in step S105. Specifically, assuming that a ray (ray from the camera to a certain point in the three-dimensional space) corresponds to each pixel of each two-dimensional image, the probability distribution of the discrete depth is predicted for each pixel. The outer product of the predicted probability distribution of the discrete depth and the vector representing the outer product feature quantity is calculated as the depth. Using the inferred depth, the coordinates of each pixel of the two-dimensional image, and the internal and external parameters of the camera, a point cloud of the viewing cone is generated for the two-dimensional image obtained by each camera. The processing of step S106 is performed by the feature quantity acquisition unit 220.

[0061] In step S107, a BEV feature is generated based on the point cloud of the viewing cone generated in step S106. The process of step S107 is performed by the feature acquisition unit 220. In step S108, a bird's-eye view image is generated based on the BEV feature. The generated bird's-eye view image is displayed on a monitor provided on the vehicle 50, for example. The process of step S108 is performed by the bird's-eye view generation unit 230. Figure 4 The processing shown is repeatedly executed at predetermined time intervals, for example, while vehicle 50 is traveling.

[0062] As described above, in this embodiment, a depth map generated based on three-dimensional coordinate information obtained using VO is used to generate a bird's-eye view. Using VO, the three-dimensional coordinates of objects around the vehicle are calculated by a geometric method based on a two-dimensional image, internal parameters of the camera, and external parameters. Therefore, a more reliable result can be obtained for the inference of the three-dimensional coordinates of objects around the vehicle. Therefore, compared with a method of generating a depth map based on three-dimensional coordinate information obtained using a machine learning model and inferring a bird's-eye view using the generated depth map, the reliability of the accuracy of the inferred bird's-eye view can be improved.

[0063] A. Second Implementation Method:

[0064] Hereinafter, a driving assistance system 1 according to a second embodiment will be described. The description will focus on the configurations that are different from the first embodiment, and the description of the configurations that are the same as the first embodiment will be omitted.

[0065] In the present embodiment, a bird's-eye view image is generated by combining the estimation result of the three-dimensional coordinates of objects around the vehicle 50 based on VO with the algorithm disclosed in the specification of U.S. Patent Application Publication No. 2023 / 0053785.

[0066] In the algorithm disclosed in the above document, first, two-dimensional images are obtained respectively through multiple cameras. Using the backbone network, the image features of each two-dimensional image are extracted. The backbone network is, for example, CNN. The extracted image features are input into the Transformer engine. The Transformer engine is a machine learning model that uses the attention mechanism. The Transformer engine is a machine learning model trained in the following way: the image features extracted from multiple two-dimensional images are used as input, the image features are fused and projected into the BEV space.

[0067] exist Figure 5 The method for estimating a bird's-eye view executed by the estimation device 100 in this embodiment is shown in FIG. Figure 5 While executing the following processing, the cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A to 300F are sequentially stored in the memory 110.

[0068] In step S201, a two-dimensional image is read from the memory 110. In step S202, VO is executed. In this embodiment, based on the two-dimensional images acquired by the cameras 300A to 300F, the three-dimensional coordinates of the object in the world coordinate system are calculated by VO. The processing of steps S201 and S202 is executed by the coordinate calculation unit 210.

[0069] In step S203, a depth map is generated using the three-dimensional coordinates in the world coordinate system calculated in step S202 and the two-dimensional image used for the calculation of the three-dimensional coordinates (the two-dimensional image obtained at time t1). As in the first embodiment, the depth of pixels other than feature points is obtained by, for example, linear interpolation using the depths of nearby pixels. The process of step S203 is performed by the feature quantity acquisition unit 220.

[0070] In step S204, the depth map obtained in step S203 and the corresponding two-dimensional image are fused. The corresponding two-dimensional image refers to the two-dimensional image used in the calculation of the three-dimensional coordinates for generating the depth map. For example, the depth map and the corresponding two-dimensional image are fused through the RGB-D fusion of the ASIF-Net algorithm (Chongyi Li, 7 others, "ASIF-Net: Attention Steered Interweave Fusion Network for RGB-D Salient Object Detection," in IEEE Transactions on Cybernetics, vol. 51, no. 1, pp. 88-100, Jan. 2021, [online], [retrieved on September 4, 2023], Internet URL: https: / / doi.org / 10.1109 / TCYB.2020.2969255). As a result, RGB-D data is generated. The processing of step S204 is performed by the feature acquisition unit 220.

[0071] In step S205, the feature quantity of the RGB-D data obtained in step S204, namely, the RGB-D feature quantity, is extracted using the backbone network. The backbone network is a machine learning model that has been learned in a manner that takes RGB-D data as input and outputs the feature quantity of RGB-D data, namely, the RGB-D data feature quantity. The RGB-D data feature quantity also includes a feature quantity related to depth. The processing of step S205 is performed by the feature quantity acquisition unit 220.

[0072] In step S206 , based on the RGB-D data feature acquired in step S205 , the BEV feature in the BEV space is extracted using the Transformer engine. The process of step S206 is executed by the feature acquisition unit 220 .

[0073] In step S207, a bird's-eye view image is generated based on the BEV feature value acquired in step S206. The generated bird's-eye view image is displayed on, for example, a monitor provided in the vehicle 50. The process of step S207 is executed by the bird's-eye view generation unit 230. Figure 5 The processing shown is repeatedly executed at predetermined time intervals, for example, while vehicle 50 is traveling.

[0074] As described above, in this embodiment, a depth map generated based on three-dimensional coordinate information obtained using a visual odometer is used to generate a bird's-eye view. Using VO, the three-dimensional coordinates of objects around the vehicle are calculated by a geometric method based on a two-dimensional image, internal parameters of the camera, and external parameters. Therefore, a result with higher reliability can be obtained for the inference of the three-dimensional coordinates of objects around the vehicle. Therefore, compared with a method of generating a depth map based on three-dimensional coordinate information obtained using a machine learning model and inferring a bird's-eye view using the generated depth map, the reliability of the accuracy of the inferred bird's-eye view can be improved.

[0075] C. Third Implementation Method:

[0076] Hereinafter, a driving assistance system 1 according to a third embodiment will be described. The description will focus on the configurations that are different from the first embodiment, and the description of the configurations that are the same as the first embodiment will be omitted.

[0077] In this embodiment, a bird's-eye view image is generated by combining the inference results of the three-dimensional coordinates of objects around the vehicle 50 based on VO in the BEVFusion algorithm (Zhijian Liu, 6 others, "BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation", CoRR, 2022).

[0078] In the BEVFusion algorithm, first, two-dimensional images are obtained separately through multiple cameras. The image feature quantities of each two-dimensional image are extracted using the encoder model for the camera. A bird's-eye view representation (BEV representation) of the image is generated using the image feature quantities extracted from multiple two-dimensional images. In addition, the laser radar feature quantities are extracted from the point cloud detected by the laser radar using the encoder model for the laser radar. A bird's-eye view representation of the laser radar is generated using the extracted laser radar feature quantities. The bird's-eye view representation of the image is combined with the bird's-eye view representation of the laser radar. Based on the combined bird's-eye view representation, an integrated bird's-eye view representation is generated using the encoder for BEV. Here, in the integrated bird's-eye view representation (unified BEV representation), both geometric and semantic information are maintained.

[0079] exist Figure 6 The method for estimating a bird's-eye view executed by the estimation device 100 in this embodiment is shown in FIG. Figure 6 The cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and the acquired two-dimensional images are output to the estimation device 100. The two-dimensional images captured by the cameras 300A to 300F are sequentially stored in the memory 110.

[0080] In step S301, a two-dimensional image is read from the memory 110. In step S302, the image feature quantity (feature quantity vector) of each two-dimensional image acquired by the cameras 300A to 300F is extracted using a machine learning model such as an encoder model for the camera. The machine learning model is a machine learning model that has been learned in a manner that takes a two-dimensional image as input and outputs a feature quantity of the two-dimensional image, i.e., an image feature quantity. The corresponding image (the two-dimensional image acquired at time t1) used for calculating the three-dimensional coordinates is input to the machine learning model. The processing of steps S301 and S302 is performed by the feature quantity acquisition unit 220.

[0081] In step S303, based on the image feature obtained in step S302, data of a bird's-eye view representation is generated. The generated data of the bird's-eye view representation represents the first BEV feature in the BEV space. Here, similar to the LSS algorithm, a ray corresponding to each pixel of each two-dimensional image (a ray from the camera to a certain point in the three-dimensional space) is assumed, and the probability distribution of the discrete depth is predicted for each pixel. The outer product of the predicted probability distribution of the discrete depth and the feature vector is calculated as the depth. Using the inferred depth, the coordinates of each pixel of the two-dimensional image, and the internal and external parameters of the camera, a frustum-shaped point cloud is generated for the two-dimensional image obtained by each camera. The first BEV feature is generated based on the point cloud of the frustum. The processing of step S303 is performed by the feature acquisition unit 220.

[0082] In step S304, VO is executed. In the present embodiment, VO is used to calculate the three-dimensional coordinates of an object in the world coordinate system based on the two-dimensional images acquired by the cameras 300A to 300F. Point cloud data is generated using the estimated three-dimensional coordinates. The processing of step S304 is executed by the coordinate calculation unit 210.

[0083] In step S305, a feature quantity, i.e., a three-dimensional feature quantity, of the point cloud data generated based on the three-dimensional coordinates obtained in step S304 is extracted using a machine learning model such as an encoder model for laser radar. The machine learning model is a machine learning model that has been learned by taking point cloud data as input and outputting three-dimensional feature quantities. The machine learning model is also referred to as a second machine learning model. The processing of step S305 is performed by the feature quantity acquisition unit 220.

[0084] In step S306, based on the three-dimensional feature quantity obtained in step S305, a machine learning model is used to generate data of a bird's-eye view representation (BEV representation). The data of the bird's-eye view representation generated here represents a second BEV feature quantity as a feature quantity in the BEV space. The machine learning model is a machine learning model that has been learned in a manner that takes the three-dimensional feature quantity as an input and outputs the second BEV feature quantity. The processing of step S306 is performed by the feature quantity acquisition unit 220.

[0085] In step S307 , the first BEV feature acquired in step S303 and the second BEV feature acquired in step S306 are merged to generate a merged feature. The process of step S307 is executed by the feature acquisition unit 220 .

[0086] In step S308, based on the fused feature quantity, a machine learning model such as an encoder for BEV is used to extract the feature quantity in the integrated bird's-eye view expression. The processing of step S308 is performed by the feature quantity acquisition unit 220. In step S309, a bird's-eye view image is generated based on the fused feature quantity. The generated bird's-eye view image is displayed on a monitor installed on the vehicle 50, for example. The processing of step S309 is performed by the bird's-eye view generation unit 230. Figure 6 The processing shown is repeatedly executed at predetermined time intervals, for example, while vehicle 50 is traveling.

[0087] As described above, in the present embodiment, the second BEV feature quantity for inferring the bird's-eye view is generated based on the three-dimensional coordinate information obtained using VO. Using VO, the three-dimensional coordinates of the objects around the vehicle are calculated by a geometric method based on a two-dimensional image, internal parameters of the camera, and external parameters. Therefore, a result with higher reliability can be obtained for the inference of the three-dimensional coordinates of the objects around the vehicle. Therefore, compared with the method of generating the second BEV feature quantity based on the three-dimensional coordinate information obtained using a machine learning model and inferring the bird's-eye view using the generated second BEV feature quantity, the reliability of the accuracy of the inferred bird's-eye view can be improved.

[0088] In addition, in this embodiment, feature points of the world coordinate system estimated by VO are used instead of point clouds detected by laser radar, so there is no need to use expensive laser radar.

[0089] D. Other implementation methods:

[0090] D1. Other implementation methods 1:

[0091] In the first embodiment, an example of using VO as a method for estimating the self-position is described, but Visual Inertial Odometry (hereinafter referred to as VIO) can also be used as a method for estimating the self-position. In VIO, a camera and an inertial sensor are used. In addition, in the configuration of the second embodiment and the third embodiment, VIO can also be used as a method for estimating the self-position. According to this method, in order to infer the three-dimensional coordinates of objects existing around the vehicle, the measurement value of the inertial measurement device is used in addition to the two-dimensional image, so that a more reliable result can be obtained for the inference of the bird's-eye view.

[0092] D2. Other implementation methods 2:

[0093] In addition, in any one of the first to third embodiments, VIO can also be used as a method for estimating the own position, and the detection value of the wheel speed sensor can also be used. For example, an IMU having a correction function using the wheel speed can be used. That is, in the present embodiment, the measurement value of the inertial measurement device and the detection value of the wheel speed sensor are used. In this case, the vehicle 50 is configured to have a wheel speed sensor. In order to infer the three-dimensional coordinates of objects existing around the vehicle, the measurement value of the inertial measurement device and the detection value of the wheel speed sensor are used in addition to the two-dimensional image, so that a more reliable result can be obtained for the inference of the bird's-eye view.

[0094] D3. Other implementation methods 3:

[0095] In the first embodiment, an example of estimating the coordinates of objects around the vehicle 50 by a feature point based method is described. Alternatively, the coordinates of objects around the vehicle 50 may be estimated by a direct method. For example, DTAM (Richard A. Newcombe, 2 others, "DTAM: Dense Tracking and Mapping in Real-Time", [online], [retrieved on September 4, 2023], Internet URL: https: / / doi.org / 10.1109 / iccv.2011.6126513) can restore the depth of all pixels in an image.

[0096] D4. Other implementation methods 4:

[0097] In the first embodiment, an example is described in which the coordinates of objects around the vehicle 50 are inferred by a feature point based method, and the depth of pixels other than the feature points is interpolated by linear interpolation. Alternatively, the depth can be interpolated by a depth interpolation method using a machine learning model (Alex Wong, 3 others, "Unsupervised Depth Completion from Visual Inertial Odometry". IEEE Robotics and Automation Letters, 5 (2), 1899-1906. [online], [retrieved on September 4, 2023], Internet URL: https: / / doi.org / 10.1109 / lra.2020.2969938).

[0098] D5. Other implementation methods 5:

[0099] In the first embodiment, it is described as follows Figure 4 The example of executing the processing sequentially as shown in FIG. Figure 4 The processes of step S102 and step S103 shown in the figure may be executed in parallel with the process of step S104.

[0100] D6. Other implementation methods 6:

[0101] In the third embodiment, it is described as follows Figure 6 The example of executing the processing sequentially as shown in FIG. Figure 6 The processes of step S302 and step S303 shown in the figure and the processes of steps S304 to S306 may be executed in parallel.

[0102] The inference device and method described in the present application may also be implemented using a dedicated computer provided by constituting the following processor and memory, wherein the processor is programmed to execute one or more functions embodied by a computer program. Alternatively, the inference device and method described in the present application may also be implemented using a dedicated computer provided by constituting a processor composed of one or more dedicated hardware logic circuits. Alternatively, the inference device and method described in the present application may also be implemented by one or more dedicated computers composed of the following combination, which is a combination of a processor and memory programmed to execute one or more functions, and a processor composed of one or more hardware logic circuits. In addition, the computer program may also be stored in a computer-readable non-transient tangible recording medium as an instruction executed by a computer.

[0103] The present application is not limited to the above-mentioned embodiments, and can be implemented by various configurations within the scope of the main purpose. For example, the technical features in the embodiments corresponding to the technical features in each method recorded in the invention content column can be appropriately replaced or combined in order to solve part or all of the above-mentioned technical problems, or to achieve part or all of the above-mentioned effects. In addition, as long as the technical feature is not described as a necessary item in this specification, it can be appropriately deleted.

Claims

1. An inference device, characterized in that: have: A coordinate calculation unit, based on two-dimensional images taken in a time series by the same camera, calculates three-dimensional coordinates of an object in the two-dimensional image using a visual odometer as a method of estimating the own position, and the coordinate calculation unit calculates the three-dimensional coordinates of an object existing around the vehicle based on each of the two-dimensional images obtained by taking the exterior of the vehicle by a plurality of cameras provided on the vehicle; a feature quantity acquisition unit that acquires the BEV feature quantity based on the three-dimensional coordinates and the two-dimensional image obtained by photographing the exterior of the vehicle by using a BEV estimation algorithm for acquiring a feature quantity in a BEV space, namely, a BEV feature quantity, wherein the BEV is a bird's-eye view; as well as The bird's-eye view generating unit generates a bird's-eye view which is an image showing a state where the vehicle is viewed from above, based on the BEV feature amount.

2. The inference device according to claim 1, wherein: In the feature quantity acquisition unit, Using the three-dimensional coordinates, generating a depth map of each of the two-dimensional images used in calculating the three-dimensional coordinates, A plurality of machine learning models that have been learned in a manner of taking the two-dimensional image captured by one of the plurality of cameras as input and outputting feature quantities of the two-dimensional image captured by the one camera, i.e., image feature quantities, are respectively adopted, and the two-dimensional images captured by the corresponding cameras are respectively input to the plurality of machine learning models to obtain the image feature quantities respectively output by the plurality of machine learning models, generating a point cloud of a viewing cone of the two-dimensional image captured by the plurality of cameras respectively using the image feature amount of the two-dimensional image and the depth map corresponding to the two-dimensional image, The BEV feature is generated based on the point cloud of the viewing cone.

3. The inference device according to claim 2, characterized in that As the self-position estimation method, a visual inertial odometry is used.

4. The inference device according to claim 3, characterized in that As the self-position estimation method, the detection value of the wheel speed sensor is also used.

5. The inference device according to claim 1, wherein: In the feature quantity acquisition unit, Using the three-dimensional coordinates, generating a depth map of each of the two-dimensional images used in calculating the three-dimensional coordinates, fusing the two-dimensional image and the depth map corresponding to the two-dimensional image to generate RGB-D data, A plurality of machine learning models that have been learned in a manner that takes the RGB-D data generated based on the two-dimensional image captured by one of the plurality of cameras as input and outputs a feature quantity of the RGB-D data captured by the one camera, that is, a feature quantity of RGB-D data, and the RGB-D data generated based on the two-dimensional image captured by the corresponding camera is input to the plurality of machine learning models, so as to obtain the feature quantities of RGB-D data outputted by the plurality of machine learning models, respectively. The BEV feature quantity is generated based on the RGB-D data feature quantity.

6. The inference device according to claim 5, characterized in that As the self-position estimation method, a visual inertial odometry is used.

7. The inference device according to claim 6, characterized in that As the self-position estimation method, the detection value of the wheel speed sensor is also used.

8. The inference device according to claim 1, wherein: In the feature quantity acquisition unit, A plurality of machine learning models that have been learned in a manner of taking the two-dimensional image captured by one of the plurality of cameras as input and outputting feature quantities of the two-dimensional image captured by the one camera, i.e., image feature quantities, are respectively adopted, and the two-dimensional images captured by the corresponding cameras are respectively input to the plurality of machine learning models to obtain the image feature quantities respectively output by the plurality of machine learning models, generating a first BEV feature quantity as a feature quantity in the BEV space based on the image feature quantity, The three-dimensional coordinates are input into a second machine learning model that has been learned in a manner that takes the three-dimensional coordinates as input and outputs a feature quantity of the three-dimensional coordinates, i.e., a three-dimensional feature quantity, to obtain the three-dimensional feature quantity output by the second machine learning model, generating a second BEV feature quantity as a feature quantity in the BEV space based on the three-dimensional feature quantity, The bird's-eye view generation unit generates the bird's-eye view based on a fusion feature obtained by fusing the first BEV feature and the second BEV feature.

9. The inference device according to claim 8, characterized in that As the self-position estimation method, a visual inertial odometry is used.

10. The inference device according to claim 9, characterized in that As the self-position estimation method, the detection value of the wheel speed sensor is also used.

11. An inference method, characterized in that: Include: A step of calculating the three-dimensional coordinates of an object in the two-dimensional image using a visual odometer as a method of estimating the vehicle's own position based on two-dimensional images taken in a time series by the same camera, wherein the three-dimensional coordinates of an object existing around the vehicle are calculated based on each of the two-dimensional images obtained by taking the exterior of the vehicle by a plurality of cameras provided on the vehicle; A step of obtaining the BEV feature quantity based on the three-dimensional coordinates and the two-dimensional image obtained by photographing the exterior of the vehicle by using a BEV inference algorithm for obtaining a feature quantity in a BEV space, namely, a BEV feature quantity, wherein the BEV is a bird's-eye view; as well as A step of generating a bird's-eye view, which is an image showing a state where the vehicle is viewed from above, based on the BEV feature amount.

Images