Estimation device and estimation method

Abstract

The invention provides an estimation device and an estimation method. An estimation device (100) is provided with: a position calculation unit (210) that, on the basis of two-dimensional images captured by the same camera (300A) and consecutive in time series, calculates three-dimensional coordinates of an object within the two-dimensional images using a visual odometer as a self-position estimation method; a position calculation unit that calculates the position and attitude of the vehicle on the basis of a two-dimensional image in which the outside of the vehicle is captured by at least one of a plurality of cameras provided on the vehicle; a feature amount acquisition unit that acquires a BEV feature amount, which is a bird's-eye view, on the basis of the two-dimensional image obtained by imaging the outside of the vehicle, using a BEV estimation algorithm for acquiring the BEV feature amount, which is a feature amount in a BEV space; and a correction unit that corrects the BEV feature amount acquired by the feature amount acquisition unit using information indicating the position and orientation. As a result, it is possible to reduce the influence of a synchronization error when adjusting the BEV feature amount.

Description

Technical Field

[0001] The present application relates to an inference device and an inference method. Background Art

[0002] In the technology described in Patent Document 1, three-dimensional spatial information (BEV feature quantity) is inferred using a machine learning model based on two-dimensional images obtained by multiple cameras mounted on a vehicle, and a bird's-eye view is generated based on the inferred three-dimensional spatial information. In the technology described in Patent Document 1, the three-dimensional spatial information (BEV feature quantity) is adjusted using the motion information of the vehicle. As the motion information of the vehicle, acceleration, speed, yaw rate, etc. detected by various sensors installed on the vehicle are used.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: U.S. Patent Application Publication No. 2023 / 0053785 Summary of the invention

[0006] These various sensors send detection values ​​to the ECU that generates the bird's-eye view image via a CAN (Controller Area Network) bus. A plurality of ECUs are connected to the CAN bus, and if the communication load of the CAN bus increases, communication delay occurs.

[0007] Here, the plurality of cameras that acquire two-dimensional images are connected to the ECU that generates the bird's-eye view image via Ethernet (registered trademark). Therefore, it can be said that there is almost no time lag between the timing when the ECU that generates the bird's-eye view image receives the image acquired by the camera and the timing when the camera transmits the image.

[0008] On the other hand, if there is a communication delay in the CAN bus, the time lag between the timing when the ECU generating the bird's-eye view image receives the detection values ​​of various sensors and the timing when the various sensors send the detection values ​​becomes larger. It is hoped that the timing when the ECU generating the bird's-eye view image obtains the two-dimensional image of the camera is the same as the timing when the sensor detection values ​​are obtained. However, due to the influence of the communication delay of the CAN bus, a synchronization error occurs when adjusting the BEV feature value. Therefore, a technology that can reduce the influence of the synchronization error when adjusting the BEV feature value is required.

[0009] According to one embodiment of the present application, an inference device is provided. The inference device comprises: a position calculation unit, which uses a visual odometer as a self-position inference method to calculate the position and posture of a vehicle based on two-dimensional images taken in a time series by the same camera, and calculates the position and posture of the vehicle based on the two-dimensional image taken by at least one of a plurality of cameras installed on the vehicle to obtain the exterior of the vehicle; a feature quantity acquisition unit, which uses an algorithm for BEV inference for obtaining feature quantities in a BEV space, i.e., BEV feature quantities, to obtain the BEV feature quantities based on the two-dimensional image taken by taking the exterior of the vehicle, the BEV being a bird's-eye view; and a correction unit, which uses information indicating the position and posture to correct the BEV feature quantities obtained by the feature quantity acquisition unit.

[0010] According to the estimation device of the above-mentioned method, the position and posture of the vehicle are estimated using the visual odometer using images obtained by a camera that is less affected by communication delay than a sensor connected to the ECU via a CAN bus, and the BEV feature quantity is corrected using information indicating the estimated position and posture. Compared with a method of correcting the BEV feature quantity using the detection value of a sensor connected to the ECU via a CAN bus, the influence of synchronization error can be reduced.

[0011] According to another embodiment of the present application, an inference method is provided, which is an inference method for inferring a bird's-eye view, comprising: a step of calculating the position and posture of a vehicle using a visual odometer as a method of inferring its own position based on two-dimensional images continuous in time series taken by the same camera, in which the position and posture of the vehicle are calculated based on the two-dimensional image taken by at least one of a plurality of cameras installed on the vehicle and obtained by taking the outside of the vehicle; a step of obtaining the BEV feature quantity based on the two-dimensional image taken by taking the outside of the vehicle and obtained by using an algorithm for BEV inference for obtaining feature quantities in the BEV space, namely, BEV feature quantities, wherein the BEV is a bird's-eye view; and a step of correcting the obtained BEV feature quantity using information representing the position and posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a block diagram showing a schematic configuration of a driving assistance system.

[0013] Figure 2 This is an explanatory diagram showing an example of camera configuration.

[0014] Figure 3 This is a diagram for explaining coordinate transformation.

[0015] Figure 4 This is a flowchart showing the method for generating a bird's-eye view in the first embodiment.

[0016] Figure 5 This is a flowchart showing a method for generating a bird's-eye view in the second embodiment.

[0017] Figure 6 This is a flowchart showing a method for generating a bird's-eye view in the third embodiment.

[0018] Figure 7 This is a flowchart showing a method for generating a bird's-eye view in the fourth embodiment.

[0019] Figure 8 This is a flowchart showing a method for generating a bird's-eye view in the fifth embodiment.

[0020] Fig. 9 This is a flowchart showing a method for generating a bird's-eye view in the seventh embodiment.

[0021] Fig.10 yes Fig. 9 Flowchart of the calculation process of the position and posture in.

[0022] Fig.11 This is a flowchart showing a method for generating a bird's-eye view in the eighth embodiment.

[0023] Fig.12 yes Fig.11 Flowchart of the calculation process of the position and posture in. DETAILED DESCRIPTION

[0024] A. First Implementation Method:

[0025] Figure 1 The driving assistance system 1 shown uses a plurality of two-dimensional images acquired in a time series by an image sensor provided on the vehicle 50 to estimate the conditions around the vehicle 50 and generate a bird's-eye view image of the vehicle 50 viewed from above. The bird's-eye view image is an image of the vehicle 50 viewed from above. For example, the driving assistance system 1 uses a plurality of two-dimensional images acquired in a time series by an image sensor provided on the vehicle 50 to display an image of the vehicle 50 viewed from above on a vehicle monitor of the vehicle 50. The vehicle 50 can travel using an ADAS (Advanced Driving Assistant System).

[0026] The driving support system 1 includes an estimation device 100 and a camera group 300 . The estimation device 100 and the camera group 300 are mounted on a vehicle 50 .

[0027] like Figure 2As shown, the camera group 300 includes six cameras 300A to 300F installed on the vehicle 50. Cameras 300A to 300F are monocular cameras capable of capturing color images. Camera 300A is disposed on the upper portion of the front windshield of the vehicle 50. Camera 300A captures a predetermined range in front of the vehicle 50. Camera 300B is disposed on the lower portion of the rearview mirror on the right side of the vehicle 50. Hereinafter, the right side of the vehicle 50 refers to the right side when viewed from a person sitting in the vehicle 50 facing the front of the vehicle 50. Camera 300B captures a predetermined range in the right oblique front of the vehicle 50.

[0028] The camera 300C is disposed at the lower part of the left rearview mirror. The left side of the vehicle 50 refers to the left side when viewed from a person sitting in the vehicle 50 facing the front of the vehicle 50. The camera 300C photographs a predetermined range in the left oblique front of the vehicle 50. The camera 300D is disposed at the right door pillar. The camera 300D photographs a predetermined range on the right side of the vehicle 50. The camera 300E is disposed at the left door pillar. The camera 300E photographs a predetermined range on the left side of the vehicle 50. The camera 300F is disposed at the center of the rear door. The camera 300F photographs a predetermined range behind the vehicle 50.

[0029] Cameras 300A to 300F capture images of a predetermined range outside vehicle 50 at a predetermined frame rate, and output the acquired images to estimation device 100. In order to facilitate technical understanding, in the embodiment, cameras 300A to 300F capture images at the same timing.

[0030] The inference device 100 is a computer including a memory 110, an input / output interface 120, and a CPU (Central Processing Unit) 150. The memory 110 and the input / output interface 120 are connected to the CPU 150 via a bus 190. For example, the functions of the inference device 100 are implemented by a driving control ECU (Electronic Control Unit) that is responsible for driving control of the vehicle 50.

[0031] Various programs and various data used in various processes executed by the inference device 100 are stored in the memory 110. Data representing a machine learning model used in various processes executed by the inference device 100 is stored in the memory 110.

[0032] The cameras 300A to 300F are connected to the input / output interface 120 via Ethernet (registered trademark).

[0033] The CPU 150 is a processor that implements various functions by executing programs stored in the memory 110. For example, the CPU 150 stores two-dimensional images received from the cameras 300A to 300F in the memory 110. In the embodiment, the CPU 150 functions as the position calculation unit 210, the feature quantity acquisition unit 220, the correction unit 225, and the bird's-eye view generation unit 230 by executing the programs stored in the memory 110.

[0034] The position calculation unit 210 calculates the position and posture of the vehicle 50 and the three-dimensional coordinates of the objects in the two-dimensional image based on the two-dimensional images that are continuous in time series, using visual odometry as a method of estimating the own position. In the present embodiment, the position calculation unit 210 calculates the position and posture of the vehicle 50 and the three-dimensional coordinates of the objects existing around the vehicle 50 based on the two-dimensional image captured by at least one of the cameras 300A to 300F, using visual odometry. The details of the processing of the position calculation unit 210 will be described later.

[0035] The feature quantity acquisition unit 220 acquires a feature quantity in a BEV (Bird's eye view) space, i.e., a BEV feature quantity, based on a two-dimensional image obtained by capturing the exterior of the vehicle 50 by the cameras 300A to 300F. The BEV feature quantity includes information on the coordinate values ​​of the point cloud in the three-dimensional space and the color of the point cloud, similarly to the point cloud data detected by a laser radar (Lidar). The processing of the feature quantity acquisition unit 220 will be described in detail later.

[0036] The correction unit 225 corrects the BEV feature amount generated by the feature amount acquisition unit 220 according to the position and posture of the vehicle 50. The details of the processing of the correction unit 225 will be described later.

[0037] The bird's-eye view generation unit 230 generates a bird's-eye view based on the BEV feature quantity. The bird's-eye view is an image showing the situation of observing the vehicle 50 from above. As a method for generating a bird's-eye view based on the BEV feature quantity, an existing method can be used. The bird's-eye view generation unit 230 can detect objects around the vehicle 50 based on the BEV feature quantity and generate a bird's-eye view showing objects around the vehicle 50. In addition, it is not necessary to include all objects around the vehicle 50 in the generated bird's-eye view. For example, the bird's-eye view generation unit 230 can also detect lanes around the vehicle 50 based on the BEV feature quantity and generate a bird's-eye view including the vehicle 50 and the lanes.

[0038] Next, the visual odometry (hereinafter referred to as VO) is described. In VO, feature points are detected in a plurality of two-dimensional images that are continuous in time series, and the position and posture of the vehicle 50 are estimated using the coordinates of the feature points in the two-dimensional images. A feature point is a point on an image that can be reliably detected.

[0039] Feature points are detected using a corner detection method. For example, the intersection of two edges is detected as a corner point. Alternatively, a point where two significant edges with different directions exist in a certain local vicinity is detected as a corner point. As a method for corner detection, for example, Harris corner detection, SHIFT (scale-invariant feature transform), SURF (Speeded up robust features), and ORB (Oriented FAST and Rotated BRIEF) can be used. In addition, most methods of corner detection can detect not only corner points but also feature points. The number of feature points detected from a two-dimensional image is, for example, 100.

[0040] For example, feature points are detected from three consecutive two-dimensional images taken by the same camera in a time series. The time when the three two-dimensional images are obtained is set as time t0, t1, and t2. Time t0 is the earliest time, and time t2 is the latest time. Feature points of the two-dimensional images obtained at time t0, t1, and t2 are detected respectively, and the coordinates and brightness values ​​of each detected feature point are recorded. The detected feature points are, for example, the corners of buildings, the outlines of traffic signs, the boundaries of the colors of traffic signs, and the corners of curbstones.

[0041] like Figure 3 As shown in , the coordinates of the detected feature points are the coordinates obtained by projecting the three-dimensional position of the object in the world coordinate system onto the image coordinate system. The relationship between the coordinates of the feature points in the two-dimensional image and the three-dimensional position of the feature points in the world coordinate system can be expressed as in the following formula (1). In formula (1), f x 、f y Indicates focal length. c x 、c y Indicates the position of the intersection of the optical axis and the projection surface. 11 ~r 33 represents the rotation matrix. t1 to t3 represent translation vectors. s represents the scale factor. The scale factor s is estimated using the results of measuring the position and posture of the camera in advance using an IMU (Inertial Measurement Unit) or the like.

[0042]

Number 1

[0043] [Math.1]

[0044]

[0045] Formula (1) can be expressed in the form of formula (2).

[0046]

Number 2

[0047] [Math.2]

[0048] su I =K[RT]X W …(2)

[0049] The matrix K in equation (2) is called the internal parameters of the camera. The matrix [RT] is called the external parameters of the camera. The external parameters of the camera are determined by the position T and posture R of the camera coordinate system. As shown in equation (2), the position u in the two-dimensional image I =[uv 1] t The position T and posture R of the camera coordinate system and the position X of the world coordinate system W =[X W Y W Z W ] t The function representation of .

[0050] When the positions of feature points corresponding to the same object or the like detected in a plurality of two-dimensional images are continuously tracked in a time series, the positions of the feature points of interest change due to the movement of the vehicle 50 , that is, the movement of the camera group.

[0051] In VO, the position and posture of the camera at time t1 and the three-dimensional coordinates of the feature points in the world coordinate system are estimated based on the positions of the feature points detected in each two-dimensional image through bundle adjustment. Specifically, the position and posture of the camera at time t1 and the three-dimensional coordinates of the feature points in the world coordinate system are calculated using the coordinates of the corresponding feature points detected in multiple two-dimensional images and the assumed position T and posture R of the camera. In addition, the calculated three-dimensional coordinates are reprojected to the image coordinate system, and the distance between the projected point and the feature point detected from the two-dimensional image (reprojection error) is estimated.

[0052] The position u of the feature point in the two-dimensional image can be obtained by the projection function as shown in formula (3). In addition, the internal parameters are set to be fixed and are not considered in formula (3). The projection function is a function that maps the coordinate system of the object space (world coordinate system) to the coordinate system of the image space (image coordinate system).

[0053]

Number 3

[0054] [Math.3]

[0055] u P =proj(R, T, X W )…(3)

[0056] The position u of the feature point detected from the two-dimensional image is defined as follows: I , and the position u of the feature point obtained by the projection function P In formula (4), u I Represents a set of coordinates of feature points detected from a two-dimensional image.

[0057]

Number 4

[0058] [Math.4]

[0059] C cam =|u I -proj(R, T, X W )|…(4)

[0060] By using the error function C represented by equation (4) cam The camera position and posture at time t1 and the three-dimensional coordinates of the feature points in the world coordinate system are inferred by minimizing the optimization process. The inferred camera position and posture are represented by a rotation matrix and a translation vector. Minimizing the error represented by equation (4) means minimizing the error between the actual coordinates of the detected feature points in the image and the coordinates of the feature points obtained by the projection function. In this way, the camera position T and posture R in the world coordinate system are inferred in VO.

[0061] Next, an algorithm used for BEV estimation in the present embodiment will be described.

[0062] In this embodiment, a bird's-eye view image is generated by combining the algorithm disclosed in the specification of U.S. Patent Application Publication No. 2023 / 0053785 with the inference result of the position and posture of the vehicle 50 based on the VO. In the algorithm disclosed in the above-mentioned document, a machine learning model with a transformer architecture is used to convert the image feature quantities corresponding to each of the multiple cameras from the image space to the BEV space, thereby obtaining the BEV feature quantity.

[0063] In the algorithm disclosed in the above document, first, two-dimensional images are obtained respectively by multiple cameras. Using the backbone network, the image features of each two-dimensional image are extracted. The backbone network is, for example, CNN. The extracted image features are input into the Transformer Engine, which is a machine learning model with a Transformer architecture. The Transformer Engine is a machine learning model that utilizes the attention mechanism. The Transformer Engine is a machine learning model trained in the following manner: that is, the image features extracted from multiple two-dimensional images are used as input, the image features are fused and projected into the BEV space.

[0064] exist Figure 4 The method for estimating a bird's-eye view executed by the estimation device 100 in this embodiment is shown in FIG. Figure 4 While executing the following processing, the cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A to 300F are sequentially stored in the memory 110.

[0065] In step S101, the feature acquisition unit 220 extracts the image feature of the two-dimensional image. Specifically, the image feature (feature vector) of the two-dimensional image acquired by the camera 300A is extracted using a machine learning model such as CNN. The image feature of the extracted two-dimensional image is stored in the memory 110. For extracting the image feature of the two-dimensional image acquired by the camera 300A, a machine learning model that has been learned in a manner that outputs the image feature of the two-dimensional image when the two-dimensional image is input is used. The machine learning model is generated by machine learning using a two-dimensional image captured by a camera installed at the same position as the camera 300A as learning data. In addition, the two-dimensional images acquired by the cameras 300B to 300F are used as input, and the image feature of the two-dimensional image is extracted respectively using the machine learning model corresponding to each camera. The image feature of the extracted two-dimensional image is stored in the memory 110.

[0066] In step S101 , for example, image feature quantities of a set of two-dimensional images may be extracted. A set of two-dimensional images refers to six two-dimensional images acquired at the same timing by the cameras 300A to 300F.

[0067] Alternatively, in step S101, image feature extraction processing may be performed on two-dimensional images that are included in a predetermined number of groups among the two-dimensional images stored in the memory 110 and that have not been subjected to image feature extraction processing. The predetermined number is, for example, "2". In this case, if the processing of step S101 is performed once, for example, the image feature of each two-dimensional image acquired by the cameras 300A to 300F at time tc1 and the image feature of each two-dimensional image acquired by the cameras 300A to 300F at time (tc1-1) are extracted. Time tc1 is the time when the two-dimensional image was acquired just before. Time tc1-1 is the time when the two-dimensional image was acquired before tc1. No two-dimensional image is acquired between time tc1-1 and time tc1.

[0068] In step S103 , a BEV feature in the BEV space is generated using the Transformer engine based on the image feature extracted in step S101 by the feature acquisition unit 220 . The generated BEV feature is stored in the memory 110 .

[0069] For example, when the image feature quantities of a group of two-dimensional images are extracted in step S101 , the BEV feature quantities are generated based on the image feature quantities of the group of two-dimensional images in step S103 .

[0070] In step S105, the position calculation unit 210 executes VO. In the present embodiment, based on the two-dimensional image obtained by the camera 300A, the position and posture of the camera 300A in the world coordinate system are calculated by VO. The calculated position and posture of the camera 300A are processed as the position and posture of the vehicle 50. In addition, the processing of step S101 and step S103 is executed in parallel with the processing of step S105.

[0071] In step S105, the position and posture of vehicle 50 at two or more consecutive times may be calculated. In this case, when the process of step S105 is executed once, for example, the position and posture of vehicle 50 at time (tc1-1) and time tc1 are calculated respectively.

[0072] In step S107, it is determined whether the correction processing can be started by the correction unit 225. The situation in which the correction processing can be started refers to the situation in which all of the following conditions (i) to (iv) are satisfied. In the following, the current frame refers to the two-dimensional image just obtained, and the previous frame refers to the two-dimensional image obtained before the current frame is obtained. No other frames are obtained between the acquisition of the current frame and the acquisition of the previous frame. (i) Based on the image feature values ​​of the current frame respectively obtained by cameras 300A to 300F, the BEV feature values ​​are generated. (ii) Based on the image feature values ​​of the previous frame respectively obtained by cameras 300A to 300F, the BEV feature values ​​are generated. (iii) The position and posture of the vehicle 50 are calculated based on the current frame. (iv) The position and posture of the vehicle 50 are calculated based on the previous frame.

[0073] If it is determined that the calibration process can be started (step S107: Yes), step S109 is executed. If the calibration process cannot be started (step S107: No), step S101 and step S105 are executed again.

[0074] In step S109, the correction unit 225 acquires the motion information of the vehicle 50. The motion information of the vehicle 50 is the position and posture of the vehicle 50 at time tc1 and the amount of change between the position and posture of the vehicle 50 at time (tc1-1).

[0075] In step S111, the BEV feature is corrected by the correction unit 225. Specifically, first, the BEV feature based on the previous frame is aligned using the motion information of the vehicle 50. Then, the corrected BEV feature is obtained by combining the aligned BEV feature with the BEV feature based on the current frame. The corrected BEV feature is stored in the memory 110. The corrected BEV feature is used in the process of estimating the position of the vehicle in the current frame, for example, when the vehicle detected in the previous frame is hidden by other vehicles and is not detected in the current frame.

[0076] In step S113 , the bird's-eye view generation unit 230 generates a bird's-eye view based on, for example, the BEV feature value of the current frame. The generated bird's-eye view image is displayed on, for example, a monitor provided in the vehicle 50 .

[0077] In step S115, it is determined whether to continue the process based on a predetermined condition. For example, while the automatic driving function is effective, it is determined to continue the process (step S115: Yes). In this case, the processes of steps S101 and S105 are executed again. In addition, if, for example, a sensor abnormality is detected, it is determined not to continue the process (step S115: No). In this case, the process ends. Figure 4 The process of step S115 is executed by the bird's-eye view generation unit 230, for example.

[0078] As described above, in this embodiment, an image obtained by a camera that is less affected by communication delay than a sensor connected to the ECU via a CAN bus is used, the position and posture of the vehicle are estimated using a visual odometer, and the BEV feature quantity is corrected using information indicating the estimated position and posture. Compared with a method of correcting the BEV feature quantity using the detection value of a sensor connected to the ECU via a CAN bus, the influence of synchronization error can be reduced.

[0079] B. Second Implementation Method:

[0080] Hereinafter, a driving assistance system 1 according to a second embodiment will be described. The description will focus on the configurations that are different from the first embodiment, and the description of the configurations that are the same as the first embodiment will be omitted.

[0081] In this embodiment, a bird's-eye view image is generated by combining the inference results of the position and posture of the vehicle 50 based on VO with the machine learning model with an attention mechanism disclosed in the UniFusion algorithm (Zequn Qin, 4 others, "UniFusion: Unified Multi-view Fusion Transformer for Spatial-Temporal Representation in Bird's-Eye-View", arXiv, [online], [retrieved on September 4, 2023], Internet URL: https: / / arxiv.org / abs / 2207.08536). In the UniFusion (unified image fusion network) algorithm, a virtual image of image feature quantities corresponding to multiple cameras respectively obtained in the past and image feature quantities corresponding to multiple cameras respectively obtained this time are generated based on the position / posture information of the vehicle 50, and the image feature quantities corresponding to multiple cameras respectively obtained this time are converted from the image space to the BEV space using the machine learning model, thereby obtaining the BEV feature quantity. In the present embodiment, by generating a virtual image using the estimation results of the position and posture of the vehicle 50 estimated by the VO, it is possible to eliminate the influence of the synchronization error between the above-mentioned camera and various sensors obtained via the CAN bus.

[0082] exist Figure 52 shows a method for estimating a bird's-eye view executed by the estimation device 100 in the present embodiment. While executing the following processing, the cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A to 300F are sequentially stored in the memory 110.

[0083] In step S201, the image feature of the two-dimensional image is extracted by the feature acquisition unit 220. Specifically, the image feature (feature vector) of the two-dimensional image acquired by the camera 300A is extracted using a machine learning model such as CNN. The image feature of the extracted two-dimensional image is stored in the memory 110. For extracting the image feature of the two-dimensional image acquired by the camera 300A, a machine learning model that has been learned in a manner that outputs the image feature of the two-dimensional image when the two-dimensional image is input is used. The machine learning model is generated by machine learning using a two-dimensional image captured by a camera installed at the same position as the camera 300A as learning data. In addition, the two-dimensional image acquired by the cameras 300B to 300F is used as input, and the image feature of the two-dimensional image is extracted respectively using the machine learning model corresponding to each camera. The image feature of the extracted two-dimensional image is stored in the memory 110.

[0084] In step S201, image feature quantities of a set of two-dimensional images may be extracted. A set of two-dimensional images refers to six two-dimensional images acquired at the same timing by the cameras 300A to 300F.

[0085] Alternatively, in step S201 , the image feature extraction process may be performed on two-dimensional images included in a predetermined number of groups among the two-dimensional images stored in the memory 110 and not subjected to the image feature extraction process.

[0086] In step S203, the position calculation unit 210 executes VO. In this embodiment, the position and posture of the camera 300A in the world coordinate system are calculated by VO based on the two-dimensional image acquired by the camera 300A. The processing of step S201 and the processing of step S203 are executed in parallel.

[0087] In step S205, it is determined whether the BEV feature generation process can be started. The case where the BEV feature generation process can be started refers to the case where all conditions (i) to (iv) are satisfied. Time tc2 is the time when the two-dimensional image was acquired just before. Time (tc2-1) is the time when the two-dimensional image was acquired before time tc2. No two-dimensional image was acquired between time (tc2-1) and time tc2. Time (tc2-2) is the time when the two-dimensional image was acquired before time tc2-1. No two-dimensional image was acquired between time (tc2-2) and time (tc2-1). (i) The image feature of each two-dimensional image acquired by cameras 300A to 300F at time tc2 is extracted. (ii) The image feature of each two-dimensional image acquired by cameras 300A to 300F at time (tc2-1) is extracted. (iii) The image feature of each two-dimensional image acquired by cameras 300A to 300F at time (tc2-2) is extracted. (iv) The position and posture of the vehicle 50 at time tc2 are calculated. (v) The position and posture of the vehicle 50 at time (tc2-1) are calculated. (vi) The position and posture of the vehicle 50 at time (tc2-2) are calculated.

[0088] If it is determined that the BEV feature value generation process can be started (step S205: Yes), step S207 is executed. If the generation process cannot be started (step S205: No), step S201 and step S203 are executed again.

[0089] In step S207, the feature acquisition unit 220 generates BEV feature values ​​in the BEV space based on the image feature values ​​of each two-dimensional image acquired by the cameras 300A to 300F and the information of the position and posture of the vehicle 50, using the attention network in the above-mentioned UniFusion algorithm. The generated BEV feature values ​​are stored in the memory 110. In the UniFusion algorithm, the BEV feature values ​​are generated based on the information obtained by converting the image feature values ​​of the two-dimensional images acquired by each camera at time (tc2-2) and time (tc2-1) into a virtual image at time tc2 using the position / posture information of the vehicle, and the image feature values ​​obtained at time tc2.

[0090] In step S209, a bird's-eye view image is generated based on the BEV feature value by the bird's-eye view generation unit 230. The generated bird's-eye view image is displayed on a monitor provided in the vehicle 50, for example.

[0091] In step S211, it is determined whether to continue the process based on a predetermined condition. For example, while the automatic driving function is effective, it is determined to continue the process (step S211: Yes). In this case, the processes of steps S201 and S203 are executed again. In addition, if, for example, a sensor abnormality is detected, it is determined not to continue the process (step S211: No). In this case, the process ends. Figure 5 The process of step S211 is executed by the bird's-eye view generation unit 230, for example.

[0092] As described above, in this embodiment, the image obtained by the camera, which is less affected by communication delay than the sensor connected to the ECU via the CAN bus, is used to estimate the position and posture of the vehicle using the visual odometer, and the BEVt feature quantity is corrected using information indicating the estimated position and posture. Compared with the method of correcting the BEV feature quantity using the detection value of the sensor connected to the ECU via the CAN bus, the influence of the synchronization error can be reduced.

[0093] C. Third Implementation Method:

[0094] Hereinafter, a driving assistance system 1 according to a third embodiment will be described. The description will focus on the configurations that are different from the first embodiment, and the description of the configurations that are the same as the first embodiment will be omitted.

[0095] In this embodiment, a bird's-eye view image is generated by combining the inference results of the position and posture of the vehicle 50 based on VO in the LSS (Lift, Splat, Shoot) algorithm (Jonah Philion, Sanja Fidler, "Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D", in European Conference on Computer Vision. Springer, 2020, pp. 194-210).

[0096] In the LSS algorithm, first, two-dimensional images are obtained by cameras 300A~300F respectively. The feature vectors of multiple two-dimensional images are extracted respectively using machine learning models such as CNN. The depth of each pixel is inferred for each two-dimensional image obtained by each camera. Since the inferred depth is blurred, it is assumed that the light corresponding to each pixel (the light from the camera to a certain point in the three-dimensional space) is assumed, and the probability distribution of the discrete depth is predicted for each pixel. The cross product of the predicted probability distribution of discrete depth and the feature vector is calculated, and a series of points are calculated as information representing the depth of the pixel. As a result, a frustum-shaped point cloud is generated for each camera. The BEV feature is generated from the point cloud of the frustum using the internal and external parameters of the camera.

[0097] exist Figure 6 The method for estimating a bird's-eye view executed by the estimation device 100 in this embodiment is shown in FIG. Figure 6 While executing the following processing, the cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A to 300F are sequentially stored in the memory 110.

[0098] In step S301, the image feature of the two-dimensional image is extracted by the feature acquisition unit 220. Specifically, the image feature (feature vector) of the two-dimensional image acquired by the camera 300A is extracted using a machine learning model such as CNN. The image feature of the extracted two-dimensional image is stored in the memory 110. For extracting the image feature of the two-dimensional image acquired by the camera 300A, a machine learning model that has been learned in a manner that outputs the image feature of the two-dimensional image when the two-dimensional image is input is used. The machine learning model is generated by machine learning using a two-dimensional image captured by a camera installed at the same position as the camera 300A as learning data. In addition, the two-dimensional image acquired by the cameras 300B to 300F is used as input, and the image feature of the two-dimensional image is extracted respectively using the machine learning model corresponding to each camera. The image feature of the extracted two-dimensional image is stored in the memory 110. In step S301, a 1×1 convolution is performed to extract the image feature (feature vector). Therefore, the size of the two-dimensional image is not changed.

[0099] In step S301, image feature quantities of a set of two-dimensional images may be extracted. A set of two-dimensional images refers to six two-dimensional images acquired at the same timing by the cameras 300A to 300F.

[0100] Alternatively, in step S301 , the image feature extraction process may be performed on two-dimensional images included in a predetermined number of groups among the two-dimensional images stored in the memory 110 and not subjected to the image feature extraction process.

[0101] In step S303, a depth map is generated. Specifically, first, VO is executed by the position calculation unit 210. Here, VO calculates the three-dimensional coordinates of objects around the vehicle 50 in the world coordinate system based on the two-dimensional images acquired by the cameras 300A to 300F.

[0102] Afterwards, the feature acquisition unit 220 generates a depth map using the three-dimensional coordinates of the feature points in the two-dimensional image in the world coordinate system and the two-dimensional image used to calculate the three-dimensional coordinates. Here, depth maps are generated for the two-dimensional images acquired by cameras 300A to 300F, respectively. In the present embodiment, the coordinates are inferred by a feature point-based method. Therefore, the coordinates of the world coordinate system are inferred only for the feature points detected from the corresponding image. Therefore, the coordinates of the world coordinate system are not inferred for pixels other than the feature points. Therefore, the depth of pixels other than the feature points can be obtained, for example, by linear interpolation using the depths of nearby pixels.

[0103] In step S303, a depth map of a set of two-dimensional images may be generated. A set of two-dimensional images refers to six two-dimensional images acquired by cameras 300A to 300F at the same time. Alternatively, depth maps of two or more sets of two-dimensional images may be generated.

[0104] In step S305, the position calculation unit 210 executes VO. In this embodiment, the position and posture of the camera 300A in the world coordinate system are calculated by VO based on the two-dimensional image acquired by the camera 300A. The processing of steps S301 and S303 is executed in parallel with the processing of step S305.

[0105] In step S307 , the feature acquisition unit 220 obtains the outer product of the image feature (feature vector) extracted from the two-dimensional image in step S301 and the depth map generated in step S303 as an outer product feature.

[0106] In step S309, similarly to the algorithm of LSS, the feature acquisition unit 220 generates a point cloud of the viewing cone based on the outer product feature acquired in step S307. Specifically, assuming that the light corresponding to each pixel of each two-dimensional image (the light from the camera to a certain point in the three-dimensional space) is a light, the probability distribution of the discrete depth is predicted for each pixel. The outer product of the predicted probability distribution of the discrete depth and the vector representing the outer product feature is calculated as the depth. Using the inferred depth, the coordinates of each pixel of the two-dimensional image, and the internal and external parameters of the camera, a point cloud of the viewing cone is generated for the two-dimensional image acquired by each camera.

[0107] In step S311 , similarly to the algorithm of LSS, the feature acquisition unit 220 generates a BEV feature based on the point cloud of the viewing cone generated in step S309 . The generated BEV feature is stored in the memory 110 .

[0108] In step S313, it is determined whether the correction process can be started by the correction unit 225. The conditions for determining whether the correction process can be started are the same as those in the first embodiment.

[0109] If it is determined that the calibration process can be started (step S313: Yes), step S315 is executed. If the calibration process cannot be started (step S313: No), step S301 and step S305 are executed again.

[0110] In step S315, the Figure 4 In step S109), the movement information of the vehicle 50 is obtained by the correction unit 225 in a similar manner.

[0111] In step S317, the Figure 4 In step S111), the BEV feature value is corrected by the correction unit 225 in the same manner.

[0112] In step S319, a bird's-eye view generating unit 230 generates a bird's-eye view based on the BEV feature value. The generated bird's-eye view image is displayed on a monitor provided in the vehicle 50, for example.

[0113] In step S321, it is determined whether to continue the process based on a predetermined condition. For example, while the automatic driving function is effective, it is determined to continue the process (step S321: Yes). In this case, the processes of steps S301 and S305 are executed again. In addition, if, for example, a sensor abnormality is detected, it is determined not to continue the process (step S321: No). In this case, the process ends. Figure 6 The process of step S321 is executed by the bird's-eye view generation unit 230, for example.

[0114] As described above, in this embodiment, an image obtained by a camera that is less affected by communication delay than a sensor connected to the ECU via a CAN bus is used, the position and posture of the vehicle are estimated using a visual odometer, and the BEV feature quantity is corrected using information indicating the estimated position and posture. Compared with a method of correcting the BEV feature quantity using the detection value of a sensor connected to the ECU via a CAN bus, the influence of synchronization error can be reduced.

[0115] D. Fourth Implementation Method:

[0116] Hereinafter, a driving assistance system 1 according to a fourth embodiment will be described. The description will focus on the configurations that are different from the first embodiment, and the description of the configurations that are the same as the first embodiment will be omitted.

[0117] In the fourth embodiment, similarly to the first embodiment, a bird's-eye view image is generated by combining the estimation result of the three-dimensional coordinates of the objects around the vehicle 50 based on VO with the algorithm disclosed in the specification of US Patent Application Publication No. 2023 / 0053785.

[0118] exist Figure 7 2 shows a method for estimating a bird's-eye view executed by the estimation device 100 in the present embodiment. While executing the following processing, the cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A to 300F are sequentially stored in the memory 110.

[0119] In step S401, similarly to the first embodiment (see Figure 4 In step S101 , the feature acquisition unit 220 extracts image features of the two-dimensional image, and the extracted image features of the two-dimensional image are stored in the memory 110 .

[0120] In step S401, image features may be extracted from a group of two-dimensional images. Alternatively, image features may be extracted from two-dimensional images stored in the memory 110 and not subjected to image feature extraction, which are included in a predetermined number of groups.

[0121] In step S403 , the feature acquisition unit 220 generates BEV feature values ​​in the BEV space using the Transformer engine based on the image feature values ​​of the two-dimensional images acquired by the cameras 300A to 300F. The generated BEV feature values ​​are stored in the memory 110 .

[0122] In step S405, VO is executed by the position calculation unit 210. In the present embodiment, the positions and postures of the cameras 300A to 300F in the world coordinate system are calculated by VO for each of the two-dimensional images acquired by the cameras 300A to 300F.

[0123] In step S405 , the position and posture of the vehicle 50 at two or more consecutive times may be calculated.

[0124] In step S407, the average value of the positions and postures of the cameras 300A to 300F is obtained. In the present embodiment, the average value of the obtained positions and postures is processed as the position and posture of the vehicle 50. In the first embodiment, the position and posture of the camera 300A are processed as the position and posture of the vehicle 50, but in the present embodiment, the average value of the positions and postures of the cameras 300A to 300F is processed as the position and posture of the vehicle 50. In addition, the processing of steps S401 and S403 is performed in parallel with the processing of steps S405 and S407.

[0125] In addition, in step S407 , the average value of the positions and postures of the cameras 300A to 300F at two or more consecutive moments may be calculated.

[0126] In step S409, it is determined whether the correction process can be started by the correction unit 225. The conditions for determining whether the correction process can be started are the same as those in the first embodiment.

[0127] In step S411, as in the first embodiment (see Figure 4 In step S109), the movement information of the vehicle 50 is obtained by the correction unit 225 in a similar manner.

[0128] In step S413, as in the first embodiment (see Figure 4 In step S111), the BEV feature value is corrected by the correction unit 225 in the same manner.

[0129] In step S415, a bird's-eye view image is generated based on the BEV feature value by the bird's-eye view generation unit 230. The generated bird's-eye view image is displayed on a monitor provided in the vehicle 50, for example.

[0130] In step S417, it is determined whether to continue the process based on a predetermined condition. For example, while the automatic driving function is effective, it is determined that the process is continued (step S417: Yes). In this case, the processes of steps S401 and S405 are executed again. In addition, if, for example, the vehicle 50 stops traveling, it is determined that the process is not continued (step S417: No). In this case, the process ends. Figure 7The process of step S417 is executed by the bird's-eye view generation unit 230, for example.

[0131] In this embodiment, the position and posture of each camera are obtained based on the two-dimensional images obtained by multiple cameras, and the average value of the positions and postures of the multiple cameras is regarded as the position and posture of the vehicle 50. For example, if only a camera such as the camera 300A that captures the front of the vehicle 50 is used, it may not be possible to detect a sufficient number of feature points outside the vehicle 50. In such a case, by obtaining the average value of the positions and postures of the multiple cameras, the estimation accuracy of the position and posture of the vehicle 50 can be further improved.

[0132] E. Fifth Implementation Method:

[0133] In the present embodiment, the position calculation unit 210 calculates the position and posture of the vehicle 50 based on two-dimensional images that are continuous in time series using Visual Inertial Odometry (VIO) as a method of estimating the own position. In this case, the vehicle 50 is assumed to include an IMU.

[0134] In VIO, the position and posture of the vehicle 50, the three-dimensional coordinates of objects around the vehicle 50, and the synchronization error between the camera and the IMU are calculated using the image obtained by the camera and the detection value of the IMU provided in the vehicle 50. As a VIO method, for example, a known technology disclosed in the literature (Tong Qin, and Shaojie Shen, "Online Temporal Calibration for Monocular Visual-Inertial Systems", IEEE International Workshop on Intelligent Robots and Systems (IROS), [online], [retrieved on September 4, 2023], Internet URL: https: / / doi.org / 10.1109 / iros.2018.8593603) can be used.

[0135] exist Figure 8 FIG. 2 shows a method for estimating a bird's-eye view executed by the estimation device 100 in this embodiment. Figure 8 In the same process as the first embodiment (see Figure 4 ) are marked with the same reference numerals. For example, if the automatic driving function is enabled, the Figure 8While executing the following processing, the cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A to 300F are sequentially stored in the memory 110.

[0136] In step S101 , similarly to the first embodiment, the feature acquisition unit 220 extracts image features (feature vectors) of the two-dimensional image acquired by the camera 300A. The extracted image features of the two-dimensional image are stored in the memory 110 .

[0137] In step S103 , similarly to the first embodiment, the feature acquisition unit 220 generates a BEV feature based on the image feature of each of the two-dimensional images acquired by the cameras 300A to 300F. The generated BEV feature is stored in the memory 110 .

[0138] In step S106, the position calculation unit 210 performs VIO. In the present embodiment, the position and posture of the camera 300A in the world coordinate system are calculated by VIO for the two-dimensional image obtained by the camera 300A. The calculated position and posture of the camera 300A are processed as the position and posture of the vehicle 50. In addition, the processing of step S101 and step S103 is performed in parallel with the processing of step S106.

[0139] In addition, in step S106 , the position and posture of the vehicle 50 at two or more consecutive times may be calculated.

[0140] The processing after step S107 is the same as that of the first embodiment, and thus the description thereof is omitted.

[0141] In the present embodiment, in order to estimate the position and attitude of the vehicle 50 , the measurement value of the inertial measurement device is used in addition to the two-dimensional image, so that the estimation of the position and attitude of the vehicle 50 can obtain a more reliable result.

[0142] F. Sixth Implementation Method:

[0143] In any one of the first to fourth embodiments, VIO can be used as a method for estimating the own position. In addition, in any one of the first to fourth embodiments, VIO with an additional wheel speed sensor can be used as a method for estimating the own position. Generally speaking, VIO cannot distinguish between changes in acceleration caused by vehicle acceleration and changes in posture after the vehicle just starts to move from a parking state, which leads to a larger error in estimating the position / posture of the vehicle. By adding a wheel speed sensor, the error in estimating the position / posture of the vehicle by VIO can be reduced. In this case, it is assumed that the vehicle 50 has a wheel speed sensor. In order to infer the position and posture of the vehicle 50, the measurement value of the inertial measurement device and the detection value of the wheel speed sensor are used in addition to the two-dimensional image, so that a more reliable result can be obtained for estimating the position and posture of the vehicle 50.

[0144] G. Seventh Implementation Method:

[0145] Hereinafter, a driving assistance system 1 according to a seventh embodiment will be described. The description will focus on the configurations that are different from the first embodiment, and the description of the configurations that are the same as the first embodiment will be omitted.

[0146] exist Fig. 9 FIG. 2 shows a method for estimating a bird's-eye view executed by the estimation device 100 in this embodiment. Fig. 9 In the same process as the first embodiment (see Figure 4 ) are denoted by the same reference numerals. While executing the following processing, the cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A to 300F are sequentially stored in the memory 110.

[0147] The processing of step S101 and step S103 is the same as that of the first embodiment, and thus the description thereof is omitted.

[0148] In step S102, the position and posture calculation process is executed. Note that the processes of step S101 and step S103 are executed in parallel with the process of step S102.

[0149] exist Fig.10 Shown in Fig. 9 Flowchart of the calculation process of the position and posture in FIG. In step S1021, VIO is executed by the position calculation unit 210. In the present embodiment, the measurement value of the inertial measurement device and the detection value of the wheel speed sensor are used.

[0150] In this embodiment, the position and posture of the camera 300A in the world coordinate system are calculated by VIO for the two-dimensional image acquired by the camera 300A. The calculated position and posture of the camera 300A are processed as the position and posture of the vehicle 50. By executing VIO, the position T and posture R of the vehicle 50 in the world coordinate system and the three-dimensional coordinates X of the set of feature points are calculated. W .

[0151] In step S1023, the offset error and the synchronization error are estimated by the position calculation unit 210. Specifically, based on the position T and posture R of the vehicle 50 and the three-dimensional coordinates X of the set of feature points W , infer the bias error of the IMU. In order to infer the bias error, a known inference method is used. And based on the position T and posture R of the vehicle 50 and the three-dimensional coordinates X of the set of feature points W , infer the synchronization error between the camera and the IMU. To infer the synchronization error, a known inference method is used.

[0152] In step S1025, the synchronization error is corrected by the position calculation unit 210. Specifically, the synchronization error estimated in step S1023 is used to select the detection value synchronized with the two-dimensional image acquired by the camera 300A from the time-series detection values ​​of the IMU.

[0153] In step S1027 , the position calculation unit 210 corrects the detection value of the IMU selected in step S1025 using the bias error estimated in step S1023 .

[0154] In step S1029, the position calculation unit 210 calculates the position and attitude of the vehicle 50 based on the detection value of the IMU whose bias error has been corrected in step S1027. The above is the calculation process of the position and attitude. In this embodiment, the position and attitude of the vehicle 50 are calculated based on the detection value of the IMU. Then, the execution Fig. 9 The processing after step S107 is shown.

[0155] like Fig. 9 As shown, in step S107, it is determined whether the correction processing can be started by the correction unit 225. The situation in which the correction processing can be started refers to the situation in which all the following conditions (i) to (iv) are satisfied. (i) Based on the image feature of the current frame respectively acquired by the cameras 300A to 300F, the BEV feature is generated. (ii) Based on the image feature of the previous frame respectively acquired by the cameras 300A to 300F, the BEV feature is generated. (iii) The position and posture of the vehicle 50 at the time of acquisition of the current frame are calculated. (iv) The position and posture of the vehicle 50 at the time of acquisition of the previous frame are calculated.

[0156] If it is determined that the calibration process can be started (step S107: Yes), step S109 is executed. If the calibration process cannot be started (step S107: No), step S101 and step S105 are executed again.

[0157] In step S109, the calibration unit 225 obtains the motion information of the vehicle 50. The motion information of the vehicle 50 is the position and posture of the vehicle 50 at the time of obtaining the current frame, and the amount of change from the position and posture of the vehicle 50 at the time of obtaining the previous frame. As described above, the position and posture of the vehicle 50 are calculated based on the measurement values ​​of the IMU.

[0158] The processing after step S111 is the same as that of the first embodiment, and thus the description thereof is omitted.

[0159] In VIO, the inference results of the position and posture of the vehicle 50 based on the two-dimensional image obtained by the camera and the inference results of the position and posture of the vehicle 50 based on the detection value of the IMU are integrated. Since the angular velocity information output by the IMU contains a bias error, the error of the inference result caused by the bias error of the IMU becomes larger when the period for inferring the position and posture involves a long time. When the two-dimensional image obtained by the camera is used for inferring the position and posture, the inference result contains an error due to the influence of the blur contained in the two-dimensional image. Therefore, when the period for inferring the position and posture is short, the accuracy of the inference result is reduced. On the other hand, since there is no bias error in the two-dimensional image obtained by the camera, the accuracy of the inference result becomes higher when the period for inferring the position and posture involves a long time.

[0160] In view of the above characteristics, in this embodiment, the bias error of the detection value of the IMU is inferred using the inference result of the VIO. After the bias error of the detection value of the IMU is corrected, the position and posture of the vehicle 50 are calculated based on the detection value of the IMU. In this way, the problems caused by the characteristics of the above sensors are eliminated.

[0161] H. Eighth Implementation Method:

[0162] Hereinafter, a driving assistance system 1 according to an eighth embodiment will be described. The description will focus on the configurations that are different from the first embodiment, and the description of the configurations that are the same as the first embodiment will be omitted.

[0163] exist Fig.11 FIG. 2 shows a method for estimating a bird's-eye view executed by the estimation device 100 in this embodiment. Fig.11 In the same process as the first embodiment (see Figure 4) are denoted by the same reference numerals. While executing the following processing, the cameras 300A to 300F capture images of a predetermined range outside the vehicle 50 at a predetermined frame rate, and output the acquired two-dimensional images to the estimation device 100. The two-dimensional images captured by the cameras 300A to 300F are sequentially stored in the memory 110.

[0164] The processing of step S101 and step S103 is the same as that of the first embodiment, and thus the description thereof is omitted.

[0165] In step S104, the position and posture calculation process is executed. Note that the processes of step S101 and step S103 are executed in parallel with the process of step S104.

[0166] exist Fig.12 Shown in Fig.11 Flowchart of the calculation process of the position and posture in step S1041. In step S1041, the position calculation unit 210 detects the moving object contained in the two-dimensional image obtained by the camera 300A and subjected to the image feature extraction process in step S101. For example, the moving object is detected using a machine learning model such as Pseudo-LiDAR. Alternatively, the moving object can be detected using a known detection method for a moving object using epipolar geometry.

[0167] In step S1043, the position calculation unit 210 performs masking processing on the two-dimensional image of the object. The two-dimensional image of the object is the two-dimensional image that has been subjected to the image feature extraction processing in step S101.

[0168] In step S1045, the position calculation unit 210 performs VIO. Here, VIO is performed based on the two-dimensional image that was masked in step S1043. By performing VIO, the position T and posture R of the vehicle 50 in the world coordinate system and the three-dimensional coordinates X of the set of feature points are calculated. W The above is the calculation process of position and posture. After that, execute Fig.11 The processing after step S107 is shown.

[0169] Fig.11 The processing after step S107 is the same as that of the first embodiment, and thus the description thereof is omitted.

[0170] When the position and posture of the vehicle 50 are estimated by VIO, it is desirable that the objects around the vehicle 50 are stationary. When other vehicles, pedestrians, etc. around the vehicle 50 are moving, there is a tendency for the error included in the estimation result of VIO to become larger. Therefore, in this embodiment, a moving object is detected in advance in a two-dimensional image, and VIO is performed based on the two-dimensional image obtained by masking the detected moving object. Therefore, the error included in the estimation result of VIO can be reduced.

[0171] I. Other implementations:

[0172] I1. Other implementations 1:

[0173] In the first embodiment, an example of estimating the coordinates of objects around the vehicle 50 by a feature point based method is described. Alternatively, the coordinates of objects around the vehicle 50 may be estimated by a direct method. For example, DTAM (Richard A. Newcombe, 2 others, "DTAM: Dense Tracking and Mapping in Real-Time", IEEE Xplore, [online], [retrieved on September 4, 2023], Internet URL: https: / / doi.org / 10.1109 / ICCV.2011.6126513) can restore the depth of all pixels in an image.

[0174] I2. Other implementation methods 2:

[0175] In the first embodiment, an example is described in which the coordinates of objects around the vehicle 50 are inferred by a feature point based method, and the depth of pixels other than the feature points is interpolated by linear interpolation. Alternatively, the depth can be interpolated by a depth interpolation method using a machine learning model (Alex Wong, 3 others, "Unsupervised Depth Completion from Visual Inertial Odometry" IEEE Xplore, [online], [retrieved on September 4, 2023], Internet URL: https: / / doi.org / 10.1109 / lra.2020.2969938).

[0176] The inference device and method described in the present application may also be implemented using a dedicated computer provided by constituting the following processor and memory, wherein the processor is programmed to execute one or more functions embodied by a computer program. Alternatively, the inference device and method described in the present application may also be implemented using a dedicated computer provided by constituting a processor composed of one or more dedicated hardware logic circuits. Alternatively, the inference device and method described in the present application may also be implemented by one or more dedicated computers composed of the following combination, which is a combination of a processor and memory programmed to execute one or more functions, and a processor composed of one or more hardware logic circuits. In addition, the computer program may also be stored in a computer-readable non-transient tangible recording medium as an instruction executed by a computer.

[0177] The present application is not limited to the above-mentioned embodiments, and can be implemented by various configurations within the scope of the main purpose. For example, the technical features in the embodiments corresponding to the technical features in each method recorded in the invention content column can be appropriately replaced or combined in order to solve part or all of the above-mentioned technical problems, or to achieve part or all of the above-mentioned effects. In addition, as long as the technical feature is not described as a necessary item in this specification, it can be appropriately deleted.

Claims

1. An inference device, characterized in that: have: a position calculation unit that calculates the position and attitude of the vehicle based on two-dimensional images that are continuous in time series and are taken by the same camera, using a visual odometer as a method of estimating the own position, the position calculation unit calculating the position and attitude of the vehicle based on the two-dimensional image taken by at least one of a plurality of cameras provided on the vehicle, of the exterior of the vehicle; a feature quantity acquisition unit that uses a BEV estimation algorithm for acquiring a feature quantity in a BEV space, namely, a BEV feature quantity, to acquire the BEV feature quantity based on the two-dimensional image obtained by photographing the exterior of the vehicle, the BEV being a bird's-eye view; as well as The correction unit corrects the BEV feature amount acquired by the feature amount acquisition unit using the information indicating the position and posture.

2. The inference device according to claim 1, wherein: In the algorithm used for the BEV inference, A plurality of machine learning models that have been learned in a manner of taking the two-dimensional images captured by corresponding cameras among a plurality of cameras installed on a vehicle as input and outputting feature quantities, i.e., image feature quantities, of the two-dimensional images captured by the corresponding cameras, are used, and the two-dimensional images captured by the corresponding cameras are respectively input to the plurality of machine learning models to obtain the image feature quantities of the two-dimensional images. Based on the image feature quantities corresponding to the plurality of cameras, feature quantities of viewing cones corresponding to the plurality of cameras are generated. The feature quantities of the viewing cones respectively corresponding to the plurality of cameras are converted into the BEV feature quantities.

3. The inference device according to claim 2, characterized in that The position calculation unit uses a visual inertial odometry as the self-position estimation method.

4. The inference device according to claim 2, characterized in that The position calculation unit further uses detection values ​​of a wheel speed sensor in calculating the position and posture of the vehicle.

5. The inference device according to claim 1, wherein: In the algorithm used for the BEV inference, A plurality of machine learning models that have been learned in a manner of taking the two-dimensional images captured by corresponding cameras among a plurality of cameras installed on a vehicle as input and outputting feature quantities, i.e., image feature quantities, of the two-dimensional images captured by the corresponding cameras, are used, and the two-dimensional images captured by the corresponding cameras are respectively input to the plurality of machine learning models to obtain the image feature quantities of the two-dimensional images. Based on the image feature quantities corresponding to the multiple cameras respectively obtained in the past and the image feature quantities corresponding to the multiple cameras respectively obtained this time, a machine learning model with an attention mechanism is used to convert the image feature quantities corresponding to the multiple cameras respectively from the image space to the BEV space, thereby obtaining the BEV feature quantities.

6. The inference device according to claim 5, characterized in that The position calculation unit uses a visual inertial odometry as the self-position estimation method.

7. The inference device according to claim 5, characterized in that The position calculation unit further uses detection values ​​of a wheel speed sensor in calculating the position and posture of the vehicle.

8. An inference method, which is an inference method for inferring a bird's-eye view, characterized in that: Include: A step of calculating the position and attitude of the vehicle using a visual odometry as a method of estimating the own position based on two-dimensional images taken in a time series by the same camera, wherein the position and attitude of the vehicle are calculated based on the two-dimensional image taken by at least one of a plurality of cameras provided on the vehicle and showing the exterior of the vehicle; A step of obtaining the BEV feature quantity based on the two-dimensional image obtained by photographing the exterior of the vehicle using an algorithm for BEV estimation for obtaining a feature quantity in a BEV space, namely, a BEV feature quantity, the BEV being a bird's-eye view; and A step of correcting the acquired BEV feature value using the information indicating the position and posture.

Images