Information processing device and information processing method

Abstract

[Problem] To provide an information processing device capable of accurate calibration even in a multi-camera setup. [Solution] An on-board information processing device (101) comprises: a map point group estimation unit (205) that, on the basis of a reference feature point estimated by a reference camera motion estimation unit (203) and the motion information of a reference camera, and a relative feature point estimated by a relative camera posture-estimation unit (204) and the motion information of a relative camera, estimates a map point group constituted by point group coordinates that are three-dimensional position information of a feature point in an environment; an optimization processing unit (206) that optimizes a reference-position posture, a relative-position posture, and a map point group so that a predetermined index becomes small; and a statistical processing unit (207) that, by statistically processing a temporary correction value representing an optimized time-series relative-position posture, determines a correction value for calibration performed on a plurality of imaging units.

Description

Information processing device and information processing method

[0001] The present invention relates to an information processing device and an information processing method.

[0002] Automated driving and driver assistance systems are used for the automated driving of vehicles and output information to the vehicle's control device to support the automated driving. Automated driving and driver assistance systems estimate the location of the vehicle on which the system is installed as its own location. To expand the scope of application of automated driving and driver assistance systems, it is important to acquire information from maps based on self-location estimation. For example, while maps for driver assistance systems have been developed for expressways, there are no plans to develop maps for general roads or residential areas such as the driver's home. Therefore, technology is needed that enables automated driving and driver assistance systems to generate maps themselves and estimate their own location.

[0003] Patent Document 1 describes a "three-dimensional map estimation device including: a first selection unit that selects one or more imaging devices from a plurality of imaging devices mounted on a moving body that capture a plurality of images; a first position and orientation estimation unit that performs position and orientation estimation for each imaging device based on the captured images of the imaging devices selected by the first selection unit and outputs a first position and orientation estimation result for each imaging device; a second position and orientation estimation unit that outputs a second position and orientation estimation result from the one or more first position and orientation estimation results; and a three-dimensional map estimation unit that estimates a three-dimensional map of the surroundings of the moving body from the second position and orientation estimation result and the image."

[0004] Japanese Patent Application Laid-Open No. 2022-166505

[0005] The invention described in Patent Document 1 uses a multi-camera system, but the technology estimates a 3D map from the time-series correspondence of each camera, leaving room for improvement in the accuracy of the estimated 3D map. Furthermore, the technology for generating a high-precision map using a multi-camera system presupposes that the multi-camera system has been calibrated, and therefore cannot accommodate changes in the state of the multi-camera system, such as when the multi-camera system is in an uncalibrated state or when the multi-camera system is subject to thermal deformation or aging. Furthermore, because the multiple camera units are mounted separately on the vehicle body, multi-camera systems tend to be more prone to calibration errors than stereo cameras.

[0006] The present invention has been made in view of the above circumstances, and has as its object to enable correct calibration of a plurality of imaging units that make up a multi-camera system.

[0007] The information processing device according to the present invention includes a reference position and attitude estimation unit that estimates a reference position and attitude of one of a plurality of imaging units mounted on a moving body as a reference imaging unit, using a plurality of reference images that the reference imaging unit captures in time series of the environment around the moving body; a relative position and attitude estimation unit that estimates a relative position and attitude of the relative imaging unit with respect to the reference position and attitude of the relative imaging unit, using relative images captured by a relative imaging unit that is at least one of the plurality of imaging units other than the reference imaging unit; a map point cloud estimation unit that estimates a map point cloud made up of point cloud coordinates that are three-dimensional position information of feature points in the environment, based on reference feature points extracted from the reference image by the reference position and attitude estimation unit and motion information of the reference imaging unit, and relative feature points extracted from the relative image by the relative position and attitude estimation unit and motion information of the relative imaging unit; an optimization processing unit that optimizes the reference position and attitude, the relative position and attitude, and the map point cloud so that a predetermined index is small; and a statistical processing unit that determines correction values ​​for calibration to be performed on the plurality of imaging units by statistically processing provisional correction values ​​that represent the time-series relative position and attitude optimized by the optimization processing unit.

[0008] According to the present invention, it is possible to correctly calibrate the multiple image capturing units that make up the multi-camera system using the determined calibration correction values. Problems, configurations, and effects other than those described above will become clear from the description of the following embodiments.

[0009] FIG. 1 is a functional block diagram showing an example of the configuration of a self-map generation and calibration device according to a first embodiment of the present invention. FIG. 1 is a functional block diagram showing an example of the processing order of functional blocks of each functional unit executed by a CPU according to the first embodiment of the present invention and the flow of each data. FIG. 2 is a flowchart showing an example of processing of each functional block according to the first embodiment of the present invention. FIG. 1 is a detailed block diagram showing an example of the configuration of a mode switching unit according to the first embodiment of the present invention. FIG. 2 is a detailed block diagram showing an example of the configuration of a reference camera motion estimator according to the first embodiment of the present invention. FIG. 3 is a detailed block diagram showing an example of the configuration of a relative camera posture estimator according to the first embodiment of the present invention. FIG. 4 is a diagram showing how a closed loop is detected from feature points of a plurality of images according to the first embodiment of the present invention. FIG. 5 is a detailed block diagram showing an example of the configuration of a map point cloud estimator according to the first embodiment of the present invention. FIG. 6 is a detailed block diagram showing an example of the configuration of a statistical processing unit according to the first embodiment of the present invention. FIG. 7 is a diagram showing an example of relative camera parameters and a multi-stage buffer according to the first embodiment of the present invention. FIG. 8 is a diagram showing how the positions of feature points change when the ignition of a vehicle according to a second embodiment of the present invention is turned off and on. FIG. 9 is a functional block diagram showing an example of the configuration of an in-vehicle information processing device according to a second embodiment of the present invention.

[0010] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functions or configurations are designated by the same reference numerals, and redundant description will be omitted. The present invention is applicable to, for example, a computing device for vehicle control capable of communicating with an on-board ECU (Electronic Control Unit) for an Advanced Driver Assistance System (ADAS) or Autonomous Driving (AD).

[0011] First Embodiment First, a configuration example and an operation example of a self-mapping and calibration device according to a first embodiment of the present invention will be described. The self-mapping and calibration device has a function of generating a self-mapping and a function of calibrating a plurality of cameras constituting a multi-camera system.

[0012] 1 is a functional block diagram showing an example configuration of the self-mapping and calibration device 100. Cameras 121 to 123 (an example of an imaging unit) provided in the self-mapping and calibration device 100, a vehicle speed sensor 161, a steering angle sensor 162, an interface 171, and an in-vehicle information processing device 101 are connected by signal lines, and various data is exchanged with the in-vehicle information processing device 101. The self-mapping and calibration device 100 is mounted on a vehicle (an example of a moving body).

[0013] The in-vehicle information processing device 101 includes a CPU (Central Processing Unit) 110, a ROM (Read Only Memory) 111, a RAM (Random Access Memory) 112, and a recording unit 113. It may be configured such that all or part of the arithmetic processing is executed by another arithmetic processing device such as an FPGA (Field Programmable Gate Array).

[0014] The CPU 110 operates as an execution unit of the in-vehicle information processing device 101 by reading and executing various programs and parameters from the ROM 111. The CPU 110 reads program code of software that realizes each function according to this embodiment from the ROM 111, loads it into the RAM 112, and executes it. Variables, parameters, etc. generated during the calculation processing of the CPU 110 are temporarily written to the RAM 112, and these variables, parameters, etc. are read by the CPU 110 as appropriate. However, an MPU (Micro Processing Unit) or a GPU (Graphics Processing Unit) may be used instead of the CPU 110, or the CPU 110 and a GPU may be used together.

[0015] The ROM 111 is a read-only storage area and stores programs described below. The programs and parameters stored in the ROM 111 are read by the CPU 110. The programs described below are loaded into the RAM 112 and executed by the CPU 110.

[0016] The RAM 112 is a readable and writable storage area, and operates as the main storage device of the in-vehicle information processing device 101. The RAM 112 stores reference camera motion parameters 209, relative camera parameters 210, map point cloud coordinates 211, high-precision map point cloud coordinates 212, time-series relative camera parameters 213, and a mode flag 214, which will be described later.

[0017] By reading and executing programs from ROM 111, CPU 110 operates as a sensor information acquisition unit 201, an odometry information acquisition unit 202, a reference camera motion estimation unit 203, a relative camera posture estimation unit 204, a map point cloud estimation unit 205, an optimization processing unit 206, a statistical processing unit 207, and a mode switching unit 208.

[0018] The ROM 111 stores camera parameter design values ​​220 and vehicle specifications 221. The camera parameter design values ​​220 store the designed positions and orientations of the cameras 121 to 123 and information specific to the cameras. That is, the camera parameter design values ​​220 store the lens distortion coefficient, the optical axis center, the focal length, and the number of pixels and dimensions of the image sensor. The vehicle specifications 221 store information specific to the vehicle.

[0019] The recording unit 113 is a non-volatile storage device such as a hard disk drive (HDD), a solid state drive (SDD), an optical disk, or a non-volatile memory, and operates as an auxiliary storage device for the in-vehicle information processing device 101. The recording unit 113 stores a calibration value 215 and a map point cloud 216. The recording unit 113 also stores an operating system (OS), various parameters, and programs for operating the in-vehicle information processing device 101.

[0020] The ROM 111 and the recording unit 113 record programs and data (not shown) necessary for the CPU 110 to operate, and are used as an example of a non-transitory storage medium that can be read by a computer and that stores programs executed by the in-vehicle information processing device 101.

[0021] The cameras 121 to 123 are attached, for example, in front of the windshield inside the vehicle cabin, and on the front right and left pillars, and capture images that cover a wide area ahead of the vehicle. The position and orientation relationship between the cameras 121 to 123 and the vehicle is stored in the ROM 111 as camera parameter design values ​​220. The method of attaching the cameras 121 to 123 is not limited to the above-described method, and other attachment methods may also be used. Furthermore, the number of cameras attached at each position does not necessarily have to be one.

[0022] The cameras 121 to 123 are equipped with lenses and image sensors (not shown). The camera parameter design values ​​220 are stored in the ROM 111 in a form including internal parameters such as the characteristics of the camera, for example, a lens distortion coefficient which is a parameter indicating lens distortion, the center of the optical axis, the focal length, and the number of pixels and dimensions of the image sensor, and external parameters which indicate the position and orientation relationship with the vehicle.

[0023] The vehicle speed sensor 161 measures the vehicle speed of the vehicle in which the in-vehicle information processing device 101 is installed, and outputs the measured vehicle speed to the in-vehicle information processing device 101. The steering angle sensor 162 measures the steering angle of the vehicle in which the in-vehicle information processing device 101 is installed, and outputs the measured steering angle to the in-vehicle information processing device 101. The in-vehicle information processing device 101 calculates the amount of movement and the direction of movement of the vehicle in which the in-vehicle information processing device 101 is installed, using the vehicle speed and steering angle, by a known dead reckoning technique.

[0024] The interface 171 is, for example, a GUI (Graphical User Interface) that accepts instruction inputs from a user. The interface 171 may accept information other than instruction inputs from a user in a form other than a GUI.

[0025] <Example of processing order of functional blocks of each functional unit and flow of each data> Fig. 2 is a functional block diagram showing an example of the processing order of functional blocks of each functional unit executed by the CPU 110 and the flow of each data. The CPU 110 is not shown in Fig. 2. Fig. 3 is a flowchart showing an example of processing of each functional block. The processing of each functional block shown in Fig. 2 will be explained by associating it with each step of the flowchart.

[0026] 2 shows the functions of a sensor information acquisition unit 201, an odometry information acquisition unit 202, a reference camera motion estimation unit 203, a relative camera attitude estimation unit 204, a map point cloud estimation unit 205, an optimization processing unit 206, a statistical processing unit 207, and a mode switching unit 208 executed by the CPU 110 as functional blocks. These functional blocks are included in the in-vehicle information processing device 101 (an example of an information processing device). Also, in FIG. 2, the processing order of the functional blocks and the processing flow between different functional blocks are shown as data flows between the functional blocks and the RAM 112, the ROM 111, and the recording unit 113. The self-mapping and calibration device 100 operates according to the configuration and processing order of these functional blocks.

[0027] First, the sensor information acquisition unit 201 acquires signals (referred to as "sensor information") output from the cameras 121 to 123 (S1 in FIG. 3). The camera 121 continuously captures images at a high frequency, for example, 30 times per second. The cameras 122 and 123 also capture images at the same frequency as the camera 121.

[0028] The sensor information acquisition unit 201 acquires sensor information including images and signals output from the cameras 121 to 123 at a constant frequency, and outputs the sensor information to the reference camera motion estimation unit 203. At this time, the sensor information acquisition unit 201 uses, for example, the camera parameter design values ​​220 read out from the ROM 111 to output an image corrected in consideration of distortion of each camera to the reference camera motion estimation unit 203. However, the timing of image correction is not limited to this, and image correction may be performed in the reference camera motion estimation unit 203 or the like at a subsequent stage. Furthermore, once the sensor information acquisition unit 201 acquires the camera parameter design values ​​220, it is not necessary to re-acquire the camera parameter design values ​​220 in subsequent processing.

[0029] The odometry information acquisition unit 202 acquires signals related to odometry estimation, such as speed and steering angle, output from the vehicle speed sensor 161 and the steering angle sensor 162 (referred to as "odometry information") (S2). The processes of steps S1 and S2 are performed in parallel. The odometry information acquisition unit 202 also continuously acquires odometry information at a high frequency and outputs it to the reference camera movement estimation unit 203. The odometry information acquisition unit 202 also reads vehicle specifications 221 from the ROM 111. The vehicle specifications 221 are information specific to the vehicle in which the self map generation and calibration device 100 is installed, and the vehicle speed, steering angle, and the like take values ​​within ranges defined by the vehicle specifications 221. Therefore, the odometry information acquisition unit 202 can determine, based on the vehicle specifications 221, whether the odometry information is within a range of values ​​that can be taken by the odometry information.

[0030] After steps S1 and S2, the reference position and orientation estimation unit (reference camera motion estimation unit 203) determines one of the multiple imaging units (cameras 121 to 123) mounted on the moving body (vehicle) as the reference imaging unit (camera 121). In the following description, the reference imaging unit (camera 121) is referred to as the "reference camera." Furthermore, the reference camera motion estimation unit 203 estimates the time-series motion of the reference camera (referred to as "reference camera motion") (S3). The reference camera motion includes time-series position and orientation changes, such as translation and rotation, of the reference camera.

[0031] The reference position and orientation estimation unit (reference camera motion estimation unit 203) estimates the reference position and orientation of the reference imaging unit (camera 121) using multiple reference images captured by the reference imaging unit (camera 121) in time series of the environment surrounding the moving body (vehicle). The reference position and orientation of the reference camera includes information on the position and orientation of the reference camera. The reference camera motion estimation unit 203 roughly estimates the time-series motion of the reference camera using a reference camera image captured by the reference camera at a previous time and a reference camera image captured by the reference camera at a current time. For example, if the camera 121 captures images at 30 frames per second, the interval between the previous time and the current time is assumed to be one frame, but the interval may be multiple frames. The reference camera motion estimation unit 203 estimates the time-series motion of the reference camera using a known technique such as the eight-point method, using the correspondence between feature points extracted from the reference camera images at the previous time and the current time. The feature points correspond to the corners of an object shown in the image, or the corners of road markings painted on the road surface.

[0032] The reference camera motion estimation unit 203 also has a function of performing high-precision reference camera motion estimation using high-precision map point cloud coordinates 212 (an example of a map point cloud) extracted from reference camera images at the previous time and the current time and odometry information output from the odometry information acquisition unit 202, and operates variably according to the situation. The high-precision map point cloud coordinates 212 extracted by the reference camera motion estimation unit 203 are output to the RAM 112. The reference camera feature points 231, which are estimated values ​​of the reference camera motion estimated by the reference camera motion estimation unit 203, are output to the RAM 112 as reference camera motion parameters 209. After the reference camera motion estimation unit 203 outputs the reference camera motion parameters 209, processing transitions to the relative camera posture estimation unit 204. An example of detailed processing by the reference camera motion estimation unit 203 will be described later with reference to FIG. 5 .

[0033] Next, the relative position and orientation estimation unit (relative camera orientation estimation unit 204) determines a relative image capture unit (camera 122 or 123) that is at least one image capture unit other than the reference image capture unit (camera 121) among the multiple image capture units (cameras 121 to 123). In the following description, the relative image capture unit (camera 122 or 123) is referred to as the "relative camera." The relative position and orientation estimation unit (relative camera orientation estimation unit 204) estimates the relative position and orientation of the relative image capture unit (camera 122 or 123) with respect to the reference position and orientation using the relative image captured by the relative image capture unit (camera 122 or 123) (S4). The relative position and orientation is the position and orientation of the relative camera with respect to the position and orientation of the reference camera, and is referred to as the "relative camera orientation."

[0034] The relative camera attitude estimation unit 204 roughly estimates the relative camera attitude between the cameras 121 and 122 by a known technique such as the eight-point method using the correspondence between the feature points of the image captured by the camera 121 and the feature points of the image captured by the camera 122. Similarly, the relative camera attitude estimation unit 204 roughly estimates the relative camera attitude between the cameras 121 and 123 by a known technique such as the eight-point method using the correspondence between the feature points of the image captured by the camera 121 and the feature points of the image captured by the camera 123.

[0035] When the relative camera posture estimation unit 204 estimates the relative camera posture of each camera, if a reliable result has been obtained by the previous time based on the relationship between the images captured by each camera, that is, a camera parameter estimate 230 (described later), then the camera parameter estimate 230 may be used instead. The estimated values ​​of the relative camera postures estimated by the relative camera posture estimation unit 204 are output to the RAM 112 as relative camera parameters 210.

[0036] Furthermore, the coordinates and feature amounts of the feature points of the relative camera image detected by the relative camera posture estimation unit 204 are output to the RAM 112 as relative camera feature points 232. The feature amounts are values ​​that each feature point has and that represent the characteristics of the feature point. After the relative camera posture estimation unit 204 outputs the coordinates and feature amounts of the feature points, processing transitions to the map point cloud estimation unit 205. Details of the relative camera posture estimation unit 204 will be described later with reference to FIG. 6.

[0037] Next, the map point cloud estimation unit (map point cloud estimation unit 205) estimates a map point cloud consisting of point cloud coordinates, which are three-dimensional position information of feature points in the environment, based on the reference feature points extracted from the reference image by the reference position and orientation estimation unit (reference camera motion estimation unit 203) and motion information of the reference imaging unit (camera 121), and the relative feature points extracted from the relative image by the relative position and orientation estimation unit (relative camera posture estimation unit 204) and motion information of the relative imaging unit (camera 122 or 123). For example, the map point cloud estimation unit 205 reads reference camera motion parameters 209, relative camera parameters 210, reference camera feature points 231, and relative camera feature points 232 from the RAM 112. Next, the map point cloud coordinates (also referred to as "map point cloud"), which are three-dimensional coordinates corresponding to each of the reference camera feature points 231 and the relative camera feature points 232, are roughly estimated by triangulation (S5). The map point cloud estimation unit 205 also has a function of substituting the high-precision map point cloud coordinates 212 for the three-dimensional coordinates when the high-precision map point cloud coordinates 212 have been obtained by the reference camera movement estimation unit 203 .

[0038] The map point cloud estimation unit 205 switches between a function of estimating a map point cloud, which is three-dimensional coordinates, by triangulation and a function of replacing the three-dimensional coordinates with high-precision map point cloud coordinates 212, depending on the situation. The three-dimensional coordinates corresponding to each feature point estimated by the map point cloud estimation unit 205 are output to the RAM 112 as map point cloud coordinates 211 (an example of a map point cloud). After each piece of information is output, processing transitions to the optimization processing unit 206. Details of the map point cloud estimation unit 205 will be described later with reference to FIG. 8.

[0039] Next, the optimization processing unit (optimization processing unit 206) optimizes the reference position and orientation, the relative position and orientation, and the map point cloud so as to reduce a predetermined index (reprojection error) (S6). For example, the optimization processing unit 206 reads out reference camera motion parameters 209, relative camera parameters 210, map point cloud coordinates 211, reference camera feature points 231, and relative camera feature points 232 from the RAM 112. Using this information read out from the RAM 112, the optimization processing unit 206 optimizes all of the reference camera motion parameters 209, the relative camera parameters 210, and the map point cloud coordinates 211 by a known optimization method so as to minimize the reprojection error. The optimization processing is applied to all of the positions and orientations of the two cameras at each time and the 3D map point cloud coordinates.

[0040] To calculate the reprojection error, the optimization processing unit 206 first projects map point cloud coordinates 211 calculated from the image captured by the reference camera onto the reference camera based on the reference camera motion parameters 209, and calculates the error between the map point cloud coordinates 211 projected onto the reference camera and the corresponding reference camera feature points 231. Next, the optimization processing unit 206 projects the map point cloud coordinates 211 calculated from the image captured by the relative camera onto the relative camera based on the relative camera parameters 210, and calculates the error between the projected map point cloud coordinates 211 and the corresponding relative camera feature points 232. Thereafter, the optimization processing unit 206 calculates the sum of the errors of the reference camera feature points 231 and the relative camera feature points 232 as the reprojection error. The reprojection error has the property of taking a smaller value as the numerical values ​​of the map point cloud coordinates 211, the reference camera motion parameters 209, and the relative camera parameters 210 are more accurate.

[0041] The optimization processing unit 206 simultaneously slightly changes all of the reference camera motion parameters 209, relative camera parameters 210, and map point cloud coordinates 211 so as to minimize the reprojection error, and optimizes each value using a known technique such as the Levenberg-Marquardt algorithm. Through this processing, the accuracy of all of the reference camera motion parameters 209, relative camera parameters 210, and map point cloud coordinates 211 is improved.

[0042] The optimization processing unit 206 also has a function of estimating high-precision reference camera motion parameters 209, relative camera parameters 210, and high-precision map point cloud coordinates 212 by using high-precision map point cloud coordinates 212 instead of the map point cloud coordinates 211. As described above, the function of estimating the map point cloud, which is three-dimensional coordinates, by triangulation and the function of substituting the high-precision map point cloud coordinates 212 for the three-dimensional coordinates are switched depending on the situation. That is, the in-vehicle information processing device 101 does not operate the high-precision map generation function when the calibration function is operating, and does not operate the calibration function when the high-precision map generation function is operating. The calibration function is a function of correcting distortion, position, etc. of images captured by each camera, and is mainly performed by the optimization processing unit 206 and the statistical processing unit 207.

[0043] When the optimization processing unit 206 estimates each parameter and the high precision map point cloud coordinates 212 using the high precision map point cloud coordinates 212, overlapping points in the high precision map point cloud coordinates 212 are overwritten and corrected or averaged, and new points are added to the high precision map point cloud coordinates 212. The reference camera motion parameters 209, relative camera parameters 210, and map point cloud coordinates 211 estimated by the optimization processing unit 206 are output to the RAM 112. After each piece of information is output, processing transitions to the statistical processing unit 207.

[0044] Next, the statistical processing unit (statistical processing unit 207) statistically processes the provisional correction values ​​that represent the time-series relative positions and orientations optimized by the optimization processing unit (optimization processing unit 206) (S7). This statistical processing determines correction values ​​(calibration values) for the calibrations to be performed on the multiple image capture units (cameras 121 to 123). For example, the statistical processing unit 207 performs statistical processing on the relative camera parameters 210 estimated by the optimization processing unit 206. The statistical processing unit 207 has a function of improving the accuracy of the relative camera parameters 210 through statistical processing and estimating calibration values ​​215. The relative camera parameters 210 that were statistically processed by the relative camera orientation estimation unit 204 in step S4 are more accurate than the values ​​obtained by the relative camera orientation estimation unit 204 through the optimization processing unit 206. However, the statistically processed relative camera parameters 210 are insufficiently accurate and unstable as calibration values ​​due to feature point errors or disturbances such as moving objects.

[0045] Therefore, in order to deal with the lack of precision of the relative camera parameters 210, the statistical processing unit 207 stores highly reliable calibration values, which are estimation results at each time, in the multi-stage buffer 130 (see FIG. 10 described later) as time-series relative camera parameters 213. The multi-stage buffer 130 is provided in the statistical processing unit 207 and is composed of multiple buffer stages.

[0046] The statistical processing unit 207 obtains a calibration value 215 by applying statistical processing such as extracting a median to the group of estimation results of the calibration values ​​stored in the multistage buffer 130. The calibration value 215 obtained by the statistical processing unit 207 is output to the RAM 112 and then recorded in the recording unit 113. After the statistical processing unit 207 outputs the calibration value 215 to the RAM 112, the processing transitions to the mode switching unit 208. Details of the statistical processing unit 207 will be described later with reference to FIG. 9 .

[0047] Next, the mode switching unit (mode switching unit 208) switches the processing modes of the reference position and attitude estimation unit (reference camera motion estimation unit 203), the relative position and attitude estimation unit (relative camera attitude estimation unit 204), and the map point cloud estimation unit to either the calibration mode or the high-precision map generation mode depending on the state of the vehicle (S8), and ends this processing. The calibration mode is a processing mode in which point cloud coordinates are estimated using the relative position and attitude and relative feature points estimated in time series. The high-precision map generation mode is a processing mode in which a map point cloud made up of point cloud coordinates is estimated using the parallax obtained in the common imaging area between the reference imaging unit (camera 121) and the relative imaging unit (camera 122 or 123). Note that the calibration mode and the high-precision map generation mode may be switched by a user input.

[0048] The mode switching unit 208 switches from the calibration mode to the high-accuracy map generation mode by basically determining when calibration is complete. The mode switching unit 208 switches the calibration mode to the map generation mode when calibration using the correction value is complete. For example, the mode switching unit 208 determines when calibration is complete based on criteria such as whether a time-series change in the calibration value 215 is smaller than a predetermined value and whether the consistency between the image feature points and the map point cloud, i.e., whether the reprojection error is smaller than a predetermined value. However, the method by which the mode switching unit 208 determines when calibration is complete is not limited to the method described herein.

[0049] On the other hand, the mode switching unit 208 switches from the high-precision map generation mode to the calibration mode basically when a decrease in calibration accuracy is suspected. Therefore, the mode switching unit (mode switching unit 208) switches the map generation mode to the calibration mode when the calibration accuracy decreases. Examples of cases where a decrease in calibration accuracy is suspected include a decrease in the number of effective pixels of the image used for distance estimation, a large reprojection error, or the passage of time since the last use, such as after parking or stopping, and the mode switching unit 208 makes a judgment based on these situations. However, the method by which the mode switching unit 208 judges the situation is not limited to the method described here.

[0050] The mode determination result by the mode switching unit 208 is output to the RAM 112 as a mode flag 214. When switching to the calibration mode, the optimization processing unit (optimization processing unit 206) and the statistical processing unit (statistical processing unit 207) calibrate the multiple image capture units (cameras 121 to 123) based on the correction values. Images captured by the multiple image capture units (cameras 121 to 123) for which calibration has been performed have correct correspondence of feature points in subsequent processing.

[0051] The mode flag 214 is expressed by two types of values: a value indicating the calibration mode, and a value indicating the high-accuracy map generation mode. After the mode flag 214 is output, the statistical processing unit 207, which is the pre-processing, is executed again based on input from the cameras 121 to 123, etc., and the mode switching unit 208 waits to switch the processing mode until a trigger is generated. Details of the mode switching unit 208 will be described later with reference to FIG. 4. The map point cloud coordinates 211 or high-accuracy map point cloud coordinates 212 and calibration values ​​215 obtained by the processing up to this point are stored as a map point cloud 216 and calibration values ​​215 in the recording unit 113 as needed.

[0052] Next, an example of detailed processing of each functional block will be described with reference to FIG. 4 and subsequent figures.

[0053] <Configuration Example of Mode Switching Unit> First, we will explain a configuration example of the mode switching unit 208. Fig. 4 is a detailed block diagram showing a configuration example of the mode switching unit 208. Arrows in Fig. 4 and subsequent figures indicate the processing order of the functional blocks and the flow of data between the functional blocks.

[0054] The mode switching unit 208 is made up of a calibration verification unit 301 and a mode output unit 308. The calibration verification unit 301 is made up of a verification determination unit 302, a time-series value verification unit 303, an error verification unit 304, a valid pixel number verification unit 305, an error verification unit 306, and a scene verification unit 307.

[0055] The verification determination unit 302 obtains the mode flag 214 from the RAM 112 and determines whether to execute the time-series value verification unit 303 and the error verification unit 304, or whether to execute the valid pixel number verification unit 305, the error verification unit 306, and the scene verification unit 307. If the mode flag 214 indicates the calibration mode, the verification determination unit 302 transitions to execution of the time-series value verification unit 303 and the error verification unit 304. If the mode flag 214 indicates the high-accuracy map generation mode, the verification determination unit 302 transitions to execution of the valid pixel number verification unit 305, the error verification unit 306, and the scene verification unit 307. Note that the mode flag 214 may store a numerical value that indicates the calibration mode when it is "0" and the high-accuracy map generation mode when it is "1," for example. Alternatively, a flag that indicates the calibration mode when the mode flag 214 is turned on and the high-accuracy map generation mode when the mode flag 214 is turned off may be stored.

[0056] The time-series value verification unit 303 references the time-series values ​​of the calibration values ​​215 from the recording unit 113 and determines whether the change in the time-series values ​​is equal to or less than a predetermined value. The time-series value verification unit 303 determines the change in the time-series values ​​based on, for example, whether the difference in the calibration values ​​for the past few frames is equal to or less than a predetermined value, or whether the standard deviation of the calibration values ​​for the past few frames is equal to or less than a certain value. If the difference in the calibration values ​​is equal to or less than the predetermined value, the time-series value verification unit 303 outputs information indicating that this difference is equal to or less than the predetermined value to the error verification unit 304 and transitions to the error verification unit 304. Note that if the difference or standard deviation of the calibration values ​​exceeds the predetermined value, the calibration mode ends without executing the error verification unit 304. If it is satisfied that the difference in the calibration values ​​for the past few frames is equal to or less than a predetermined value, or the standard deviation of the calibration values ​​for the past few frames is equal to or less than a certain value, the mode is confirmed.

[0057] The error verification unit 304 determines whether the reprojection error is equal to or less than a predetermined value. The reprojection error is calculated based on the reference camera motion parameters 209, the relative camera parameters 210, the reference camera feature points 231, the relative camera feature points 232, and the map point cloud coordinates 211, and is similar to that described for the optimization processing unit 206. If the reprojection error is equal to or less than the predetermined value, the error verification unit 304 outputs information indicating that the reprojection error is equal to or less than the predetermined value to the mode output unit 308 together with the determination result of the time-series value verification unit 303, and transitions to the mode output unit 308.

[0058] The valid pixel number verification unit 305 compares the number of valid pixels between the relative cameras with a predetermined value and verifies whether the number of valid pixels is equal to or less than the predetermined value, thereby determining whether calibration has been successful. In the high-precision map generation mode, the valid pixel number verification unit 305 performs parallelization processing of the relative cameras using the relative camera parameters 210, and further performs distance measurement in a stereo camera manner, thereby performing high-precision distance measurement. In the block matching processing performed when performing high-precision distance measurement, pixels whose parallax reliability is equal to or greater than a predetermined value are called valid pixels.

[0059] The number of valid pixels tends to be relatively high if the calibration is successful, and conversely, tends to be relatively low if the calibration is unsuccessful. Therefore, the valid pixel number verification unit 305 determines that the calibration has failed if the number of valid pixels is equal to or less than a predetermined value, and determines that the calibration has succeeded if the number of valid pixels exceeds the predetermined value. If the valid pixel number verification unit 305 determines that the calibration has failed, the processing of the high-precision map generation mode ends, and the calibration mode starts.

[0060] The error verification unit 306 verifies whether the reprojection error is equal to or less than a predetermined value. The reprojection error is calculated based on the reference camera motion parameters 209, the relative camera parameters 210, the reference camera feature points 231, the relative camera feature points 232, and the map point cloud coordinates 211, and is similar to that described for the optimization processing unit 206. If the reprojection error is equal to or less than the predetermined value, the error verification unit 306 outputs information indicating that the reprojection error is equal to or less than the predetermined value to the scene verification unit 307, and transitions to the scene verification unit 307.

[0061] The scene verification unit 307 determines the calibration status based on, for example, whether the scene representing the current vehicle situation is a scene in which time has passed since the last time the in-vehicle information processing device 101 was used, such as after the vehicle has been parked or stopped, or the actual elapsed time since the vehicle ignition was turned on. In a situation in which a calibration error is likely to occur due to changes in vehicle temperature, the vehicle temperature often changes over time. For this reason, the scene verification unit 307 determines that a calibration error may have occurred if more time than a predetermined value has passed since the last calibration value was updated. Alternatively, the scene verification unit 307 may determine that a calibration error may have occurred even if the vehicle has remained in the same place, such as when parked or stopped. The scene verification unit 307 outputs the scene verification result to the mode output unit 308 and transitions to the mode output unit 308.

[0062] The mode output unit 308 outputs a mode flag 214 indicating either the calibration mode or the high-accuracy map generation mode based on the results of the time-series value verification unit 303 and the error verification unit 304, or the results of the valid pixel number verification unit 305, the error verification unit 306, and the scene verification unit 307. When both the time-series value verification unit 303 and the error verification unit 304 determine that the calibration accuracy is high and that the calibration has been successful, the mode output unit 308 outputs information indicating the high-accuracy map generation mode as the mode flag 214 to transition to the high-accuracy map generation mode.

[0063] On the other hand, if the mode output unit 308 determines that there is a possibility that a calibration error has occurred in any of the valid pixel number verification unit 305, the error verification unit 306, or the scene verification unit 307, it outputs information indicating the calibration mode as the mode flag 214. If the scene verification unit 307 verifies that the scene is one in which more time than a predetermined value has passed since the last calibration value was updated, or one in which the vehicle has remained in the same place for a long time, such as while parking, the mode output unit 308 outputs information indicating the calibration mode as the mode flag 214.

[0064] <Configuration Example of Reference Camera Motion Estimation Unit> FIG. 5 is a detailed block diagram showing a configuration example of the reference camera motion estimation unit 203. As shown in FIG.

[0065] The reference camera motion estimation unit 203 is composed of a mode determination unit 401, a feature point extraction unit 402, a feature amount calculation unit 403, a time-series corresponding point detection unit 404, a reference camera motion estimation unit 405, a high-precision map point acquisition unit 406, and a high-precision camera motion estimation unit 407.

[0066] The mode determination unit 401 acquires the mode flag 214 and determines whether the mode flag 214 indicates the calibration mode or the high-accuracy map generation mode. If the calibration mode is determined, the reference camera motion estimation unit 203 executes the feature point extraction unit 402, the feature amount calculation unit 403, the time-series corresponding point detection unit 404, and the reference camera motion estimation unit 405. If the high-accuracy map generation mode is determined, the reference camera motion estimation unit 203 executes the high-accuracy map point acquisition unit 406 and the high-accuracy camera motion estimation unit 407.

[0067] (Calibration Mode) The feature point extraction unit 402 extracts feature points representing image features from the reference camera image. The feature point extraction unit 402 can realize the process of extracting feature points using known techniques such as Harris feature point extraction or FAST (Features from Accelerated Segment Test). The feature points extracted by the feature point extraction unit 402 are recorded in the RAM 112 as reference camera feature points 231. After the feature points are extracted, the process transitions to the feature amount calculation unit 403.

[0068] The feature amount calculation unit 403 calculates feature amounts representing the characteristics of all feature points extracted by the feature point extraction unit 402. Feature amounts are also called descriptors, and are obtained by known techniques such as BRIEF (Binary Robust Independent Elementary Features) or ORB (Oriented FAST and Rotated BRIEF). The feature point extraction unit 402 calculates feature amounts for all feature points extracted from images captured by the reference camera. After calculating the feature amounts, the process transitions to the time-series corresponding point detection unit 404.

[0069] The time-series corresponding point detection unit 404 calculates time-series feature point correspondences of the reference camera. Time-series feature point correspondences refer to feature points representing the same object, where feature points obtained from an image captured by the reference camera at a previous time correspond to feature points obtained from an image captured at a current time. A feature point search method for feature point correspondences is realized by searching for feature points from the images captured by the reference camera that have feature amounts closest to those calculated by the feature amount calculation unit 403. In this case, although not reflected in FIG. 5 , the time-series corresponding point detection unit 404 may limit the search area estimated from the vehicle motion using odometry information or the like. After the time-series corresponding point detection unit 404 has calculated correspondences for all feature points, the process transitions to the reference camera motion estimation unit 405.

[0070] The reference camera motion estimation unit 405 estimates the reference camera motion using the feature points extracted by the feature point extraction unit 402 and the time-series correspondence of the feature points detected by the time-series correspondence point detection unit 404. The reference camera motion can be calculated from the correspondence of the feature points using a known technique such as the 8-point method. The reference camera motion estimated by the reference camera motion estimation unit 405 is output to the RAM 112 as the reference camera motion parameters 209.

[0071] (High-Precision Map Generation Mode) When the mode determination unit 401 determines that the mode is the high-precision map generation mode, the reference camera motion estimation unit 203 executes the high-precision map point acquisition unit 406 and the high-precision camera motion estimation unit 407 .

[0072] The high-precision map point acquisition unit 406 acquires the high-precision map point cloud coordinates 212 obtained in the high-precision map generation mode from the RAM 112. The acquired high-precision map point cloud coordinates 212 are output to the high-precision camera motion estimation unit 407, and the process transitions to the high-precision camera motion estimation unit 407.

[0073] The high-precision camera motion estimation unit 407 estimates the reference camera motion with high precision using the high-precision map point cloud coordinates 212. The high-precision camera motion estimation unit 407 estimates the motion of the reference camera by comparing the high-precision map point cloud coordinates 212 at the previous time with the high-precision map point cloud coordinates 212 at the current time. The estimation can be realized by a known technique for performing point cloud matching, such as ICP (Iterative Closest Point). The reference camera motion obtained by point cloud matching is output as reference camera motion parameters 209.

[0074] <Configuration Example of Relative Camera Posture Estimation Unit> FIG. 6 is a detailed block diagram showing a configuration example of the relative camera posture estimation unit 204. As shown in FIG.

[0075] The relative camera posture estimation unit 204 estimates relative camera parameters 210 that represent the relative camera posture. The relative camera posture estimation unit 204 is configured with a mode determination unit 501, a feature point extraction unit 502, a feature amount calculation unit 503, a time-series corresponding point detection unit 504, a relative image capture unit corresponding point detection unit 505, a closed-loop detection unit 506, a relative camera posture estimation unit 507, and a calibration value acquisition unit 508.

[0076] The mode determination unit 501 acquires the mode flag 214 from the RAM 112 and determines whether the mode flag 214 indicates the calibration mode or the high-accuracy map generation mode. In the calibration mode, the mode determination unit 501 executes the feature point extraction unit 502, the feature amount calculation unit 503, the time-series corresponding point detection unit 504, the relative image capture unit corresponding point detection unit 505, the closed-loop detection unit 506, and the relative camera posture estimation unit 507. In the high-accuracy map generation mode, the mode determination unit 501 executes the calibration value acquisition unit 508.

[0077] (Calibration Mode) The feature point extraction unit 502 extracts feature points that represent the features of an image. The process of extracting feature points by the feature point extraction unit 502 can be realized by known techniques such as Harris feature point extraction or FAST. The feature point extraction unit 502 extracts feature points from all images (referred to as "all camera images") captured by all cameras (for example, cameras 121 to 123). The feature points extracted by the feature point extraction unit 502 are recorded in the RAM 112 as relative camera feature points 232. After the feature point extraction unit 502 extracts the feature points, the process transitions to the feature amount calculation unit 503.

[0078] The feature amount calculation unit 503 calculates feature amounts representing the characteristics of all feature points extracted by the feature point extraction unit 502. The feature amount calculation unit 503 calculates feature amounts for all feature points extracted from all camera images. After the feature amount calculation unit 503 calculates the feature points, the process transitions to the time-series corresponding point detection unit 504.

[0079] The time-series corresponding point detection unit 504 calculates time-series feature point correspondences for all camera images. The search for corresponding points is realized by searching for points with the closest feature amounts obtained by the feature amount calculation unit 503. At this time, although not reflected in FIG. 6 , the time-series corresponding point detection unit 504 may use odometry information or the like to limit the search area estimated from the vehicle motion. After the time-series corresponding point detection unit 504 has calculated the correspondence relationships for all feature points, the process transitions to the relative image capture unit corresponding point detection unit 505.

[0080] As described above, the reference camera motion estimation unit 203 has already extracted the feature points of the reference camera and calculated the feature amounts of the reference camera. Therefore, the relative camera posture estimation unit 204 may extract feature points of relative cameras excluding the reference camera and calculate the feature amounts of the reference camera, without targeting all camera images. The relative camera posture estimation unit 204 may read the feature points and feature amounts of the reference camera from the RAM 112 as necessary and use them in processing.

[0081] The relative image capture unit corresponding point detection unit 505 calculates the relative correspondence of feature points of all the cameras (called "relative image capture unit corresponding points") using images captured by all the cameras at the same time. The relative image capture unit corresponding points are searched for from all the camera images to find points with the closest feature amounts obtained by the feature amount calculation unit 503. After the relative image capture unit corresponding point detection unit 505 has found the correspondence of all the feature points, the process transitions to the closed-loop detection unit 506.

[0082] The relative position and orientation estimation unit (relative camera orientation estimation unit 204) selects feature points from which the relative position and orientation can be estimated, based on the state of a loop formed by time-series corresponding points of images captured by the reference imaging unit (camera 121) at the same time and time-series corresponding points of images captured by the relative imaging unit (camera 122 or 123). For example, the closed-loop detection unit 506 checks the accuracy of the detected feature points and the accuracy of the correspondence between the feature points, based on the correspondence between the time series feature points of all the camera images obtained by the time-series corresponding point detection unit 504 and the correspondence between the relative feature points between all the cameras obtained by the relative imaging unit corresponding point detection unit 505.

[0083] If the correspondence between feature points is correct, a closed loop is formed in a shape ranging from a rectangle to a parallelogram or a trapezoid. The area in which a closed loop is detected is the same subject captured at a location where the angles of view of both the reference camera and the relative camera overlap. Conversely, if a closed loop is not detected and there are feature points with no correspondence, the closed loop detection unit 506 determines that the coordinates or correspondence of the feature points are incorrect and discards the relevant feature points.

[0084] Here, how a closed loop is detected from feature points will be described with reference to Fig. 7. Fig. 7 is a diagram showing how a closed loop is detected from feature points of a plurality of images.

[0085] Camera 121 is installed on the left side of the vehicle, and camera 122 is installed on the right side of the vehicle. Both cameras 121 and 122 capture images of the area ahead of the vehicle. The upper side of Fig. 7 shows two images PL(T) and PR(T) captured by cameras 121 and 122 at time T (an example of the current time). The lower side of Fig. 7 shows two images PL(T-1) and PR(T-1) captured by cameras 121 and 122 at time T-1 (an example of the previous time). In Fig. 7, the images are shown stretched vertically to make it easier to identify corresponding points.

[0086] In each image, the feature points extracted by the feature point extraction unit 402 and the feature points extracted by the feature point extraction unit 502 are indicated by black dots. For example, the feature point F11(T) at the edge of the road in the upper left of the image PL(T) captured at time T corresponds to the feature point F12(T) at the edge of the road in the upper left of the image PR(T). The feature points F11(T) and F12(T) are connected by a dashed line L1(T), which indicates that the feature points F11(T) and F12(T) correspond to each other. Similarly, the feature point F11(T-1) at the edge of the road in the upper left of the image PL(T-1) captured at time T-1 corresponds to the feature point F12(T-1) at the edge of the road in the upper left of the image PR(T-1). The feature points F11(T-1) and F12(T-1) are connected by a dashed line L1(T-1), which indicates that the feature points F11(T-1) and F12(T-1) correspond to each other.

[0087] The closed-loop detection unit 506 detects a closed loop LP, indicated by an arrow in the figure, connecting the feature points F11(T), F12(T), F11(T-1), and F12(T-1). Similarly, it is determined whether closed-loop LPs can be detected for feature points at other positions in the image. By detecting the closed-loop LP, it is determined that a correspondence can be established between the feature points of each image. The above-described closed-loop detection process is also performed on the images captured by the cameras 121 and 123.

[0088] Returning to the explanation of Fig. 6, the closed-loop detection unit 506 performs the closed-loop detection process shown in Fig. 7 and outputs feature points for which it has determined that the correspondence is correct to the relative camera posture estimation unit 507. After the closed-loop detection unit 506 outputs feature points for which it has determined that the correspondence is correct, the process transitions to the relative camera posture estimation unit 507.

[0089] The relative camera posture estimation unit 507 estimates the relative camera posture of the relative camera with respect to the reference camera based on the feature points obtained by the closed-loop detection unit 506 and the correspondence between the feature points. The relative camera posture can be calculated using a known technique such as the eight-point method. The obtained relative camera posture is output as the relative camera parameters 210.

[0090] (High-accuracy map generation mode) When the mode determination unit 501 determines that the high-accuracy map generation mode is selected, the calibration value acquisition unit 508 is executed. The calibration value acquisition unit 508 sets the calibration values ​​215 obtained by the calibration as the relative camera parameters 210.

[0091] <Configuration Example of Map Point Cloud Estimation Unit> FIG. 8 is a detailed block diagram showing a configuration example of the map point cloud estimation unit 205. As shown in FIG.

[0092] The map point cloud estimation unit 205 is composed of a mode determination unit 601, a feature point acquisition unit 602, a reference camera motion acquisition unit 603, a relative camera attitude acquisition unit 604, a map point cloud calculation unit 605, a feature point acquisition unit 606, a parallelization unit 607, and a map point cloud calculation unit 608.

[0093] The mode determination unit 601 obtains the mode flag 214 from the RAM 112 and determines whether the mode flag 214 indicates the calibration mode or the high-accuracy map generation mode. In the calibration mode, the mode determination unit 601 executes the feature point acquisition unit 602, the reference camera motion acquisition unit 603, the relative camera attitude acquisition unit 604, and the map point cloud calculation unit 605. In the high-accuracy map generation mode, the mode determination unit 601 executes the feature point acquisition unit 606, the parallelization unit 607, and the map point cloud calculation unit 608.

[0094] (Calibration Mode) The feature point acquisition unit 602 acquires the reference camera feature points 231 and the relative camera feature points 232 from the RAM 112. After the feature point acquisition unit 602 acquires the reference camera feature points 231 and the relative camera feature points 232, the unit transitions to the reference camera movement acquisition unit 603.

[0095] The reference camera movement acquisition unit 603 acquires the reference camera movement parameters 209 from the RAM 112. After the reference camera movement acquisition unit 603 acquires the reference camera movement parameters 209, the process transitions to a relative camera attitude acquisition unit 604.

[0096] The relative camera attitude acquisition unit 604 acquires the relative camera parameters 210 from the RAM 112. The relative camera parameters 210 include information on time-series changes in the attitude of the relative camera with respect to the reference camera. After the relative camera attitude acquisition unit 604 acquires the relative camera parameters 210, the process transitions to the map point cloud calculation unit 605.

[0097] The map point cloud calculation unit 605 calculates a map point cloud, which is map point cloud coordinates 211, using the reference camera feature points 231, the relative camera feature points 232, the reference camera movement parameters 209, and the relative camera parameters 210 acquired from the RAM 112 by the feature point acquisition unit 602, the reference camera movement acquisition unit 603, and the relative camera attitude acquisition unit 604. The map point cloud calculation unit 605 calculates the map point cloud according to the principle of triangulation, using the correspondence between the feature points and the camera attitudes of the reference camera and the relative camera.

[0098] Furthermore, the map point cloud calculation unit 605 calculates the distances by triangulation for the feature points that are commonly captured by the relative cameras, using the relative camera feature points 232, the correspondence between the relative camera feature points 232, and the relative camera parameters 210. Thereafter, the map point cloud calculation unit 605 estimates the three-dimensional coordinates of the relative camera feature points 232 based on the calculated distances.

[0099] Similarly, the map point cloud calculation unit 605 calculates the distances by triangulation for feature points that are commonly captured during the time-series reference camera movements, using the reference camera feature points 231, the correspondence between the reference camera feature points 231, and the reference camera movement parameters 209. The map point cloud calculation unit 605 then estimates the three-dimensional coordinates of the reference camera feature points 231 based on the calculated distances. The three-dimensional coordinates of the reference camera feature points 231 and the relative camera feature points 232 calculated by the map point cloud calculation unit 605 are also called map point cloud coordinates. These map point cloud coordinates are output to the RAM 112 as map point cloud coordinates 211.

[0100] (High-Precision Map Generation Mode) When the mode determination unit 601 determines that the high-precision map generation mode is selected, the feature point acquisition unit 606, the rectification unit 607, and the map point cloud calculation unit 608 are executed.

[0101] The feature point acquisition unit 606 acquires the relative camera feature points 232 from the RAM 112. After the feature point acquisition unit 606 acquires the relative camera feature points 232, the process transitions to the parallelization unit 607.

[0102] The rectification unit 607 performs rectification processing of the relative cameras and outputs a rectified image. For example, the rectification unit 607 acquires the calibration value 215 from the recording unit 113 via the RAM 112, and performs rectification processing based on the relative relationship of the cameras using the calibration value 215 to generate a rectified image. For the rectification processing, hardware capable of high-speed calculation may be used. The rectification unit 607 outputs the rectified image to the map point cloud calculation unit 608, and transitions to the map point cloud calculation unit 608.

[0103] The map point cloud calculation unit 608 calculates a high-precision map point cloud using the rectified image calculated by the rectification unit 607. The map point cloud calculation unit 608 searches for feature points corresponding to the relative camera feature points 232 on the rectified image and calculates the coordinates of the corresponding feature points. At this time, the map point cloud calculation unit 608 calculates distances from the correspondences and parallax of the feature points and estimates three-dimensional coordinates corresponding to the feature points from the obtained distances. The three-dimensional coordinates estimated by the map point cloud calculation unit 608 through searching on the rectified image using the calibration values ​​215 have improved accuracy compared to three-dimensional coordinates estimated without using the rectified image.

[0104] The map point cloud calculation unit 608 outputs the estimated three-dimensional coordinates to the RAM 112 as high-precision map point cloud coordinates 212. Furthermore, when searching for corresponding points on the rectified image, the map point cloud calculation unit 608 calculates the reliability of the corresponding point search. The map point cloud calculation unit 608 labels feature points whose reliability is equal to or greater than a predetermined value as valid pixels and outputs them to the RAM 112. Information on the feature points labeled as valid pixels is used by the mode switching unit 208.

[0105] FIG. 9 is a detailed block diagram showing an example of the configuration of the statistical processing unit 207.

[0106] The statistical processing unit 207 is composed of a multistage time series value storage unit 701 and a statistical value calculation unit 702. The time series value storage unit (multistage time series value storage unit 701) stores provisional correction values ​​acquired in time series in a buffer (multistage buffer 130 shown in FIG. 10 to be described later). The provisional correction values ​​are values ​​stored in each buffer of the multistage buffer 130 based on highly reliable estimated values ​​shown in FIG. 10 to be described later. The multistage time series value storage unit 701 has a buffer function of acquiring relative camera parameters 210 from the RAM 112 and storing the relative camera parameters 210 in the multistage buffer 130 only when the reprojection error calculated by the optimization processing unit 206 is smaller than a predetermined value.

[0107] The multistage time-series value storage unit 701 sequentially stores the relative camera parameters 210 and statistical values ​​in the multistage buffer 130 for the relative camera parameters 210 obtained in a time series manner.

[0108] The statistical value calculation unit (statistical value calculation unit 702) performs statistical processing on the provisional correction values ​​stored in the buffer (multi-stage buffer 130) to calculate correction values ​​(calibration values). For example, the statistical value calculation unit 702 performs statistical processing on the relative camera parameters 210 stored in the multi-stage time-series value storage unit 701 to estimate highly accurate calibration values. After a predetermined number or more values ​​have been stored in the last-stage multi-stage buffer 130, the statistical value calculation unit 702 obtains an estimated value by calculating a statistical value such as a median or a mode for the last-stage buffer values. The statistical value calculation unit 702 outputs the estimated value to the RAM 112 as the calibration value 215 and then records it in the recording unit 113.

[0109] Fig. 10 is a diagram showing an example of the relative camera parameters 210 and the multi-stage buffer 130. The upper part of Fig. 10 shows a time-series change in the yaw angle of the relative camera as an example of the relative camera parameters 210. In this diagram, the horizontal axis of the relative camera parameters 210 represents the number of frames [frames] and the vertical axis represents the yaw angle [deg], and as the number of frames increases, the relative camera parameter 210 for the yaw angle is shown to take on a constant value of y6.

[0110] 10 shows an example of the configuration of the multistage buffer 130. The multistage buffer 130 is made up of multiple buffer stages including a first-stage buffer. After the first-stage buffer of the multistage buffer 130 is filled with the relative camera parameters 210, statistical values ​​such as the median and mode calculated from the first-stage buffer are stored in the second and subsequent stages of the multistage buffer 130.

[0111] The relative camera parameters 210 estimated by the optimization processing unit 206 tend to be unstable due to low accuracy. Therefore, the multi-stage time-series value storage unit 701 stores only highly reliable relative camera parameters 210, i.e., highly reliable yaw angle estimates indicated by circles in the figure, in the multi-stage buffer 130. In a given frame, many highly reliable yaw angle estimates are detected. For example, when the current time is time t, the yaw angle is calculated as y6 at time t-3 going back from time t, and the yaw angle is calculated as y6 at time t-7 going back even further. The calculated highly reliable yaw angle estimates are stored in the first-stage buffer. Furthermore, the yaw angle is calculated as y7 at time t-29 in the past, and as y8 at time t-30 in the past, and the calculated yaw angles are stored in the first-stage buffer. Note that, since no highly reliable estimates were detected at past times (e.g., t-1, t-2), the highly reliable estimates at those times are not stored in the first-stage buffer.

[0112] When multiple yaw angle values ​​are stored in the first-stage buffer, the median value y6 of those yaw angles is stored in the second-stage buffer. This is also shown in the upper part of FIG. 10, where the yaw angle converges to y6. Note that it is desirable that the yaw angle value stored in the second-stage buffer be a value calculated using a statistical method. The statistical value calculated by the statistical value calculation unit 702 from the yaw angles stored in the second-stage buffer becomes the calibration value.

[0113] In FIG. 10, the yaw angle has been described as an example of the relative camera parameter 210, but the roll angle and pitch angle may also be stored in the multi-stage buffer 130 in the same way, and calibration values ​​may be calculated.

[0114] In the in-vehicle information processing device 101 according to the first embodiment described above, a relative camera attitude estimation unit 204 and a statistical processing unit 207 used for multi-camera calibration are added to the functional block that performs general map generation. Therefore, the in-vehicle information processing device 101 is capable of both generating a high-precision map and performing multi-camera calibration, and can deal with the occurrence of calibration errors. Furthermore, the in-vehicle information processing device 101 shares its core processing (the map point cloud estimation unit 205 and the optimization processing unit 206) with calibration. Therefore, after calibration, the in-vehicle information processing device 101 can generate a high-precision map that makes use of the multi-camera by changing some of the processing (distance measurement, attitude estimation method). Furthermore, by sharing the core processing with calibration, the in-vehicle information processing device 101 can suppress an increase in processing load.

[0115] In this way, the in-vehicle information processing device 101 estimates point cloud coordinates of a map point cloud generated based on feature points contained in multiple images captured by multiple multi-cameras attached to the vehicle, and performs optimization processing to generate a map point cloud consisting of correction values ​​of the point cloud coordinates. Then, calibration of the multiple cameras is performed using correction values ​​determined by statistically processing the correction values ​​of the relative positions of the multiple cameras. Therefore, even if the mounting positions of the cameras are changed from their original mounting positions, the cameras are calibrated at the changed positions, so a highly accurate map point cloud can be created using images captured by each camera.

[0116] Furthermore, the self map generation and calibration device 100 uses several common functional blocks to achieve both the function of generating a self map and the function of performing calibration. This eliminates the need to provide independent functional blocks for the function of generating a self map and the function of performing calibration, thereby simplifying the configuration of the self map generation and calibration device 100.

[0117] Second Embodiment Next, a configuration example and an operation example of an in-vehicle information processing device according to a second embodiment of the present invention will be described with reference to Fig. 11 and Fig. 12. The in-vehicle information processing device according to the second embodiment has a function of performing calibration when the vehicle is started.

[0118] Fig. 11 is a diagram showing how the positions of feature points change when the vehicle ignition is off and on. The upper part of Fig. 11 shows examples of feature points detected when the vehicle ignition is off. This diagram shows a camera 121, the camera 121 capturing an image of a feature point f1 of a subject in front of the vehicle, and an example of an image P captured by the camera 121. The image P shows an example of a feature point f11 that is located at a position corresponding to the feature point f1 and detected from the image P. When the vehicle ignition is off, there is a correspondence between the feature point f1 and the feature point f11, as connected by a dashed line.

[0119] There are situations where calibration errors occur in images taken by a camera attached to a vehicle, for example, when the vehicle is left parked for a long time. A situation where calibration errors occur is expected, for example, when the camera housing itself is deformed by heat, which increases the influence of parts that change the optical characteristics of the lens. Another example is when, in a parking lot in the middle of summer, the interior of a vehicle is cold immediately after parking, but the vehicle heats up due to the outside air or sunlight after the air conditioner is turned off, causing distortion in the pillars.

[0120] An example of feature points detected when the vehicle ignition is on is shown in the lower part of Fig. 11. Because time has passed since the vehicle ignition was turned off, the optical characteristics of the lens of the camera 121 have changed, and a positional deviation g1 occurs between the position of feature point f12 detected from the image P when the ignition was on and the position of feature point f11 detected from the image P when the ignition was off. Therefore, the in-vehicle information processing device 101A according to the second embodiment shown in Fig. 12 corrects the position of feature point f12 by the positional deviation g1 when the ignition is on, thereby suppressing the influence of the positional deviation g1 caused by the change in the optical characteristics of the lens of the camera 121.

[0121] 12 is a functional block diagram showing an example of the configuration of an in-vehicle information processing device 101A according to the second embodiment. The in-vehicle information processing device 101A has a configuration in which a vehicle-stopped map point cloud and corresponding point acquisition unit 801 and a startup calibration unit 802 are added to the in-vehicle information processing device 101 of the self map generation and calibration device 100 according to the first embodiment. The in-vehicle information processing device 101A according to the second embodiment aims to perform calibration early using the generated map point cloud 216.

[0122] The sensor information acquisition unit 201 acquires sensor information from the cameras 121 to 123 and acquires camera parameter design values ​​220 from the ROM 111. The odometry information acquisition unit 202 acquires vehicle specifications 221 from the ROM 111. Note that since the processing according to the second embodiment is performed while the vehicle is stopped, there is no output from the vehicle speed sensor 161 and the steering angle sensor 162 to the odometry information acquisition unit 202.

[0123] The reference camera motion estimation unit 203 inputs and outputs reference camera motion parameters 209 and high-precision map point cloud coordinates 212 to the RAM 112. The high-precision map point cloud coordinates 212 are information generated in high-precision map generation mode before the ignition is turned off. The reference camera motion estimation unit 203 also outputs the reference camera motion parameters 209, which are the result of estimating that no reference camera motion has been performed because the vehicle is stopped, to the RAM 112.

[0124] The relative camera posture estimation unit 204 outputs relative camera parameters 210, which are estimated values ​​of the relative camera posture, to the RAM 112. When the ignition is turned on, the relative camera posture estimation unit 204 also outputs the relative camera parameters 210 to the startup calibration unit 802.

[0125] The map point cloud estimation unit 205 inputs and outputs map point cloud coordinates 211 and high-precision map point cloud coordinates 212 to the RAM 112. The map point cloud estimation unit 205 calculates a reprojection error using the map point cloud coordinates 211. The map point cloud estimation unit 205 also estimates reference camera motion parameters 209, relative camera parameters 210, and high-precision map point cloud coordinates 212.

[0126] The optimization processing unit 206 generates a stopped-state map point cloud 812 when the vehicle is stopped, using the map point cloud information estimated by the map point cloud estimation unit 205 (map point cloud coordinates 211 and high-precision map point cloud coordinates 212). The optimization processing unit 206 also generates a stopped-state feature point 811, which is a feature point of an image captured by the reference camera and a feature point of an image captured by the relative camera when the vehicle is stopped. The stopped-state feature point 811 corresponds to the feature point f1 in FIG. 11 . The optimization processing unit 206 also generates a stopped-state map point cloud 812 based on the stopped-state feature point 811 when the vehicle is stopped. The optimization processing unit 206 also has a function of estimating high-precision reference camera motion parameters 209, relative camera parameters 210, and high-precision map point cloud coordinates 212 using the high-precision map point cloud coordinates 212 instead of the map point cloud coordinates 211.

[0127] For startup calibration, the stationary-time map point cloud and corresponding point acquisition unit 801 records the map point cloud 216 read out from the RAM 112 by the optimization processing unit 206 in the recording unit 113. Furthermore, in addition to the map point cloud 216 stored in the recording unit 113 during normal operation, the stationary-time map point cloud and corresponding point acquisition unit 801 acquires stationary-time feature points 811 and stationary-time map point cloud 812 from the optimization processing unit 206 and records them in the recording unit 113.

[0128] The vehicle-stopped map point cloud and corresponding point acquisition unit 801 also acquires, from the statistical processing unit 207, a calibration value 215 calculated when the vehicle is stopped, and stores the calibration value 215 in the recording unit 113. In addition to the various pieces of information according to the first embodiment, the recording unit 113 records the calibration value 215, the map point cloud 216, the vehicle-stopped feature points 811, and the vehicle-stopped map point cloud 812 generated when the vehicle is stopped. The vehicle-stopped map point cloud and corresponding point acquisition unit 801 needs to be activated in a scene where a long period of time is expected to pass, such as when the vehicle is parked. For example, the vehicle-stopped map point cloud and corresponding point acquisition unit 801 operates when it is triggered by the ignition being turned off after the vehicle is stopped. However, the vehicle-stopped map point cloud and corresponding point acquisition unit 801 may be activated by other factors.

[0129] The startup calibration unit 802 calibrates the multiple imaging units (cameras 121 to 123) when the moving body (vehicle) starts up, using the map point cloud and feature points generated when the moving body (vehicle) stops. For example, the startup calibration unit 802 operates in response to being triggered by the ignition being turned on when the vehicle starts operating. The startup calibration unit 802 performs calibration early in a situation where a calibration error occurs due to the passage of time, such as when the vehicle is parked or stopped.

[0130] Upon starting operation, the startup calibration unit 802 reads out the vehicle-stopped map point cloud 812, vehicle-stopped feature points 811, and calibration values ​​215 stored in the recording unit 113. The startup calibration unit 802 compares the coordinates of the feature points extracted from the images captured by each camera with the vehicle-stopped feature points 811. The startup calibration unit 802 detects whether or not a positional deviation g1 shown in FIG. 11 has occurred, and if a positional deviation g1 has occurred, corrects the camera attitude so that the vehicle-stopped map point cloud 812 matches the coordinates of the feature points extracted by each camera. By correcting the camera attitude, calibration at vehicle startup is performed.

[0131] In the in-vehicle information processing device 101A according to the second embodiment described above, the stopped-state map point cloud 812 generated when the ignition is off and the corresponding stopped-state feature points 811 of the multiple images captured by each camera are recorded in the recording unit 113. The next time the ignition is turned on, the in-vehicle information processing device 101A reads the stopped-state map point cloud 812 from the recording unit 113, and performs startup calibration if the elapsed time exceeds a predetermined value. In this way, the in-vehicle information processing device 101A can complete calibration in a short time by utilizing the corresponding stopped-state feature points 811 of the multiple images captured by the multi-cameras when the ignition is off as initial values ​​of the relative camera attitudes.

[0132] Although the calibration at the start of operation has been described as being performed by the startup calibration unit 802, similar to the processing according to the first embodiment, the calibration may be performed by the optimization processing unit 206 and the statistical processing unit 207. In this case, the startup calibration unit 802 outputs an instruction to the optimization processing unit 206 and the statistical processing unit 207 to perform the calibration.

[0133] The present invention is not limited to the above-described embodiments, and various other applications and modifications are possible without departing from the spirit of the present invention as defined in the claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Other embodiments conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0134] Furthermore, although the self-mapping and calibration device 100 is described as an example of a device mounted on a vehicle, it may also be mounted on a moving object such as an autonomously controlled robot, an airborne drone, or heavy construction machinery.

[0135] The self-mapping and calibration device 100 may also include an input / output interface (not shown), and a program may be loaded from another device via the input / output interface and a medium available to the self-mapping and calibration device 100 as needed. Here, the medium refers to, for example, a storage medium detachable from the input / output interface, or a communication medium, such as a wired, wireless, or optical network, or a carrier wave or digital signal propagating through the network. Some or all of the functions implemented by the program may be implemented by a hardware circuit or FPGA. Each of the above-described components and functions may be implemented by software, with a processor interpreting and executing a program that implements each function. Information such as the program, table, and file that implements each function may be recorded in a memory, a recording device such as a hard disk or SSD, or a recording medium such as an IC card, SD card, or DVD. Furthermore, the control lines and information lines shown are those considered necessary for explanation, and not all control lines and information lines are necessarily shown in the actual product. In reality, it can be assumed that almost all components are interconnected.

[0136] 100... Self-map generation and calibration device, 101... In-vehicle information processing device, 110... CPU, 111... ROM, 112... RAM, 113... Recording unit, 121 to 123... Camera, 161... Vehicle speed sensor, 162... Steering angle sensor, 171... Interface, 201... Sensor information acquisition unit, 202... Odometry information acquisition unit, 203... Reference camera motion estimation unit, 204... Relative camera attitude estimation unit, 205... Map point cloud estimation unit, 206... Optimization processing unit, 207... Statistical processing unit, 208... Mode switching unit

Claims

1. A reference position and attitude estimation unit that estimates a reference position and attitude of a reference imaging unit using a plurality of reference images captured by the reference imaging unit in time series of an environment around the moving body, using one of a plurality of imaging units mounted on a moving body as a reference imaging unit; a relative position and attitude estimation unit that estimates a relative position and attitude of the relative imaging unit with respect to the reference position and attitude using a relative image captured by a relative imaging unit which is at least one imaging unit other than the reference imaging unit among the plurality of imaging units; a map point cloud estimation unit that estimates a map point cloud consisting of point cloud coordinates which are three-dimensional position information of the feature points in the environment, based on reference feature points extracted from the reference image by the reference position and attitude estimation unit, motion information of the reference imaging unit, relative feature points extracted from the relative image by the relative position and attitude estimation unit, and motion information of the relative imaging unit; and an optimization processing unit that optimizes the reference position and attitude, the relative position and attitude, and the map point cloud so as to reduce a predetermined index. a statistical processing unit that determines correction values ​​for calibration to be performed on the plurality of imaging units by statistically processing provisional correction values ​​that represent the time-series relative positions and orientations optimized by the optimization processing unit.

2. The information processing device according to claim 1, wherein the relative position and orientation estimation unit selects feature points from which the relative position and orientation can be estimated based on the formation state of a loop consisting of time-series corresponding points of the image captured by the reference imaging unit and time-series corresponding points of the image captured by the relative imaging unit at the same time.

3. The information processing device according to claim 2, wherein the statistical processing unit includes a time series value storage unit that stores the provisional correction values ​​acquired in time series in a buffer, and a statistical value calculation unit that performs statistical processing on the provisional correction values ​​stored in the buffer to calculate the correction values.

4. An information processing device as described in claim 3, further comprising a mode switching unit that switches the processing modes of the reference position and attitude estimation unit, the relative position and attitude estimation unit, and the map point cloud estimation unit between a calibration mode in which the point cloud coordinates are estimated using the relative position and attitude and the relative feature points estimated in time series, and a map generation mode in which the map point cloud consisting of the point cloud coordinates is estimated using the parallax obtained in a common imaging area between the reference imaging unit and the relative imaging unit.

5. The information processing device according to claim 4, wherein the mode switching unit switches the calibration mode to the map generation mode when calibration using the correction value is completed, and switches the map generation mode to the calibration mode when calibration accuracy decreases.

6. The information processing device according to claim 5, wherein the optimization processing section and the statistical processing section perform calibration of the plurality of imaging sections based on the correction value.

7. The information processing device according to claim 6, wherein the map point cloud and the feature points generated when the moving object stops are used in calibration of the multiple imaging units, which is performed when the moving object starts up.

8. An information processing method comprising: a process of estimating a reference position and orientation of a reference imaging unit using a plurality of reference images captured in time series by the reference imaging unit of an environment around the moving body, the reference imaging unit being one of a plurality of imaging units mounted on the moving body; a process of estimating a relative position and orientation of the relative imaging unit with respect to the reference position and orientation using relative images captured by a relative imaging unit which is at least one imaging unit other than the reference imaging unit among the plurality of imaging units; a process of estimating a map point cloud consisting of point cloud coordinates which are three-dimensional position information of the feature points in the environment, based on reference feature points extracted from the reference images, motion information of the reference imaging unit, relative feature points extracted from the relative images, and motion information of the relative imaging unit; a process of optimizing the reference position and orientation, the relative position and orientation, and the map point cloud so as to reduce a predetermined index; and a process of determining correction values ​​for calibration to be performed on the plurality of imaging units by statistically processing provisional correction values ​​which represent the optimized time-series relative position and orientation.

Images