Information processing device and information processing method

Abstract

To provide an information processing device capable of correctly performing calibration even in a multi-camera.SOLUTION: An on-vehicle information processing device 101 includes: a map point group estimation part 205 for estimating a map point group comprising a point group coordinate which is three-dimensional position information of a feature point in an environment on the basis of a reference feature point estimated by a reference camera motion estimation part 203 and motion information of a reference camera and a relative feature point estimated by a relative camera attitude estimation part 204 and motion information of a relative camera; an optimization processing part 206 for optimizing a reference position attitude, a relative position attitude, and the map point group so as to make a predetermined index smaller; and a statistical processing part 207 for determining a correction value of calibration executed with respect to a plurality imaging parts by statistically processing a temporary correction value representing the optimized time-series relative position attitude.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] Automated driving and driver assistance systems are used for the automated driving of vehicles and output information to the vehicle's control device to support the automated driving. Automated driving and driver assistance systems estimate the location of the vehicle on which the system is installed as its own location. To expand the scope of application of automated driving and driver assistance systems, it is important to acquire information from maps based on self-location estimation. For example, while maps for driver assistance systems have been developed for expressways, there are no plans to develop maps for general roads or residential areas such as the driver's home. Therefore, technology is needed that enables automated driving and driver assistance systems to generate maps themselves and estimate their own location.

[0003] In response to this, Patent Document 1 describes a "3D map estimation device including: a first selection unit that selects one or more imaging devices from among a plurality of imaging devices mounted on a moving body that capture a plurality of images; a first position and orientation estimation unit that performs position and orientation estimation for each imaging device based on the captured images of the imaging devices selected by the first selection unit and outputs the first position and orientation estimation result for each imaging device; a second position and orientation estimation unit that outputs a second position and orientation estimation result from the one or more first position and orientation estimation results; and a 3D map estimation unit that estimates a 3D map of the surroundings of the moving body from the second position and orientation estimation result and the image." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-166505 Summary of the Invention [Problem to be solved by the invention]

[0005] The invention described in Patent Document 1 uses multiple cameras, but the technology estimates a 3D map from the time-series correspondence of each camera, leaving room for improvement in the accuracy of the estimated 3D map. Furthermore, the technology for generating a highly accurate map using multiple cameras presupposes that the cameras have been calibrated, and therefore cannot accommodate changes in the state of the cameras, such as when the cameras are uncalibrated or when the cameras are subject to thermal deformation or aging. Furthermore, because multiple camera units are attached to the vehicle body at different distances, multi-cameras tend to be more prone to calibration errors than stereo cameras.

[0006] The present invention has been made in view of the above circumstances, and has as its object to enable correct calibration of a plurality of cameras constituting a multi-camera system. [Means for solving the problem]

[0007] The information processing device according to the present invention includes a reference position and attitude estimation unit that estimates a reference position and attitude of one of a plurality of imaging units mounted on a moving body as a reference imaging unit, using a plurality of reference images that the reference imaging unit captures in time series of the environment around the moving body; a relative position and attitude estimation unit that estimates a relative position and attitude of the relative imaging unit with respect to the reference position and attitude of the relative imaging unit, using relative images captured by a relative imaging unit that is at least one of the plurality of imaging units other than the reference imaging unit; a map point cloud estimation unit that estimates a map point cloud made up of point cloud coordinates that are three-dimensional position information of feature points in the environment, based on reference feature points extracted from the reference image by the reference position and attitude estimation unit and motion information of the reference imaging unit, and relative feature points extracted from the relative image by the relative position and attitude estimation unit and motion information of the relative imaging unit; an optimization processing unit that optimizes the reference position and attitude, the relative position and attitude, and the map point cloud so that a predetermined index is small; and a statistical processing unit that determines correction values ​​for calibration to be performed on the plurality of imaging units by statistically processing provisional correction values ​​that represent the time-series relative position and attitude optimized by the optimization processing unit. [Effects of the Invention]

[0008] According to the present invention, it becomes possible to correctly calibrate the multiple cameras constituting the multi-camera system using the determined calibration correction values. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a functional block diagram showing an example of the configuration of a self-mapping and calibration device according to a first embodiment of the present invention. [Figure 2] 2 is a functional block diagram showing an example of the processing order of functional blocks of each functional unit executed by a CPU according to the first embodiment of the present invention and the flow of each piece of data. FIG. [Figure 3] 4 is a flowchart showing an example of processing of each functional block according to the first embodiment of the present invention. [Figure 4] 3 is a detailed block diagram showing an example of the configuration of a mode switching unit according to the first embodiment of the present invention. FIG. [Figure 5] 2 is a detailed block diagram showing an example of the configuration of a reference camera motion estimating unit according to the first embodiment of the present invention. FIG. [Figure 6] FIG. 2 is a detailed block diagram showing an example of the configuration of a relative camera posture estimation unit according to the first embodiment of the present invention. [Figure 7] FIG. 2 is a diagram showing how a closed loop is detected from feature points of a plurality of images according to the first embodiment of the present invention. [Figure 8] FIG. 2 is a detailed block diagram showing an example of the configuration of a map point cloud estimation unit according to the first embodiment of the present invention. [Figure 9] FIG. 2 is a detailed block diagram showing an example of the configuration of a statistical processing unit according to the first embodiment of the present invention. [Figure 10] 3A to 3C are diagrams illustrating examples of relative camera parameters and multi-stage buffers according to the first embodiment of the present invention. [Figure 11] FIG. 10 is a diagram showing how the positions of feature points change when the ignition of a vehicle according to a second embodiment of the present invention is turned off and on. [Figure 12] FIG. 10 is a functional block diagram showing an example of the configuration of an in-vehicle information processing device according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functions or configurations are designated by the same reference numerals, and redundant description will be omitted. The present invention is applicable to, for example, a computing device for vehicle control capable of communicating with an on-board ECU (Electronic Control Unit) for an Advanced Driver Assistance System (ADAS) or Autonomous Driving (AD).

[0011] [First embodiment] First, a configuration example and an operation example of a self-mapping and calibration device according to a first embodiment of the present invention will be described. The self-mapping and calibration device has a function of generating a self-mapping and a function of calibrating a plurality of cameras constituting a multi-camera system.

[0012] FIG. 1 is a functional block diagram showing an example of the configuration of a self-mapping and calibration device 100. As shown in FIG. The cameras 121 to 123 (an example of an imaging unit) provided in the self-mapping and calibration device 100, the vehicle speed sensor 161, the steering angle sensor 162, the interface 171, and the in-vehicle information processing device 101 are connected by signal lines, and various data is exchanged with the in-vehicle information processing device 101. The self-mapping and calibration device 100 is mounted on a vehicle (an example of a moving body).

[0013] The in-vehicle information processing device 101 includes a CPU (Central Processing Unit) 110, a ROM (Read Only Memory) 111, a RAM (Random Access Memory) 112, and a recording unit 113. It may be configured so that all or part of the arithmetic processing is executed by another arithmetic processing device such as an FPGA (Field Programmable Gate Array).

[0014] The CPU 110 operates as an execution unit of the in-vehicle information processing device 101 by reading and executing various programs and parameters from the ROM 111. The CPU 110 reads program code of software that realizes each function according to this embodiment from the ROM 111, loads it into the RAM 112, and executes it. Variables, parameters, etc. generated during the calculation processing of the CPU 110 are temporarily written to the RAM 112, and these variables, parameters, etc. are read by the CPU 110 as appropriate. However, an MPU (Micro Processing Unit) or a GPU (Graphics Processing Unit) may be used instead of the CPU 110, or the CPU 110 and a GPU may be used together.

[0015] The ROM 111 is a read-only storage area and stores programs, which will be described later. The programs and parameters stored in the ROM 111 are read by the CPU 110. The programs, which will be described later, are loaded into the RAM 112 and executed by the CPU 110.

[0016] The RAM 112 is a readable / writable storage area and operates as the main storage device of the in-vehicle information processing device 101. The RAM 112 stores reference camera motion parameters 209, relative camera parameters 210, map point cloud coordinates 211, high-precision map point cloud coordinates 212, time-series relative camera parameters 213, and a mode flag 214, which will be described later.

[0017] By reading and executing programs from ROM 111, CPU 110 operates as a sensor information acquisition unit 201, an odometry information acquisition unit 202, a reference camera motion estimation unit 203, a relative camera posture estimation unit 204, a map point cloud estimation unit 205, an optimization processing unit 206, a statistical processing unit 207, and a mode switching unit 208.

[0018] The ROM 111 stores camera parameter design values ​​220 and vehicle specifications 221. The camera parameter design values ​​220 store the designed positions and orientations of the cameras 121 to 123 and information specific to the cameras. That is, the lens distortion coefficient, the optical axis center, the focal length, the number of pixels and dimensions of the image pickup element are stored. The vehicle specifications 221 store information specific to the vehicle.

[0019] The recording unit 113 is a non-volatile storage device such as a hard disk drive (HDD), a solid state drive (SDD), an optical disk, or a non-volatile memory, and operates as an auxiliary storage device for the in-vehicle information processing device 101. The recording unit 113 stores calibration values ​​215 and a map point cloud 216. The recording unit 113 also stores an operating system (OS), various parameters, and programs for operating the in-vehicle information processing device 101.

[0020] The ROM 111 and the recording unit 113 record programs and data (not shown) necessary for the CPU 110 to operate, and are used as an example of a non-transitory storage medium that can be read by a computer and that stores programs executed by the in-vehicle information processing device 101.

[0021] The cameras 121 to 123 are attached, for example, in front of the windshield inside the vehicle, and on the front right and left pillars, and capture images that cover a wide area ahead of the vehicle. The position and orientation relationship between the cameras 121 to 123 and the vehicle is stored in the ROM 111 as camera parameter design values ​​220. The method of attaching the cameras 121 to 123 is not limited to the above-mentioned method, and other attachment methods may also be used. Furthermore, the number of cameras attached at each position does not necessarily have to be one.

[0022] The cameras 121 to 123 are equipped with lenses and image sensors (not shown). The camera parameter design values ​​220 are stored in the ROM 111 in a form including internal parameters such as the characteristics of the camera, for example, a lens distortion coefficient which is a parameter indicating lens distortion, the center of the optical axis, the focal length, the number of pixels and dimensions of the image sensor, and external parameters which indicate the position and attitude relationship with the vehicle.

[0023] The vehicle speed sensor 161 measures the speed of the vehicle in which the on-vehicle information processing device 101 is installed, and outputs the measured vehicle speed to the on-vehicle information processing device 101 . The steering angle sensor 162 measures the steering angle of the vehicle in which the on-vehicle information processing device 101 is installed, and outputs the measured steering angle to the on-vehicle information processing device 101 . The in-vehicle information processing device 101 calculates the amount of movement and the direction of movement of the vehicle in which the in-vehicle information processing device 101 is installed by a known dead reckoning technique using the vehicle speed and steering angle.

[0024] The interface 171 is, for example, a GUI (Graphical User Interface) that accepts instruction inputs from a user. The interface 171 may accept information other than instruction inputs from a user in a form other than a GUI.

[0025] <Example of processing order of functional blocks in each functional unit and flow of each data> 2 is a functional block diagram showing an example of the order of processing of the functional blocks of each functional unit and the flow of each piece of data executed by the CPU 110. In FIG. 2, the illustration of the CPU 110 is omitted. 3 is a flowchart showing an example of the processing of each functional block. The processing of each functional block shown in FIG. 2 will be explained in association with each step of the flowchart.

[0026] 2 shows the functions of a sensor information acquisition unit 201, an odometry information acquisition unit 202, a reference camera motion estimation unit 203, a relative camera attitude estimation unit 204, a map point cloud estimation unit 205, an optimization processing unit 206, a statistical processing unit 207, and a mode switching unit 208 executed by the CPU 110 as functional blocks. These functional blocks are configured to be included in an in-vehicle information processing device 101 (an example of an information processing device). FIG. 2 also shows the processing order of the functional blocks, the flow of data between the functional blocks, and between the functional blocks and the RAM 112, the ROM 111, and the recording unit 113. The self-mapping and calibration device 100 operates according to the configuration and processing order of these functional blocks.

[0027] First, the sensor information acquisition unit 201 acquires signals (referred to as "sensor information") output from the cameras 121 to 123 (S1 in FIG. 3). The camera 121 captures images continuously at a high frequency, for example, 30 times per second. The cameras 122 and 123 also capture images at the same frequency as the camera 121.

[0028] The sensor information acquisition unit 201 acquires sensor information including images and signals output from the cameras 121 to 123 at a constant frequency, and outputs the sensor information to the reference camera motion estimation unit 203. At this time, the sensor information acquisition unit 201 uses, for example, the camera parameter design values ​​220 read out from the ROM 111 to output an image corrected in consideration of distortion of each camera, to the reference camera motion estimation unit 203. However, the timing of image correction is not limited to this, and image correction may be performed in the reference camera motion estimation unit 203 or the like at a subsequent stage. Furthermore, once the sensor information acquisition unit 201 acquires the camera parameter design values ​​220, it is not necessary to re-acquire the camera parameter design values ​​220 in subsequent processing.

[0029] The odometry information acquisition unit 202 acquires signals (referred to as "odometry information") related to odometry estimation, such as speed and steering angle, output from the vehicle speed sensor 161 and the steering angle sensor 162 (S2). The processes of steps S1 and S2 are performed in parallel. The odometry information acquisition unit 202 also continuously acquires odometry information at a high frequency and outputs it to the reference camera motion estimation unit 203. The odometry information acquisition unit 202 also reads vehicle specifications 221 from the ROM 111. The vehicle specifications 221 are information specific to the vehicle in which the self map generation and calibration device 100 is installed, and the vehicle's speed, steering angle, etc. take values ​​within ranges defined by the vehicle specifications 221. Therefore, the odometry information acquisition unit 202 can determine, based on the vehicle specifications 221, whether the odometry information is within a range of values ​​that can be taken by the odometry information.

[0030] After steps S1 and S2, the reference position and orientation estimation unit (reference camera motion estimation unit 203) determines one of the multiple imaging units (cameras 121-123) mounted on the moving body (vehicle) as the reference imaging unit (camera 121). In the following description, the reference imaging unit (camera 121) is referred to as the "reference camera." Furthermore, the reference camera motion estimation unit 203 estimates the time-series motion of the reference camera (referred to as the "reference camera motion") (S3). The reference camera motion includes time-series position and orientation changes such as translation and rotation of the reference camera.

[0031] The reference position and orientation estimation unit (reference camera motion estimation unit 203) estimates the reference position and orientation of the reference imaging unit (camera 121) using multiple reference images captured by the reference imaging unit (camera 121) in time series of the environment around the moving body (vehicle). The reference position and orientation of the reference camera includes information on the position and orientation of the reference camera. The reference camera motion estimation unit 203 roughly estimates the time-series motion of the reference camera using a reference camera image captured by the reference camera at a previous time and a reference camera image captured by the reference camera at a current time. For example, if the camera 121 captures images at 30 frames per second, the interval between the previous time and the current time is assumed to be one frame, but the interval may be multiple frames. The reference camera motion estimation unit 203 estimates the time-series motion of the reference camera using a known technique such as the eight-point method, using the correspondence between feature points extracted from the reference camera images at the previous time and the current time. The feature points correspond to the corners of an object shown in the image, or the corners of road markings painted on the road surface.

[0032] The reference camera motion estimation unit 203 also has a function of performing high-precision reference camera motion estimation using high-precision map point cloud coordinates 212 (an example of a map point cloud) extracted from reference camera images at the previous time and the current time and odometry information output from the odometry information acquisition unit 202, and operates variably according to the situation. The high-precision map point cloud coordinates 212 extracted by the reference camera motion estimation unit 203 are output to the RAM 112. The reference camera feature points 231, which are estimated values ​​of the reference camera motion estimated by the reference camera motion estimation unit 203, are output to the RAM 112 as reference camera motion parameters 209. After the reference camera motion estimation unit 203 outputs the reference camera motion parameters 209, the process transitions to the relative camera posture estimation unit 204. An example of detailed processing by the reference camera motion estimation unit 203 will be described later with reference to FIG. 5.

[0033] Next, the relative position and orientation estimation unit (relative camera orientation estimation unit 204) determines a relative image capturing unit (camera 122 or 123) that is at least one image capturing unit other than the reference image capturing unit (camera 121) among the multiple image capturing units (cameras 121 to 123). In the following description, the relative image capturing unit (camera 122 or 123) is referred to as a "relative camera." The relative position and orientation estimation unit (relative camera orientation estimation unit 204) estimates the relative position and orientation of the relative image capturing unit (camera 122 or 123) with respect to the reference position and orientation using a relative image captured by the relative image capturing unit (camera 122 or 123) (S4). The relative position and orientation is the position and orientation of the relative camera with respect to the position and orientation of the reference camera, and is referred to as a "relative camera orientation."

[0034] Relative camera attitude estimation unit 204 roughly estimates the relative camera attitude between camera 121 and camera 122 by a known technique such as the 8-point method using the correspondence between feature points in the image captured by camera 121 and feature points in the image captured by camera 122. Similarly, relative camera attitude estimation unit 204 roughly estimates the relative camera attitude between camera 121 and camera 123 by a known technique such as the 8-point method using the correspondence between feature points in the image captured by camera 121 and feature points in the image captured by camera 123.

[0035] When the relative camera attitude estimation unit 204 estimates the relative camera attitude of each camera, if a reliable result has been obtained by the previous time based on the relationship between the images captured by each camera, that is, a camera parameter estimate 230 (described later), then the relative camera attitude estimation unit 204 may substitute the camera parameter estimate 230. The estimated values ​​of the relative camera attitudes estimated by the relative camera attitude estimation unit 204 are output to the RAM 112 as relative camera parameters 210.

[0036] Furthermore, the coordinates and feature amounts of the feature points of the relative camera image detected by the relative camera attitude estimation unit 204 are output to the RAM 112 as relative camera feature points 232. The feature amounts are values ​​that each feature point has and that represent the characteristics of the feature point. After the relative camera attitude estimation unit 204 outputs the coordinates and feature amounts of the feature points, the process transitions to the map point cloud estimation unit 205. Details of the relative camera attitude estimation unit 204 will be described later with reference to FIG. 6.

[0037] Next, the map point cloud estimation unit (map point cloud estimation unit 205) estimates a map point cloud consisting of point cloud coordinates, which are three-dimensional position information of feature points in the environment, based on the reference feature points extracted from the reference image by the reference position and orientation estimation unit (reference camera motion estimation unit 203) and motion information of the reference imaging unit (camera 121), and the relative feature points extracted from the relative image by the relative position and orientation estimation unit (relative camera posture estimation unit 204) and motion information of the relative imaging unit (camera 122 or 123). For example, the map point cloud estimation unit 205 reads reference camera motion parameters 209, relative camera parameters 210, reference camera feature points 231, and relative camera feature points 232 from the RAM 112. Next, the map point cloud coordinates (also referred to as "map point cloud"), which are three-dimensional coordinates corresponding to each of the reference camera feature points 231 and the relative camera feature points 232, are roughly estimated by triangulation (S5). The map point cloud estimation unit 205 also has a function of substituting the high-precision map point cloud coordinates 212 for the three-dimensional coordinates when the high-precision map point cloud coordinates 212 have been obtained by the reference camera movement estimation unit 203 .

[0038] The map point cloud estimation unit 205 switches between a function of estimating a map point cloud, which is three-dimensional coordinates, by triangulation and a function of replacing the three-dimensional coordinates with high-precision map point cloud coordinates 212, depending on the situation. The three-dimensional coordinates corresponding to each feature point estimated by the map point cloud estimation unit 205 are output to the RAM 112 as map point cloud coordinates 211 (an example of a map point cloud). After outputting each piece of information, processing transitions to the optimization processing unit 206. Details of the map point cloud estimation unit 205 will be described later with reference to FIG. 8.

[0039] Next, the optimization processor (optimization processor 206) optimizes the reference position and orientation, the relative position and orientation, and the map point cloud so as to reduce a predetermined index (reprojection error) (S6). For example, the optimization processor 206 reads out reference camera motion parameters 209, relative camera parameters 210, map point cloud coordinates 211, reference camera feature points 231, and relative camera feature points 232 from the RAM 112. Using this information read out from the RAM 112, the optimization processor 206 optimizes all of the reference camera motion parameters 209, the relative camera parameters 210, and the map point cloud coordinates 211 by a known optimization method so as to minimize the reprojection error. The optimization process is applied to the positions and orientations of the two cameras at each time and all of the 3D map point cloud coordinates.

[0040] To calculate the reprojection error, the optimization processing unit 206 first projects map point cloud coordinates 211 calculated from the image captured by the reference camera onto the reference camera based on the reference camera motion parameters 209, and calculates the error between the map point cloud coordinates 211 projected onto the reference camera and the corresponding reference camera feature points 231. Next, the optimization processing unit 206 projects the map point cloud coordinates 211 calculated from the image captured by the relative camera onto the relative camera based on the relative camera parameters 210, and calculates the error between the projected map point cloud coordinates 211 and the corresponding relative camera feature points 232. Thereafter, the optimization processing unit 206 calculates the sum of the errors of the reference camera feature points 231 and the relative camera feature points 232 as the reprojection error. The reprojection error has the property of taking a smaller value as the numerical values ​​of the map point cloud coordinates 211, the reference camera motion parameters 209, and the relative camera parameters 210 are more accurate.

[0041] The optimization processing unit 206 simultaneously slightly changes all of the reference camera motion parameters 209, relative camera parameters 210, and map point cloud coordinates 211 so as to minimize the reprojection error, and optimizes each value using a known technique such as the Levenberg-Marquardt algorithm. This processing improves the accuracy of all of the reference camera motion parameters 209, relative camera parameters 210, and map point cloud coordinates 211.

[0042] The optimization processing unit 206 also has a function of estimating high-precision reference camera motion parameters 209, relative camera parameters 210, and high-precision map point cloud coordinates 212 by using high-precision map point cloud coordinates 212 instead of the map point cloud coordinates 211. As described above, the function of estimating map point clouds, which are three-dimensional coordinates, by triangulation and the function of substituting high-precision map point cloud coordinates 212 for three-dimensional coordinates are switched between operating depending on the situation. That is, the in-vehicle information processing device 101 does not operate the high-precision map generation function when the calibration function is operating, and does not operate the calibration function when the high-precision map generation function is operating. The calibration function is a function of correcting distortion, position, etc. of images captured by each camera, and is mainly performed by the optimization processing unit 206 and the statistical processing unit 207.

[0043] When the optimization processing unit 206 estimates each parameter and the high precision map point cloud coordinates 212 using the high precision map point cloud coordinates 212, overlapping points in the high precision map point cloud coordinates 212 are overwritten and corrected or averaged, and new points are added to the high precision map point cloud coordinates 212. The reference camera motion parameters 209, relative camera parameters 210, and map point cloud coordinates 211 estimated by the optimization processing unit 206 are output to the RAM 112. After each piece of information is output, processing transitions to the statistical processing unit 207.

[0044] Next, the statistical processing unit (statistical processing unit 207) statistically processes the provisional correction values ​​that represent the time-series relative positions and orientations optimized by the optimization processing unit (optimization processing unit 206) (S7). This statistical processing determines correction values ​​(calibration values) for the calibration to be performed on the multiple imaging units (cameras 121 to 123). For example, the statistical processing unit 207 performs statistical processing on the relative camera parameters 210 estimated by the optimization processing unit 206. The statistical processing unit 207 has a function of improving the accuracy of the relative camera parameters 210 by the statistical processing and estimating calibration values ​​215. The relative camera parameters 210 that were statistically processed by the relative camera orientation estimation unit 204 in step S4 are more accurate than the values ​​obtained by the relative camera orientation estimation unit 204 by the optimization processing unit 206. However, the statistically processed relative camera parameters 210 are insufficiently accurate and unstable as calibration values ​​due to feature point errors or disturbances such as moving objects.

[0045] Therefore, in order to deal with the lack of precision of the relative camera parameters 210, the statistical processing unit 207 stores highly reliable calibration values, which are estimation results at each time, in the multistage buffer 130 (see FIG. 10 described later) as time-series relative camera parameters 213. The multistage buffer 130 is provided in the statistical processing unit 207 and is composed of multiple buffer stages.

[0046] The statistical processing unit 207 obtains a calibration value 215 by applying statistical processing such as extracting a median to the group of estimation results of the calibration values ​​stored in the multi-stage buffer 130. The calibration value 215 obtained by the statistical processing unit 207 is output to the RAM 112. After the statistical processing unit 207 outputs the calibration value 215, the processing transitions to the mode switching unit 208. Details of the statistical processing unit 207 will be described later with reference to FIG. 9.

[0047] Next, the mode switching unit (mode switching unit 208) switches the processing mode of the reference position and attitude estimation unit (reference camera motion estimation unit 203), the relative position and attitude estimation unit (relative camera attitude estimation unit 204), and the map point cloud estimation unit to either the calibration mode or the high-accuracy map generation mode according to the state of the vehicle (S8), and ends this processing. The calibration mode is a processing mode in which point cloud coordinates are estimated using the relative position and attitude and relative feature points estimated in time series. The high-accuracy map generation mode is a processing mode in which a map point cloud consisting of point cloud coordinates is estimated using the parallax obtained in the common imaging area between the reference imaging unit (camera 121) and the relative imaging unit (camera 122 or 123). Note that the calibration mode and the high-accuracy map generation mode may be switched by a user input.

[0048] The mode switching unit 208 switches from the calibration mode to the high-accuracy map generation mode by basically determining when calibration is complete. The mode switching unit 208 switches the calibration mode to the map generation mode when calibration using a correction value is complete. For example, the mode switching unit 208 determines when calibration is complete based on criteria such as whether a time-series change in the calibration value 215 is smaller than a predetermined value and whether the consistency between the image feature points and the map point cloud, i.e., whether the reprojection error is smaller than a predetermined value. However, the method by which the mode switching unit 208 determines when calibration is complete is not limited to the method described here.

[0049] On the other hand, the mode switching unit 208 switches from the high-accuracy map generation mode to the calibration mode basically when a decrease in calibration accuracy is suspected. For this reason, the mode switching unit (mode switching unit 208) switches the map generation mode to the calibration mode when the calibration accuracy is decreased. Cases where a decrease in calibration accuracy is suspected include, for example, a decrease in the number of effective pixels of the image used for distance estimation, a large reprojection error, or the passage of time since the last use, such as after parking or stopping, and the mode switching unit 208 makes a judgment based on these situations. However, the method by which the mode switching unit 208 judges the situation is not limited to the method described here.

[0050] The mode determination result by the mode switching unit 208 is output to the RAM 112 as a mode flag 214. When switching to the calibration mode, the optimization processing unit (optimization processing unit 206) and the statistical processing unit (statistical processing unit 207) calibrate the multiple image capturing units (cameras 121 to 123) based on the correction value. Images captured by the multiple image capturing units (cameras 121 to 123) for which calibration has been performed have correct correspondence of feature points in subsequent processing.

[0051] The mode flag 214 is expressed by two types of values: a value indicating a calibration mode, and a value indicating a high-accuracy map generation mode. After the mode flag 214 is output, the statistical processing unit 207, which is a pre-processing step, is executed again based on input from the cameras 121 to 123, etc., and the mode switching unit 208 waits to switch the processing mode until a trigger is generated. Details of the mode switching unit 208 will be described later with reference to FIG. 4. The map point cloud coordinates 211 or high-accuracy map point cloud coordinates 212 and calibration values ​​215 obtained by the processing up to this point are stored as a map point cloud 216 and calibration values ​​215 in the recording unit 113 as needed.

[0052] Next, an example of detailed processing of each functional block will be described with reference to FIG. 4 and subsequent figures.

[0053] <Configuration example of mode switching section> First, an example of the configuration of the mode switching unit 208 will be described. Fig. 4 is a detailed block diagram showing an example of the configuration of the mode switching unit 208. Arrows in Fig. 4 and subsequent figures indicate the processing order of the functional blocks and the flow of data between the functional blocks.

[0054] The mode switching unit 208 is made up of a calibration verification unit 301 and a mode output unit 308 . The calibration verification unit 301 includes a verification determination unit 302 , a time-series value verification unit 303 , an error verification unit 304 , a valid pixel number verification unit 305 , an error verification unit 306 , and a scene verification unit 307 .

[0055] The verification determination unit 302 obtains the mode flag 214 from the RAM 112 and determines whether to execute the time-series value verification unit 303 and the error verification unit 304, or whether to execute the valid pixel number verification unit 305, the error verification unit 306, and the scene verification unit 307. If the mode flag 214 indicates the calibration mode, the verification determination unit 302 transitions to execution of the time-series value verification unit 303 and the error verification unit 304. If the mode flag 214 indicates the high accuracy map generation mode, the verification determination unit 302 transitions to execution of the valid pixel number verification unit 305, the error verification unit 306, and the scene verification unit 307. Note that the mode flag 214 may store a value that indicates the calibration mode when it is "0" and the high accuracy map generation mode when it is "1," for example. Alternatively, a flag that indicates the calibration mode when the mode flag 214 is turned on and the high accuracy map generation mode when the mode flag 214 is turned off may be stored.

[0056] The time-series value verification unit 303 references the time-series values ​​of the calibration values ​​215 from the recording unit 113 and determines whether the change in the time-series values ​​is equal to or less than a predetermined value. The time-series value verification unit 303 determines the change in the time-series values ​​based on, for example, whether the difference in the calibration values ​​for the past few frames is equal to or less than a predetermined value, or whether the standard deviation of the calibration values ​​for the past few frames is equal to or less than a certain value. If the difference in the calibration values ​​is equal to or less than the predetermined value, the time-series value verification unit 303 outputs information indicating that this difference is equal to or less than the predetermined value to the error verification unit 304 and transitions to the error verification unit 304. Note that if the difference or standard deviation of the calibration values ​​exceeds the predetermined value, the calibration mode ends without executing the error verification unit 304. If it is satisfied that the difference in the calibration values ​​for the past few frames is equal to or less than a predetermined value, or the standard deviation of the calibration values ​​for the past few frames is equal to or less than a certain value, the mode is confirmed.

[0057] The error verification unit 304 determines whether the reprojection error is equal to or less than a predetermined value. The reprojection error is calculated based on the reference camera motion parameters 209, relative camera parameters 210, reference camera feature points 231, relative camera feature points 232, and map point cloud coordinates 211, and is similar to that described for the optimization processing unit 206. If the reprojection error is equal to or less than the predetermined value, the error verification unit 304 outputs information indicating that the reprojection error is equal to or less than the predetermined value to the mode output unit 308 together with the determination result of the time-series value verification unit 303, and transitions to the mode output unit 308.

[0058] The valid pixel number verification unit 305 compares the number of valid pixels between the relative cameras with a predetermined value and verifies whether the number of valid pixels is equal to or less than the predetermined value, thereby determining whether calibration has been successful. In high-precision map generation mode, the valid pixel number verification unit 305 performs parallelization processing of the relative cameras using the relative camera parameters 210, and further performs distance measurement in a stereo camera manner, thereby performing high-precision distance measurement. In the block matching processing performed when performing high-precision distance measurement, pixels whose disparity reliability is equal to or greater than a predetermined value are called valid pixels.

[0059] The number of valid pixels tends to be relatively high if calibration is successful, and conversely, tends to be relatively low if calibration is unsuccessful. Therefore, the valid pixel number verification unit 305 determines that calibration has failed if the number of valid pixels is equal to or less than a predetermined value, and determines that calibration has succeeded if the number of valid pixels exceeds the predetermined value. If the valid pixel number verification unit 305 determines that calibration has failed, processing in the high precision map generation mode ends, and calibration mode begins.

[0060] The error verification unit 306 verifies whether the reprojection error is equal to or less than a predetermined value. The reprojection error is calculated based on the reference camera motion parameters 209, the relative camera parameters 210, the reference camera feature points 231, the relative camera feature points 232, and the map point cloud coordinates 211, and is the same as that described for the optimization processing unit 206. If the reprojection error is equal to or less than the predetermined value, the error verification unit 306 outputs information indicating that the reprojection error is equal to or less than the predetermined value to the scene verification unit 307, and transitions to the scene verification unit 307.

[0061] The scene verification unit 307 determines the calibration status based on, for example, whether the scene representing the current vehicle situation is a scene in which time has passed since the last time the in-vehicle information processing device 101 was used, such as after the vehicle has been parked or stopped, or the actual elapsed time since the vehicle ignition was turned on. In a situation in which calibration errors are likely to occur due to changes in vehicle temperature, the vehicle temperature often changes over time. For this reason, the scene verification unit 307 determines that a calibration error may have occurred if more time than a predetermined value has passed since the last calibration value was updated. Alternatively, the scene verification unit 307 may determine that a calibration error may have occurred even if the vehicle has remained in the same place, such as when parked or stopped. The scene verification unit 307 outputs the scene verification result to the mode output unit 308 and transitions to the mode output unit 308.

[0062] The mode output unit 308 outputs a mode flag 214 indicating either the calibration mode or the high-accuracy map generation mode based on the results of the time-series value verification unit 303 and the error verification unit 304, or the results of the valid pixel number verification unit 305, the error verification unit 306, and the scene verification unit 307. When both the time-series value verification unit 303 and the error verification unit 304 determine that the calibration accuracy is high and that the calibration is successful, the mode output unit 308 outputs information indicating the high-accuracy map generation mode as the mode flag 214 to transition to the high-accuracy map generation mode.

[0063] On the other hand, if the mode output unit 308 determines that there is a possibility that a calibration error has occurred in any of the valid pixel number verification unit 305, the error verification unit 306, or the scene verification unit 307, it outputs information indicating the calibration mode as the mode flag 214. If the scene verification unit 307 verifies that the scene is one in which more time than a predetermined value has passed since the last calibration value was updated, or one in which the vehicle has remained in the same place for a long time, such as while parking, the mode output unit 308 outputs information indicating the calibration mode as the mode flag 214.

[0064] <Configuration example of reference camera motion estimation unit> FIG. 5 is a detailed block diagram showing an example of the configuration of the reference camera motion estimating unit 203. As shown in FIG.

[0065] The reference camera motion estimation unit 203 is composed of a mode determination unit 401, a feature point extraction unit 402, a feature amount calculation unit 403, a time-series corresponding point detection unit 404, a reference camera motion estimation unit 405, a high-precision map point acquisition unit 406, and a high-precision camera motion estimation unit 407.

[0066] The mode determination unit 401 acquires the mode flag 214 and determines whether the mode flag 214 indicates the calibration mode or the high-accuracy map generation mode. If the calibration mode is determined, the reference camera motion estimation unit 203 executes the feature point extraction unit 402, the feature amount calculation unit 403, the time-series corresponding point detection unit 404, and the reference camera motion estimation unit 405. If the high-accuracy map generation mode is determined, the reference camera motion estimation unit 203 executes the high-accuracy map point acquisition unit 406 and the high-accuracy camera motion estimation unit 407.

[0067] (Calibration mode) The feature point extraction unit 402 extracts feature points representing image features from the reference camera image. The feature point extraction unit 402 can realize the process of extracting feature points using known techniques such as Harris feature point extraction or FAST (Features from Accelerated Segment Test). The feature points extracted by the feature point extraction unit 402 are recorded in the RAM 112 as reference camera feature points 231. After the feature points are extracted, the process transitions to the feature amount calculation unit 403.

[0068] The feature amount calculation unit 403 calculates feature amounts representing the characteristics of all feature points extracted by the feature point extraction unit 402. Feature amounts are also called descriptors, and are obtained by known techniques such as BRIEF (Binary Robust Independent Elementary Features) or ORB (Oriented FAST and Rotated BRIEF). The feature point extraction unit 402 calculates feature amounts for all feature points extracted from the image captured by the reference camera. After calculating the feature amounts, the process transitions to the time-series corresponding point detection unit 404.

[0069] The time-series corresponding point detection unit 404 calculates time-series feature point correspondence of the reference camera. Time-series feature point correspondence refers to feature points representing the same object, where feature points obtained from an image captured by the reference camera at a previous time correspond to feature points obtained from an image captured at the current time. A feature point search method for feature point correspondence is realized by searching for feature points from the image captured by the reference camera that have feature amounts closest to those calculated by the feature amount calculation unit 403. At this time, although not reflected in FIG. 5 , the time-series corresponding point detection unit 404 may use odometry information or the like to limit the search area estimated from the vehicle motion. After the time-series corresponding point detection unit 404 has calculated correspondence relationships for all feature points, the process transitions to the reference camera motion estimation unit 405.

[0070] The reference camera motion estimation unit 405 estimates the reference camera motion using the feature points extracted by the feature point extraction unit 402 and the time-series correspondence of the feature points detected by the time-series corresponding point detection unit 404. The reference camera motion can be calculated from the correspondence of the feature points using a known technique such as the 8-point method. The reference camera motion estimated by the reference camera motion estimation unit 405 is output to the RAM 112 as the reference camera motion parameters 209.

[0071] (High-precision map generation mode) When the mode determination unit 401 determines that the high-accuracy map generation mode is selected, the reference camera motion estimation unit 203 executes the high-accuracy map point acquisition unit 406 and the high-accuracy camera motion estimation unit 407 .

[0072] The high-precision map point acquisition unit 406 acquires the high-precision map point cloud coordinates 212 obtained in the high-precision map generation mode from the RAM 112. The acquired high-precision map point cloud coordinates 212 are output to the high-precision camera motion estimation unit 407, and the process transitions to the high-precision camera motion estimation unit 407.

[0073] The high-precision camera motion estimation unit 407 estimates the reference camera motion with high precision using the high-precision map point cloud coordinates 212. The high-precision camera motion estimation unit 407 estimates the motion of the reference camera by comparing the high-precision map point cloud coordinates 212 at the previous time with the high-precision map point cloud coordinates 212 at the current time. The estimation can be realized by a known technique for performing point cloud matching, such as ICP (Iterative Closest Point). The reference camera motion obtained by point cloud matching is output as reference camera motion parameters 209.

[0074] <Configuration example of relative camera posture estimation unit> FIG. 6 is a detailed block diagram showing an example of the configuration of the relative camera posture estimation unit 204. As shown in FIG.

[0075] The relative camera posture estimation unit 204 estimates relative camera parameters 210 that represent the relative camera posture. The relative camera posture estimation unit 204 includes a mode determination unit 501, a feature point extraction unit 502, a feature amount calculation unit 503, a time-series corresponding point detection unit 504, a relative image capture unit corresponding point detection unit 505, a closed-loop detection unit 506, a relative camera posture estimation unit 507, and a calibration value acquisition unit 508.

[0076] The mode determination unit 501 obtains the mode flag 214 from the RAM 112 and determines whether the mode flag 214 indicates the calibration mode or the high-accuracy map generation mode. In the calibration mode, the mode determination unit 501 executes the feature point extraction unit 502, the feature amount calculation unit 503, the time-series corresponding point detection unit 504, the relative image capture unit corresponding point detection unit 505, the closed-loop detection unit 506, and the relative camera posture estimation unit 507. In the high-accuracy map generation mode, the mode determination unit 501 executes the calibration value acquisition unit 508.

[0077] (Calibration mode) The feature point extraction unit 502 extracts feature points that represent the features of an image. The process of extracting feature points by the feature point extraction unit 502 can be realized by known techniques such as Harris feature point extraction or FAST. The feature point extraction unit 502 extracts feature points from all images (referred to as "all camera images") captured by all cameras (for example, cameras 121 to 123). The feature points extracted by the feature point extraction unit 502 are recorded in the RAM 112 as relative camera feature points 232. After the feature point extraction unit 502 extracts the feature points, the process transitions to the feature amount calculation unit 503.

[0078] The feature amount calculation unit 503 calculates feature amounts that represent the characteristics of all feature points extracted by the feature point extraction unit 502. Feature amounts are calculated for all feature points extracted from all camera images. After the feature amount calculation unit 503 calculates the feature points, the process transitions to the time-series corresponding point detection unit 504.

[0079] The time-series corresponding point detection unit 504 calculates time-series feature point correspondences for all camera images. The search for corresponding points is realized by searching for points with the closest feature amounts obtained by the feature amount calculation unit 503. At this time, although not reflected in FIG. 6, the time-series corresponding point detection unit 504 may use odometry information or the like to limit the search area estimated from the vehicle motion. After the time-series corresponding point detection unit 504 has calculated the correspondence relationships for all feature points, the process transitions to the relative image capture unit corresponding point detection unit 505.

[0080] As described above, the reference camera motion estimation unit 203 has already extracted the feature points of the reference camera and calculated the feature amounts of the reference camera. Therefore, the relative camera posture estimation unit 204 may extract feature points of relative cameras excluding the reference camera and calculate the feature amounts of the reference camera, without targeting all camera images. The relative camera posture estimation unit 204 may read the feature points and feature amounts of the reference camera from RAM 112 as necessary and use them in processing.

[0081] The relative image capture unit corresponding point detection unit 505 calculates the relative correspondence of feature points of all the cameras (called "relative image capture unit corresponding points") using images captured by all the cameras at the same time. The relative image capture unit corresponding points are searched for from all the camera images to find points with the closest feature amounts obtained by the feature amount calculation unit 503. After the relative image capture unit corresponding point detection unit 505 has found the correspondence of all the feature points, the process transitions to the closed-loop detection unit 506.

[0082] The relative position and orientation estimation unit (relative camera orientation estimation unit 204) selects feature points from which the relative position and orientation can be estimated, based on the state of a loop formed by time-series corresponding points of images captured by the reference imaging unit (camera 121) at the same time and time-series corresponding points of images captured by the relative imaging unit (camera 122 or 123). For example, the closed-loop detection unit 506 checks the accuracy of the detected feature points and the accuracy of the correspondence between the feature points, based on the correspondence between the time series feature points of all the camera images obtained by the time-series corresponding point detection unit 504 and the correspondence between the relative feature points between all the cameras obtained by the relative imaging unit corresponding point detection unit 505.

[0083] If the correspondence between feature points is correct, a closed loop is formed in a shape that ranges from a rectangle to a parallelogram or a trapezoid. The area in which a closed loop is detected is the same subject captured at a location where the angles of view of both the reference camera and the relative camera overlap. Conversely, if a closed loop is not detected and there are feature points with no correspondence, the closed loop detection unit 506 determines that the coordinates or correspondence of the feature points are incorrect and discards the relevant feature points.

[0084] Here, how a closed loop is detected from feature points will be described with reference to FIG. FIG. 7 is a diagram showing how a closed loop is detected from feature points of a plurality of images.

[0085] Camera 121 is installed on the left side of the vehicle, and camera 122 is installed on the right side of the vehicle. Both cameras 121 and 122 capture images of the area ahead of the vehicle. The upper part of FIG. 7 shows two images PL(T) and PR(T) captured by cameras 121 and 122 at time T (an example of the current time). The lower part of FIG. 7 shows two images PL(T-1) and PR(T-1) captured by cameras 121 and 122 at time T-1 (an example of the previous time). In FIG. 7, the images are shown stretched vertically to make it easier to identify corresponding points.

[0086] In each image, feature points extracted by feature point extraction unit 402 and feature points extracted by feature point extraction unit 502 are indicated by black dots. For example, feature point F11(T) at the edge of the road in the upper left of image PL(T) captured at time T corresponds to feature point F12(T) at the edge of the road in the upper left of image PR(T). Feature points F11(T) and F12(T) are connected by a dashed line L1(T), indicating that feature points F11(T) and F12(T) correspond to each other. Similarly, feature point F11(T-1) at the edge of the road in the upper left of image PL(T-1) captured at time T-1 corresponds to feature point F12(T-1) at the edge of the road in the upper left of image PR(T-1). The feature points F11(T-1) and F12(T-1) are connected by a dashed line L1(T-1), which indicates that the feature points F11(T-1) and F12(T-1) correspond to each other.

[0087] The closed-loop detection unit 506 detects a closed-loop LP, indicated by an arrow in the figure, connecting the feature points F11(T), F12(T), F11(T-1), and F12(T-1). Similarly, it is determined whether closed-loop LPs can be detected for feature points at other positions in the image. By detecting the closed-loop LP, it is determined that the feature points of each image have a corresponding relationship. The above-described closed-loop detection process is also performed on images captured by the cameras 121 and 123.

[0088] Returning to the explanation of Figure 6. 7, and outputs feature points for which it is determined that the correspondence is correct to the relative camera posture estimation unit 507. After the closed loop detection unit 506 outputs feature points for which it is determined that the correspondence is correct, it transitions to the relative camera posture estimation unit 507.

[0089] The relative camera posture estimation unit 507 estimates the relative camera posture of the relative camera with respect to the reference camera based on the feature points obtained by the closed-loop detection unit 506 and the correspondence between the feature points. The relative camera posture can be calculated using a known technique such as the eight-point method. The obtained relative camera posture is output as the relative camera parameters 210.

[0090] (High-precision map generation mode) If the mode determination unit 501 determines that the high accuracy map generation mode is selected, the calibration value acquisition unit 508 is executed. The calibration value acquisition unit 508 sets the calibration values ​​215 obtained by the calibration as the relative camera parameters 210.

[0091] <Configuration example of map point cloud estimation unit> FIG. 8 is a detailed block diagram showing an example of the configuration of the map point cloud estimation unit 205.

[0092] The map point cloud estimation unit 205 is composed of a mode determination unit 601, a feature point acquisition unit 602, a reference camera motion acquisition unit 603, a relative camera attitude acquisition unit 604, a map point cloud calculation unit 605, a feature point acquisition unit 606, a parallelization unit 607, and a map point cloud calculation unit 608.

[0093] The mode determination unit 601 obtains the mode flag 214 from the RAM 112 and determines whether the mode flag 214 indicates the calibration mode or the high-accuracy map generation mode. In the calibration mode, the mode determination unit 601 executes a feature point acquisition unit 602, a reference camera motion acquisition unit 603, a relative camera attitude acquisition unit 604, and a map point cloud calculation unit 605. In the high-accuracy map generation mode, the mode determination unit 601 executes a feature point acquisition unit 606, a parallelization unit 607, and a map point cloud calculation unit 608.

[0094] (Calibration mode) The feature point acquisition unit 602 acquires the reference camera feature points 231 and the relative camera feature points 232 from the RAM 112. After the feature point acquisition unit 602 acquires the reference camera feature points 231 and the relative camera feature points 232, the process transitions to a reference camera movement acquisition unit 603.

[0095] The reference camera movement acquisition unit 603 acquires the reference camera movement parameters 209 from the RAM 112. After the reference camera movement acquisition unit 603 acquires the reference camera movement parameters 209, the process transitions to a relative camera attitude acquisition unit 604.

[0096] The relative camera attitude acquisition unit 604 acquires the relative camera parameters 210 from the RAM 112. The relative camera parameters 210 include information on time-series attitude changes of the relative camera with respect to the reference camera. After the relative camera attitude acquisition unit 604 acquires the relative camera parameters 210, the process transitions to the map point cloud calculation unit 605.

[0097] The map point cloud calculation unit 605 calculates a map point cloud, which is map point cloud coordinates 211, using the reference camera feature points 231, relative camera feature points 232, reference camera motion parameters 209, and relative camera parameters 210 acquired from the RAM 112 by the feature point acquisition unit 602, reference camera movement acquisition unit 603, and relative camera attitude acquisition unit 604. The map point cloud calculation unit 605 calculates the map point cloud according to the principle of triangulation, using the correspondence between the feature points and the camera attitudes of the reference camera and the relative camera.

[0098] Furthermore, the map point cloud calculation unit 605 calculates the distances by triangulation for the feature points that are commonly captured by the relative cameras, using the relative camera feature points 232, the correspondence between the relative camera feature points 232, and the relative camera parameters 210. Thereafter, the map point cloud calculation unit 605 estimates the three-dimensional coordinates of the relative camera feature points 232 based on the calculated distances.

[0099] Similarly, the map point cloud calculation unit 605 calculates the distances by triangulation for feature points that are commonly captured during time-series reference camera movements, using the reference camera feature points 231, the correspondence between the reference camera feature points 231, and the reference camera movement parameters 209. The map point cloud calculation unit 605 then estimates the three-dimensional coordinates of the reference camera feature points 231 based on the calculated distances. The three-dimensional coordinates of the reference camera feature points 231 and the relative camera feature points 232 calculated by the map point cloud calculation unit 605 are also called map point cloud coordinates. These map point cloud coordinates are output to the RAM 112 as map point cloud coordinates 211.

[0100] (High-precision map generation mode) When the mode determination unit 601 determines that the high-accuracy map generation mode is selected, the feature point acquisition unit 606, the rectification unit 607, and the map point cloud calculation unit 608 are executed.

[0101] The feature point acquisition unit 606 acquires the relative camera feature points 232 from the RAM 112. After the feature point acquisition unit 606 acquires the relative camera feature points 232, the process transitions to the parallelization unit 607.

[0102] The rectification unit 607 performs rectification processing of the relative cameras and outputs a rectified image. For example, the rectification unit 607 acquires the calibration value 215 from the RAM 112, performs rectification processing based on the relative relationship of the cameras using the calibration value 215, and generates a rectified image. For the rectification processing, hardware capable of high-speed calculation may be used. The rectification unit 607 outputs the rectified image to the map point cloud calculation unit 608, and transitions to the map point cloud calculation unit 608.

[0103] The map point cloud calculation unit 608 calculates a high-precision map point cloud using the rectified image calculated by the rectification unit 607. The map point cloud calculation unit 608 searches for feature points corresponding to the relative camera feature points 232 on the rectified image and calculates the coordinates of the corresponding feature points. At this time, the map point cloud calculation unit 608 calculates the distance from the correspondence and parallax of the feature points and estimates the 3D coordinates corresponding to the feature points from the obtained distance. The 3D coordinates estimated by the map point cloud calculation unit 608 by searching on the rectified image using the calibration value 215 have improved accuracy compared to 3D coordinates estimated without using the rectified image.

[0104] The map point cloud calculation unit 608 outputs the estimated three-dimensional coordinates to the RAM 112 as high-precision map point cloud coordinates 212. Furthermore, when searching for corresponding points on the rectified image, the map point cloud calculation unit 608 calculates the reliability of the corresponding point search. The map point cloud calculation unit 608 labels feature points whose reliability is equal to or greater than a predetermined value as valid pixels and outputs them to the RAM 112. Information on the feature points labeled as valid pixels is used by the mode switching unit 208.

[0105] FIG. 9 is a detailed block diagram showing an example of the configuration of the statistical processing unit 207.

[0106] The statistical processing unit 207 is composed of a multi-stage time series value storage unit 701 and a statistical value calculation unit 702 . The time series value storage unit (multistage time series value storage unit 701) stores provisional correction values ​​acquired in time series in a buffer (multistage buffer 130 shown in FIG. 10 to be described later). The provisional correction values ​​are values ​​stored in each buffer of the multistage buffer 130 based on highly reliable estimated values ​​shown in FIG. 10 to be described later. The multistage time series value storage unit 701 has a buffer function of acquiring relative camera parameters 210 from RAM 112 and storing the relative camera parameters 210 in the multistage buffer 130 only when the reprojection error calculated by the optimization processing unit 206 is smaller than a predetermined value.

[0107] The multistage time series value storage unit 701 sequentially stores the relative camera parameters 210 and statistical values ​​in the multistage buffer 130 for the relative camera parameters 210 obtained in time series.

[0108] The statistical value calculation unit (statistical value calculation unit 702) performs statistical processing on the provisional correction values ​​stored in the buffer (multistage buffer 130) to calculate correction values ​​(calibration values). For example, the statistical value calculation unit 702 performs statistical processing on the relative camera parameters 210 stored in the multistage time-series value storage unit 701 to estimate highly accurate calibration values. After a predetermined number or more values ​​have been stored in the last-stage multistage buffer 130, the statistical value calculation unit 702 obtains an estimated value by taking a statistical value such as the median or mode for the last-stage buffer values. The statistical value calculation unit 702 outputs the estimated value to the RAM 112 as the calibration value 215.

[0109] FIG. 10 is a diagram showing an example of the relative camera parameters 210 and the multi-stage buffer 130. As shown in FIG. The upper part of Fig. 10 shows how the yaw angle of the relative camera changes over time as an example of the relative camera parameter 210. In this figure, the horizontal axis of the relative camera parameter 210 is the number of frames [frames] and the vertical axis is the yaw angle [deg], and as the number of frames increases, the relative camera parameter 210 for the yaw angle is shown to take on a constant value of y6.

[0110] An example of the configuration of the multi-stage buffer 130 is shown below 10. The multi-stage buffer 130 is configured with multiple stages, including one stage. After the first stage buffer of the multi-stage buffer 130 is filled with relative camera parameters 210, statistical values ​​such as the median and mode calculated from the first stage buffer are stored in the second and subsequent stages of the multi-stage buffer 130.

[0111] The relative camera parameters 210 estimated by the optimization processor 206 tend to be unstable due to low accuracy. Therefore, the multi-stage time-series value storage unit 701 stores only highly reliable relative camera parameters 210, i.e., highly reliable estimates of the yaw angle, indicated by a circle in the figure. In a given frame, many highly reliable estimates of the yaw angle are detected. For example, when the current time is time t, the yaw angle is calculated as y6 at time t-3 going back from time t, and the yaw angle is calculated as y6 at time t-7 going back even further. The calculated highly reliable estimates of the yaw angle are stored in the first-stage buffer. Furthermore, at time t-29 in the past, the yaw angle is calculated as y7, and at time t-30 in the past, the yaw angle is calculated as y8, and the calculated yaw angles are stored in the first-stage buffer. Since no highly reliable estimates were detected at past times (e.g., t-1, t-2), the highly reliable estimates at those times are not stored in the first-stage buffer.

[0112] When multiple yaw angle values ​​are stored in the first-stage buffer, the median value y6 of those yaw angles is stored in the second-stage buffer. This is also shown in the upper part of FIG. 10, where the yaw angle converges to y6. Note that the yaw angle value stored in the second-stage buffer is preferably a value calculated using a statistical method. The statistical value calculated by the statistical value calculation unit 702 from the yaw angles stored in the second-stage buffer becomes the calibration value.

[0113] In FIG. 10, the yaw angle has been described as an example of the relative camera parameter 210, but the roll angle and pitch angle may also be stored in the multi-stage buffer 130 in the same way, and calibration values ​​may be calculated.

[0114] In the in-vehicle information processing device 101 according to the first embodiment described above, a relative camera attitude estimation unit 204 and a statistical processing unit 207 used for multi-camera calibration are added to the functional block where general map generation is performed. Therefore, the in-vehicle information processing device 101 is capable of both processing for generating a high-precision map and processing for performing multi-camera calibration, and can deal with the occurrence of calibration errors. Furthermore, the in-vehicle information processing device 101 shares its core processing (the map point cloud estimation unit 205 and the optimization processing unit 206) with calibration. Therefore, after calibration, the in-vehicle information processing device 101 can generate a high-precision map making use of the multi-camera by changing some of the processing (distance measurement, attitude estimation method). Furthermore, by sharing the core processing with calibration, the in-vehicle information processing device 101 can suppress an increase in processing load.

[0115] In this way, the in-vehicle information processing device 101 estimates point cloud coordinates of a map point cloud generated based on feature points contained in multiple images captured by multiple multi-cameras attached to a vehicle, and performs optimization processing to generate a map point cloud consisting of corrected point cloud coordinate values. Then, calibration of the multiple cameras is performed using correction values ​​determined by statistically processing the correction values ​​of the relative positions of the multiple cameras. Therefore, even if the camera mounting positions are changed from their original positions, the camera calibration is performed at the changed positions, so a highly accurate map point cloud can be created using images captured by each camera.

[0116] Furthermore, the self map generation and calibration device 100 uses several common functional blocks to achieve both the function of generating a self map and the function of performing calibration. This eliminates the need to provide independent functional blocks for the function of generating a self map and the function of performing calibration, thereby simplifying the configuration of the self map generation and calibration device 100.

[0117] [Second embodiment] Next, a configuration example and an operation example of an in-vehicle information processing device according to a second embodiment of the present invention will be described with reference to Fig. 11 and Fig. 12. The in-vehicle information processing device according to the second embodiment has a function of performing calibration when the vehicle is started.

[0118] FIG. 11 is a diagram showing how the positions of the feature points change when the vehicle ignition is turned off and on. The upper part of Fig. 11 shows an example of feature points detected when the vehicle ignition is turned off. This figure shows a camera 121, the state in which the camera 121 captures an image of a feature point f1 of a subject in front of the vehicle, and an example of an image P captured by the camera 121. An example of a feature point f11 detected from the image P is shown in the image P, which is located at a position corresponding to the feature point f1. When the vehicle ignition is turned off, there is a correspondence between the feature point f1 and the feature point f11, which is connected by a dashed line.

[0119] Calibration errors may occur in images captured by a camera mounted on a vehicle due to the passage of time while the vehicle is parked. Examples of situations where calibration errors may occur include when the camera housing itself is deformed by heat, which increases the influence of changes in the optical characteristics of the lens. Another example is when, in a parking lot in the middle of summer, the interior of a vehicle is cold immediately after parking, but the vehicle heats up due to the outside air or sunlight after the air conditioner is turned off, causing distortion in the pillars.

[0120] An example of feature points detected when the vehicle ignition is on is shown in the lower part of Fig. 11. As time has passed since the vehicle ignition was turned off, the optical characteristics of the lens of the camera 121 have changed, and a positional shift g1 occurs between the position of feature point f12 detected from the image P when the ignition was on and the position of feature point f11 detected from the image P when the ignition was off. Therefore, the in-vehicle information processing device 101A according to the second embodiment shown in Fig. 12 corrects the position of feature point f12 by the positional shift g1 when the ignition is on, thereby suppressing the influence of the positional shift g1 caused by the change in the optical characteristics of the lens of the camera 121.

[0121] 12 is a functional block diagram showing an example of the configuration of an on-board information processing device 101A according to the second embodiment. The on-board information processing device 101A has a configuration in which a stopped-state map point cloud and corresponding point acquisition unit 801 and a startup calibration unit 802 are added to the on-board information processing device 101 of the self map generation and calibration device 100 according to the first embodiment. The on-board information processing device 101A according to the second embodiment aims to perform calibration early using the generated map point cloud 216.

[0122] The sensor information acquisition unit 201 acquires sensor information from the cameras 121 to 123, and acquires camera parameter design values ​​220 from the ROM 111. The odometry information acquisition unit 202 acquires vehicle specifications 221 from the ROM 111. Note that since the processing according to the second embodiment is performed while the vehicle is stopped, there is no output from the vehicle speed sensor 161 and the steering angle sensor 162 to the odometry information acquisition unit 202.

[0123] The reference camera motion estimation unit 203 inputs and outputs reference camera motion parameters 209 and high precision map point cloud coordinates 212 to the RAM 12. The high precision map point cloud coordinates 212 are information generated in high precision map generation mode before the ignition is turned off. Furthermore, the reference camera motion estimation unit 203 outputs the reference camera motion parameters 209, which are the result of estimating that no reference camera motion has been performed because the vehicle is stopped, to the RAM.

[0124] The relative camera attitude estimation unit 204 outputs relative camera parameters 210, which are estimated values ​​of the relative camera attitude, to the RAM 112. When the ignition is turned on, the relative camera attitude estimation unit 204 also outputs the relative camera parameters 210 to the startup calibration unit 802.

[0125] The map point cloud estimation unit 205 inputs and outputs map point cloud coordinates 211 and high-precision map point cloud coordinates 212 to the RAM 112. The map point cloud estimation unit 205 calculates a reprojection error using the map point cloud coordinates 211. The map point cloud estimation unit 205 also estimates reference camera motion parameters 209, relative camera parameters 210, and high-precision map point cloud coordinates 212.

[0126] The optimization processing unit 206 generates a stopped-time map point cloud 812 when the vehicle is stopped, using the map point cloud information (map point cloud coordinates 211 and high-precision map point cloud coordinates 212) estimated by the map point cloud estimation unit 205. The optimization processing unit 206 also generates a stopped-time feature point 811, which is a feature point of an image captured by the reference camera and a feature point of an image captured by the relative camera when the vehicle is stopped. The stopped-time feature point 811 corresponds to the feature point f1 in FIG. 11 . The optimization processing unit 206 also generates a stopped-time map point cloud 812 based on the stopped-time feature point 811 when the vehicle is stopped. The optimization processing unit 206 also has a function of estimating high-precision reference camera motion parameters 209, relative camera parameters 210, and high-precision map point cloud coordinates 212, using the high-precision map point cloud coordinates 212 instead of the map point cloud coordinates 211.

[0127] For startup calibration, the stationary-time map point cloud and corresponding point acquisition unit 801 records the map point cloud 216 read out from the RAM 112 by the optimization processing unit 206 in the recording unit 113. Furthermore, in addition to the map point cloud 216 stored in the recording unit 113 during normal operation, the stationary-time map point cloud and corresponding point acquisition unit 801 acquires stationary-time feature points 811 and stationary-time map point cloud 812 from the optimization processing unit 206 and records them in the recording unit 113.

[0128] The vehicle-stopped map point cloud and corresponding point acquisition unit 801 also acquires, from the statistical processing unit 207, a calibration value 215 calculated when the vehicle is stopped, and stores the calibration value 215 in the recording unit 113. In addition to the various pieces of information according to the first embodiment, the recording unit 113 records the calibration value 215, the map point cloud 216, the vehicle-stopped feature points 811, and the vehicle-stopped map point cloud 812 generated when the vehicle is stopped. The vehicle-stopped map point cloud and corresponding point acquisition unit 801 needs to be activated in a scene where a long period of time is expected to pass, such as when the vehicle is parked. For example, the vehicle-stopped map point cloud and corresponding point acquisition unit 801 operates when it is triggered by the ignition being turned off after the vehicle is stopped. However, the vehicle-stopped map point cloud and corresponding point acquisition unit 801 may be activated by other factors.

[0129] The startup calibration unit 802 calibrates the multiple imaging units (cameras 121-123) when the moving body (vehicle) starts up, using the map point cloud and feature points generated when the moving body (vehicle) stops. For example, the startup calibration unit 802 operates in response to being triggered by the ignition being turned on when the vehicle starts to operate. The startup calibration unit 802 performs calibration early in a situation where a calibration error occurs due to the passage of time, such as when the vehicle is parked or stopped.

[0130] The startup calibration unit 802, which has started operating, reads out the stopped-vehicle map point cloud 812, stopped-vehicle feature points 811, and calibration values ​​215 stored in the recording unit 113. The startup calibration unit 802 compares the coordinates of the feature points extracted from the images captured by each camera with the stopped-vehicle feature points 811. The startup calibration unit 802 detects whether or not a positional deviation g1 shown in FIG. 11 has occurred, and if a positional deviation g1 has occurred, corrects the camera attitude so that the stopped-vehicle map point cloud 812 matches the coordinates of the feature points extracted by each camera. By correcting the camera attitude, calibration at vehicle startup is performed.

[0131] In the in-vehicle information processing device 101A according to the second embodiment described above, the stopped-state map point cloud 812 generated when the ignition is off and the corresponding stopped-state feature points 811 of the multiple images captured by each camera are recorded in the recording unit 113. The next time the ignition is turned on, the in-vehicle information processing device 101A reads the stopped-state map point cloud 812 from the recording unit 113, and performs startup calibration if the elapsed time exceeds a predetermined value. In this way, the in-vehicle information processing device 101A can complete calibration in a short time by using the corresponding stopped-state feature points 811 of the multiple images captured by the multi-cameras when the ignition is off as initial values ​​of the relative camera attitudes.

[0132] Although the calibration at the start of operation has been described as being performed by the startup calibration unit 802, similar to the processing according to the first embodiment, the calibration may be performed by the optimization processing unit 206 and the statistical processing unit 207. In this case, the startup calibration unit 802 outputs an instruction to the optimization processing unit 206 and the statistical processing unit 207 to perform the calibration.

[0133] It should be noted that the present invention is not limited to the above-described embodiment, and it goes without saying that various other applications and modifications are possible without departing from the gist of the present invention as set forth in the claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Other aspects conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0134] Furthermore, although the self-mapping and calibration device 100 has been described as an example of a device mounted on a vehicle, it may also be mounted on a moving object such as an autonomously controlled robot, an airborne drone, or heavy construction machinery.

[0135] The self-mapping and calibration device 100 may also include an input / output interface (not shown), and a program may be loaded from another device via the input / output interface and a medium available to the self-mapping and calibration device 100 as needed. Here, the medium refers to, for example, a storage medium detachable from the input / output interface, or a communication medium, such as a wired, wireless, or optical network, or a carrier wave or digital signal propagating through the network. Some or all of the functions implemented by the program may be implemented by a hardware circuit or FPGA. Each of the above configurations, functions, etc. may also be implemented by software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function may be recorded on a memory, a recording device such as a hard disk or SSD, or a recording medium such as an IC card, SD card, or DVD. In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0136] 100... Self-map generation and calibration device, 101... In-vehicle information processing device, 110... CPU, 111... ROM, 112... RAM, 113... Recording unit, 121 to 123... Camera, 161... Vehicle speed sensor, 162... Steering angle sensor, 171... Interface, 201... Sensor information acquisition unit, 202... Odometry information acquisition unit, 203... Reference camera motion estimation unit, 204... Relative camera attitude estimation unit, 205... Map point cloud estimation unit, 206... Optimization processing unit, 207... Statistical processing unit, 208... Mode switching unit

Claims

1. a reference position and orientation estimation unit that estimates a reference position and orientation of a reference imaging unit by using a plurality of reference images captured in time series by the reference imaging unit of a surrounding environment of the moving body; and a relative position and orientation estimation unit that estimates a relative position and orientation of the relative imaging unit with respect to the reference position and orientation using a relative image captured by a relative imaging unit that is at least one imaging unit other than the reference imaging unit among the plurality of imaging units; a map point cloud estimation unit that estimates a map point cloud made up of point cloud coordinates that are three-dimensional position information of the feature points in the environment, based on reference feature points extracted from the reference image by the reference position and orientation estimation unit and motion information of the reference imaging unit, relative feature points extracted from the relative image by the relative position and orientation estimation unit, and motion information of the relative imaging unit; an optimization processing unit that optimizes the reference position and orientation, the relative position and orientation, and the map point cloud so as to reduce a predetermined index; a statistical processing unit that determines correction values ​​for calibration to be performed on the plurality of imaging units by statistically processing provisional correction values ​​that represent the time-series relative positions and orientations optimized by the optimization processing unit. Information processing device.

2. The relative position and orientation estimation unit selects feature points from which the relative position and orientation can be estimated, based on a state of formation of a loop constituted by time-series corresponding points of the image captured by the reference imaging unit and time-series corresponding points of the image captured by the relative imaging unit at the same time. The information processing device according to claim 1 .

3. The statistical processing unit includes a time series value storage unit that stores the provisional correction values ​​acquired in time series in a buffer; a statistical value calculation unit that performs statistical processing on the provisional correction value stored in the buffer to calculate the correction value; The information processing device according to claim 2 .

4. and a mode switching unit that switches the processing modes of the reference position and attitude estimation unit, the relative position and attitude estimation unit, and the map point cloud estimation unit between a calibration mode in which the point cloud coordinates are estimated using the relative position and attitude and the relative feature points estimated in time series, and a map generation mode in which the map point cloud consisting of the point cloud coordinates is estimated using a parallax obtained in a common imaging area between the reference imaging unit and the relative imaging unit. The information processing device according to claim 3 .

5. The mode switching unit switches the calibration mode to the map generation mode when calibration using the correction value is completed, and switches the map generation mode to the calibration mode when calibration accuracy is reduced. The information processing device according to claim 4.

6. The optimization processing unit and the statistical processing unit perform calibration of the plurality of imaging units based on the correction value. The information processing device according to claim 5 .

7. The map point cloud and the feature points generated when the moving object stops are used in the calibration of the plurality of image capturing units, which is executed when the moving object starts up. The information processing device according to claim 6.

8. a process of estimating a reference position and orientation of a reference imaging unit, using a plurality of reference images captured in time series by the reference imaging unit of a plurality of imaging units mounted on a moving body, the reference imaging unit being one of the plurality of imaging units mounted on the moving body; a process of estimating a relative position and orientation of the relative imaging unit with respect to the reference position and orientation using a relative image captured by a relative imaging unit that is at least one imaging unit other than the reference imaging unit among the plurality of imaging units; a process of estimating a map point cloud including point cloud coordinates that are three-dimensional position information of the feature points in the environment, based on reference feature points extracted from the reference image, motion information of the reference imaging unit, relative feature points extracted from the relative image, and motion information of the relative imaging unit; optimizing the reference position and orientation, the relative position and orientation, and the map point cloud so as to reduce a predetermined index; and determining correction values ​​for calibration to be performed on the plurality of imaging units by statistically processing the provisional correction values ​​that represent the optimized time-series relative positions and orientations. Information processing methods.

Images