Map creation device, map creation method, and map creation program

Abstract

To reduce the computational complexity of repeated calculation in calculating a three-dimensional positions of feature points in a plurality of images obtained by imaging different spots.SOLUTION: A map creation device 10 comprises: an odometry information calculation unit 11C for calculating odometry information indicating a moving amount of a vehicle; an initial value calculation unit 11D for calculating, from the odometry information of the vehicle, an initial value of a homography matrix between a plurality of images obtained by imaging different spots by an in-vehicle camera; an optimum value calculation unit 11E for, from the calculated initial value and brightness values of pixels included in road surface regions designated with respect to the plurality of images, calculating an optimum value of the homography matrix by repeated calculation; a camera position orientation calculation unit 11F for calculating, by decomposing the optimum value, the amount of change of camera position and the amount of change of camera orientation regarding the in-vehicle camera; and a three-dimensional position calculation unit 11H for calculating, from the amount of change of camera position and the amount of change of camera orientation, three-dimensional positions of feature points in the plurality of images.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a map creation device, a map creation method, and a map creation program.

Background Art

[0002] Conventionally, there has been a technique of photographing a surrounding environment with a camera mounted on a vehicle and estimating the position and orientation of the vehicle from the photographed image (see, for example, Non-Patent Document 1).

[0003] According to the technique described in Non-Patent Document 1, as shown in FIG. 9, when two different points are photographed by a camera installed at the rear of the vehicle, a plurality of images as shown in FIG. 10 are obtained. The road surface area is extracted from these plurality of images, and an optimal value of the homography matrix between the plurality of images is obtained by performing iterative calculations based on the luminance of the extracted road surface area. The ESM (Efficient Second order Minimisation) algorithm is used as the algorithm for iterative calculations. Then, the amount of change in the position and the amount of change in the orientation of the vehicle are calculated from the optimal value of the homography matrix.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technique described in Non-Patent Document 1, the amount of calculation for iterative calculation when obtaining the optimal value of the homography matrix is large, and cost reduction is not possible. Therefore, it is desired to reduce the amount of calculation for iterative calculation.

[0006] The present disclosure has been made in view of the above circumstances, and an object thereof is to provide a map creation device, a map creation method, and a map creation program that can reduce the amount of calculation for iterative calculation when calculating the three-dimensional positions of feature points in a plurality of images obtained by photographing different points.

Means for Solving the Problems

[0007] In order to achieve the above object, a map creation device according to a first aspect is mounted on a vehicle, and an image acquisition unit that acquires a plurality of images obtained by photographing different points from an in-vehicle camera that photographs the periphery of the vehicle, an odometry information calculation unit that calculates odometry information indicating the movement amount of the vehicle, an initial value calculation unit that calculates an initial value of a homography matrix between the plurality of images from the odometry information of the vehicle, and an optimal value calculation unit that repeatedly calculates an optimal value of the homography matrix from the initial value and the luminance values of each pixel included in a road surface area specified for the plurality of images, a camera position and orientation calculation unit that decomposes the optimal value and calculates a change amount of the camera position and a change amount of the camera orientation for the in-vehicle camera, and a three-dimensional position calculation unit that calculates the three-dimensional positions of feature points in the plurality of images from the change amount of the camera position and the change amount of the camera orientation.

[0008] Further, a map creation device according to a second aspect is the map creation device according to the first aspect, wherein the camera position and orientation calculation unit further calculates an estimated value of a road surface normal vector, which is a vector in the normal direction of the road surface viewed from the in-vehicle camera, by decomposing the optimal value, and when an error represented by an angle between the estimated value of the road surface normal vector and a value of the road surface normal vector obtained in advance by calibration of the in-vehicle camera is less than a threshold value, a utilization determination unit that determines to use the change amount of the camera position and the change amount of the camera orientation is further provided.

[0009] Further, in the map creation device according to the third aspect, in the map creation device according to the second aspect, when the utilization determination unit determines that the error is equal to or greater than the threshold value, the utilization determination unit determines not to use the amount of change in the camera position and the amount of change in the camera attitude.

[0010] Furthermore, in order to achieve the above object, a map creation method according to a fourth aspect is mounted on a vehicle and acquires a plurality of images taken at different points from an in-vehicle camera that captures the periphery of the vehicle, calculates odometry information indicating the amount of movement of the vehicle, calculates an initial value of a homography matrix between the plurality of images from the odometry information of the vehicle, repeatedly calculates an optimal value of the homography matrix from the initial value and the luminance values of each pixel included in a road surface area specified for the plurality of images, decomposes the optimal value, calculates the amount of change in the camera position and the amount of change in the camera attitude for the in-vehicle camera, and calculates the three-dimensional position of feature points in the plurality of images from the amount of change in the camera position and the amount of change in the camera attitude.

[0011] Furthermore, in order to achieve the above object, a map creation program according to a fifth aspect causes a computer to execute acquiring a plurality of images taken at different points from an in-vehicle camera that is mounted on a vehicle and captures the periphery of the vehicle, calculating odometry information indicating the amount of movement of the vehicle, calculating an initial value of a homography matrix between the plurality of images from the odometry information of the vehicle, repeatedly calculating an optimal value of the homography matrix from the initial value and the luminance values of each pixel included in a road surface area specified for the plurality of images, decomposing the optimal value, calculating the amount of change in the camera position and the amount of change in the camera attitude for the in-vehicle camera, and calculating the three-dimensional position of feature points in the plurality of images from the amount of change in the camera position and the amount of change in the camera attitude.

Advantages of the Invention

[0012] According to the technology of the present disclosure, when calculating the three-dimensional positions of feature points in a plurality of images obtained by photographing different locations, it is possible to reduce the amount of repetitive calculations, which has the effect of reducing the amount of repetitive calculations.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0014] Hereinafter, with reference to the drawings, an example of a mode for implementing the technology of the present disclosure will be described in detail. Note that components and processes with the same functions, operations, and functions are given the same reference numerals throughout the drawings, and redundant descriptions may be omitted as appropriate. Each drawing only schematically shows the technology of the present disclosure to such an extent that it can be sufficiently understood. Therefore, the technology of the present disclosure is not limited to only the illustrated examples. Further, in this embodiment, descriptions of configurations not directly related to the technology of the present disclosure and well-known configurations may be omitted.

[0015] The map creation device according to this embodiment relates to the initialization of a map in the case of creating a point cloud map in the framework of Visual SLAM (Simultaneous Localization and Mapping) technology using an in-vehicle camera. In map initialization, it is first necessary to obtain the amount of change in the position and the amount of change in the orientation of the in-vehicle camera from images taken at two different points. However, in a scene where there are few feature points detectable from the images, it is difficult to accurately obtain the amount of change in the position and the amount of change in the orientation of the in-vehicle camera from the feature points. For this reason, the amount of change in the position and the amount of change in the orientation of the in-vehicle camera are obtained from the luminance values of each pixel in the road surface area.

[0016] FIG. 1 is a diagram showing an example of the configuration of a map creation system 100 according to this embodiment.

[0017] As shown in FIG. 1, the map creation system 100 according to this embodiment includes a map creation device 10, a wheel speed sensor 20, a steering angle sensor 21, and an in-vehicle camera 22.

[0018] The in-vehicle camera 22 is mounted on the vehicle and captures the surroundings of the vehicle. The in-vehicle camera 22 only needs to be installed in a state where it can capture the road surface, and the installation location in the vehicle is not particularly limited. For example, a monocular camera is applied to the in-vehicle camera 22, but it is not limited thereto, and a stereo camera or the like may be used.

[0019] Specifically, the in-vehicle camera 22 is a monocular camera provided on the upper part of the vehicle or the like, and captures the peripheral areas in front of and behind the vehicle. The in-vehicle camera 22 is provided, for example, near the substantially central part in the vehicle width direction, and is arranged such that the optical axis of the in-vehicle camera 22 faces slightly downward from the horizontal direction. The in-vehicle camera 22 is communicably connected to the map creation device 10 and sends the captured image to the map creation device 10.

[0020] The wheel speed sensor 20 detects the wheel speeds of the four wheels of the vehicle. The wheel speed sensor 20 sends the detected wheel speeds to the map creation device 10. An encoder of a normal wheel is usually used for the wheel speed sensor 20, but in a vehicle equipped with a motor such as a hybrid vehicle, an encoder of the motor may be used. The encoder of the motor is desirable because it has higher detection accuracy than the encoder of the wheel.

[0021] The steering angle sensor 21 detects the steering angle of the vehicle. The steering angle sensor 21 sends the detected steering angle to the map creation device 10.

[0022] The map creation device 10 according to the present embodiment calculates an initial value of a homography matrix between a plurality of images from the odometry information of the vehicle, and calculates an optimal value of the homography matrix using the calculated initial value. Thereby, the calculation amount of the iterative calculation can be reduced. The map creation device 10 according to the present embodiment is assumed for in-vehicle use. In the case of in-vehicle use, since the processing capabilities of resources such as a processor and a memory are relatively low, by reducing the calculation amount of the iterative calculation, the load on the resources is reduced and a greater effect can be obtained.

[0023] Specifically, the map creation device 10 may be realized as a part of an ECU (Electronic Control Unit) which is a vehicle control computer, or may be realized as an in-vehicle computer separate from the ECU.

[0024] The map creation device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an input / output interface (I / O) 14, a storage unit 15, and an external interface (external I / F) 16.

[0025] The CPU 11, ROM 12, RAM 13, and I / O 14 are each connected via a bus. To the I / O 14, each functional unit including the storage unit 15 and the external I / F 16 is connected. These functional units are made to be able to communicate with each other with the CPU 11 via the I / O 14.

[0026] The control unit is constituted by the CPU 11, ROM 12, RAM 13, and I / O 14. The control unit may be configured as a sub-control unit that controls some operations of the map creation device 10, or may be configured as a part of the main control unit that controls the entire operation of the map creation device 10. For a part or all of each block of the control unit, for example, an integrated circuit such as an LSI (Large Scale Integration) or an IC (Integrated Circuit) chipset is used. Individual circuits may be used for the above-mentioned respective blocks, or circuits in which some or all are integrated may be used. The above-mentioned respective blocks may be provided integrally, or some blocks may be provided separately. Also, in each of the above-mentioned blocks, a part thereof may be provided separately. For the integration of the control unit, not limited to LSI, a dedicated circuit or a general-purpose processor may be used.

[0027] As the storage unit 15, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, etc. are used. In the storage unit 15, the map creation program 15A according to the present embodiment is stored. Note that this map creation program 15A may be stored in the ROM 12.

[0028] The map creation program 15A may be pre-installed in the map creation device 10, for example. The map creation program 15A may be realized by storing it in a non-volatile storage medium, distributing it via a network, and appropriately installing it in the map creation device 10. Examples of non-volatile storage media include CD-ROM (Compact Disc Read Only Memory), magneto-optical disk, HDD, DVD-ROM (Digital Versatile Disc Read Only Memory), flash memory, memory card, etc.

[0029] The external I / F 16 is an interface for communicably connecting to each of the wheel speed sensor 20, the steering angle sensor 21, and the in-vehicle camera 22.

[0030] The CPU 11 of the map creation device 10 according to the present embodiment functions as each part shown in FIG. 2 by writing the map creation program 15A stored in the storage unit 15 into the RAM 13 and executing it.

[0031] FIG. 2 is a block diagram showing an example of the functional configuration of the map creation device 10 according to the present embodiment.

[0032] As shown in FIG. 2, the CPU 11 of the map creation device 10 according to the present embodiment functions as an image acquisition unit 11A, a sensor information acquisition unit 11B, an odometry information calculation unit 11C, an initial value calculation unit 11D, an optimum value calculation unit 11E, a camera position and orientation calculation unit 11F, a utilization determination unit 11G, and a three-dimensional position calculation unit 11H.

[0033] The image acquisition unit 11A acquires a plurality of images from the in-vehicle camera 22. The image acquisition unit 11A sends the acquired plurality of images to the optimum value calculation unit 11E. The plurality of images are, for example, images taken at two different points.

[0034] The sensor information acquisition unit 11B acquires the wheel speed detected by the wheel speed sensor 20 and the steering angle detected by the steering angle sensor 21. The sensor information acquisition unit 11B sends the acquired wheel speed and steering angle to the odometry information calculation unit 11C.

[0035] The odometry information calculation unit 11C calculates odometry information indicating the movement amount of the vehicle based on the wheel speed and steering angle from the sensor information acquisition unit 11B. Specifically, the odometry information calculation unit 11C calculates the traveling distance of the vehicle based on the wheel speed and calculates the turning radius of the vehicle based on the steering angle.

[0036] FIG. 3 is a diagram showing an example of the change amount of the position and the change amount of the yaw angle of the vehicle according to the present embodiment.

[0037] As shown in FIG. 3, the odometry information calculation unit 11C calculates the position change amount (ΔX v , ΔY v ) and the yaw angle change amount Δθ v of the vehicle (reference point P1) in the vehicle coordinate system from the traveling distance and turning radius of the vehicle. The odometry information calculation unit 11C sends the calculated position change amount (ΔX v , ΔY v ) and the yaw angle change amount Δθ v to the initial value calculation unit 11D as the odometry information of the vehicle.

[0038] The initial value calculation unit 11D calculates the initial value of the homography matrix between a plurality of images based on the odometry information of the vehicle from the odometry information calculation unit 11C. Note that homography refers to projecting one plane onto another plane using projective transformation. Specifically, the initial value calculation unit 11D calculates the translation vector t v , ΔY v representing the position change of the in-vehicle camera 22 from the position change amount (ΔX c ), and calculates the rotation matrix R v representing the attitude change of the in-vehicle camera 22 from the yaw angle change amount Δθ c .

[0039] FIG. 4 is a diagram showing an example of the positional relationship between the in-vehicle camera 22 and the road surface according to the present embodiment.

[0040] As shown in FIG. 4, the in-vehicle camera 22 is installed at a height h from the road surface. When the vehicle moves from one point to another point, as odometry information, the above-described position change amounts (△X v , △Y v ), and the yaw angle change amount △θ v are calculated. Then, a translation vector t v , △Y v ) is calculated from the position change amounts (△X c ), and a rotation matrix R v is calculated from the yaw angle change amount △θ c . The road surface normal vector n is a vector in the normal direction of the road surface as viewed from the in-vehicle camera 22, and its magnitude is 1.

[0041] The initial value calculation unit 11D calculates an initial value G0 of the homography matrix using the following equation (1).

[0042] G0 = K(R c + t c n T / h)K -1 ···(1)

[0043] However, K represents the internal parameter matrix of the in-vehicle camera 22, R c represents a rotation matrix. t c represents a translation vector, n T represents the transposed matrix of the value n of the road surface normal vector, and h represents the installation height of the in-vehicle camera 22 from the road surface. Note that, for the internal parameter matrix K, the road surface normal vector n, and the installation height h, values obtained in advance by calibration of the in-vehicle camera 22 are used.

[0044] The initial value calculation unit 11D sends the initial value G0 calculated by the above equation (1) to the optimum value calculation unit 11E.

[0045] The optimal value calculation unit 11E repeatedly calculates the optimal value of the homography matrix from the initial value from the initial value calculation unit 11D and the luminance values of the pixels included in the road surface area specified for the plurality of images from the image acquisition unit 11A. Specifically, as an example, as shown in FIG. 5, a road surface area is specified in the image, and the optimal value G of the homography matrix is calculated using the ESM algorithm, which is an example of iterative calculation. OPT is calculated.

[0046] FIG. 5 is a diagram showing an example of corresponding road surface areas among a plurality of images according to the present embodiment. FIG. 5(A) shows an image before vehicle movement, and FIG. 5(B) shows an image after vehicle movement.

[0047] In the image before vehicle movement shown in FIG. 5(A), a road surface area R1 is specified. In the image after vehicle movement shown in FIG. 5(B), a road surface area R2 calculated using the initial value G0 of the homography matrix and a road surface area R3 calculated using the optimal value G of the homography matrix are included. OPT are included.

[0048] According to the example of FIG. 5, the initial value G0 of the homography matrix between two images is calculated from the odometry information of the vehicle, and the optimal value G of the homography matrix is calculated using the calculated initial value G0. That is, by calculating the initial value G0 of the homography matrix from the odometry information of the vehicle, the initial value G0 is set to a value closer to the optimal value G compared to the case where odometry information is not used. Therefore, the number of iterations of the iterative calculation can be reduced. The specific calculation method of the optimal value G of the homography matrix will be described later. OPT is calculated. That is, by calculating the initial value G0 of the homography matrix from the odometry information of the vehicle, the initial value G0 is set to a value closer to the optimal value G compared to the case where odometry information is not used. OPT For this reason, the number of iterations of the iterative calculation can be reduced. The specific calculation method of the optimal value G of the homography matrix will be described later. OPT will be described later.

[0049] The camera position and orientation calculation unit 11F decomposes the optimal value from the optimal value calculation unit 11E and calculates the change amount of the camera position and the change amount of the camera orientation for the in-vehicle camera 22. The change amount of the camera position is represented as an estimated value t of the translation vector est , and the change amount of the camera orientation is represented as an estimated value R of the rotation matrix estIt is represented as follows. Further, the camera position and orientation calculation unit 11F decomposes the optimal value from the optimal value calculation unit 11E to calculate an estimated value of the road surface normal vector. Specifically, the optimal value G OPT is decomposed as shown in the following formula (2).

[0050] G OPT =K(R est +t est n est T / h)K -1 ···(2)

[0051] However, K represents the internal parameter matrix of the in-vehicle camera 22, and R est represents a rotation matrix (estimated value) representing the change amount of the camera orientation. t est represents a translation vector (estimated value) representing the change amount of the camera position, and n est T represents the transposed matrix of the estimated value n est of the road surface normal vector, and h represents the installation height of the in-vehicle camera 22 from the road surface.

[0052] The utilization determination unit 11G determines to use the change amount of the camera position and the change amount of the camera orientation in a subsequent process (that is, the three-dimensional position calculation process) when the error represented by the angle between the estimated value n est of the road surface normal vector and the value n of the road surface normal vector obtained in advance by calibration of the in-vehicle camera 22 is less than the threshold value. On the other hand, when the above error is greater than or equal to the threshold value, the utilization determination unit 11G determines not to use the change amount of the camera position and the change amount of the camera orientation in the subsequent process. In the case of not using, the optimal value of the homography matrix is recalculated from the images taken at two different points. The threshold value can be set to an appropriate value in the range greater than 0 degrees and less than or equal to 5 degrees. Note that the utilization determination unit 11G is not essential, and a configuration that does not include the utilization determination unit 11G may be used. In this case, the change amount of the camera position and the change amount of the camera orientation calculated by the camera position and orientation calculation unit 11F are directly used in the subsequent process.

[0053] When the three-dimensional position calculation unit 11H determines that it is to be used by the usage determination unit 11G, it calculates the three-dimensional positions of the feature points in a plurality of images from the amount of change in the camera position and the amount of change in the camera orientation. Specifically, the three-dimensional position calculation unit 11H calculates the three-dimensional position of the feature point based on the principle of triangulation from the positions of the feature points associated between images at two different points, the amount of change in the camera position, and the amount of change in the camera orientation.

[0054] Next, with reference to FIG. 6, the operation of the map creation device 10 according to the present embodiment will be described.

[0055] FIG. 6 is a flowchart showing an example of the processing flow by the map creation program 15A according to the present embodiment.

[0056] First, when the map creation device 10 receives an instruction to start map creation processing, the map creation program 15A is started by the CPU 11, and the following steps are executed.

[0057] In step S101 of FIG. 6, the CPU 11 acquires, as an example, a plurality of images taken at two different points from the in-vehicle camera 22 as shown in FIG. 5 above.

[0058] In step S102, the CPU 11 acquires the wheel speed from the wheel speed sensor 20 and the steering angle from the steering angle sensor 21 as sensor information, respectively.

[0059] In step S103, the CPU 11 calculates, as an example, odometry information indicating the amount of movement of the vehicle based on the wheel speed and the steering angle acquired in step S102 as shown in FIG. 3 above. Specifically, the CPU 11 calculates the travel distance of the vehicle based on the wheel speed and calculates the turning radius of the vehicle based on the steering angle. Then, from the travel distance and the turning radius of the vehicle, the amount of change in the position (△X v , △Y v ) of the vehicle (reference point P1) in the vehicle coordinate system and the amount of change in the yaw angle △θ v are calculated.

[0060] In step S104, the CPU 11 calculates an initial value of the homography matrix between a plurality of images based on the odometry information of the vehicle calculated in step S103. Specifically, as an example, as shown in FIG. 4 described above, the CPU 11 calculates a translation vector t v representing the position change of the in-vehicle camera 22 from the position change amounts (△X v , △Y v ), calculates a rotation matrix R c representing the attitude change of the in-vehicle camera 22 from the yaw angle change amount △θ v c , and calculates an initial value G0 of the homography matrix using the above formula (1). v , calculates a rotation matrix R c representing the attitude change of the in-vehicle camera 22 from the yaw angle change amount △θ v c , and calculates an initial value G0 of the homography matrix using the above formula (1).

[0061] In step S105, the CPU 11 repeatedly calculates the optimal value of the homography matrix from the initial value calculated in step S104 and the luminance values of each pixel included in the specified road surface area for the plurality of images acquired in step S101, as an example, as shown in FIG. 5 described above. Here, with reference to FIGS. 7 and 8, the homography matrix optimal value calculation process in step S105 will be specifically described.

[0062] FIG. 7 is a flowchart showing an example of the flow of the homography matrix optimal value calculation process according to the present embodiment, and is a subroutine of step S105 in FIG. 6. Further, FIG. 8 is a diagram showing an example of the image I * before movement and the tracking area.

[0063] In step S111 of FIG. 7, the CPU 11 designates a tracking area (synonymous with the road surface area) for the image I * before movement, as an example, as shown in FIG. 8, and calculates a luminance gradient matrix J I* I* , and Jacobian matrices J W W , J G G .

[0064] Specifically, as shown in FIG. 8, the number of pixels in the tracking area is set to n (= n u × n v ). The luminance gradient matrix J I* I* is the image I *The luminance (value from 0 to 255) of each pixel in the tracking area is calculated using the following formula.

[0065] JPEG0007714436000001.jpg3946

[0066] However,

[0067] JPEG0007714436000002.jpg1093

[0068] is. J I*ui represents the horizontal luminance gradient of the i-th pixel, and J I*vi represents the vertical luminance gradient of the i-th pixel.

[0069] The Jacobian matrix J W is calculated from the coordinates of each pixel in the tracking area in the image I * before movement using the following formula.

[0070] JPEG0007714436000003.jpg4145

[0071] However, the coordinates of each pixel in the tracking area are

[0072] JPEG0007714436000004.jpg1356

[0073] represented as. At this time,

[0074] JPEG0007714436000005.jpg3868

[0075] is.

[0076] The Jacobian matrix J G is calculated from the basis A i (i = 1 to 8) using the following formula.

[0077] JPEG0007714436000006.jpg779

[0078] Here, [A i v is a 9-row and 1-column vector rearranged row by row.

[0079] JPEG0007714436000007.jpg53164

[0080] In step S112, the CPU 11 substitutes the initial value G0 into the estimated value G^ of the homography matrix (^ is directly above G, the same hereinafter), and substitutes 1 into the iteration count (number of repetitions) n ite for 1.

[0081] In step S113, the CPU 11 calculates the luminance gradient matrix J I of the tracking area in the image I after movement.

[0082] Specifically, the coordinates in the image I after movement are calculated by the following formula.

[0083] JPEG0007714436000008.jpg27114

[0084] However, the coordinates in the image I after movement are

[0085] JPEG0007714436000009.jpg1281

[0086] represented as.

[0087] The luminance gradient matrix J I is calculated from the luminance of each pixel in the tracking area in the image I after movement by the following formula.

[0088] JPEG0007714436000010.jpg3938

[0089] However,

[0090] ​ JPEG0007714436000011.jpg1085

[0091] It is. J Iui indicates the horizontal luminance gradient of the i-th pixel, and J Ivi indicates the vertical luminance gradient of the i-th pixel.

[0092] In step S114, the CPU 11 calculates the parameter x (an 8-row 1-column vector) of the homography matrix.

[0093] Specifically, the parameter x is calculated by the following formula.

[0094] JPEG0007714436000012.jpg1247

[0095] Here, J esm is the Jacobian matrix and is calculated by the following formula.

[0096] JPEG0007714436000013.jpg1585

[0097] On the other hand, y is the luminance difference vector and is expressed by the following formula.

[0098] JPEG0007714436000014.jpg1280

[0099] Here, yi is the luminance I of the i-th pixel after movement i and the luminance I of the i-th pixel before movement i * and is calculated by the following formula.

[0100] JPEG0007714436000015.jpg1543

[0101] In step S115, the CPU 11 updates the estimated value Ĝ of the homography matrix by the following formula.

[0102] JPEG0007714436000016.jpg3045

[0103] Set the above G to a new G^.

[0104] In step S116, the CPU 11 determines whether the end condition is satisfied, that is, whether iteration (repetition) is required. If the end condition is satisfied, that is, if it is determined that iteration (repetition) is not required (in the case of an affirmative determination), the process proceeds to step S117. If the end condition is not satisfied, that is, if it is determined that iteration (repetition) is required (in the case of a negative determination), the process returns to step S113 and the process is repeated.

[0105] Specifically, let the root mean square of the square of the current luminance difference be y curr Then, y curr is represented by the following equation.

[0106] JPEG0007714436000017.jpg4373

[0107] Let the upper limit number of iterations be n max (for example, 100), and let the convergence determination threshold be ε (for example, 10 -5 ).

[0108] n ite = 1, substitute the root mean square of the square of the current luminance difference y prev into the root mean square of the square of the previous luminance difference y curr , add 1 to the number of iterations n ite and return to step S113.

[0109] 1 < n ite < n max If, y prev - y curr > ε, it is determined that convergence has not occurred, and the root mean square of the square of the current luminance difference y prev is substituted into the root mean square of the square of the previous luminance difference y curr and the number of iterations nite Add 1 to it and return to step S113. On the other hand, y prev -y curr If it is ≤ ε, it is determined that convergence has occurred, and the process proceeds to step S117.

[0110] n ite =n max In the case of, the process proceeds to step S117.

[0111] In step S117, the CPU 11 adopts the estimated value G^ of the homography matrix as the optimal value G OPT and returns to step S106 in FIG. 6.

[0112] Returning to FIG. 6, in step S106, the CPU 11, as an example, uses the above formula (2) to decompose the optimal value G calculated in step S105 to calculate the change amount of the camera position, the change amount of the camera attitude, and the estimated value of the road surface normal vector. OPT

[0113] In step S107, the CPU 11 determines whether the change amount of the camera position and the change amount of the camera attitude can be used. If it is determined that they can be used (in the case of an affirmative determination), the process proceeds to step S108. If it is determined that they cannot be used (in the case of a negative determination), the process returns to step S101 and the process is repeated. Specifically, the CPU 11 determines that the change amount of the camera position and the change amount of the camera attitude can be used in a subsequent process when the error represented by the angle between the estimated value n of the road surface normal vector and the value n of the road surface normal vector obtained in advance by calibration of the in-vehicle camera 22 is less than the threshold value. On the other hand, when the above error is greater than or equal to the threshold value, it is determined that the change amount of the camera position and the change amount of the camera attitude cannot be used in a subsequent process. If they are not used, the optimal value of the homography matrix is recalculated from the images taken at two different points. est

[0114] ​​In step S108, the CPU 11 calculates the three-dimensional positions of the feature points in the plurality of images from the amount of change in the camera position and the amount of change in the camera orientation, and ends a series of processes by the local map creation program 15A. Specifically, the CPU 11 calculates the three-dimensional position of the feature point based on the principle of triangulation from the positions of the feature points associated between images at two different points, the amount of change in the camera position, and the amount of change in the camera orientation.

[0115] As described above, according to this embodiment, by calculating the initial value of the homography matrix from the vehicle's odometry information, the iterative calculation can start from a value close to the optimal value of the homography matrix. Therefore, the number of iterations of the iterative calculation is reduced.

[0116] Also, when the road surface texture (pattern) is scarce, the error of the homography matrix becomes large, so the error of the road surface normal obtained from the homography matrix may also become large. The error of the road surface normal is obtained by comparing it with the road surface normal obtained from the camera installation direction. When the error of the road surface normal is large, it is determined that the obtained amount of change in the camera position and the amount of change in the camera orientation cannot be used, and the homography matrix can be recalculated from the images taken at two other points.

[0117] In each of the above embodiments, the processor refers to a processor in a broad sense, including a general-purpose processor (for example, CPU: Central Processing Unit, etc.) and a dedicated processor (for example, GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0118] Also, the operations of the processor in each of the above embodiments may be performed not only by a single processor but also by a plurality of physically separated processors cooperating with each other. Also, the order of each operation of the processor is not limited to the order described in each of the above embodiments, and may be changed as appropriate.

[0119] As described above, the map creation device according to the embodiment has been illustrated and described. The embodiment may be in the form of a program for causing a computer to execute the functions of each part included in the map creation device. The embodiment may be in the form of a computer-readable non-transitory storage medium storing these programs.

[0120] In addition, the configuration of the map creation device described in the above embodiment is an example, and may be changed according to the situation within the scope not departing from the gist.

[0121] Also, the flow of the program processing described in the above embodiment is an example, and unnecessary steps may be deleted, new steps may be added, or the processing order may be changed within the scope not departing from the gist.

[0122] Also, in the above embodiment, the case where the processing according to the embodiment is realized by software configuration using a computer by executing a program has been described, but the present invention is not limited to this. The embodiment may be realized by, for example, a hardware configuration or a combination of a hardware configuration and a software configuration.

Explanation of Reference Numerals

[0123] 10 Map creation device 11 CPU 11A Image acquisition unit 11B Sensor information acquisition unit 11C Odometry information calculation unit 11D Initial value calculation unit 11E Optimal value calculation unit 11F Camera position and orientation calculation unit 11G Utilization determination unit 11H Three-dimensional position calculation unit 12 ROM 13 RAM 14 I / O 15 Memory unit 15A Map creation program 16 External I / F 20 Wheel speed sensor 21 Steering angle sensor 22 In-vehicle camera 100 Map creation system

Claims

1. An image acquisition unit that is mounted on a vehicle and acquires a plurality of images captured at different points from an in-vehicle camera that captures the periphery of the vehicle; An odometry information calculation unit that calculates odometry information indicating the movement amount of the vehicle; An initial value calculation unit that calculates an initial value of a homography matrix between the plurality of images from the odometry information of the vehicle; An optimal value calculation unit that repeatedly calculates an optimal value of the homography matrix from the initial value and the luminance values of each pixel included in a road surface area specified for the plurality of images; A camera position and orientation calculation unit that decomposes the optimal value to calculate a change amount of the camera position and a change amount of the camera orientation for the in-vehicle camera, and decomposes the optimal value to calculate an estimated value of a road surface normal vector that is a vector in the normal direction of the road surface viewed from the in-vehicle camera; A utilization determination unit that determines to use the change amount of the camera position and the change amount of the camera orientation calculated by the camera position and orientation calculation unit when an error represented by an angle between the estimated value of the road surface normal vector and a value of the road surface normal vector obtained in advance by calibration of the in-vehicle camera is less than a threshold; A three-dimensional position calculation unit that calculates the three-dimensional positions of feature points in the plurality of images from the change amount of the camera position and the change amount of the camera orientation when determined to be used by the utilization determination unit; A map creation device comprising the above.

2. The utilization determination unit determines not to use the change amount of the camera position and the change amount of the camera orientation when the error is greater than or equal to the threshold. The map creation device according to Claim 1.

3. Acquire a plurality of images captured at different points from an in-vehicle camera that is mounted on a vehicle and captures the periphery of the vehicle, Calculate odometry information indicating the movement amount of the vehicle, Calculate an initial value of a homography matrix between the plurality of images from the odometry information of the vehicle, Repeatedly calculate an optimal value of the homography matrix from the initial value and the luminance values of each pixel included in a road surface area specified for the plurality of images, Decompose the optimal value to calculate a change amount of the camera position and a change amount of the camera orientation for the in-vehicle camera, and decompose the optimal value to calculate an estimated value of a road surface normal vector that is a vector in the normal direction of the road surface viewed from the in-vehicle camera, When it is determined to use the calculated amount of change in the camera position and the amount of change in the camera attitude when the error represented by the angle between the estimated value of the road surface normal vector and the value of the road surface normal vector obtained in advance by calibration of the in-vehicle camera is less than the threshold value, When it is determined to use, calculating the three-dimensional positions of the feature points in the plurality of images from the amount of change in the camera position and the amount of change in the camera attitude, A map creation method executed by a computer.

4. A plurality of images taken at different points are acquired from an in-vehicle camera mounted on a vehicle and photographing the periphery of the vehicle, Odometry information indicating the amount of movement of the vehicle is calculated, An initial value of the homography matrix between the plurality of images is calculated from the odometry information of the vehicle, The optimal value of the homography matrix is calculated by iterative calculation from the initial value and the luminance values of the respective pixels included in the road surface area specified for the plurality of images, The optimal value is decomposed to calculate the amount of change in the camera position and the amount of change in the camera attitude for the in-vehicle camera, and the optimal value is decomposed to calculate an estimated value of the road surface normal vector, which is a vector in the normal direction of the road surface as viewed from the in-vehicle camera, When it is determined to use the calculated amount of change in the camera position and the amount of change in the camera attitude when the error represented by the angle between the estimated value of the road surface normal vector and the value of the road surface normal vector obtained in advance by calibration of the in-vehicle camera is less than the threshold value, When it is determined to use, calculating the three-dimensional positions of the feature points in the plurality of images from the amount of change in the camera position and the amount of change in the camera attitude, A map creation program for causing a computer to execute.

Images