Line sensor camera position and orientation estimation device and position and orientation estimation method

Abstract

To provide a position and attitude estimating apparatus and a position and attitude estimating method for a line sensor camera capable of accurately estimating a position and an attitude of the line sensor camera using a photographing target which is simply installed and whose coordinates are unknown.SOLUTION: The position and attitude estimating apparatus includes: a line sensor camera 3; a photographing target 5 having a plurality of markers and photographed by the line sensor camera 3; a first distance measuring instrument 7 and a second distance measuring instrument 9 disposed at different positions from each other and measuring a distance to the photographing target 5 based on reference coordinates; and an estimating apparatus body 11 for estimating a position and an attitude of the line sensor camera 3 based on image data photographed by the line sensor camera 3 and distance measuring data measured by the first distance measuring instrument 7 and the second distance measuring instrument 9.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a position and orientation estimation device and a position and orientation estimation method for a line sensor camera. [Background technology]

[0002] When performing various image analyses using a line sensor camera, it is necessary to determine and calibrate the internal parameters of the line sensor camera, such as the focal length and distortion coefficient of the lens, as well as external parameters such as the position and optical axis orientation of the line sensor camera. Furthermore, in order to apply it to three-dimensional measurement (stereo vision) using images, it is necessary to determine the position and orientation of the line sensor camera in advance.

[0003] In the calibration method for a line sensor camera described in Patent Document 1, an L-shaped 3D marker with known coordinates is photographed, and initial internal and external parameters are input to an evaluation function which is obtained by adding an error function that takes lens distortion into account and a value that takes internal parameters into account. Subsequently, for all detected points and all images, each parameter is adjusted so that the evaluation function, which is obtained by adding the sum of the error functions and a value that takes internal parameters into account, is minimized, and the internal and external parameters are calculated by a nonlinear solution method.

[0004] Patent Document 2 describes a method for adjusting the optical axis of a line sensor camera, in which an adjustment image is displayed on two display devices positioned at different locations in the front-to-back and left-to-right directions, and this image is captured by two line sensor cameras positioned at different locations in the left-to-right directions. The optical axis is then adjusted by adjusting the position of the displayed image and the position and angle of the line sensor cameras. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2016-218815 [Patent Document 2] Japanese Patent Publication No. 2019-20314 [Overview of the project] [Problems that the invention aims to solve]

[0006] The calibration method described in Patent Document 1 assumes that the coordinates of the L-shaped 3D marker are known, requiring that the coordinates of the 3D marker be determined in advance. In this case, there is a risk that the method may not be applicable when using a simple shooting target based on arbitrary reference coordinates. The optical axis adjustment method described in Patent Document 2 adjusts the optical axis based on the image display, but the position and orientation of the line sensor camera are not calculated during this process.

[0007] This invention has been made in view of the circumstances described above, and the problem that this invention aims to solve is to provide a line sensor camera position and orientation estimation device and method that can accurately estimate the position and orientation of a line sensor camera using a shooting target that is simply installed and whose coordinates are unknown. [Means for solving the problem]

[0008] To solve the above problems, the present invention employs the following means. In other words, the line sensor camera position and orientation estimation device of the present invention comprises a line sensor camera, a target being photographed by the line sensor camera having a plurality of markers, a first distance measuring device and a second distance measuring device arranged at different positions from each other and measuring the distance to the target being photographed based on reference coordinates, and an estimation device body that estimates the position and orientation of the line sensor camera based on image data captured by the line sensor camera and distance measurement data measured by the first distance measuring device and the second distance measuring device, wherein the estimation device body comprises a measurement coordinate calculation unit that calculates the image coordinates of the target being photographed from the image data, a true value coordinate calculation unit that calculates the true value coordinates of the target being photographed in the reference coordinates from the distance measurement data, and a nonlinear solution unit that calculates the position and orientation of the line sensor camera from the image coordinates and the true value coordinates using a nonlinear solution method.

[0009] The line sensor camera position and orientation estimation device of the present invention measures the distance to the target using a first distance measuring device and a second distance measuring device, calculates the true coordinates of the target in reference coordinates from this distance measurement data, and calculates the position and orientation of the line sensor camera based on these true coordinates. Therefore, it is possible to estimate the position and orientation of the line sensor camera using a simply installed target whose coordinates are unknown.

[0010] The method for estimating the position and orientation of a line sensor camera according to the present invention includes: photographing a target having a plurality of markers with a line sensor camera; measuring the distance to the target based on reference coordinates using a first distance measuring device and a second distance measuring device arranged at different positions from each other; and estimating the position and orientation of the line sensor camera based on image data captured by the line sensor camera and distance measurement data measured by the first distance measuring device and the second distance measuring device, wherein the estimation of the position and orientation of the line sensor camera includes: calculating the image coordinates of the target from the image data; calculating the true coordinates of the target at the reference coordinates from the distance measurement data; and calculating the position and orientation of the line sensor camera from the image coordinate data and the true coordinate data using a nonlinear solution method.

[0011] The line sensor camera position and orientation estimation method of the present invention measures the distance to the target using a first distance measuring device and a second distance measuring device, calculates the true coordinates of the target in reference coordinates from this distance measurement data, and calculates the position and orientation of the line sensor camera based on these true coordinates. Therefore, the position and orientation of the line sensor camera can be estimated using a simply installed target whose coordinates are unknown. [Effects of the Invention]

[0012] According to the present invention, a line sensor camera position and orientation estimation device and method can be provided that can accurately estimate the position and orientation of a line sensor camera using a shooting target that is simply installed and whose coordinates are unknown. [Brief explanation of the drawing]

[0013] [Figure 1] This is a schematic diagram showing the configuration of a line sensor camera position and orientation estimation device in an embodiment of the present invention. [Figure 2]It is a schematic diagram showing the configuration of a position and orientation estimation device for a line sensor camera in an embodiment of the present invention. [Figure 3] It is a schematic diagram showing the configuration of a position and orientation estimation device for a line sensor camera in an embodiment of the present invention. [Figure 4] It is a flowchart for explaining the operation of a position and orientation estimation device for a line sensor camera in an embodiment of the present invention. [Figure 5] It is a functional block diagram of an estimation device main body of a position and orientation estimation device for a line sensor camera in an embodiment of the present invention. [Figure 6] It is a schematic diagram for explaining the calculation of true value coordinates of a position and orientation estimation device for a line sensor camera in an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. (First Embodiment) FIG. 1 is a schematic diagram showing the configuration of a position and orientation estimation device for a line sensor camera in an embodiment of the present invention. As shown in FIG. 1, the position and orientation estimation device 1 for a line sensor camera in the present embodiment includes a line sensor camera 3, a photographing target 5, a first distance measuring device 7, a second distance measuring device 9, and an estimation device main body 11. The line sensor camera 3, the photographing target 5, the first distance measuring device 7, the second distance measuring device 9, and the estimation device main body 11 are connected by wire or wirelessly and are capable of transmitting and receiving data to and from each other.

[0015] The first distance measuring device 7 and the second distance measuring device 9 are positioned at different locations on the X-axis in the reference coordinate system shown in Figure 1, which is defined by the X-axis and Y-axis from the origin O(0,0). As shown in the figure, the shooting target 5 is a rod-shaped body with multiple black and white markers, and the spacing between each black and white marker is measured in advance and is known. The shooting target 5 is positioned within the plane of this two-dimensional reference coordinate system, such that the center c of the shooting target 5 is located at an arbitrary position on the Y-axis of the reference coordinate system. In this case, it is preferable that the shooting target 5 is positioned so as to be approximately parallel to the X-axis within the plane of the reference coordinate system.

[0016] The measurement directions of the first distance measuring device 7 and the second distance measuring device 9 to the target shooting target 5 are set to be parallel to the Y-axis of the reference coordinate system. In other words, the first distance measuring device 7 and the second distance measuring device 9 measure the distance from the X-axis of the reference coordinate system in which they are positioned to the target shooting target 5. The method for measuring the distance from the X-axis of the reference coordinate system to the first distance measuring device 7 and the second distance measuring device 9 is not particularly limited, but for example, laser distance measuring devices can be used.

[0017] The line sensor camera 3 can be installed at any position where the target 5 can be photographed. The image data of the target 5 captured by the installed line sensor camera 3, and the distance measurement data from the X-axis of the reference coordinates to the target 5 measured by the first distance measuring instrument 7 and the second distance measuring instrument 9 are sent to the estimation device body 11. Based on this data, the estimation device body 11 estimates the position and orientation of the line sensor camera 3.

[0018] Figure 5 is a functional block diagram of the estimation device body 11. As shown in Figure 5, the estimation device body 11 of this embodiment includes a storage unit 13, an input unit 15, a measurement coordinate calculation unit 17, a true value coordinate calculation unit 19, and a nonlinear solution unit 21. The input unit 15 receives image data captured by the line sensor camera 3 and distance measurement data measured by the first distance measuring instrument 7 and the second distance measuring instrument 9, and sends it to the storage unit 13 for storage. The measurement coordinate calculation unit 17 calculates the image coordinates of the marker of the shooting target 5 from the image data, and sends the calculated marker image coordinates as image coordinate data to the storage unit 13 for storage. The true value coordinate calculation unit 19 calculates the true value coordinates of the marker at the reference coordinates of the shooting target 5 from the distance measurement data, and sends the calculated true value coordinates as true value coordinate data to the storage unit 13 for storage. The nonlinear solution unit 21 calculates the position and orientation of the line sensor camera 3 from the image coordinate data and the true coordinate data using a nonlinear solution method, and sends the calculated position and orientation to the storage unit 13 for storage as camera position parameters. Here, the position of the line sensor camera 3 is the position of the origin of the optical axis of the lens of the line sensor camera 3 in the reference coordinate system, and the orientation of the line sensor camera 3 is the angle expressed using the rotation matrix of the reference coordinate system to represent the direction of the optical axis.

[0019] The estimation device body 11 is, for example, an information processing device such as a personal computer. The input unit 15, the measurement coordinate calculation unit 17, the true value coordinate calculation unit 19, and the nonlinear solution unit 21 may be software or programs executed by the CPU or GPU of this information processing device. The storage unit 13 may be a storage device such as a hard disk or flash memory located inside or outside this information device. The estimation device body 11 functionally possesses the configuration described above by having the information processing device execute processing based on a pre-set program or the like.

[0020] Figure 4 is a flowchart illustrating the operation of the line sensor camera position and orientation estimation device 1, configured as described above. The operation of the line sensor camera position and orientation estimation device 1 of this embodiment will be explained with reference to this flowchart. The following are the operations of each of steps S01 - S04. (S01) The imaging target 5 is installed at one or a plurality of different positions on the Y-axis of the reference coordinates, and each is imaged by the line sensor camera 3. (S02) Perform black-and-white marker detection processing of the imaging target 5 from the image data obtained in S01, and calculate the black-and-white marker coordinates (image coordinates) on the image. (S03) The distances to the imaging target 5 are measured by the first distance measuring device 7 and the second distance measuring device 9 arranged on the X-axis of the reference coordinates with different positions, and based on this distance measurement information, the coordinates (true value coordinates) of the black-and-white marker in the reference coordinates are calculated. (S04) Based on the image coordinates and the true value coordinates, the camera position parameters (the position and orientation of the line sensor camera) from the reference coordinates are calculated by the non-linear solution method in Patent Document 1.

[0021] Next, the processing of step S03 described above will be explained. FIG. 6 is a schematic diagram for explaining the calculation of the true value coordinates in step S03. The calculation of the true value coordinates X n , Y n will be explained below. (1) In the true value coordinate calculation process, the following values are used. Y L1 : The distance from the right distance measuring device (the second distance measuring device 9) to the imaging target 5 Y L2 : The distance from the left distance measuring device (the first distance measuring device 7) to the imaging target 5 X L1 : The installation position of the right distance measuring device (the distance from the origin O of the reference coordinates) of the second distance measuring device 9 X L2 : The installation position of the left distance measuring device (the distance from the origin O of the reference coordinates) of the first distance measuring device 7 W m : The width between the white band markers of the target M C : The marker number at the center of the target M n : The marker sequence number for coordinate calculation (numbered in order from the left end of the imaging target 5)

[0022] (2) Calculate each value using the following formula. • Position P within the target of the marker to be used for coordinate calculation n (Distance from the center) Note: P c The (center position of the shooting target) is the reference position, so its value is "0".

[0023]

number

[0024] • The distance between the distance measuring devices installed on the left and right sides W

[0025]

number

[0026] • Height difference (H) between the left and right distance measuring devices and the target. Note: In the following [Equation 3], the upper equation represents the case where the right side is high, and the lower equation represents the case where the right side is low.

[0027]

number

[0028] • Height Y of the target's central point reference c Note: In [Equation 4] below, the upper equation is for the case where the right side is higher, and the lower equation is for the case where the right side is lower. If the left and right heights are the same, the target is horizontal, and either the left or right height is adopted. In this embodiment, for simplicity, the first distance measuring device 7 and the second distance measuring device 9 are assumed to be positioned equidistant from the origin O of the reference coordinate system.

[0029]

number

[0030] • The slope β of the target is determined using the inverse tangent function based on the length and width.

[0031]

number

[0032] • Marker coordinates (X n ,Y n The true coordinates are calculated. The true coordinates of the marker on target 5 can be obtained using the following formula [Equation 6].

[0033]

number

[0034] Next, we will explain the process of step S04 in the flowchart of Figure 4. In step S04, the position and orientation of the line sensor camera 3 are calculated using the nonlinear solution method of Patent Document 1. Here, we will only explain the core elements of the nonlinear solution method of Patent Document 1.

[0035] Here, the mathematical model used for the nonlinear solution is shown in equation [Equation 7] below. In Figure 1, since the line sensor camera 3 only captures one line, the imaging range becomes the reference coordinate plane in three-dimensional space. As mentioned above, the marker coordinates of the target 5 on this reference coordinate plane are expressed as true coordinates obtained in step S03 above, using the X and Y axes of the reference coordinates. The camera coordinate system (one-dimensional image coordinate system) of the line sensor camera 3 is represented by the u axis, which is orthogonal to the axis w in the direction of the optical axis of the line sensor camera 3 and parallel to the reference coordinate plane of the imaging range. Furthermore, the rotation matrix between the reference coordinate system and the camera coordinate system is R (with one axis of rotation and an angle of θ), and the translation vector is t (a two-dimensional vector with elements t1 and t2).

[0036]

number

[0037] Here, s is the scaling factor, f is the focal length, c is the principal point coordinate, u is the position in the camera coordinate system as actually observed (measured coordinate in pixel coordinates), and X and Y are the positions in the reference coordinate system (true coordinates of the reference coordinate system of the target being photographed).

[0038] Equation [Equation 7] above is a mathematical model for converting from reference coordinates (2D plane coordinates) to camera coordinates (1D image coordinates) of line sensor camera 3. However, equation [Equation 7] above does not take into account the distortion of the lens of line sensor camera 3. Therefore, by defining (x,y) as in equation [Equation 8] below, the distortion can be represented as in equations [Equation 9] to [Equation 11] below. Note that k1 to k3 are radial distortion coefficients.

[0039]

number

number

number

number

[0040] We want to expand the above equations [Equation 7] to [Equation 11], but since we cannot directly substitute x'' into equation [Equation 11], we substitute the following equation [Equation 12] into equation [Equation 10]. Substituting equations [Equation 7] to [Equation 9] into equation [Equation 10] results in the following equation [Equation 13].

[0041]

number

number

[0042] When performing calibration with this camera model, the goal is to minimize the error in the unit focal length plane, and the internal and external parameters are determined using the Levenberg-Marquardt method so that the following equation [Equation 14] is minimized.

[0043]

number

[0044] Here, in equation [Equation 14] above, M is the number of times an image is captured, and N is the number of feature points acquired in each capture (the number of detected markers on the target 5). Equation [Equation 14] above is the sum of the error function based on equation [Equation 13] above over the total number of captured images M and the total number of detected points N. The position and orientation of the line sensor camera 3 can be calculated from the translation vector t and rotation matrix R obtained by the Levenberg-Marquard method such that equation [Equation 14] is minimized.

[0045] As described above, the line sensor camera position and orientation estimation device and method in this embodiment acquire the image coordinates of a marker from an image of the shooting target 5 captured by the line sensor camera 3. To obtain the true coordinates of the shooting target 5, the distance to the shooting target 5 is measured using the first distance measuring instrument 7 and the second distance measuring instrument 9. The true coordinates of the shooting target 5 in reference coordinates are calculated from this distance measurement data. Based on these true coordinates and the image coordinates of the marker, the position and orientation of the line sensor camera 3 are calculated using a nonlinear solution method. Therefore, the position and orientation of the line sensor camera 3 at any position can be easily and accurately estimated using a simply installed shooting target 5 whose coordinates are unknown. For estimating the position and orientation of the line sensor camera 3, multiple arrangements of the shooting target 5 can be set and estimation can be performed on multiple captured images, so it is possible to improve the accuracy of the estimation as needed. Since the distance to the shooting target 5 is measured using the first distance measuring instrument 7 and the second distance measuring instrument 9, the device configuration simplifies the installation conditions of the shooting target 5.

[0046] (Second Embodiment) Figure 2 is a schematic diagram showing the configuration of the line sensor camera position and orientation estimation device 10 according to this embodiment. This embodiment differs from the first embodiment in that, in addition to line sensor camera 3, line sensor camera 3a is added to the line sensor camera used to estimate position and orientation. Other components common to the first embodiment are denoted by the same reference numerals in the figure and their descriptions are omitted. In this embodiment, the process of estimating the position and orientation of line sensor camera 3 in the first embodiment is also performed on line sensor camera 3a, and the positions and orientations of the two line sensor cameras 3 and 3a are acquired. In this embodiment, in addition to the effects of the first embodiment, the relative positions of the two cameras can be determined from the positions and orientations of the two line sensor cameras 3 and 3a, so it can be applied to stereo vision (three-dimensional measurement using images).

[0047] (Third embodiment) Figure 3 is a schematic diagram showing the configuration of the line sensor camera position and orientation estimation device 20 according to this embodiment. This embodiment differs from the first embodiment in that the line sensor cameras used to estimate position and orientation are replaced by line sensor cameras 31, 32, ..., 3 instead of line sensor camera 3. n The point is that n cameras are arranged. Other components common to the first embodiment are denoted by the same reference numerals in the figure and their descriptions are omitted. In this embodiment, the process of estimating the position and orientation of line sensor camera 3 in the first embodiment is performed for line sensor cameras 31, 32, ..., 3 n This was implemented on n units, and the n line sensor cameras 31, 32, ..., 3 n The position and orientation are obtained. In this embodiment, in addition to the effects of the first embodiment, line sensor cameras 31, 32, ..., 3 n By determining the relative positions of the cameras from the positions and orientations of the n cameras, it is possible to apply this to multi-view stereo, which enables more accurate three-dimensional reconstruction. [Explanation of Symbols]

[0048] 1, 10, 20 Line sensor camera position and orientation estimation device 3, 3a, 31...3n Line Sensor Camera 5. Target to photograph 7. First distance measuring device 9. Second distance measuring device 11 Estimation device main body 17 Measurement Coordinate Calculation Unit 19 True Coordinate Calculation Unit 21 Nonlinear Solution Section

Claims

1. Line sensor camera and A target having multiple markers and being photographed by the line sensor camera, A first distance measuring device and a second distance measuring device are arranged at different positions from each other and measure the distance to the target being photographed based on reference coordinates. The system includes an estimation device body that estimates the position and orientation of the line sensor camera based on image data captured by the line sensor camera and distance measurement data measured by the first distance measuring device and the second distance measuring device. The estimation device body is, A measurement coordinate calculation unit that calculates the image coordinates of the target being photographed from the image data, A true coordinate calculation unit calculates the true coordinates of the target object in the reference coordinates from the distance measurement data, A nonlinear solution unit calculates the position and orientation of the line sensor camera from the image coordinates and the true value coordinates using a nonlinear solution method. A position and orientation estimation device for a line sensor camera, equipped with the necessary components.

2. A position and attitude estimation device, To photograph a target having multiple markers using a line sensor camera, The distance to the target being photographed is measured based on reference coordinates using a first distance measuring device and a second distance measuring device, which are positioned at different locations from each other. This includes estimating the position and orientation of the line sensor camera based on image data captured by the line sensor camera and distance measurement data measured by the first distance measuring device and the second distance measuring device, Estimating the position and orientation of the line sensor camera is: Calculating the image coordinates of the target to be photographed from the aforementioned image data, To calculate the true coordinates of the target object in the reference coordinate system from the distance measurement data. The position and orientation of the line sensor camera are calculated from the image coordinates and the true value coordinates using a nonlinear solution method. A method for estimating the position and orientation of a line sensor camera, including the above.

Images