Image processing system

Abstract

To provide an image processing system that more appropriately combines images captured by a plurality of cameras even when a plurality of vehicles rotate relative to each other.SOLUTION: An image processing system that is communicatively connected to a mobile device having a first camera and a second camera provided on at least two vehicles having a connection portion includes: an image acquisition unit that acquires a first image captured by a first camera and a second image captured by a second camera; a measurement information acquisition unit that acquires a rotation amount of at least one of the connecting portions; and a synthesis processing unit that, on the basis of inter-camera external parameters indicating a relative position and a relative orientation between the first camera and the second camera, a position and an orientation of a rotation axis of a connecting portion located between the first camera and the second camera, and a rotation amount acquired for the connection portion located between the first camera and the second camera, synthesizes the first image and the second image.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a technique for synthesizing captured images. [Background technology]

[0002] Conventionally, a mobile device in which a plurality of vehicles are connected to each other so as to be capable of relative rotation has been known. Such a mobile device is used for inspecting facilities that cannot be easily accessed by workers, and each of the plurality of vehicles is equipped with a camera. Images captured by the plurality of cameras are synthesized into one composite image, which allows the worker to easily check the condition of the inspection target in a wide field of view.

[0003] As a related technique, an image processing method is known that harmonizes images captured by a first camera and a second camera covering the same road space at different times (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2023-31307 A Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present invention is to provide a technique for more appropriately combining images captured by a plurality of cameras even when a plurality of vehicles rotate relative to one another. [Means for solving the problem]

[0006] One embodiment is an image processing device communicatively connected to a mobile device having at least two vehicles, at least one coupling part that couples the at least two vehicles in the direction of travel and is rotatable around a rotation axis, and a first camera and a second camera that are provided on different vehicles, and the image acquisition part acquires a first captured image captured by the first camera and a second captured image captured by the second camera, a measurement information acquisition part acquires an amount of rotation of the at least one coupling part, and a synthesis processing part that synthesizes the first captured image and the second captured image based on the relative position and relative attitude between the first camera and the second camera, inter-camera external parameters indicating the position and orientation of the rotation axis of the coupling part located between the first camera and the second camera, and the amount of rotation acquired for the coupling part located between the first camera and the second camera. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic plan view showing a configuration of a moving device according to an embodiment. [Diagram 2] 1 is a schematic side view showing a configuration of a moving device according to an embodiment. [Diagram 3] FIG. 1 is a block diagram showing a configuration of a control system according to an embodiment. [Figure 4] 1 is a block diagram showing a hardware configuration of an image processing apparatus according to an embodiment; [Diagram 5] 1 is a block diagram showing a functional configuration of an image processing apparatus according to an embodiment; [Figure 6] FIG. 2 is a diagram illustrating a world coordinate system according to the embodiment. [Figure 7] FIG. 2 is a diagram illustrating a camera coordinate system according to the embodiment. [Figure 8] FIG. 2 is a diagram illustrating a moving device coordinate system according to an embodiment. [Figure 9] FIG. 2 is a diagram showing a pinhole camera model according to an embodiment. [Figure 10] 5A and 5B are diagrams showing marker patterns according to the embodiment; [Figure 11]6 is a flowchart showing an operation of a first adjustment process according to the embodiment. [Figure 12] 10 is a flowchart showing an operation of a second adjustment process according to the embodiment. [Figure 13] FIG. 2 is a diagram illustrating a model space according to the embodiment. [Figure 14] FIG. 13 is a diagram showing a captured image to which no mask processing is performed according to the embodiment. [Figure 15] FIG. 13 is a diagram showing a captured image on which mask processing has been performed according to the embodiment. [Figure 16] 10 is a flowchart showing the operation of a synthesis process according to the embodiment. [Figure 17] 11A and 11B are diagrams illustrating a composite image using rotation information according to the embodiment. [Figure 18] FIG. 13 is a diagram showing a composite image without using rotation information according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0009] (1 Mobile device) (1.1 Configuration of the mobile device) The moving device according to the present embodiment will be described below. Figures 1 and 2 are a schematic plan view and a schematic side view, respectively, showing the configuration of the moving device according to the present embodiment.

[0010] As shown in Figs. 1 and 2, the moving device 1 comprises three vehicles 10A to 10C and two coupling units 12 that couple them together. Each of the three vehicles 10A to 10C comprises two wheels 11. Vehicle 10A is located at the front of the moving device 1 in the straight-ahead direction, vehicle 10C is located at the rear of the moving device 1 in the straight-ahead direction, and vehicle 10B is located between vehicles 10A and 10C. Vehicles 10A and 10B are coupled together by coupling unit 12, and vehicles 10B and 10C are coupled together by coupling unit 12. In the following description, when there is no need to distinguish between the three vehicles 10A to 10C, they will be referred to as vehicle 10.

[0011] Two wheels 11 provided on one vehicle 10 are provided so as to rotate about a substantially coaxial rotation axis. Two connecting parts 12 connect the two vehicles 10 so as to be rotatable about a rotation axis that faces a direction perpendicular to the rotation axis direction of the wheels 11 and the straight-ahead direction of the mobile device 1. This allows the mobile device 1 to easily change its traveling direction.

[0012] Vehicle 10A is equipped with one camera 13, vehicle 10B is equipped with two cameras 13, and vehicle 10C is equipped with four cameras 13. All seven cameras 13 capture images of the forward side in the traveling direction of the moving device 1. The camera 13 equipped on vehicle 10A is provided at approximately the center in the width direction of vehicle 10A, and its imaging direction is inclined toward the ground contact surface side of two wheels 11 equipped on vehicle 10A with respect to the straight traveling direction. The two cameras 13 equipped on vehicle 10B are provided so as to be spaced apart from each other in the width direction of vehicle 10B, and their imaging directions are inclined toward the outer side in the width direction of vehicle 10B with respect to the straight traveling direction, and are inclined toward the ground contact surface side of two wheels 11 equipped on vehicle 10B. Of the four cameras 13 provided on the vehicle 10C, two cameras 13 are provided spaced apart from each other in the width direction of the vehicle 10C, and their imaging directions are inclined outward in the width direction of the vehicle 10C with respect to the straight-ahead direction, and inclined toward the ground contact surfaces of the two wheels 11 provided on the vehicle 10C. In addition, the two cameras 13 provided on each side in the width direction of the vehicle 10C are provided at different positions from each other in a direction perpendicular to the straight-ahead direction and the width direction of the vehicle 10C.

[0013] (1.2 Control System Configuration) The configuration of a control system for controlling the moving device according to this embodiment will be described below. Fig. 3 is a block diagram showing the configuration of the control system according to this embodiment.

[0014] As shown in Fig. 3, the control system according to this embodiment includes a control device 2 and an image processing device 3. The control device 2 is communicatively connected to drive devices 14 and sensors 15 provided in the mobile device 1. The image processing device 3 is communicatively connected to five cameras 13. The control device 2 and the image processing device 3 may be mounted on the mobile device 1 or may be separated from the mobile device 1 so as to be communicatively connected thereto.

[0015] The driving devices 14 include motors that rotate each of the six wheels of the mobile device 1. The sensors 15 include a Global Navigation Satellite System (GNSS) sensor as a sensor that measures the position of the mobile device 1, and an encoder as a sensor that measures the amount of rotation of each of the two connecting parts 12. Note that the sensors 15 do not necessarily need to include a GNSS sensor, and may include a sensor that can measure the position of the mobile device 1 that can be expressed in a world coordinate system (described later) instead of the GNSS sensor.

[0016] The control device 2 controls the driving devices 14 in response to instructions from an operator of the moving device 1. The image processing device 3 acquires the position of the moving device 1 and the amount of rotation of each of the two connecting parts 12 via the control device 2, and transforms and combines images captured by the seven cameras 13 based on this information. Note that the image processing device 3 may acquire the position of the moving device 1 and the amount of rotation of each of the two connecting parts 12 directly, without going through the control device 2.

[0017] (1.3 Configuration of image processing device) The hardware configuration and functional configuration of the image processing device according to this embodiment will be described below. Figures 4 and 5 are block diagrams showing the hardware configuration and functional configuration of the image processing device according to this embodiment, respectively.

[0018] As shown in FIG. 4, the image processing device 3 includes hardware such as a central processing unit (CPU) 31, a random access memory (RAM) 32, a storage device 33, and an input / output I / F .

[0019] The CPU 31 and RAM 32 work together to execute various functions, and the storage device 33 stores various data used in processes executed by the various functions. The input / output I / F 34 is an interface for the image processing device 3 to communicate with external devices, and images captured by the seven cameras 13, the measured positions of the mobile device 1, and the respective rotation amounts of the two connecting parts 12 are input to the image processing device 3 via the input / output I / F 34.

[0020] As shown in FIG. 5, the image processing device 3 includes functions such as an image acquisition unit 301, a measurement information acquisition unit 302, an adjustment processing unit 303, a mask processing unit 304, and a synthesis processing unit 305.

[0021] The image acquisition unit 301 acquires captured images captured by the seven cameras 13. The measurement information acquisition unit 302 acquires the measurement position of the mobile device 1 and the rotation amount of each of the two connecting parts 12 as measurement information. As calibration, the adjustment processing unit 303 executes a first adjustment process for adjusting external parameters that define the relative position and relative orientation between the two cameras 13, and a second adjustment process for adjusting external parameters that define the device position and device orientation between the cameras 13 and the mobile device 1. The parameters and the first and second adjustment processes will be described in detail later.

[0022] The mask processing unit 304 executes mask processing, which will be described later, on the captured images acquired by the image acquisition unit 301. The synthesis processing unit 305 generates a synthetic image in which the captured images masked by the mask processing unit 304 are projected onto a model space, based on the external parameters adjusted by the adjustment processing unit 303 and the measurement information acquired by the measurement information acquisition unit 302. The mask processing and the model space will be described in detail later.

[0023] (2 coordinate systems) The coordinate systems used by the image processing device according to this embodiment will be described. Such coordinate systems include a world coordinate system based on the ground, a camera coordinate system based on the camera, and a mobile device coordinate system based on the mobile device. In the following description, the coordinate systems that express coordinate values ​​are indicated by superscripts, and the subscripts w denotes the world coordinate system, and the subscript c Let denote the camera coordinate system.

[0024] (2.1 World Coordinate System) The world coordinate system according to this embodiment will now be described with reference to Fig. 6, which is a diagram showing the world coordinate system according to this embodiment.

[0025] As shown in Fig. 6, the world coordinate system has an origin at an arbitrary position on the ground and has mutually perpendicular X-axis, Y-axis, and Z-axis. The X-axis and Y-axis are oriented horizontally, and the Z-axis is oriented vertically downward.

[0026] (2.2 Camera Coordinate System) The camera coordinate system according to this embodiment will be described below with reference to Fig. 7, which is a diagram showing the camera coordinate system according to this embodiment.

[0027] 7, the camera coordinate system has an origin at the position of the camera 13, more specifically, the position of the image element of the camera 13, and has an X-axis, a Y-axis, and a Z-axis that are orthogonal to each other. The Z-axis points forward in the optical axis direction of the camera 13 (shooting direction), the X-axis points to the right with respect to the shooting direction, and the Y-axis points downward with respect to the shooting direction.

[0028] (2.3 Movement device coordinate system) The mobile device coordinate system according to this embodiment will be described below with reference to Fig. 8, which is a diagram showing the mobile device coordinate system according to this embodiment.

[0029] 8, the mobile device coordinate system has an origin set to the position of any one of the cameras 13 projected onto the ground in a first adjustment process described in detail later, and has an X-axis, a Y-axis, and a Z-axis that are perpendicular to each other. The Z-axis points forward in the traveling direction of the mobile device 1, the X-axis points to the right with respect to the traveling direction, and the Y-axis points downward with respect to the traveling direction.

[0030] (2.4 Image Processing Library) The image processing device 3 according to this embodiment uses OpenCV as an image processing library. In OpenCV, the direction of raster scan is from the upper left to the lower right, so that in the camera coordinate system, the X axis faces the right direction and the Y axis faces the downward direction, as described above. In addition, since OpenCV uses a right-handed system, the Z axis is the depth direction. The orientations of the camera coordinate system and the mobile device coordinate system are aligned with this. The arrangement of the feature point positions of the marker pattern, which will be described later, is also the same as that of the raster scan, and the coordinate axes when expressing the feature point positions are also the same. When the marker pattern is placed on the ground, the Z axis in the coordinate axes when expressing the feature point positions faces vertically downward, so that the Z axis of the world coordinate system also faces vertically downward.

[0031] (3) Captured images Camera coordinate system position X (bold) c The projection of the above onto a captured image will now be described.

[0032] (3.1 Camera model) As a method of projecting onto a captured image according to this embodiment, a pinhole camera model is considered. Fig. 9 is a diagram showing the pinhole camera model according to this embodiment.

[0033] As shown in Fig. 9, in the pinhole camera model, position X (bold) in real space c is projected onto the projection surface PP through a small pinhole PH.

[0034] The projection position (u,v) is calculated as follows:

[0035]

number

[0036]

number

[0037] where f x ,f y are the focal lengths in the x and y directions, and c x ,c y is the principal point position.

[0038] (3.2 Distortion Correction) The camera 13 captures the light that passes through the lens. The use of the lens causes distortion in the image. The projection position (u', v') on the image when the distortion is taken into account is expressed as follows:

[0039]

number

[0040] Here, k1, k2, k3, k4, k5, k6, p1, and p2 are distortion correction coefficients.

[0041] (3.3 Internal parameters) The internal parameters according to this embodiment are the focal lengths f x ,f y and distortion correction coefficients k1, k2, k3, k4, k5, k6, p1, and p2. The internal parameters are calculated based on a captured image of a marker pattern, which will be described in detail later. In this embodiment, the internal parameters are estimated by minimizing a reprojection error using OpenCV. Here, the reprojection error is the difference between a position detected on a captured image and a projection position calculated by using the parameters.

[0042] (4 External parameters) The external parameters used in the coordinate transformation will now be described.

[0043] A given coordinate position represented by a given coordinate system can be transformed into a coordinate position represented by another coordinate system using a rotation matrix and a translation vector as shown in the following equation: Note that the following equation shows, as an example, a transformation from a coordinate position in the world coordinate system to a coordinate position in the camera coordinate system.

[0044]

number

[0045] Here, R (in bold) is the rotation matrix, and T (in bold) is the translation vector. The subscripts and superscripts attached to R and T represent the coordinate systems before and after the transformation, respectively.

[0046] The external parameters collectively refer to the rotation matrix and the translation vector.

[0047] (4.1 Rotation matrix) A rotation matrix is ​​a 3x3 orthonormal matrix with a determinant of 1. Because a rotation matrix has three independent components due to these properties, it can be described by three parameters. A rotation matrix can be expressed as follows using the zenith angle θ, azimuth angle φ, and rotation amount ψ as independent parameters.

number

[0048] (4.2 Translation Vector) The translation vector T has components t in each axis direction. x ,t y ,t z It is described as:

number

[0049] (5 Rotation axis of connecting part) (5.1 Effects of rotation) As described above, the mobile device 1 is composed of three vehicles 10, and the two connectors 12 that connect them are configured to be rotatable. This causes the mobile device 1 to deform, and this deformation may cause a deviation between captured images when the captured images are combined by the multiple cameras 13. To address this deviation, the adjustment processing unit 303 corrects the external parameters based on the amount of rotation of the two connectors 12. As described above, these amounts of rotation are measured by the encoders included in the sensors 15.

[0050] (5.2 Rotational Axis Parameters) The rotation axis of the connecting part 12 can be expressed by the position X (bold) through which the rotation axis passes and the unit vector n (bold) indicating the axial direction of the rotation axis. X (bold) and n (bold) each have three components, as shown in the following equation.

[0051]

number

[0052] (5.3 Correction of external parameters) As described above, the movement device 1 according to this embodiment has two connecting parts 12. Therefore, if the positions of the rotation axes of the two connecting parts 12 are known, it becomes possible to correct the external parameters according to the amount of rotation.

[0053] 1 and 2, a rotation axis exists between the camera 13 provided on vehicle 10A, the two cameras 13 provided on vehicle 10B, and the four cameras 13 provided on vehicle 10C. No rotation axis exists between the two cameras 13 provided on vehicle 10B and between the four cameras 13 provided on vehicle 10C. Combining of images captured by multiple cameras 13 provided on the same vehicle 10 is not affected by the rotation axis.

[0054] Let us consider how the external parameters between two coordinate systems A and B are affected by two rotations. Let the rotation amounts of the two rotation axes be τ1 and τ2, and let R (bold) be the external parameters between the cameras when τ1 = τ2 = 0. 0AB ,T(bold) 0A B The first rotation axis in the A coordinate system is X (bold)1 A Through n(bold)1 A , and the second rotation axis is X (bold)2 A Through n(bold)2 A Let it be parallel to .

[0055] As a calculation procedure, we consider the effect of only rotating the first of the two rotation axes, and then consider the effect of rotation on the second rotation axis. When rotating on the first rotation axis, the external parameters between the camera 13 and the second rotation axis change. The value after the change is indicated by R (bold). 1A B ,T(bold) 1A B ,X(bold)2' A ,n(bold)2' A Then, it can be calculated as follows:

[0056]

number

[0057] Next, the second axis of rotation is rotated to R (bold) 1A B ,T(bold) 1A B The value after the change is R (bold). 2A B ,T(bold) 2A B Then, it can be calculated as follows:

[0058]

number

[0059] (6 Calibration) Calibration is a process for determining external parameters prior to image synthesis. Calibration includes a first adjustment process for adjusting external parameters between different cameras 13, and a second adjustment process for adjusting external parameters between one camera 13 and the mobile device 1.

[0060] (6.1 Markers) The marker pattern used in this embodiment will be described below. Fig. 10 is a diagram showing the marker pattern according to this embodiment.

[0061] As a calibration, the adjustment processing unit 303 calculates external parameters based on feature point positions in captured images of a marker pattern captured by two cameras 13 that capture two adjacent captured images in a composite image. In this embodiment, the calibration is performed using a marker pattern MP called ChArUco as shown in Fig. 10. This marker pattern MP is a combination of a chessboard-like pattern in which black and white squares are arranged alternately, and a marker for AR.

[0062] The adjustment processing unit 303 detects the corner positions in the chessboard pattern as feature points. Here, the corner positions are the positions of lattice points where two white cells WH and two black cells BK are gathered, in other words, the vertex positions shared by the corners of four cells forming a square. In a chessboard pattern that does not include a marker for AR, the entire marker pattern needs to be captured in order to identify the corner positions. On the other hand, with the marker pattern MP, by combining the detection results of the marker MK for AR, it becomes possible to identify the corner positions even when only a part of the marker pattern MP is captured. In addition, since the chessboard pattern has a higher detection accuracy than the marker MK for AR, the combination of the chessboard pattern and the marker MK for AR makes it possible to identify and detect the corner positions with higher accuracy.

[0063] In this embodiment, calibration is performed on the seven cameras 13 provided on the mobile device 1, but at this time, two of the cameras 13 need to capture the marker pattern MP simultaneously. Because it may be difficult to capture the entire marker pattern MP with two cameras due to the arrangement of the seven cameras 13, the marker pattern MP is more advantageous than a chessboard-like pattern that does not include AR markers MK. In addition, the adjustment processing unit 303 uses OpenCV to recognize the AR markers MK and detect corner positions, and detects the corner positions with sub-pixel accuracy.

[0064] (6.2 Inter-camera calibration) The adjustment processing unit 303 calculates extrinsic parameters between two cameras 13 whose imaging ranges at least partially overlap. The adjustment processing unit 303 calculates extrinsic parameters based on the marker pattern MP for each of the two cameras 13 based on two captured images in which the marker pattern MP is captured, and calculates extrinsic parameters between the two cameras based on the two calculated extrinsic parameters.

[0065] (6.2.1 Camera Combinations) According to the seven cameras 13 provided on the mobile device 1, there are combinations of two cameras 13 whose imaging areas do not overlap at least partially due to their imaging directions. Such external parameters between two cameras 13 cannot be directly obtained by calibration. However, for example, when the external parameters between the first camera 13 and the second camera 13 and the external parameters between the first camera 13 and the third camera 13 are known, the external parameters between the second camera 13 and the third camera 13 can be calculated using the two known external parameters. From this, if a calibration is performed between a specific combination of cameras, the external parameters between all cameras can be obtained. The calculation of the external parameters based on such two external parameters will be described later in the section "Supplementary Note on Combination of External Parameters."

[0066] (6.2.2 Imaging Procedure) The procedure for capturing images used in the inter-camera calibration according to this embodiment is as follows: The following capturing procedure applies to all combinations of two cameras 13 whose imaging areas at least partially overlap.

[0067] 1. An operator places a marker pattern MP in a position that falls within the imaging areas of two of the multiple cameras 13 required for calibration. 2. An operator captures an image of the marker pattern MP multiple times, for example, about 10 times, with two cameras 13 at different positions and orientations. The captured image is associated with an identifier that uniquely identifies the camera 13 that captured the image, and the amount of rotation of the rotation axis between the two cameras 13 at the time of capturing the image. 3. If a connecting portion 12 is present between the two cameras 13 of interest, the worker repeats steps 1 and 2 by varying the amount of rotation of the connecting portion 12. 4. The operator selects a combination of other cameras 13 required for calibration, and performs steps 1 to 3.

[0068] (6.2.3 First Adjustment Process) The first adjustment process according to this embodiment will be described. FIG. 11 is a flowchart showing the operation of the first adjustment process according to this embodiment. It is assumed that all captured images required for calculating the external parameters have already been acquired in FIG. 11. It is also assumed that initial values ​​are set in advance for the external parameters of each combination of the multiple cameras 13 for which the external parameters are to be calculated in FIG. It is also assumed that the external parameters of the camera 13 provided in the leading vehicle 10A and each of the other six cameras 13 are to be calculated in FIG. 11.

[0069] As shown in FIG. 11, the adjustment processing unit 303 detects all corner positions captured in each of all captured images required to calculate external parameters between the first camera 13, which is the camera 13 provided on the vehicle 10A, and each of the other six cameras 13, which is the second camera 13 (S101).

[0070] Next, the adjustment processing unit 303 calculates external parameters based on the marker pattern MP for all cameras 13 based on all corner positions detected in all captured images (S102), and selects one unselected camera 13 among the cameras 13 other than the first camera 13 as the second camera 13 (S103).

[0071] Next, the adjustment processing unit 303 initializes the external parameters of the first camera 13 and the second camera 13 by the external parameters set in advance. Here, the external parameters of the first camera 13 and the second camera 13 are θ, φ, ψ, t x ,t y ,t z , X(bold), n(bold), where X(bold), n(bold) are the position and orientation of the rotation axis existing between the first camera 13 and the second camera 13. The rotation matrix R and translation vector T in these external parameters are expressed by the following equations with the amount of rotation τ as a parameter.

[0072]

number

[0073] The adjustment processing unit 303 calculates external parameters of the first camera 13 and the second camera 13 using the amount of rotation τ at the time of capturing associated with the captured image.

[0074] Next, the adjustment processing unit 303 calculates a corner position in the image captured by the first camera 13 in the coordinate system of the second camera 13 based on the external parameters of the first camera 13 and the second camera 13, and calculates the distance between the position on the image captured by the second camera 13 to which the calculated corner position is projected and the corner position in the marker pattern MP on the image captured by the second camera 13 as a reprojection error (S105).

[0075] Next, the adjustment processing unit 303 determines whether the sum of squares of the reprojection errors calculated for all corner positions is equal to or smaller than a predetermined threshold value (S106).

[0076] If the sum of squares is not equal to or less than the threshold (S106, NO), the adjustment processing unit 303 adjusts the external parameters of the first camera 13 and the second camera 14 (S107), and calculates the reprojection error again (S105).

[0077] On the other hand, if the sum of squares is less than or equal to the threshold value (S106, YES), the adjustment processing unit 303 determines the adjusted external parameters as the external parameters of the first camera 13 and the second camera 13 (S108), and determines whether all of the cameras 13 have been selected as the second camera (S109).

[0078] If all cameras 13 have been selected (S109, YES), the first adjustment process is terminated, and if all cameras 13 have not been selected (S109, NO), the adjustment processing unit 303 again selects one unselected camera 13 from among the cameras 13 other than the first camera 13 as the second camera 13 (S103).

[0079] In such a first adjustment process, OpenCV is used for the processes of steps S101 and S102. Moreover, the processes of steps S105 to S107 are parameter estimation by the least squares method. Here, if the error calculation includes nonlinearity, the parameters cannot be obtained analytically, and need to be obtained by numerical calculation. Since the reprojection error calculation includes trigonometric functions, the external parameters cannot be obtained analytically. Therefore, in the first adjustment process according to this embodiment, the external parameters are adjusted using the Gauss-Newton method, which is one of the numerical calculation methods.

[0080] (6.3 Calibration between the moving device and the camera) The adjustment processing unit 303 calculates the external parameters of the moving device 1 and the camera 13. At this time, the moving device 1 is controlled to move in a straight line, and the measurement information acquisition unit 302 acquires measurement information by the sensors 15. The adjustment processing unit 303 calculates the external parameters of the moving device 1 and one of the cameras 13 based on the measurement information. The external parameters of the moving device 1 and the remaining camera 13 can be calculated using the external parameters between the two cameras 13 described above. In this embodiment, the position of the leading vehicle 10A is set as the reference point of the moving device 1, and the external parameters of the camera 13 provided on the vehicle 10A and the moving device 1 are calculated.

[0081] (6.3.1 Shooting Procedure) The procedure for capturing an image used for calibrating the mobile device 1 and the camera 13 according to this embodiment is as follows.

[0082] 1. A worker places a marker pattern MP on the ground. 2. While moving the mobile device 1, one camera 13 photographs the marker pattern MP multiple times.

[0083] (6.3.2 Second Adjustment Process) The second adjustment process according to this embodiment will be described. Fig. 12 is a flowchart showing the operation of the second adjustment process according to this embodiment. Note that in Fig. 12, it is assumed that all captured images required for calculating the external parameters have already been acquired.

[0084] 12, first, the adjustment processing unit 303 selects an unselected captured image from among captured images required for calibration between the moving device 1 and the camera 13 (S201). Next, the adjustment processing unit 303 calculates external parameters of the camera 13 that captured the selected captured image based on the marker position in the marker pattern MP on the selected captured image (S202), and calculates the position of the camera 13 based on the calculated external parameters (S203).

[0085] Next, the adjustment processing unit 303 calculates the position of the mobile device 1 by projecting the calculated position of the camera 13 onto the ground (S204), and determines whether or not all captured images have been selected (S205).

[0086] If all the captured images have not been selected (S205, NO), the adjustment processing unit 303 again selects an unselected captured image (S201).

[0087] On the other hand, if all the captured images have been selected (S205, YES), the adjustment processing unit 303 calculates an approximate straight line as the trajectory of the mobile device 1 based on the positions of the mobile device 1 calculated for each of all the captured images (S206).

[0088] Next, the adjustment processing unit 303 sets a mobile device coordinate system based on the calculated trajectory of the mobile device 1 (S207). Here, the adjustment processing unit 303 sets the mobile device coordinate system for the mobile device 1 so that the Z axis coincides with the trajectory of the mobile device 1.

[0089] Next, the adjustment processing unit 303 calculates the external parameters of the moving device 1 for each of all the selected captured images (S208), and calculates the external parameters of the moving device 1 and the camera 13 based on each of the external parameters calculated for each of all the captured images and the external parameters of the camera 13 (S209).

[0090] Next, the adjustment processing unit 303 calculates the average value of the external parameters of the moving device 1 and the camera 13 calculated for each of all captured images, and sets this average value as the external parameters of the moving device 1 and the camera 13 (S210), and the second adjustment processing is terminated.

[0091] (7. Combining captured images) (7.1 Preparation for synthesis) (7.1.1 Model Space) The virtual reproduction of the space in which the mobile device 1 travels while capturing images with the seven cameras 13 is called the model space. The image processing device 3 according to this embodiment reproduces the texture of the surroundings of the mobile device 1 by projecting the captured images into the model space. As shown in Fig. 13, the model space according to this embodiment reproduces a space in which a T-shaped steel bar exists.

[0092] (7.1.2 Virtual Camera) A virtual camera whose position and orientation can be freely set is set in the model space. The image captured by this virtual camera of the model space becomes a composite image that is finally output by the image processing device 3. The virtual camera is set as a simple pinhole camera with no lens distortion.

[0093] (7.2 Masking) Depending on conditions such as the reflection of lighting, the image on the captured image may look different from the actual one. The seven cameras 13 are located at different positions and have different orientations, and the positional relationship with the lighting changes as the mobile device 1 moves, so the conditions such as the reflection of lighting differ for each captured image. When images under such different conditions are synthesized, the boundary line of the synthesis may be noticeable.

[0094] The positional relationship between each of the seven cameras 13 and the lighting is also different from one another. This may result in the possibility of combining captured images captured under different lighting conditions. In such a case, the boundary line of the composite is clearly visible. In order to composite captured images more naturally, it is necessary to make the boundary line less noticeable. Therefore, the portion of the captured image where the lighting is strongly reflected is not used for the composite. The mask processing unit 304 executes a mask process for the captured image, which applies a mask to separate the portion that can be used for the composite from the portion that cannot.

[0095] Such masking processing makes it possible to hide the reflection of the vehicle 10. Fig. 14 and Fig. 15 are diagrams showing a captured image without masking processing according to this embodiment and a captured image with masking processing, respectively. The captured image shown in Fig. 14 shows the wheels 11 of the moving device 1 reflected in it, but in the captured image shown in Fig. 15, that part is masked.

[0096] (7.3 Images used for synthesis) The image acquisition unit 301 periodically acquires captured images captured by the seven cameras 13 while the mobile device 1 is moving. The measurement information acquisition unit 302 acquires measurement information from the sensors 15 at the timing when the captured image is acquired. The synthesis processing unit 305 calculates the position of the mobile device 1 at the time when the captured image was acquired based on the acquired measurement information, and sets external parameters of the mobile device 1. The external parameters of the seven cameras 13 are also calculated based on the external parameters of the mobile device 1.

[0097] (7.4 Compositing) The compositing process according to this embodiment will be described. Fig. 16 is a flowchart showing the operation of the compositing process according to this embodiment. Note that the operation of the compositing process shown in Fig. 16 is only one period of the periodically executed operations, and only shows the operation of mapping one captured image of seven captured images to be processed onto a model space.

[0098] As shown in FIG. 16, the synthesis processing unit 305 selects an unselected pixel position on the projection plane of the virtual camera (S301), calculates a ray to be projected onto the selected pixel position (S302), and calculates the intersection position where the calculated ray intersects with a surface in model space (S303).

[0099] Next, the composition processing unit 305 calculates the pixel position of the captured image projected onto the calculated intersection position (S304). Here, the composition processing unit 305 calculates external parameters for the model space of the camera 13 that captured the captured image to be processed based on the measurement results related to the movement of the mobile device 1 included in the measurement information and the rotation amounts of each of the two connecting units 12, and converts the intersection position into a position in the camera coordinate system of the camera 13 using the external parameters to calculate the pixel position of the captured image.

[0100] Next, the synthesis processing unit 305 sets the pixel value of the calculated pixel position of the captured image to the pixel value of the pixel position on the projection plane of the selected virtual camera (S305), and determines whether all pixel positions have been selected (S306).

[0101] If all pixel positions have not been selected (S306, NO), the synthesis processing unit 305 again selects an unselected pixel position on the projection plane of the virtual camera (S301).

[0102] On the other hand, if all pixel positions have been selected (S306, YES), the synthesis process for one captured image in one period is completed.

[0103] (8 comparisons) A composite image obtained by a synthesis process using the rotation amount of the connecting portion 12 is compared with a composite image obtained by a synthesis process not using the rotation amount. Fig. 17 is a diagram showing a composite image using rotation information according to the present embodiment. Fig. 18 is a diagram showing a composite image not using rotation information according to the embodiment.

[0104] In Fig. 17 and Fig. 18, the area enclosed by the dashed line is a cable mold stretched across the floor. This mold is configured in a straight line. In the composite image shown in Fig. 17, the captured images are combined so that the mold is linear, just like the actual mold. In contrast, in the composite image shown in Fig. 18, the mold is not linear, but is combined in a shifted state. This result shows that combining captured images using the amount of rotation is effective.

[0105] (9 Supplementary information on rotation matrices) A supplementary note regarding rotation matrices.

[0106] Rotation can be described by the axis of rotation and the amount of rotation. The axis of rotation is considered to be a unit vector that passes through the origin (if the axis of rotation does not pass through the origin, it is sufficient to perform a parallel translation before and after the rotation). Here we consider rotations that move the coordinate axes. When considering the movement of a point within a coordinate system, we consider the inverse transformation. Rotations are expressed by a rotation matrix R, and their inverse transformation R -1 is R -1 =R T This is because the rotation matrix R is an orthogonal matrix with determinant 1. The unit vector n representing the axis of rotation is expressed as follows using the zenith angle θ and the azimuth angle φ.

[0107]

number

[0108] (When θ=0) When θ=0, the rotation is around the z-axis.

[0109]

number

[0110] (When θ ≠ 0) When θ ≠ 0, consider a coordinate system based on the rotation axis (XYZ coordinate system). n (bold) is the Z axis, and the X axis is taken in the opposite direction to the position where the z axis is projected onto a plane perpendicular to n (bold).

[0111] The position of the z-axis projected onto a plane perpendicular to n (bold) is given by the following equation:

[0112]

number

[0113] Since this position and n are orthogonal,

[0114]

number

[0115] From this formula, α = -n z The unit vector representing the X-axis is

[0116]

number

[0117] The unit vector representing the Y-axis can be calculated as the cross product of two unit vectors,

[0118]

number

[0119] The matrix M (bold type) consisting of three unit vectors is

[0120]

number

[0121] This matrix M (bold type) represents the rotation matrix that transforms the XYZ coordinate system into the xyz coordinate system, and its inverse matrix M (bold type) T is the matrix that transforms the xyz coordinate system into the XYZ coordinate system. The rotation matrix R (in bold) that we are looking for rotates around the Z axis by ψ, so it can be written as follows:

[0122]

number

[0123] Calculating equation (A5) gives the following equation:

[0124]

number

[0125] If θ = 0 is substituted into equation (A6), it becomes the same as equation (A2), and therefore equation (A6) always holds true.

[0126] (10 Supplementary information regarding changes in external parameters) We will now provide additional information regarding changes in external parameters.

[0127] Let us consider how the external parameters change when, for example, the camera 13 moves. The two coordinate systems are represented as coordinate system A and coordinate system B. Coordinate system A is fixed, and coordinate system B moves.

[0128] The initial external parameter is R (bold) 0A B ,T(bold) 0A B In this case, the following relationship is satisfied.

[0129]

number

[0130] X (bold) A and X (bold) B are the same position in each coordinate system. X (bold) is added according to the movement of the coordinate system. A or X (bold) B The external parameters are calculated by considering the correspondence as if either one of them, for example the one that is easier to calculate, had changed. X (bold)' A or X' B Let the external parameters after the change be R (bold). 1A B ,T(bold) 1A B Then, it can be expressed by one of the following formulas:

[0131]

number

[0132] (10.1 dX(bold) A (In the case of translation of dX(bold) A In the case of translation, the position of interest in the A coordinate system moves with the B coordinate system by dX (bold). A From the B coordinate system, the position of interest is X (bold). B has not changed since.

[0133]

number

[0134] Substituting equation (A10) into equation (A8) and rearranging the equation gives the following equation.

[0135]

number

[0136] Comparing equation (A11) with equation (A7) gives the following result:

[0137]

number

[0138] When rotating around the axis of rotation that passes through the origin of the A coordinate system, the position of interest in the A coordinate system is considered to have rotated together with the B coordinate system. When viewed from the B coordinate system, the position of interest is X (bold). B The axis of rotation is the unit vector

[0139]

number

[0140] Consider the following rotation matrix:

[0141]

number

[0142] The position after rotation as seen from the A coordinate system is expressed as follows.

[0143]

number

[0144] Substituting equation (A14) into equation (A8) gives the following equation.

[0145]

number

[0146] Comparing equation (A15) with equation (A7) we get the following result:

[0147]

number

[0148] (10.2 Rotation around an axis passing through any point in the A coordinate system) When rotating around an axis passing through any point in the A coordinate system, the axis of rotation is X A Passing through, the unit vector

[0149]

number

[0150] In this case, first -X (bold) A Then, this rotation can be treated as a rotation through the origin. After the rotation, X (bold) A The translation is performed.

[0151] Consider the following rotation matrix:

[0152]

number

[0153] Combining equations (A12) and (A16) gives the following result:

[0154]

number

[0155] (10.3 dX(bold) B (In the case of translation of dX(bold) B In the case of translation, the position of interest does not move when viewed from the A coordinate system. When viewed from the B coordinate system, X (bold) B -dX (bold) B It appears that the movement of

[0156]

number

[0157] Substituting this into equation (A9) and transforming the equation gives us the following:

[0158]

number

[0159] Comparing equation (A20) with equation (A7) we get the following result:

[0160]

number

[0161] (10.4 Rotation around the axis passing through the origin of the B coordinate system) When rotating around the axis of rotation that passes through the origin of the B coordinate system, the position of interest is considered to not move when viewed from the A coordinate system. The axis of rotation is the unit vector

[0162]

number

[0163] Consider the following rotation matrix:

[0164]

number

[0165] In the B coordinate system, X (bold) B appears to have rotated by -τ.

[0166]

number

[0167] Substituting equation (A23) into equation (A9) and transforming the equation gives the following:

[0168]

number

[0169] Comparing equation (A24) with equation (A7) gives the following result:

[0170]

number

[0171] (10.5 Rotation around an axis passing through any point in the B coordinate system) When rotating around an axis passing through any point in the B coordinate system, the axis of rotation is X (bold). B through unit vector n (bold) BThe direction is , and the amount of rotation is τ. As with rotation around an axis of rotation that passes through an arbitrary point in the A coordinate system, it is sufficient to translate through the origin and then translate again after rotation, but the final translation will be a translation amount influenced by the rotation. It is also possible to calculate the translation amount after rotation, but here we will express the axis of rotation in the A coordinate system and apply the following formula to formula (A18).

[0172]

number

[0173] (11 Supplementary note on external parameter synthesis) We will now provide additional information on the synthesis of external parameters.

[0174] For example, consider the case where an external parameter between the cameras 13 is calculated from two external parameters.

[0175] Here, we consider the external parameters between the three cameras 13 (camera A, camera B, and camera C). The external parameter R between cameras A and B A B ,T A B and the external parameter R between the camera AC A C ,T A C If we know the extrinsic parameters R between cameras B and C, B C ,T B C Request.

[0176] Coordinate transformation is performed according to the external parameters as follows:

[0177]

number

[0178] Equation (A28) can be transformed as follows:

[0179]

number

[0180] Substitute equation (A30) into equation (A29).

[0181]

number

[0182] From equation (A31), it can be seen that the external parameters are expressed as follows:

[0183]

number

[0184] The embodiments of the present invention are presented as examples and are not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0185] 1. Mobile Device 3. Image Processing Device 10 Vehicles 13. Camera 15 Sensors 301 Image Acquisition Unit 302 Measurement information acquisition section 305 Synthesis Processing Unit

Claims

1. An image processing device is communicatively connected to a moving device including at least two vehicles, at least one coupling section that couples the at least two vehicles in a traveling direction and is rotatable around a rotation axis, and a first camera and a second camera that are provided on different vehicles, an image acquisition unit that acquires a first captured image captured by the first camera and a second captured image captured by the second camera; A measurement information acquisition unit that acquires a rotation amount of the at least one connecting portion; a synthesis processing unit that synthesizes the first captured image and the second captured image based on inter-camera extrinsic parameters indicating a relative position and a relative orientation between the first camera and the second camera, a position and a direction of a rotation axis of a connecting portion located between the first camera and the second camera, and an amount of rotation acquired for the connecting portion located between the first camera and the second camera; and An image processing device comprising:

2. an adjustment processing unit that adjusts a first external parameter based on an adjustment captured image obtained by capturing a pattern in which feature points are two-dimensionally arranged by a first camera and a second camera, and an amount of rotation of a connection portion located between the first camera and the second camera at the time when the adjustment captured image is captured; The image processing apparatus according to claim 1 , wherein the synthesis processing unit synthesizes the first captured image and the second captured image based on the adjusted first external parameter.

3. 3. The image processing device according to claim 2, wherein the adjustment processing unit calculates a position of a feature point in the image captured by the first camera in a coordinate system of the second camera based on the first external parameters, and adjusts the first external parameters so that a distance between a position on the image captured by the second camera onto which the calculated feature point is projected and a feature point identical to the calculated feature point on the image captured by the second camera is minimized.

4. the adjustment processing unit calculates, as a position of the mobile device, a position of the first camera projected onto the ground with respect to the pattern for each of a plurality of captured images in which the first camera captures the pattern while moving the mobile device; calculates an approximation curve based on the calculated positions of the mobile device as a trajectory of the mobile device; and calculates second external parameters indicating a relative position and a relative attitude between the mobile device and the first camera based on the calculated restriction; 4. The image processing apparatus according to claim 2, wherein the synthesis processing unit synthesizes the first captured image and the second captured image based on the first external parameter and the second external parameter.

5. The image processing device described in claim 4, characterized in that the adjustment processing unit sets a coordinate system of the moving device so that one axis coincides with the calculated trajectory, calculates a third external parameter indicating a relative position and relative orientation of the moving device with respect to the pattern for each of a plurality of captured images of the pattern captured by the first camera while moving the moving device, and calculates the second external parameter based on the calculated plurality of third external parameters.

6. The image processing device described in claim 5, characterized in that the adjustment processing unit calculates a plurality of fourth external parameters indicating a device position and relative orientation of the first camera with respect to the pattern for each of a plurality of captured images of the pattern captured by the first camera while moving the moving device, and calculates the second external parameter based on each of the calculated fourth external parameters and each of the plurality of third external parameters.

7. 7. The image processing device according to claim 6, wherein the adjustment processing unit calculates, as the second external parameter, an average value of external parameters calculated based on each of a plurality of third external parameters and each of the plurality of fourth external parameters.

Images