Single camera calibration

Abstract

The invention relates to a method (100) for determining extrinsic parameters of a camera, the method comprising the steps of: starting (102) a movement of the camera, detecting (104) a first raw image having a first patch and a second patch parallel to the first patch at a first point in time and a second raw image having a third patch and a fourth patch parallel to the third patch at a second point in time by the moving camera, wherein the distance between the first and the second patch is the same as the distance between the third and the fourth patch, detecting (108) a reference position A of at least one image feature in the first patch, a reference position C of at least one image feature position in the second patch, an offset position B of the at least one feature of the first patch in the third patch and an offset position D of the at least one feature of the second patch in the fourth patch, applying (110) a epipolar model based on the reference positions A and C, the offset positions B and D and a determined distance covered by the moving camera between the first point in time and the second point in time, and determining (112) extrinsic parameters of the camera as an output of the epipolar model.

Description

Technical Field

[0001] This invention relates to a calibration method and system for a single camera device, a calibration system for a single camera device that determines the external parameters of a camera device installed on a vehicle, a computer program unit, and a computer-readable medium. Background Technology

[0002] Typically, extrinsic parameters—such as pitch, yaw, roll, and altitude—are calculated using predefined targets. The camera to be calibrated inspects a pattern on a target of known size. The resulting image is processed by an extrinsic parameter solver to calculate the camera's extrinsic parameters. For good results, the target object must be extremely well-maintained, and conditions similar to those around a laboratory, along with bright lighting, must be provided. Any variation in lighting or the cleanliness of the target object will affect the accuracy of the calibration measurements, which is time-consuming, labor-intensive, and therefore expensive. Summary of the Invention

[0003] Therefore, it may be necessary to reduce the workload and cost of calibrating the camera equipment.

[0004] This task is addressed by the subject matter of the appended independent claims. Embodiments are provided in the dependent claims and the following description and drawings.

[0005] The embodiments also relate to a calibration method for a single camera device, a calibration system for a single camera device, a calibration purpose for a single camera device, a computer program unit, and a computer-readable medium.

[0006] Furthermore, it must be noted that all embodiments of the present invention relate to a method that can be implemented in the described order of steps; however, this is not necessarily the only or necessary order of steps for the method. Unless otherwise explicitly stated below, the methods described herein may be performed in an order of steps other than the published order without departing from the respective method embodiments.

[0007] Technical terms are used according to their common sense. If a particular term expresses a specific meaning, its definition will be given in the context in which it is used below.

[0008] According to a first aspect, a method for determining the extrinsic parameters of a camera device is provided. The method includes the following steps: First, the camera device is moved. In a second step, a first original image having a first patch and a second patch parallel to the first patch is detected at a first time point, and a second original image having a third patch and a fourth patch parallel to the third patch is detected at a second time point, wherein the distance between the first patch and the second patch is the same as the distance between the third patch and the fourth patch. In a next step, reference positions A for at least one image feature in the first patch, reference positions C for at least one image feature in the second patch, offset positions B of at least one feature of the first patch in the third patch, and offset positions D of at least one feature of the second patch in the fourth patch are detected. In a subsequent step, an epipolar model (or "epidial / epidial geometry model") is applied based on reference positions A and C, offset positions B and D, and the detected distance traveled / traveled by the moving camera device between the first and second time points; finally, the extrinsic parameters of the camera device are determined as the output of the epipolar model.

[0009] In other words, a moving camera captures images that map portions of the road. Patches 1 and 3 are parallel or nearly parallel to patches 2 and 4, respectively. Assuming patches 1 and 3 are to the left of the camera's orientation, the second and fourth patches are to the right. The first image with patches 1 and 2 and the second image with patches 3 and 4 can be detected within a short time interval, so the left patches and their image features partially overlap. The same applies to the right patches. Since these images are detected by a single camera, the distance between the left and right patches is almost equidistant in each image. The detected features represent a point in the camera's coordinate system for each image, resulting in four points: A, B, C, and D. These coordinates, along with the actual distances between A and B, constitute the input to the epipolar model, which will be explained in further detail below. This input can geometrically determine the direction of the camera's movement, or derive linear motion, or rotation if the motion is not linear. Furthermore, the height of the camera can be determined, thereby allowing the detection of all necessary extrinsic parameters.

[0010] The term "patch" / "image patch" is used to describe the image display content of an image. It can be any moving texture under the camera's viewfinder, i.e., a region of interest. If the camera is mounted on a vehicle, the patch could be, for example, the asphalt structure of a road.

[0011] According to one embodiment, the motion of the camera device is linear, circular, or both. Since nonlinear motion is effectively divided into linear motion and rotation, the actual (“real”) path can be arbitrary. In purely linear motion, rotation is not detected or is barely detected. A true circular path results in both linear motion and rotation.

[0012] According to one embodiment, the camera device is a fisheye camera device. Detection of reference position A, offset position B, reference position C, and offset position D includes ground plane mapping of first, second, third, and fourth patches to obtain corresponding patches as two-dimensional ground plane images in a virtual plane using a predefined image format. The ground plane has associated configurable parameters, such as patch width and height in meters in the real world, its projection, i.e., width and length in pixels, and an offset determining the distance from the camera device reference frame to the ground in front of the vehicle. The fisheye camera device can use a spherical model. This means that for each pixel in the fisheye image, a point is reserved on a unit sphere. The sphere has a Cartesian coordinate system with its origin at O ​​and axes x, y, z, referred to in this patent document as the camera device coordinate system. For ground plane mapping, the algorithm selects two patches from an actual plane, such as a road surface. The selected patch, having, for example, 128x256 pixels, represents a virtual plane onto which the actual plane or road surface is projected.

[0013] According to one embodiment, the first, second, third, and fourth patches in the virtual plane are preprocessed, wherein the preprocessing includes subtracting the average brightness value and applying a Gaussian window, wherein the Gaussian window has a maximum value at the center of the image and a minimum value at the edge of the image.

[0014] In other words, the input for preprocessing is a ground plane image with road patches in a virtual plane, which has a predefined format. Preprocessing is performed on each of the four ground plane images or each of the four ground plane patches separately. The average brightness value of the image is subtracted from the original image, and then the resulting image is multiplied by a Gaussian window image. The Gaussian window image suppresses pixels near the image edges and highlights features near the image center.

[0015] According to one embodiment, the offset position B of the third patch relative to the reference position A of the first patch, and the offset position D of the fourth patch relative to the reference position C of the second patch, are determined by performing phase correlation.

[0016] According to one embodiment, the phase correlation includes a Fast Fourier Transform (FFT). Reference positions A and C can be determined by autocorrelation of the image, or equivalently by determining the image center, while offset positions B and D are determined by the phase correlation of the first patch with the third patch, and the phase correlation of the second patch with the fourth patch. More specifically, the correlation is achieved by performing a Fourier transform on the image features and multiplying the corresponding images after this transform. The reference position is the image center because associating the image with a non-shifted overhang vertex is equivalent to autocorrelation and results in a peak at the image center.

[0017] According to one embodiment, reference positions A and C, and bias positions B and D, are relative positions, and the biases are determined using a gradient-based similarity metric. In other words, the biases are determined from the phase-correlation output (Fast Fourier Transform (FFT)), and the gradients of the first image patch (i.e., the first and second patches) and the gradients of the second image patch (i.e., the third and fourth patches) are computed. The gradient of the second image patch is shifted. Then, the dot product between the gradients of the second and first image patches is computed.

[0018] The resulting image is blurred or filtered to obtain the pixel coordinates, i.e., the positions, with the highest similarity.

[0019] According to one embodiment, the detection of offset positions B and D further includes determining the ground plane unit vectors from the camera device to reference positions A and C, and to offset positions B and D. In this processing stage, only the orientation of the camera device needs to be considered; that is, the orientation from the origin of the camera device's coordinate system to positions A, B, C, and D is known. Although the offsets are known, the absolute positions in the coordinate system are unknown. Therefore, unit vectors are determined, not vectors from the origin to the positions. The offsets provide one of several constraints for determining the extrinsic parameters in the following steps.

[0020] According to one embodiment, applying the epipolar model includes using ground plane element vectors to determine, in a first sub-step, a first direction V from reference position A to offset position C, which is the same direction from reference position B to offset position D, and a second direction W from reference position A to reference position B, which is the same direction from offset position C to offset position D, determining the normal vector N = V x W for the first direction V and the second direction W, and determining the height H of the camera device by determining the vectors from the origin to the actual positions A', B', C' and D' corresponding to image positions A, B, C and D respectively.

[0021] According to another aspect, a single-camera device calibration system is provided, comprising a camera device controller configured to control the camera device to be calibrated, such that, during movement, the camera device detects a first original image having a first patch and a second patch parallel to the first patch at a first time point, and a second original image having a third patch and a fourth patch parallel to the third patch at a second time point, wherein the distance between the first and second patches is the same as the distance between the third and fourth patches. The single-camera device calibration system further includes a processing module configured to detect a reference position A of at least one image feature in the first patch, a reference position C of at least one image feature position in the second patch, an offset position B of at least one feature of the first patch in the third patch, and an offset position D of at least one feature of the second patch in the fourth patch. The processing module is further configured to apply an epipolar model to the detection based on reference positions A and C, offset positions B and D, and the determined distance traversed by the moving camera device between the first and second time points, and is further configured to determine the extrinsic parameters of the camera device as the output of the epipolar model.

[0022] According to one embodiment, the system further includes a camera device controlled by a camera device controller, wherein the camera device is mounted on a vehicle.

[0023] According to another aspect, a vehicle is provided, the vehicle including the single camera device calibration system described above for camera device calibration.

[0024] On the other hand, the aforementioned single camera calibration system is used to determine the external parameters of a camera device installed on a vehicle.

[0025] In one example, a data processing system, such as a computer, is provided for performing the steps of the method. The data processing system may include circuitry without programmable logic, or may be a microcontroller, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), or any other programmable logic device of a class known to those skilled in the art.

[0026] According to another aspect, a computer program unit including instructions is provided, which, when executed by a computer, cause the computer to perform the steps of the method. The computer program unit may be part of a computer program, but it may also be a complete program itself. For example, the computer program unit can be used to update an existing computer program to implement the present invention.

[0027] According to another aspect, a computer-readable medium including instructions is provided, which, when executed by a computer, cause the computer to perform the method steps. The computer-readable medium may be a storage medium, such as a USB (Universal Serial Bus) stick, optical disc, DVD, data medium, hard disk, integrated chip medium of an embedded system, or other medium capable of storing the aforementioned program units.

[0028] Therefore, the algorithm solves the problem of estimating the extrinsic parameters of a vehicle's camera system at the end of the production line (EOL), at a service station, and under normal driving conditions. The algorithm requires a camera system with known intrinsic parameters, such as computer-aided design (CAD) values ​​for a rear-mounted camera, including its X-axis and Y-axis positions and rotations around the X, Y, and Z axes. Furthermore, the algorithm requires the vehicle's speed, steering, and suspension signals, as well as images detected by the camera system and the corresponding signals described above.

[0029] The algorithm is extremely simple and can be implemented on embedded platforms with limited runtime and memory resources. There is no need to maintain a dedicated target or pattern for the calibration of a single camera unit, nor is it necessary to provide a dedicated space or special surrounding environment for calibration. Since the camera unit can see moving images, it can be calibrated while traveling to the end of the production line (EOL) or on a conveyor belt. The algorithm can be adjusted for operation of a panoramic surround-view system with four cameras.

[0030] These and other aspects of the invention will become apparent and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0031] Exemplary embodiments of the present invention are described below with reference to the following figures:

[0032] Figure 1 A flowchart is shown for a method of determining the external parameters of a camera device;

[0033] Figure 2 A schematic diagram of a single camera device calibration system is shown.

[0034] Figure 3 A diagram illustrating the mapping of the ground plane is shown;

[0035] Figure 4 A diagram illustrating the calculation of epipolar lines is shown;

[0036] Figure 5 This diagram illustrates the patch preprocessing before frequency domain spectrum analysis.

[0037] Figure 6 A flowchart is shown for the frequency domain spectral analysis and the calculation of the bias between the two patches;

[0038] Figure 7 An illustration of patch movement is shown;

[0039] Figure 8 A schematic diagram of the ground plane vector from the camera device to the patch feature point is shown;

[0040] Figure 9 A schematic diagram of a hypothetical tire is shown;

[0041] Figure 10 An explanatory diagram showing two points on the spherical surface of the camera device is provided.

[0042] Figure 11 The diagram shows the points on the sphere of the camera device and the geometry of the center of the camera device projected onto the ground plane, as well as the normal vector across the plane;

[0043] Figure 12 A diagram showing the movement of the camera device is provided.

[0044] The accompanying drawings are merely illustrative and are used only to illustrate embodiments of the present invention. In principle, identical or equivalent parts have the same reference numerals. Detailed Implementation

[0045] Figure 1 The diagram illustrates a flowchart of a method 100 for determining the extrinsic parameters of a camera device, the method comprising the following steps. In a first step 102, the camera device is initiated to move. In a second step 104, two raw images are successively detected at two different locations using the moving camera device. At a first time point, a first raw image with a first patch and a second patch parallel to the first patch is detected; at a second time point, a second raw image with a third patch and a fourth patch parallel to the third patch is detected, wherein the distance between the first and second patches is the same as the distance between the third and fourth patches. In a next step 108, a reference position A for at least one image feature is detected in the first patch; a reference position C for at least one image feature location is detected in the second patch; an offset position B for at least one feature of the first patch is detected in the third patch; and an offset position D for at least one feature of the second patch is detected in the fourth patch. In a further step 110, an epipolar model is applied based on reference positions A and C, offset positions B and D, and the determined distance traversed by the moving camera device between the first and second time points. In the final step 112, the external parameters of the camera device are determined as the output of the epipolar model.

[0046] Figure 2The system 200 shown is configured to determine extrinsic parameters. It includes a camera controller 220 that controls a camera device 202 and a processing module 230. The single-camera calibration system 200 can be installed, for example, in a vehicle for “online” calibration of the vehicle-mounted camera device. The camera device 202 can be accessed via cable or wirelessly. The processing module 230 includes a ground plane mapping module 204, into which the raw image 202 is input and converted into a predefined format. The raw image can be, for example, an image of a road with two patch areas on which a vehicle is moving. The ground plane mapping module 204 can, for example, project a raw image 302 of a road plane patch measuring 1 meter x 1.5 meters onto the image, or... Figure 3 As shown, on a patch 312 with a size of 256*512 pixels, a resolution of 1 / 256 = 0.0039 meters per pixel along the length of the patch can be obtained. This step is performed on two images detected at two subsequent time points, each image including two patches of the road. Figure 3 As shown, the first image includes, for example, a left patch 302 and a right patch 304 relative to the optical axis (x-axis) of the camera device 322 at a first time point, and two additional road patches at a second time point. Figure 3 (Not shown in the image). Figure 3 The demonstration further shows a vehicle 300 with a rear-view camera 322 that detects images in this example, as well as other cameras 324 located on the left, right, and front sides of the vehicle 300. These other cameras can also capture images similar to those captured by camera 322, and thus these images can also be processed according to the described method.

[0047] At every moment (and in short time intervals), the rear wheel center is as follows: Figure 4 As shown, it rotates around the center of the circle in which the vehicle moves. (Refer to...) Figure 2 and Figure 4 In the proposed method, the values ​​of the following parameters between two arbitrary time points are provided by the vehicle autonomous motion module 210. The epipolar line length, i.e., the length of vector 404, is used to determine the height of the camera device; the angle between epipolar line 404 and the direction of travel 402 is used to determine the direction of travel 402 and the vehicle angle 406 via the epipolar line, and therefore, this angle is also the rotation angle of the camera device. This angle 406 is used to compensate for the rotation of the spherical image. Inputs to the vehicle autonomous motion module 210, such as vehicle speed, wheel angle, or suspension data, are provided by the vehicle sensors 208, for example, via the Controller Area Network (CAN) bus. The values ​​of these parameters are provided to the ground plane vector module 212, which will be described further below.

[0048] Reference Figure 2 and Figure 5 In a further processing step performed in the bias determination module 206, patch 504, obtained by subtracting the average value from patch 502, can be multiplied by the Gaussian window 506, for example... Figure 5 Patch 502 and other patches in the image are preprocessed to suppress pixels near the edges of patch 504 and to make features closer to the image center more prominent by assigning higher weights. Then, the resulting patch 508 is used as input for a spectral analysis based on Fast Fourier Transform (FFT) to determine the bias.

[0049] For reference Figure 6 By applying a Fast Fourier Transform (FFT), features of a first left-side patch 602 detected at a first time point can be matched with features of a second left-side patch 604 detected at a second time point. As a result, a bias between the features of the first patch 602 and the second patch 604 can be determined, corresponding to the distance traveled by the vehicle between the first and second time points. The method... Figure 6 A more detailed explanation follows. A phase correlation is implemented by transforming the preprocessed images 602 and 604 using a Fast Fourier Transform (FFT). In 606, their spectra are multiplied, and the product is inversely transformed (IFFT), resulting in a phase-correlated image in 608. After applying the Inverse Fast Fourier Transform (IFFT) to the resulting spectrum and obtaining a pixel-domain image, the resulting image 610 is adjusted to obtain the centered zero-frequency component. This simply means that when the two patches are phase-correlated without any bias, the unbiased result is placed at the center of the image. A target mask defining the prediction search window is used to find the maximum correlation in 612, i.e., the shifted pixel location. The reason for applying the target mask 612 will be explained later. Pixel peaks in the phase-correlated image provide information about the movement between the two patches 602 and 604. The appearance of patch features changes with image scale and camera mounting position. This can lead to multiple spurious high correlation values ​​throughout the image. To overcome this problem, peak values ​​are predicted, considering only the region near the predicted pixel location. To do this, a target mask is computed, which is weighted exponentially around the predicted location of the center pixel of the first patch. This target mask is multiplied by the phase-correlated image, and the peak values ​​of the resulting image are used to calculate the pixel shift between the two patches. As mentioned above, since no shifted correlation would result in a peak pixel at the image center, the image center is the reference location associated with the shifted peak pixel. Therefore, the pixel shift is obtained by subtracting the center pixel coordinates from the phase-correlated peak pixel coordinates.

[0050] Reference Figure 7To take into account more diverse features in patches 602 and 604, a gradient-based similarity metric was also used. Therefore, Figure 7 Patch 702, as the first road patch, is shifted by a calculated pixel shift, resulting in a shifted patch 704. This shifted first patch 704 is compared to a third patch 706, which refers to the same patch to the left of, for example, at the second time point. The gradient images of the shifted first patch 704 and the third patch 706 at the second time point are obtained and processed with a Gaussian blur filter. Finally, they are multiplied. The pixel coordinates of the peaks are considered feature points, which can be correlated with the image center of the third patch 706. That is, the corresponding pixel coordinates of the first patch can be easily calculated by subtracting the calculated pixel shift from the feature points.

[0051] The above method is applied to two patches extracted from the left side at two different time points and two patches extracted from the right side of the camera device at these two different time points. For example... Figure 8 As shown, this yields four feature points and essentially four camera ray 801, 802, 803, and 804 associated with these features. Each camera ray is a unit vector corresponding to one of the four feature points in the camera reference frame. Camera rays 801 and 802 are calculated from the left side of the camera, using two relatively shifted patches, while 803 and 804 are calculated from the right side.

[0052] The epipolar model used to calculate extrinsic parameters will now be explained. This model is based on the aforementioned unit vectors, whose origins are on the camera lens and the spherical camera assembly, thus intersecting the spherical surface of the camera assembly at points A, B, C, and D.

[0053] For the epipolar model proposed herein, the spherical camera device can be mounted on a vehicle. The direction “upward” is defined as the normal to the road surface (if the road is sloping). This direction is simply a point on the sphere as seen from the lens of the camera device, which serves as the origin. Another important direction is the forward direction. It is parallel to or lies on the long axis of the vehicle, on the plane of the lane surface, and therefore coincides with the direction of vehicle movement, provided the vehicle is moving forward in a straight line. We refer to these directions as the “upward direction” and the “forward direction.” They are represented by two points in the spherical camera device. These two points, together with the height of the camera device, constitute all the required extrinsic parameters.

[0054] There is also a rightward direction, which is simply a cross product of forward × upward. The three vectors—upward, forward, and rightward—form a matrix M as a column of vectors in the camera device coordinate system. This matrix M, combined with the current camera device height, constitutes the output of the single camera device calibration algorithm. Matrix M can be designated as the "camera device rotation matrix".

[0055] With the vehicle rotating at a constant angle, the Ackermann steering model is applied to demonstrate how the camera position changes due to the vehicle's current speed and steering. The application of the Ackermann steering model is as follows: Figure 9 As shown. If vehicle 902 turns with a constant steering attitude, the question arises as to what circular trajectories all four tires will trace. According to the Ackermann steering model, a tire 904 at the center of the front axle is introduced as a hypothetical tire. Based on information about the vehicle's steering, the steering angle of this hypothetical tire 904 can be determined. It should be noted that the center tire, inner tire, and outer tire run on different circular trajectories. We know the angle turned by the central hypothetical tire, which is the steering angle (δ). This angle can be used to calculate the radius (r) using tan(δ) = L / r, where L is the distance between the front and rear axles. If the arc length (s) is known, the radius can also be provided using the equation of a circle, s = rδ.

[0056] In the following text, we will explain the epipolar model for calculating extrinsic parameters and its applications.

[0057] Figure 10 and 11 Explain how to determine the direction of two parallel segments as seen from the spherical camera device. Figure 10 This illustrates that when observing two points A and B on a spherical camera device, the exact point cannot be determined. The exact point of A can be either A' or A'", while the exact point of B can be either B' or B'". Even knowing the length of the segment A'B', it is impossible to determine point A' or point B'.

[0058] However, these three known points O, A, and B define a plane passing through them. This plane is defined by its orthogonal vectors, which can be found as the cross product of two vectors in the plane, namely the normal vector N(O, A, B) = OA × OB. We assume that... Figure 11 As shown, two parallel segments are defined by real points A'C' and B'D'. These points are displayed as points AC and BC in a spherical camera setup. The direction of A'C' is orthogonal to N(O, A, C). It is also orthogonal to N(O, B, D) (since it is the same as the direction B'D'). Therefore, the real-world direction A'C' is parallel to the cross product N(O, A, C) × N(O, B, D), which is available in the camera setup's coordinate system.

[0059] Our conclusion is that parallel segments A'C' and B'D' (described as AC and BD in the camera device) are both parallel to vector V = (OA × OC) × (OB × OD). This is verified by checking the dot product (·) of the vectors.

[0060] If V·AC>0, then V and A'C' have the same direction.

[0061] If V·AC < 0, then V and A'C' are in opposite directions.

[0062] In parallelogram A'C'D'B', the point ACDB in the spherical image can be used to construct a three-dimensional solid. Using the formula for parallel segments mentioned above, we can find the two directions of the parallelogram as follows: V = (OA × OC) × (OB × OD) and W = (OA × OB) × (OC × OD). The normal of the parallelogram is their intersection V × W. We define the outer normal N as follows:

[0063] If (V×W)·OA<0, then N=V×W / normal(V×W);

[0064] If (V×W)·OA>0, then N=-V×W / normal(V×W).

[0065] Here, OA can be safely replaced by OB, OC, or OD.

[0066] Now consider the projection O' of O onto the plane of the parallelogram. If H is expressed as the length of O'O, then it can be expressed as...

[0067] OA' = OA * H / (-N·OA)

[0068] OB' = OB * H / (-N·OB)

[0069] OC' = OC * H / (-N·OC)

[0070] OD' = OD * H / (-N·OD)

[0071] Therefore, normal (B'A') = normal (OA*H / (-N·OA) - OB*H / (-N·OB))

[0072] Therefore, H = normal (B'A') / normal (OA / (-N·OA) - OB / (-N·OB))

[0073] Therefore, knowing the length of one side of a parallelogram (in our example, the length of A'B'), we can determine the spatial positions of all four vertices of the parallelogram using the coordinates of the camera device.

[0074] In the simple first case 1, the camera device moves linearly without rotation. Assume the camera device is as follows: Figure 12 As shown, the camera moves parallel to the road in a straight line without rotation, thus moving a vector V between the two frames. We further assume that two points are tracked, so that points A and C on the road in the first frame appear as B and D in the second frame. Instead of the camera moving by a magnitude V, we can assume that the camera is stationary, but the road moves by a magnitude -V. Points A and C represent real points A' and C' on the road. Then, if the road moves by a magnitude -V, points A' and C' will appear at new positions B' and D', so B' = A' - V and D' = C' - V. Points B' and D' are treated as B and D in the spherical camera setup. Figure 12 As shown, points A', C', B' = A'-V and D' = C'-V form a parallelogram.

[0075] Now, let's assume the vehicle moves in a straight line without rotation in the first scenario. Then, vector V is the vector representing the vehicle's linear motion. Since the velocity and time between two image frames are known in the CAN (Controller Area Network) data, we know the length of V. Therefore, we can use the formula derived above for the parallelogram to calculate the direction of V, the normal N, and the distance H from the camera to the road. These parameters define the extrinsic parameters to be determined in this simple scenario.

[0076] In the second scenario, the camera moves in a rotational manner along a known axis and a known angle. If we compensate for this rotation, i.e., rotate the camera back, this scenario simplifies to the previous one. The camera rotates about an axis OZ orthogonal to the road. In this case, OZ is known in the camera coordinate system. If we define the known rotation angle occurring between two frames as CarAngleOfRotation, the rotation matrix about the known axis OZ_iter at the known angle is described by the Rodrigues formula, and its calculation result is:

[0077] Rodrigues(OZ_iter*CarAngleOfRotation).

[0078] Now let's consider how to compensate for the rotation. Note that the camera device does indeed move in a curved path, but since we are only concerned with two positions, we can assume that the camera device moves only in a straight line along the epipolar vector between these two positions. The motion of the camera device between two frames can be described in two stages: the first stage is the straight line motion along the epipolar vector V without rotation. The second stage involves rotating the camera device in the new position (rotating to CarAngleOfRotation). After the first stage of movement, points A' and C' become B' = A' - V and D' = C' - V, exactly as in the previous section, with the four points forming a parallelogram. Then, in the second stage of movement, points B and D rotate in the opposite direction by Rodrigues(OZ_iter * CarAngleOfRotation). This can be interpreted by considering that if the camera device rotates to the right, the visible image will rotate to the left. After the entire movement and the first and second steps described above are completed, we can see points B_ and D_ in the spherical image, meaning they are available for calculation. However, we want to obtain B and D that are not directly available for use.

[0079] OB=Rodrigues(OZ*CarAngleOfRotation)*OB_;

[0080] OD=Rodrigues(OZ*CarAngleOfRotation)*OD_;

[0081] Then we move into the first case 1, whereby we find the epipolar line V, and can use the fact that the length of the epipolar line is known since the movement of the vehicle and the camera device is known, to calculate the normal N and the height of the camera device.

[0082] Please note that we are calculating the normal N. If all conditions are fully met, then N should equal OZ.

[0083] Since we know the angle between the epipolar line V and the direction of travel, we can find the direction of travel by rotating V around N to this known angle. From this, we conclude that we can obtain the upward direction from N, the direction of travel, and the height of the camera device, thus obtaining the extrinsic parameters.

[0084] We propose an optimal estimation apparatus that provides more accurate extrinsic parameter measurements in the calibration method. Due to physical installation deviations of the camera device, the distribution of calibration measurements becomes wider, thus affecting accuracy. The optimal estimation apparatus employs a time-varying sample calibration history and iteratively adjusts the extrinsic samples to a convergent output by applying sample statistics and comprehensive error analysis.

[0085] By studying the accompanying drawings, this patent document, and the appended claims, those skilled in the art can understand and implement other variations of the disclosed embodiments when practicing the claimed invention. In the claims, the word "comprising" does not exclude other parts or steps, and the indefinite articles "a" or "an" do not exclude several. A single processor or other unit can perform the function of a given part or step in several claims. The fact that specific measures are implemented in mutually unrelated dependent claims does not imply that these measures cannot be combined in an advantageous manner. Computer programs may be stored / distributed on suitable media, such as optical storage media or solid-state media provided or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method (100) for determining the external parameters of a camera device, comprising the following steps: The movement of the camera device begins (102); The moving camera device detects a first original image with a first patch and a second patch parallel to the first patch at a first time point and a second original image with a third patch and a fourth patch parallel to the third patch at a second time point (104), wherein the distance between the first patch and the second patch is the same as the distance between the third patch and the fourth patch. Detect (108) the reference position A of at least one image feature in the first patch, the reference position C of at least one image feature in the second patch, the offset position B of at least one feature in the first patch in the third patch, and the offset position D of at least one feature in the second patch in the fourth patch; Based on reference positions A and C, offset positions B and D, and the determined distance traversed by the moving camera device between the first and second time points, the (110) epipolar model is applied; and The external parameters of the camera device are determined (112) as the output of the epipolar model.

2. The method (100) according to claim 1, wherein, The movement of the camera device is linear, circular, or a combination of both; wherein the distance traveled by the moving camera device is determined by external sensors and / or measurement data.

3. The method (100) according to claim 1 or 2, wherein, The camera device is a fisheye camera device, wherein the step of detecting (108) reference position A, offset position B, reference position C and offset position D includes mapping the first, second, third and fourth patches to obtain the corresponding patches as two-dimensional ground plane images in a predefined image format in a virtual plane.

4. The method (100) according to claim 3, wherein, The first, second, third, and fourth patches in the virtual plane are preprocessed, wherein the preprocessing includes subtracting the average brightness value and applying a Gaussian window, wherein the Gaussian window has a maximum value at the center of the image and a minimum value at the edge of the image.

5. The method (100) according to claim 1 or 2, wherein, The offset position B of the third patch relative to the reference position A of the first patch and the offset position D of the fourth patch relative to the reference position C of the second patch are determined by performing a phase correlation.

6. The method (100) according to claim 5, wherein, The phase correlation includes Fast Fourier Transform (FFT).

7. The method (100) according to claim 1 or 2, wherein, The reference positions A and C, as well as the offset positions B and D, are relative positions, and the offsets are determined using a gradient-based similarity metric.

8. The method (100) according to claim 1 or 2, wherein, Positions A, B, C, and D are their locations in the coordinate system. Wherein, the origin of the coordinate system coincides with the lens of the camera device; and The step of detecting (108) reference position A, offset position B, reference position C and offset position D also includes determining the ground plane unit vector from the origin of the coordinate system to reference positions A and C and to offset positions B and D.

9. The method (100) according to claim 8, wherein, The steps of applying the (110) epipolar model include: using ground plane cell vectors to determine, in a first sub-step, a first direction V from reference position A to offset position C and a second direction W from reference position A to reference position B, wherein the first direction V is the same as the direction from reference position B to offset position D and the second direction W is the same as the direction from offset position C to offset position D; determining the normal vector N = V x W for the first direction V and the second direction W; and determining the height H of the camera device by determining vectors from the origin to the actual positions A', B', C' and D' corresponding to image positions A, B, C and D respectively.

10. A single camera device calibration system (200) for calibrating camera devices, comprising: A camera device controller (220) is configured to control a camera device to be calibrated such that the camera device detects a first original image having a first patch and a second patch parallel to the first patch at a first time point during movement, and detects a second original image having a third patch and a fourth patch parallel to the third patch at a second time point, wherein the distance between the first patch and the second patch is the same as the distance between the third patch and the fourth patch. Processing module, the processing module being configured to: Detect the reference position A of at least one image feature in the first patch, the reference position C of at least one image feature in the second patch, the offset position B of at least one feature in the first patch in the third patch, and the offset position D of at least one feature in the second patch in the fourth patch; and configure for Based on reference positions A and C, offset positions B and D, and the determined distance traversed by the moving camera device between the first and second time points, an epipolar model is applied; and configuration is used for... The external parameters of the camera device are determined as the output of the epipolar model.

11. The single camera device calibration system (200) for calibrating a camera device according to claim 10, wherein, The system also includes a camera device controlled by a camera device controller, wherein the camera device is mounted on a vehicle (300).

12. A vehicle (300) including a single camera device calibration system (200) for calibrating a camera device as described in claim 10 or 11.

13. The single camera device calibration system (200) according to claim 11 is used for determining the external parameters of a camera device mounted on a vehicle (300).

14. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method (100) of any one of claims 1 to 9.

15. A computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method (100) of any one of claims 1 to 9.

Images