Camera calibration method, device, controller, vehicle and storage medium

Abstract

The camera calibration method, device, controller, vehicle and storage medium provided in the embodiments of the present disclosure are used to obtain a set of orientation views collected by a panoramic vision system on a road in each current period and perform image recognition fine-tuning processing; the fine-tuning processing includes: using the current internal parameter and the current external parameter of each camera to perform distortion correction and perspective conversion on each orientation view, and extracting lane lines based on edge features; performing deviation correction processing based on the front view bird's-eye view and the rear view bird's-eye view to obtain the fine-tuned front view bird's-eye view and the fine-tuned rear view bird's-eye view, wherein the deviation correction processing is to calculate a perspective transformation deviation matrix by using the coordinate conversion relationship between the reference points in the front / rear view and the same-named points in the left view and the right view to fine-tune the front view bird's-eye view / rear view bird's-eye view; and performing an iterative calibration action to iteratively update the fine-tuning result; and the calibration is determined to be successful when the iteration meets the calibration target. The calibration can be conveniently performed when the vehicle is located in an actual road, and the calibration is efficient and convenient.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a camera calibration method, device, controller, vehicle, and storage medium. Background Art

[0002] In traditional driving and parking scenarios, drivers usually rely on rearview mirrors and reversing cameras for assistance. However, this creates a large number of blind spots, which results in a limited understanding of the vehicle's surroundings and can lead to adverse consequences such as vehicle collisions.

[0003] The emergence of panoramic vision systems addresses this issue. These systems typically use cameras (also known as cameras) mounted on the front, rear, left, and right sides of a vehicle. These systems capture four directional views and stitch together overlapping images to create a bird's-eye view, effectively eliminating blind spots.

[0004] Because cameras in panoramic vision systems typically use fisheye lenses, which can cause image distortion, they also require a transformation from the image coordinate system to the real-world coordinate system. Therefore, after the panoramic vision system is installed on the vehicle, camera calibration is performed to determine intrinsic and extrinsic parameters. Common calibration methods include using a black and white checkerboard grid. The resulting calibration parameters, such as intrinsic and extrinsic parameters, are stored in the controller for practical use.

[0005] However, after calibration, the vehicle is likely to undergo structural changes in real-world scenarios. This can occur due to assembly errors, installation errors after after-sales camera replacement or adjustment, or camera shift caused by gaps in the vehicle over time. These can all lead to deviations from the original calibration extrinsic parameters. In particular, extrinsic parameters correspond to the positional relationship between the camera coordinate system and the world coordinate system. Deviations from these extrinsic parameters can lead to inaccurate stitched panoramic views, misaligned stitching, and localized distortion. However, recalibration requires a return trip to the factory, which is very inconvenient. Summary of the Invention

[0006] In view of the above-mentioned shortcomings of the prior art, the purpose of the present disclosure is to provide a camera calibration method, device, controller, vehicle and storage medium to solve the problems in the related art.

[0007] A first aspect of the present disclosure provides a camera calibration method, which is applied to the calibration of cameras in a panoramic vision system mounted on a vehicle, wherein the panoramic vision system includes cameras arranged in front, rear, left and right directions, and the method includes: in each current cycle, obtaining a set of azimuth views acquired by the panoramic vision system on the road and performing image recognition fine-tuning processing; the set of azimuth views includes: a front view, a left view, a right view and a rear view; wherein the image recognition fine-tuning processing includes: using the current internal parameters and current external parameters of each camera, performing distortion correction and a first perspective transformation of the bird's-eye view on each of the azimuth views to obtain a set of converted bird's-eye views; extracting lane lines in each bird's-eye view; performing deviation correction processing based on the front bird's-eye view and the rear bird's-eye view in the set of bird's-eye views to obtain a fine-tuned front bird's-eye view and a fine-tuned rear bird's-eye view, including: a pair of parallel lane lines in the front bird's-eye view / rear bird's-eye view At least two reference points are selected above, and at least two points of the same name on at least one lane line are selected in the left view and the right view respectively; based on the image coordinate transformation relationship between each reference point and its points of the same name, a perspective transformation deviation matrix of the front bird's-eye view / rear bird's-eye view representing the image coordinate transformation relationship is calculated; based on the perspective transformation deviation matrix, the current external parameter is corrected and applied to the first perspective transformation to obtain a fine-tuned front bird's-eye view / fine-tuned rear bird's-eye view; in response to the group of bird's-eye views obtained by the image recognition fine-tuning processing not meeting the calibration target, an iterative calibration action is performed, including: iteratively updating the fine-tuned front bird's-eye view and the fine-tuned rear bird's-eye view in the group of bird's-eye views within a preset time length or a preset number of iteration rounds, and judging whether the updated group of bird's-eye views meets the calibration target; in response to the group of bird's-eye views obtained by the image recognition fine-tuning processing meeting the calibration target, the calibration is determined to be successful.

[0008] In an embodiment of the first aspect, the method further includes: in response to the group of bird's-eye views obtained by the image recognition fine-tuning process not meeting the calibration target, performing an iterative calibration action, including: iteratively updating the fine-tuned front bird's-eye view and the fine-tuned rear bird's-eye view in the group of bird's-eye views within a preset time length or a preset number of iteration rounds, and judging whether the updated group of bird's-eye views meet the calibration target; in response to the calibration target being met, determining that the calibration is successful; in response to the calibration target not being met after performing the iterative calibration action, but the image can be recognized from the finally updated group of bird's-eye views. If the lane line features are not met after the iterative calibration action is performed, the iterative calibration action is re-executed according to the finally updated set of bird's-eye views; in response to the calibration target not being met after the iterative calibration action is performed, and the lane line features cannot be identified from the finally updated set of bird's-eye views, a set of azimuth views is re-collected to perform the image recognition fine-tuning process; in response to the situation that the iterative calibration action based on a set of bird's-eye views after the image recognition fine-tuning process and the iterative calibration action according to each set of re-collected azimuth views still does not meet the calibration target for a preset time or a preset number of collections, the calibration is determined to have failed.

[0009] In an embodiment of the first aspect, the extracting of lane lines in each bird's-eye view based on each of the edge features includes: based on filtering conditions related to the shape and / or scale features of the lane lines, respectively extracting image feature points that meet the filtering conditions in each edge to obtain a feature point vector set for each edge of the lane line; and performing a straight line fitting method based on the feature point vector set of each edge to obtain two edges of the lane line.

[0010] In an embodiment of the first aspect, the calibration target includes: a maximum pixel difference between image feature points in each edge is less than a preset threshold.

[0011] In an embodiment of the first aspect, the reference points on the pair of parallel lane lines are arranged in pairs, and the two reference points in the pair have the same longitudinal coordinate.

[0012] In an embodiment of the first aspect, the method for selecting the reference point and the same-name point includes: selecting the middle point in the point set on the lane line with the lowest coordinate value discreteness or lower than a preset discreteness threshold as the reference point and the same-name point.

[0013] In an embodiment of the first aspect, the perspective transformation deviation matrix of the front-view bird's-eye view / rear-view bird's-eye view representing the image coordinate transformation relationship between each reference point and its homonymous point is calculated, including: jointly establishing a relationship equation between the image ordinate of each reference point and the image ordinate and image abscissa of the homonymous point, and a relationship equation between the image abscissa of each reference point and the image ordinate and image abscissa of the homonymous point; wherein the coefficients in the relationship equations are parameters in the perspective transformation deviation matrix; and obtaining the perspective transformation deviation matrix by solving the jointly established relationship equations to obtain the parameters in each perspective transformation deviation matrix.

[0014] According to a second aspect of the present disclosure, there is provided a camera calibration device for calibrating cameras in a panoramic vision system mounted on a vehicle, wherein the panoramic vision system includes cameras arranged in front, rear, left and right directions; the device includes: an image recognition fine-tuning processing module for acquiring a set of azimuth views acquired by the panoramic vision system on the road in each current cycle and performing image recognition fine-tuning processing; the set of azimuth views includes: a front view, a left view, a right view and a rear view; wherein the image recognition fine-tuning processing includes: using the current internal parameters and the current external parameters of each camera, performing distortion correction and a first perspective transformation of a bird's-eye view on each of the azimuth views to obtain a set of transformed bird's-eye views; extracting lane lines in each bird's-eye view; performing deviation correction processing based on the front bird's-eye view and the rear bird's-eye view in the set of bird's-eye views to obtain a fine-tuned front bird's-eye view and a fine-tuned rear bird's-eye view, including: a pair of parallel lanes in the front bird's-eye view / rear bird's-eye view At least two reference points are selected on the line, and at least two points of the same name are selected on at least one lane line in the left view and the right view respectively; based on the image coordinate transformation relationship between each reference point and its points of the same name, a perspective transformation deviation matrix of the front bird's-eye view / rear bird's-eye view representing the image coordinate transformation relationship is calculated; based on the perspective transformation deviation matrix, the current external parameter is corrected and applied to the first perspective transformation to obtain a fine-tuned front bird's-eye view / fine-tuned rear bird's-eye view; a calibration result judgment module is used to perform an iterative calibration action in response to the group of bird's-eye views obtained by the image recognition fine-tuning processing not meeting the calibration target, including: iteratively updating the fine-tuned front bird's-eye view and the fine-tuned rear bird's-eye view in the group of bird's-eye views within a preset time length or a preset number of iteration rounds, and judging whether the updated group of bird's-eye views meets the calibration target; in response to the group of bird's-eye views obtained by the image recognition fine-tuning processing meeting the calibration target, judging that the calibration is successful.

[0015] A third aspect of the present disclosure provides a controller, comprising: a memory and a processor; the memory stores program instructions, and the processor is configured to execute the program instructions to implement the camera calibration method as described in any one of the first aspects.

[0016] A fourth aspect of the present disclosure provides a vehicle, comprising: an onboard panoramic vision system; and a controller as described in the third aspect, communicatively connected to the onboard panoramic vision system.

[0017] A fifth aspect of the present disclosure provides a computer-readable storage medium storing program instructions, wherein the program instructions are executed to perform the camera calibration method according to any one of the first aspects.

[0018] As described above, the disclosed embodiments provide a camera calibration method, apparatus, controller, vehicle, and storage medium. In each current cycle, a set of azimuth views acquired by the panoramic vision system on the road are acquired and image recognition fine-tuning processing is performed. The fine-tuning processing includes: utilizing the current intrinsic and extrinsic parameters of each camera to perform distortion correction and perspective transformation on each azimuth view, and extracting lane lines based on edge features; performing deviation correction processing based on the front and rear bird's-eye views to obtain a fine-tuned front and rear bird's-eye view. The deviation correction processing is to calculate the perspective transformation deviation matrix using the coordinate transformation relationship between the reference points in the front / rear view and the same-name points in the left and right views to fine-tune the front and rear bird's-eye views; and performing iterative calibration actions to iteratively update the fine-tuning results. The calibration is determined to be successful when the iterations meet the calibration target. This allows convenient correction of calibration parameter errors when the vehicle is on the actual road, which is efficient and convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG1(a), FIG1(b), FIG1(c) and FIG1(d) are schematic diagrams showing scenes formed by different lane lines when a vehicle is traveling on a road in one embodiment of the present disclosure.

[0020] Figure 2 A flowchart illustrating a camera calibration method according to an embodiment of the present disclosure is shown.

[0021] Figure 3 A schematic diagram showing a specific implementation flow of step S201 in an embodiment of the present disclosure is shown.

[0022] Figure 4 A schematic diagram showing the principle of conversion from world coordinate system to pixel coordinate system.

[0023] Figure 5 An example image showing distortion correction and a first perspective transformation in an embodiment of the present disclosure.

[0024] Figure 6 A schematic diagram showing an image processing result obtained after edge features are extracted from a set of transformed orientation views obtained by distortion correction and the first perspective transformation in one embodiment of the present disclosure.

[0025] Figure 7 A schematic diagram showing lane line extraction based on edge features in one embodiment of the present disclosure.

[0026] FIG8( a ) is a schematic diagram showing the result of selecting two points of the same name on a lane line in a left-view bird's-eye view in one embodiment of the present disclosure.

[0027] FIG8( b ) is a schematic diagram showing the result of selecting two points of the same name on a lane line in a right-view bird's-eye view in one embodiment of the present disclosure.

[0028] FIG8( c ) is a schematic diagram showing the result of selecting four reference points on two lane lines in the front view in one embodiment of the present disclosure.

[0029] Figure 9 A module schematic diagram showing a camera calibration device according to an embodiment of the present disclosure is shown.

[0030] Figure 10 A schematic diagram showing the structure of a controller in one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] The following describes the embodiments of the present disclosure through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present disclosure from the information disclosed in this disclosure. The present disclosure can also be implemented or applied through different specific embodiments. The details of the present disclosure can also be modified or changed according to different viewpoints and application modules without departing from the spirit of the present disclosure. It should be noted that the embodiments and features in the embodiments of the present disclosure can be combined with each other unless there is a conflict.

[0032] The following is a detailed description of the embodiments of the present disclosure with reference to the accompanying drawings so that those skilled in the art can easily implement the present disclosure. The present disclosure can be embodied in many different forms and is not limited to the embodiments described herein.

[0033] Throughout the present disclosure, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present disclosure. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or a group of embodiments or examples. Furthermore, those skilled in the art may combine and integrate different embodiments or examples, and features of different embodiments or examples, as described in the present disclosure, without conflicting requirements.

[0034] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the context of this disclosure, "a group" means two or more, unless otherwise specifically defined.

[0035] In order to clearly describe the present disclosure, components not related to the description are omitted, and the same or similar components throughout the specification are denoted by the same reference numerals.

[0036] Throughout this specification, when a device is said to be "connected" to another device, this includes not only "direct connection" but also "indirect connection" with other elements interposed therebetween. Furthermore, when a device is said to "include" a certain component, unless otherwise stated, this does not exclude the inclusion of other components but rather implies that the device may include other components.

[0037] Although the terms first, second, etc. are used in this document to represent various elements in some examples, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, the first interface and the second interface, etc. are represented. Furthermore, as used in this document, the singular forms "one," "an," and "the" are intended to also include the plural forms, unless there is a contrary indication in the context. It should be further understood that the terms "comprise" and "include" indicate the presence of the described features, steps, operations, elements, modules, projects, types, and / or groups, but do not exclude the presence, occurrence, or addition of one or a group of other features, steps, operations, elements, modules, projects, types, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B, or C" or "A, B, and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B, and C." Exceptions to this definition only occur when the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0038] The technical terms used herein are intended only to refer to specific embodiments and are not intended to limit the present disclosure. The singular form used herein also includes the plural form, unless the statement explicitly indicates otherwise. The term "comprising" as used in this specification specifies specific features, regions, integers, steps, operations, elements, and / or components, and does not exclude the presence or addition of other features, regions, integers, steps, operations, elements, and / or components.

[0039] Although not defined differently, all terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art to which this disclosure belongs. Terms defined in commonly used dictionaries are additionally interpreted as having meanings consistent with relevant technical literature and the current message. Unless otherwise defined, they should not be overly interpreted as ideal or highly formalized meanings.

[0040] Surround vision systems are widely used in vehicles, effectively helping users eliminate blind spots during parking and driving, providing a safer and more convenient driving experience. However, while surround vision systems are pre-calibrated to determine their parameters before leaving the vehicle, in actual use, variations in the surround vision system structure, such as assembly errors and vehicle structural damage, can cause these calibration parameters to deviate from the actual scene, resulting in inaccurate panoramic views or misaligned stitching.

[0041] In view of this, an embodiment of the present disclosure provides a camera calibration method that can be used to correct calibration parameters that produce errors, so as to overcome the above-mentioned problems and obtain an accurate panoramic view.

[0042] like Figure 1(a) to Figure 1(d) , which is a schematic diagram showing an application scenario of a camera calibration method in one embodiment of the present disclosure.

[0043] The camera calibration method can be applied to roads with multiple parallel lane lines. The number of lane lines corresponds to the number of lanes. For example, a four-lane road in one direction has three parallel lane lines, while a two-lane road in one direction has two parallel lane lines. The type of lane lines is related to road driving regulations. For example, a dashed line indicates a lane line that can be crossed, while a solid line indicates a lane line that cannot be crossed.

[0044] Figure 1(a) to Figure 1(d) Each combination of lane lines on both sides of a vehicle on a road is shown. Figure 1(a) shows a scene with dashed lane lines on both sides of the vehicle, which can correspond to the middle lane of a one-way multi-lane road. Figure 1(b) shows a scene with solid lane lines on the left side of the vehicle and dashed lane lines on the right side, which can correspond to the middle lane in the right-hand lane of a two-way multi-lane road. Figure 1(c) shows a scene with dashed lane lines on the left side of the vehicle and solid lane lines on the right side, which can correspond to the middle lane in the left-hand lane of a two-way multi-lane road. Figure 1(d) shows a scene with solid lane lines on both sides of the vehicle, that is, lanes where lane changes are not allowed on both sides.

[0045] The vehicle shown in the figure may be equipped with a panoramic vision system. The panoramic vision system may include a front camera, a rear camera, a left camera, and a right camera, respectively mounted on the front, rear, left, and right sides of the vehicle. The system captures azimuth views from the outside of the corresponding directions, namely the front view, rear view, left view, and right view. The camera calibration method described above can be executed while the vehicle is driving or parked on the road to complete the correction of the original calibration parameters.

[0046] like Figure 2 FIG. 1 is a flow chart showing a camera calibration method according to an embodiment of the present disclosure, wherein the dashed boxes and corresponding arrows represent optional items.

[0047] exist Figure 2 In the embodiment, the camera calibration method includes:

[0048] Step S201: In each current cycle, a set of orientation views acquired by the panoramic vision system on the road is obtained and image recognition fine-tuning processing is performed.

[0049] In some embodiments, step S201 may be triggered by a preset trigger condition, which includes but is not limited to a software switch or a hard switch.

[0050] Specifically, the set of orientation views can be acquired synchronously. The set of orientation views includes: a front view, a left view, a right view, and a rear view. In some embodiments, n sets of orientation views may be acquired and stored in step S201, with each set of orientation views retrieved and used one by one in subsequent steps. Because each camera in each orientation uses a wide-angle lens, such as a fisheye lens, to capture images with the largest possible viewing angle, resulting in lens distortion (such as barrel distortion from a fisheye lens), the image recognition fine-tuning process first performs distortion correction on the acquired distorted orientation views. To address calibration parameter deviations caused by possible assembly or damage to the vehicle, such as external parameters describing the positional relationship between the world coordinate system and the camera coordinate system, the camera coordinate system is the camera coordinate system of each camera in the panoramic vision system, while the world coordinate system can be set to the vehicle coordinate system. Furthermore, a perspective transformation error matrix is ​​derived based on the image coordinate transformation relationship between the same-name points on the lane lines between the different orientation views. This matrix is ​​then used to correct the calibration parameters to obtain a more accurate front / rear view after fine-tuning.

[0051] Therefore, if Figure 3 FIG. 2 is a flow chart showing a specific implementation process of step S201 in an embodiment of the present disclosure. Step S201 may specifically include:

[0052] Step S301: using the current intrinsic parameters and current extrinsic parameters of each camera, performing distortion correction and a first perspective transformation of a bird's-eye view on each of the azimuth views to obtain a set of transformed bird's-eye views.

[0053] In some embodiments, the current internal parameters and the current external parameters are calibration parameters obtained by initial calibration after the vehicle-mounted panoramic vision system is installed on the vehicle.

[0054] To understand intrinsic and extrinsic parameters, you need to know four coordinate systems: the world coordinate system, the camera coordinate system, the image physical coordinate system, and the image pixel coordinate system.

[0055] World coordinate system: used to represent the absolute coordinates of spatial objects, expressed as (Xw, Yw, Zw).

[0056] Camera coordinate system: The optical center of the camera is the origin of the coordinate system (the optical center of the camera can be understood as the geometric center of the camera lens), and is expressed as (Xc, Yc, Zc), where the Xc and Yc axes are parallel to the Xw and Yw axes of the image coordinate system, the optical axis of the camera is the Zc axis, and the coordinate system satisfies the right-hand rule.

[0057] Image physical coordinate system: The coordinate origin is at the center of the CCD image plane, the x and y axes are parallel to the (u, v) axes of the image pixel coordinate system, and the coordinates are expressed as (x, y).

[0058] Image pixel coordinate system: This represents the projection of a three-dimensional object onto the image plane. Pixels are discretized, with the origin at the upper left corner of the CCD image plane. The u-axis is parallel to the CCD plane and points horizontally to the right, while the v-axis is perpendicular to the u-axis and points downward. Coordinates are expressed as (u, v). The image width is W and the height is H.

[0059] Intrinsic parameters are used to transform from the camera coordinate system to the image coordinate system, and from the image coordinate system to the pixel coordinate system. These parameters include 1 / dx, 1 / dy, r, u0, v0, and f. Because the original orientation view captured by the camera may be distorted, the pre-calibrated current intrinsic parameters are used to convert the actual image coordinates into pixel coordinates in the undistorted image. Extrinsic parameters are used to transform from the world coordinate system to the camera coordinate system. These parameters typically include the rotation matrix R and the translation vector t.

[0060] For details, please refer to Figure 4 As shown, a schematic diagram showing the principle of converting the world coordinate system to the pixel coordinate system is shown. Specifically, the camera coordinate system and the world coordinate system are just rigid body transformations of rotation (parameter matrix R) and translation (parameter matrix t) between three-dimensional coordinate systems with different center origins. R and t are the external parameters of the camera. The camera external parameters can be used to convert the coordinate points (Xw, Yw, Zw) in the world coordinate system to the coordinate points (Xc, Yc, Zc) in the camera coordinate system. The coordinate points (Xc, Yc, Zc) in the camera coordinate system can be converted to the coordinate points (x, y) in the image coordinate system (also called the image physical coordinate system) of the image captured by the camera through the matrix of the center projection. x, y are measured in physical dimensions. The coordinate points (x, y) in the image coordinate system can be converted to the coordinate points (u, v) in the pixel coordinate system of the image through the discretized parameter matrix. u, v are measured in pixels.

[0061] Based on the above, the following formula (1) shows the coordinate conversion from the world coordinate system to the pixel coordinate system using external parameters and internal parameters:

[0062]

[0063] That is, the formula (2) for converting the world coordinate system to the image coordinate system is:

[0064]

[0065] Among them, the integrated As internal reference, For external reference.

[0066] The current extrinsic parameters of each camera can be used to calculate the first perspective transformation from the directional view after distortion correction of the camera's perspective to the bird's-eye view. In some embodiments, a homography transformation matrix H from the coordinate system of the directional view to the coordinate system of the bird's-eye view is determined during calibration. It can be understood that the homography matrix is ​​actually determined based on the current extrinsic parameters of the camera, because the homography matrix is ​​equivalent to converting the image coordinates in a directional view to the world coordinate system and then to the coordinate system of the bird's-eye view using the intrinsic and extrinsic parameters of the corresponding camera. In other words, the first perspective transformation is actually achieved using the current extrinsic parameters.

[0067] like Figure 5 Figure 2 shows example images demonstrating distortion correction and a first perspective transformation in accordance with an embodiment of the present disclosure. The four front, back, left, and right views on the left exhibit barrel distortion. After distortion correction based on the current intrinsic parameters of each camera's orientation and a first perspective transformation based on the current extrinsic parameters, the four bird's-eye views on the right, representing the front, back, left, and right directions, exhibit distortion elimination. These four bird's-eye views can be stitched together based on overlapping pixels to create a panoramic image.

[0068] Step S302: extract edge features from each bird's-eye view based on a line detection algorithm.

[0069] In some embodiments, the acquired distortion-corrected and first perspective-transformed azimuth view can be converted into a grayscale image, filtered, and then edge features can be extracted using a line detection algorithm. For example, the line detection algorithm can be any one of a Hough transform, an LSD (line segment detection) line detection algorithm, an FLD line detection algorithm, an EDlines line detection algorithm, and a deep learning algorithm for detecting lines.

[0070] The extracted edge features are actually image feature point sets. Since lane lines are straight lines, the edge features extracted by the line detection algorithm will contain the image feature point sets of lane lines. Figure 6 Shown are the image processing results obtained after extracting edge features from a set of transformed azimuth views that are dedistorted and converted to a bird's-eye view.

[0071] Step S303: extracting lane lines in each bird's-eye view based on the edge features.

[0072] exist Figure 6As can be seen in the figure, the extracted edge features include some non-lane line features, such as greenery. Therefore, the inherent features of lane lines can be used to filter image feature points related to lane lines to form a point set. The lane line corresponding to the coordinate position of this point set can then be determined in the transformed orientation view.

[0073] In some embodiments, the inherent features of the lane lines are contained in the prior information, such as shape features and / or scale features. For example, the shape features may include lane lines being parallel to each other, parallel to the vehicle body, or parallel to the direction of travel of the vehicle. The scale features may include: the width of the lane lines being a certain preset size, the spacing between adjacent lane lines being a certain preset size, the lane lines being parallel to each other and parallel to the vehicle, etc. Therefore, after obtaining the edge features, based on the filtering conditions related to the lane line shape and / or scale features, the image feature points that meet the filtering conditions are extracted from each edge to obtain a feature point vector set for each edge of the lane line. Figure 7 In the example shown, the left side shows the four orientation images obtained after edge feature extraction, and the right side shows the result of extracting lane lines in the converted orientation views through edge feature correspondence. Since each lane line contains two edges, one is the inner edge close to the inside of the lane, and the other is the outer edge close to the outside of the lane, or it can also be determined according to the left and right edges in the figure. The width of each lane line is the distance between the edges on both sides of the lane line (as indicated by arrows C and D). If the detected points meet the above-mentioned screening conditions of road shape and / or scale features, such as parallel vehicle body or parallel vehicle driving direction (judged by whether the angle is 0, some angle deviation can be allowed), and the image feature points on both sides of the edges have a mutual width less than a preset threshold (such as 10 to 15 cm), they are placed in the image feature point set of each edge for lane line extraction.

[0074] A more specific embodiment is given to illustrate the process of extracting lane lines. The image after edge extraction can be denoised first, for example, by binarization, opening and closing intervals, and other operations. Then, based on the method that the lane line edges (each lane line includes two edges) are parallel (can be judged based on being parallel to the vehicle body or having an angle of 0 degrees with the direction of vehicle travel (a certain angle deviation is allowed) and the distance between the two edges (lane line width) is a specified distance (such as 10cm-15cm), the image feature points corresponding to each edge are extracted respectively to obtain a feature point vector set for each edge. In some embodiments, if multiple lane lines are preliminarily identified, such as more than 2, then for the left and right views, in order to reduce the amount of calculation, only one lane line can be taken. In the front and rear views, two lane lines are taken for the lane lines in the left and right images, respectively.

[0075] After obtaining each set of feature point vectors, a straight line fit can be performed on each feature point vector to obtain the fitted two lane edges. For example, the straight line fit can be weighted straight line fit, where the M pixels of the lane edge are fitted into small line segments, and then the fitted line segments are merged into a single straight line. Of course, in other embodiments, other straight line fit methods can also be used, not limited to weighted straight line fit.

[0076] Step S304: performing deviation correction processing based on the front bird's-eye view and the rear bird's-eye view in the group of bird's-eye views to obtain a fine-tuned front bird's-eye view and a fine-tuned rear bird's-eye view.

[0077] Step S304 specifically includes:

[0078] Step S3041: Select at least two reference points on a pair of parallel lane lines in the front bird's-eye view / rear bird's-eye view, and select at least two points of the same name on at least one lane line in the left view and the right view.

[0079] In some embodiments, the method for selecting reference points and homonymous points includes: selecting the middle point in the set of points on the lane line with the lowest coordinate value discreteness or below a preset discreteness threshold as the reference point and homonymous point. For example, in the image feature point set of a certain edge, the middle point with relatively smaller discreteness of the horizontal coordinate value (x value) and the vertical coordinate value (y value) is selected as the optimal point, which is used as the reference point and homonymous point. Since the deviation between the optimal point and the true point is smaller, it can be closer to the true edge line, making the result of the deviation correction processing more accurate. In some embodiments, after determining the reference point of the front view, the positions of the homonymous points of the left and right views are determined based on the position (x, y values) of the reference point in the coordinate system (i.e., it is assumed that the point of the front view coincides with the points of the left view and the right view, respectively). If there is no way to find a homonymous point with coordinates for some reference points, the closest point can be found, such as the point with the closest y distance. If there are multiple points with the same y distance, the coordinate difference between the two points in the x direction is further determined, and the point with the smallest x coordinate difference is selected as the homonymous point.

[0080] In some embodiments, reference points on different lane lines may be arranged in pairs, and the paired reference points may have the same vertical coordinate.

[0081] As shown in Figures 8(a), 8(b) and 8(c), Figure 8(c) shows the results of taking four points A, B, C and D on the two lane lines in the front bird's-eye view. Figure 8(a) shows the same-named point A' of A and the same-named point B' of B on a lane line in the left bird's-eye view (actually the same as the left one of the two lane lines in the front view). Figure 8(b) shows the same-named point C' of C and the same-named point D' of D on a lane line in the right bird's-eye view (actually the same as the right one of the two lane lines in the front view).

[0082] Step S3042: Based on the image coordinate transformation relationship between each reference point and its synonymous point, calculate the perspective transformation deviation matrix of the front view bird's eye view / back view bird's eye view representing the image coordinate transformation relationship.

[0083] Due to assembly errors, damage, and other factors, the relative positions of the left and right cameras to the front and rear cameras may change, causing the camera's current extrinsic parameters to be inaccurate. Therefore, the current extrinsic parameters of the front and rear cameras are calibrated by obtaining the actual positional relationship between the reference points and their corresponding points in the front and left views, i.e., the perspective transformation deviation matrix.

[0084] Continue Figures 8(a) to 8(c) In this example, points A, B, C, and D are considered to be in the (u, v) coordinate system. Points A', B', C', and D' are considered to be in the (x, y) coordinate system. Bias correction is to solve the perspective transformation matrix from (x, y) coordinates to (u, v) coordinates.

[0085] The general transformation formula for perspective transformation is formula (3):

[0086]

[0087] In formula (3):

[0088] u and v are the original images, and the parameter w is equal to 1

[0089] The image coordinates obtained by perspective transformation are x, y, where:

[0090] Equation 1:

[0091]

[0092] Perspective Matrix Denoted as M.

[0093] In the matrix a of formula (3) 33It is mainly used to assist in calculations, and its value is 1. Therefore, there are a total of 8 variables that need to be solved, that is, 8 equations (4 coordinate pairs) are required to solve. Therefore, in this step, at least 4 coordinate pairs are required, namely A's x-A', A's y-A', B's x-B', B's y-B', C's x-C', C's y-C', D's x-D', and D's y-D'. By jointly establishing the relationship equations between the image ordinate of each reference point and the image ordinate and image horizontal coordinates of the same-name point, as well as the relationship equations between the image horizontal coordinates of each reference point and the image ordinate and image horizontal coordinates of the same-name point; wherein the coefficients in the relationship equations are the parameters in the perspective transformation deviation matrix.

[0094] Correspondingly, the expressions of x and y after transformation are:

[0095] Equation 2:

[0096]

[0097] Given four point pairs A(u1, v1), B(u2, v2), C(u3, v3), D(u4, v4) and A'(x1, y1), B'(x2, y2), C'(x3, y3), D'(x4, y4), we can get equation group 3:

[0098]

[0099] The perspective transformation deviation matrix M can be obtained by solving equation group 3.

[0100] Similar to the above principle, by taking 4 reference points in the rear view and 2 points of the same name in the left and right views, a system of equations is established to solve the perspective transformation deviation matrix N.

[0101] Step S3043: Correcting the current extrinsic parameter based on the perspective transformation deviation matrix is ​​applied to the first perspective transformation to obtain a fine-tuned front-view bird's-eye view / a fine-tuned rear-view bird's-eye view.

[0102] Exemplarily, the M can be multiplied by the original current external parameter to obtain the deviation-corrected external parameter. When the deviation-corrected external parameter of the camera is used to apply the directional view of the corresponding camera to the first perspective transformation into a bird's-eye view, the fine-tuned forward bird's-eye view of the forward camera can be obtained.

[0103] Back to Figure 2In the method flow, step S202: Determine whether the obtained set of bird's-eye views meets the calibration target. If so, proceed to step S203; if not, proceed to step S204. In some embodiments, the calibration target can be determined based on the accuracy of lane line recognition. For example, the maximum difference between pixel coordinates on a set of points along a lane line edge is no greater than a preset threshold, such as 2 pixels.

[0104] Step S203: In response to the group of bird's-eye views obtained through the image recognition fine-tuning process meeting the calibration target, it is determined that the calibration is successful.

[0105] Step S204: In response to the set of bird's-eye views obtained from the image recognition fine-tuning process not meeting the calibration targets, an iterative calibration process is performed, including iteratively updating the fine-tuned front and rear bird's-eye views within the set of bird's-eye views for a preset duration or a preset number of iterations. In each iteration, step S202 is invoked based on the updated set of bird's-eye views to determine whether the calibration targets are met. If so, the calibration is considered successful. If not, the process proceeds to step S205.

[0106] Among them, through the iterative calibration action, the perspective transformation deviation matrices M and N can be updated to obtain the corresponding fine-tuned front view bird's-eye view and fine-tuned rear view bird's-eye view. The so-called fine-tuning means that during the iteration process, the fine-tuned front view / rear view bird's-eye view generated in the current cycle will change compared to the fine-tuned front view / rear view bird's-eye view of the previous cycle, so as to gradually correct possible coordinate misalignment.

[0107] Step S205: In each iteration round, determine whether the iterative calibration action reaches a preset time length or a preset number of rounds;

[0108] Step S206: Determine whether the iterative calibration action reaches a preset duration or a preset number of rounds; if so, proceed to step S207; if not, continue the iterative calibration action.

[0109] Step S207: Determine whether lane features can be identified from the set of bird's-eye views. If so, proceed to step S208; if not, proceed to step S209. In some embodiments, the inability to capture lane features means that no point set that meets the requirements (e.g., can fit the lane edge) can be found in the processed image.

[0110] Step S208: In response to not meeting the calibration target after executing the iterative calibration action, but lane line features can be identified from the final updated set of bird's-eye views, re-execute the iterative calibration action based on the final updated set of bird's-eye views.

[0111] Step S209: In response to the calibration target not being met after executing the iterative calibration action, and the lane line features cannot be recognized from the finally updated set of bird's-eye views, a new set of azimuth views is collected to perform the image recognition fine-tuning process, i.e., calling steps S201 and S202;

[0112] If the target is not met, the iterative calibration process continues. If the updated set of bird's-eye views during the iterative calibration process meets the calibration target, the calibration is considered successful. This process is then repeated. If the calibration target is still not met after the iterative calibration process, a new set of azimuth views is collected and the above process is repeated.

[0113] For example, a calibration failure condition can be set here, such as re-collecting a set of azimuth views and performing the above process and still not meeting the calibration target for a preset field of view or a preset number of rounds, then it can be determined as a failure. Therefore, the following steps can also be included:

[0114] Step S210: Determine whether a set of bird's-eye views based on each set of re-collected azimuth views after image recognition fine-tuning and iterative calibration still fails to meet the calibration target after iterative calibration; if not, continue the current process; if so, determine that the calibration has failed.

[0115] For example, the iterative calibration process described above involves performing image recognition fine-tuning on a set of views A collected during each cycle (e.g., 3 seconds) (i.e., obtaining a fine-tuned bird's-eye view based on the front and rear views of A in step S202), generating a new set of fine-tuned bird's-eye views A1. The next cycle continues with image recognition fine-tuning on A1 to generate A2, and so on, resulting in An, where n is a natural number. If, within a preset duration (e.g., 3 minutes, or a preset number of rounds), any of An (e.g., A1, A2, etc.) meets the calibration target (e.g., the maximum pixel coordinate difference within the set of points along the spliced ​​lane line edges is within 2 pixels), the calibration is successful and ends. If An obtained within 3 minutes does not meet the calibration target but still meets the characteristics of the collected lane line, the iterative process continues for the final An obtained within 3 minutes, and the iterative calibration process is performed for another 3 minutes. During this process, each updated set of fine-tuned bird's-eye views is evaluated to determine whether it meets the calibration target, and the calibration process ends; otherwise, the calibration process continues. Alternatively, if An obtained within 3 minutes does not meet the calibration target and Ap no longer meets the lane line feature, a new set of orientation views B will be collected. Image recognition and fine-tuning of B will be performed on B each cycle to generate Bn images. If the automatic calibration target is met within 3 minutes, calibration is successful and ends. Similarly, if Bn still does not meet the calibration target and Bn images no longer meet the lane line feature, a new image C will be collected, and this cycle will repeat. If the orientation views collected within a certain period of time or a certain number of times fail to meet the calibration target, the calibration will be displayed as failed and the process will exit.

[0116] like Figure 9 FIG2 shows a schematic diagram of a camera calibration device according to an embodiment of the present disclosure. The camera calibration device is used to calibrate cameras in a vehicle-mounted panoramic vision system, which includes cameras arranged in front, rear, left, and right directions. It should be noted that the principles and technical implementation of the camera calibration device can be referenced to the camera calibration method described in the previous embodiment, and therefore will not be repeated in this embodiment.

[0117] The camera calibration device 900 includes:

[0118] The image recognition fine-tuning processing module 901 is used to obtain a set of azimuth views acquired by the panoramic vision system on the road in each current cycle and perform image recognition fine-tuning processing; the set of azimuth views includes: front view, left view, right view and rear view; wherein, the image recognition fine-tuning processing includes: using the current internal parameters and current external parameters of each camera, performing distortion correction and first perspective transformation of the bird's-eye view on each azimuth view to obtain a set of transformed bird's-eye views; extracting the lane lines in each bird's-eye view; performing deviation correction processing based on the front bird's-eye view and the rear bird's-eye view in the set of bird's-eye views The method comprises the steps of: selecting at least two reference points on a pair of parallel lane lines of the front bird's-eye view / the rear bird's-eye view, and selecting at least two points of the same name on at least one lane line in the left view and the right view; calculating a perspective transformation deviation matrix of the front bird's-eye view / the rear bird's-eye view representing the image coordinate transformation relationship based on an image coordinate transformation relationship between each reference point and its points of the same name; and correcting the current extrinsic parameter based on the perspective transformation deviation matrix and applying the correction to a first perspective transformation to obtain the fine-tuned front bird's-eye view / the fine-tuned rear bird's-eye view.

[0119] The calibration result determination module 902 is configured to determine that the calibration is successful in response to the group of bird's-eye views obtained by the image recognition fine-tuning process meeting the calibration target.

[0120] It should be noted that in Figure 9 The various functional modules in the embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented in whole or in part in the form of a program instruction product. The program instruction product includes one or a group of program instructions. When the program instruction instructions are loaded and executed on a computer, the process or function according to the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium.

[0121] and, Figure 9 The devices disclosed in the embodiments can be implemented using other module division methods. The device embodiments shown above are merely illustrative. For example, the module division is merely a logical functional division. In actual implementation, other division methods may be used, such as a group of modules or modules that can be combined or dynamically integrated into another system, or some features that can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between devices or modules shown or discussed can be through some interface, and the indirect coupling or communication connection between devices or modules can be electrical or other forms.

[0122] in addition, Figure 9 Each functional module and submodule in the embodiments may be dynamically integrated into a single processing component, each module may exist physically independently, or two or more modules may be dynamically integrated into a single component. The aforementioned dynamic components may be implemented in hardware or as software functional modules. If the aforementioned dynamic components are implemented as software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0123] It should be noted that the processes or methods represented by the flowcharts of the above embodiments of the present disclosure can be understood as modules, segments, or portions of code that include one or more sets of executable instructions configured to implement specific logical functions or steps of a process. Furthermore, the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may be performed in a different order than that shown or discussed, including performing functions substantially simultaneously or in reverse order depending on the functions involved.

[0124] For example, Figure 2 The order of the steps in the method embodiments may be changed in specific scenarios and is not limited to the above.

[0125] In another embodiment of the present application, a vehicle is provided, comprising: an onboard panoramic vision system, namely, including front, rear, left, and right cameras; and a controller communicatively connected to the onboard panoramic vision system. The controller is configured to execute the camera calibration method described above.

[0126] like Figure 10 As shown, a schematic diagram of the structure of the controller in one embodiment of the present disclosure is shown.

[0127] The controller 1000 includes a bus 1001, a processor 1002, and a memory 1003. The processor 1002 and the memory 1003 can communicate with each other via the bus 1001. The memory 1003 can store program instructions. The processor 1002 executes the program instructions in the memory 1003 to implement the steps in the camera calibration method.

[0128] Bus 1001 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of illustration, although only one thick line is used in the figure, this does not mean that there is only one bus or only one type of bus.

[0129] In some embodiments, the processor 1002 may be implemented as a central processing unit (CPU), a microprocessor unit (MCU), a system on a chip (SoC), or a field programmable gate array (FPGA). The memory 1003 may include volatile memory, such as random access memory (RAM), for temporarily storing data while running programs.

[0130] The memory 1003 may also include a non-volatile memory (non-volatile memory) for data storage, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state disk (SSD).

[0131] In some embodiments, the controller 1000 may further include a communicator 1004. The communicator 1004 is used to communicate with the outside. In a specific example, the communicator 1004 may include one or a group of wired and / or wireless communication circuit modules. For example, the communicator 1004 may include one or more of a wired network card, a USB module, a serial interface module, etc. The wireless communication protocols followed by the wireless communication module include, for example, near field communication (NFC) technology, infrared (IR) technology, Global System for Mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth (BT), Global Navigation Satellite System (GNSS), etc. One or more of the following.

[0132] An embodiment of the present disclosure may further provide a computer-readable storage medium storing program instructions, which implement the camera calibration method in any of the previous embodiments when the program instructions are executed.

[0133] That is, the method steps in the above embodiments are implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or are implemented as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium downloaded via a network and to be stored in a local recording medium, so that the method represented herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA).

[0134] In summary, the disclosed embodiments provide a camera calibration method, device, controller, vehicle, and storage medium. In each current cycle, a set of azimuth views captured by the panoramic vision system on the road are acquired and image recognition fine-tuning processing is performed. The fine-tuning processing includes: utilizing the current intrinsic and extrinsic parameters of each camera to perform distortion correction and perspective transformation on each azimuth view, and extracting lane lines based on edge features; performing deviation correction processing based on the front and rear bird's-eye views to obtain a fine-tuned front and rear bird's-eye view. The deviation correction processing is to calculate the perspective transformation deviation matrix using the coordinate transformation relationship between the reference points in the front / rear view and the same-name points in the left and right views to fine-tune the front and rear bird's-eye views; and performing iterative calibration actions to iteratively update the fine-tuning results. The calibration is determined to be successful when the iterations meet the calibration target. This allows for convenient correction of calibration parameter errors when the vehicle is on the actual road, which is efficient and convenient.

[0135] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical concepts disclosed herein shall be covered by the claims of this disclosure.

Claims

1. A camera calibration method, characterized in that: The method is applied to calibrating cameras in a vehicle-mounted panoramic vision system, wherein the panoramic vision system includes cameras arranged in front, rear, left, and right directions. The method includes: In each current cycle, a set of azimuth views acquired by the panoramic vision system on the road is obtained and image recognition fine-tuning processing is performed; the set of azimuth views includes: a front view, a left view, a right view and a rear view; wherein, the image recognition fine-tuning processing includes: using the current internal parameters and current external parameters of each camera, performing distortion correction and a first perspective transformation of the bird's-eye view on each of the azimuth views to obtain a set of converted bird's-eye views, and extracting the lane lines in each bird's-eye view; performing deviation correction processing based on the front bird's-eye view and the rear bird's-eye view in the set of bird's-eye views to obtain the front and rear bird's-eye view after fine-tuning The method comprises: selecting at least two reference points on a pair of parallel lane lines in the front bird's-eye view / rear bird's-eye view, and selecting at least two points of the same name on at least one lane line in the left view and the right view; calculating a perspective transformation deviation matrix of the front bird's-eye view / rear bird's-eye view representing the image coordinate transformation relationship based on an image coordinate transformation relationship between each reference point and its points of the same name; and correcting the current extrinsic parameter based on the perspective transformation deviation matrix and applying the correction to a first perspective transformation to obtain the fine-tuned front bird's-eye view / rear bird's-eye view. In response to the set of bird's-eye views obtained by the image recognition fine-tuning process meeting the calibration target, determining that the calibration is successful; Also includes: In response to the set of bird's-eye views obtained from the image recognition fine-tuning process not meeting the calibration target, performing an iterative calibration action, including: iteratively updating the fine-tuned front view bird's-eye view and the fine-tuned rear view bird's-eye view in the set of bird's-eye views within a preset time duration or a preset number of iteration rounds, and determining whether the updated set of bird's-eye views meets the calibration target; In response to the calibration target being met, determining that the calibration is successful; In response to not meeting the calibration target after executing the iterative calibration action, but being able to identify lane features from the finally updated set of bird's-eye views, re-executing the iterative calibration action based on the finally updated set of bird's-eye views; In response to the calibration target not being met after executing the iterative calibration action, and the lane line features cannot be identified from the finally updated set of bird's-eye views, a set of azimuth views is re-collected to perform the image recognition fine-tuning process; in response to the situation that a set of bird's-eye views according to each set of azimuth views re-collected after the image recognition fine-tuning process and the iterative calibration action still does not meet the calibration target after the iterative calibration action for a preset time or a preset number of collections, the calibration is determined to have failed.

2. The camera calibration method according to claim 1, wherein: Extracting lane lines in each bird's-eye view includes: Based on the line detection algorithm, edge features are extracted for each bird's-eye view; Extracting lane lines in each bird's-eye view based on the edge features, including: extracting image feature points that meet the filtering conditions from each edge based on filtering conditions related to lane line shape and / or scale features, to obtain a feature point vector set for each lane line edge; A straight line fitting method is performed based on the feature point vector set of each edge to obtain the two edges of the lane line.

3. The camera calibration method according to claim 1, wherein: The calibration target includes: the maximum pixel difference between each image feature point in each edge is less than a preset threshold.

4. The camera calibration method according to claim 1, wherein: The reference points on the pair of parallel lane lines are arranged in pairs, and the two reference points in the pair have the same vertical coordinate.

5. The camera calibration method according to claim 1, wherein: The method for selecting the reference point and the same-name point includes: selecting the middle point in the point set on the lane line with the lowest coordinate value dispersion or lower than a preset dispersion threshold as the reference point and the same-name point.

6. The camera calibration method according to claim 1, wherein: The step of calculating the perspective transformation deviation matrix of the front-view bird's-eye view image / the rear-view bird's-eye view image representing the image coordinate transformation relationship based on the image coordinate transformation relationship between each reference point and its synonymous point includes: Simultaneously establish a relationship equation between the image ordinate of each reference point and the image ordinate and image abscissa of the same-name point, and a relationship equation between the image abscissa of each reference point and the image ordinate and image abscissa of the same-name point; wherein the coefficients in the relationship equations are parameters in the perspective transformation deviation matrix; The perspective transformation deviation matrix is ​​obtained by solving the simultaneous relationship equations to obtain parameters in each perspective transformation deviation matrix.

7. A camera calibration device, characterized in that: The device is used to calibrate cameras in a vehicle-mounted panoramic vision system, wherein the panoramic vision system includes cameras arranged in front, rear, left, and right directions; the device includes: An image recognition fine-tuning processing module is used to obtain a set of azimuth views acquired by the panoramic vision system on the road in each current cycle and perform image recognition fine-tuning processing; the set of azimuth views includes: a front view, a left view, a right view, and a rear view; wherein the image recognition fine-tuning processing includes: using the current internal parameters and current external parameters of each camera, performing distortion correction and a first perspective transformation of the bird's-eye view on each of the azimuth views to obtain a set of converted bird's-eye views, and extracting the lane lines in each bird's-eye view; performing deviation correction processing based on the front bird's-eye view and the rear bird's-eye view in the set of bird's-eye views Obtaining a fine-tuned front bird's-eye view and a fine-tuned rear bird's-eye view, including: selecting at least two reference points on a pair of parallel lane lines in the front bird's-eye view / rear bird's-eye view, and selecting at least two points of the same name on at least one lane line in the left view and the right view; calculating a perspective transformation deviation matrix of the front bird's-eye view / rear bird's-eye view representing the image coordinate transformation relationship based on an image coordinate transformation relationship between each reference point and its points of the same name; correcting the current extrinsic parameter based on the perspective transformation deviation matrix, and applying the correction to a first perspective transformation to obtain the fine-tuned front bird's-eye view / rear bird's-eye view; The calibration result determination module determines that the calibration is successful in response to the group of bird's-eye views obtained by the image recognition fine-tuning process meeting the calibration target; and performs an iterative calibration action in response to the group of bird's-eye views obtained by the image recognition fine-tuning process not meeting the calibration target, including: iteratively updating the fine-tuned front view bird's-eye view and the fine-tuned rear view bird's-eye view in the group of bird's-eye views within a preset time period or a preset number of iteration rounds, and determining whether the updated group of bird's-eye views meets the calibration target; In response to the calibration target being met, determining that the calibration is successful; In response to not meeting the calibration target after executing the iterative calibration action, but being able to identify lane features from the finally updated set of bird's-eye views, re-executing the iterative calibration action based on the finally updated set of bird's-eye views; In response to the calibration target not being met after executing the iterative calibration action, and the lane line features cannot be identified from the finally updated set of bird's-eye views, a set of azimuth views is re-collected to perform the image recognition fine-tuning process; in response to the situation that a set of bird's-eye views according to each set of azimuth views re-collected after the image recognition fine-tuning process and the iterative calibration action still does not meet the calibration target after the iterative calibration action for a preset time or a preset number of collections, the calibration is determined to have failed.

8. A controller, characterized in that: include: Memory and processor; The memory stores program instructions, and the processor is configured to execute the program instructions to implement the camera calibration method according to any one of claims 1 to 7.

9. A vehicle, characterized in that: include: Vehicle-mounted panoramic vision system; The controller according to claim 8 is communicatively connected to the vehicle-mounted surround view system.

10. A computer-readable storage medium, characterized in that Program instructions are stored, and the program instructions are executed to perform the camera calibration method according to any one of claims 1 to 6.

Images