Calibration method and device for external parameters of camera and terminal equipment

Abstract

The invention is suitable for the technical field of camera calibration, and provides a camera external parameter calibration method and device, and terminal equipment, and the method comprises the steps: carrying out the lane line detection of image data collected by a target camera through an initial external parameter and an initial internal parameter, so as to determine a detected lane line point set corresponding to each lane line in the image data; re-projecting map data corresponding to the field of view of the target camera to a plane coordinate system corresponding to the image data according to the initial external reference and the initial internal reference so as to determine a reference lane line point set corresponding to each lane line; and according to the difference between the detected lane line point set corresponding to each lane line and the reference lane line point set, performing iterative optimization on the initial external parameter to determine the target external parameter of the target camera. Therefore, real-time online automatic optimization of the external parameters of the camera is realized, manual participation is not needed, the calibration efficiency is improved, and the calibration accuracy is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of camera calibration, and in particular, relates to a camera extrinsic parameter calibration method, apparatus, terminal device and computer-readable storage medium. Background Art

[0002] Camera extrinsic calibration is a key step in application areas such as autonomous driving. The process of establishing a geometric model of camera imaging and matching points in the image with points on the surface of a spatial object is called camera extrinsic calibration. Camera extrinsic calibration is crucial in the process of restoring the three-dimensional information of an object in a two-dimensional image.

[0003] In related technologies, the imaging position of a calibration plate on the image plane is usually used to calculate the internal and external parameters of a camera. However, since a vehicle may bump while driving, or a camera may move for various reasons when installed on the roadside, using static calibration results as the camera's external parameters not only has high labor costs and low efficiency, but also has large errors and low accuracy. Summary of the invention

[0004] The embodiments of the present application provide a camera extrinsic parameter calibration method, apparatus, terminal device and storage medium, which can solve the problem of using static calibration results as camera extrinsic parameters, which not only has high labor costs and low efficiency, but also has large errors and low accuracy.

[0005] In a first aspect, an embodiment of the present application provides a method for calibrating camera extrinsic parameters, including: obtaining image data acquired by a target camera with initial extrinsic parameters and initial intrinsic parameters and map data corresponding to the field of view of the target camera, wherein the field of view of the target camera includes at least one lane line; performing lane line detection on the image data to determine a detected lane line point set corresponding to each lane line contained in the image data; reprojecting the map data into a plane coordinate system corresponding to the image data based on the initial extrinsic parameters and the initial intrinsic parameters to determine a reference lane line point set corresponding to each lane line; iteratively optimizing the initial extrinsic parameters based on the difference between the detected lane line point set corresponding to each lane line and the reference lane line point set to determine the target extrinsic parameters of the target camera.

[0006] In a possible implementation manner of the first aspect, the detected lane line point set includes M detected lane line points, the reference lane line point set includes N reference lane line points, and M and N are integers greater than 1; accordingly, the iterative optimization of the initial extrinsic parameters according to the difference between the detected lane line point set corresponding to each lane line and the reference lane line point set to determine the target extrinsic parameters of the target camera includes:

[0007] Determine the jth reference lane line point that matches the ith detected lane line point according to the distances between the ith detected lane line point and the M reference lane line points, where i is an integer greater than or equal to 1 and less than or equal to M, and j is an integer greater than or equal to 1 and less than or equal to N;

[0008] Determine the distance deviation and angle deviation corresponding to the i-th detected lane line point according to the coordinates of the i-th detected lane line point and the coordinates of the j-th reference lane line point;

[0009] According to the distance deviation and angle deviation corresponding to each detected lane line point, the loss value corresponding to the initial external parameter is determined;

[0010] The initial extrinsic parameters are iteratively optimized according to the loss value to determine the target extrinsic parameters.

[0011] Optionally, in another possible implementation of the first aspect, determining the loss value corresponding to the initial extrinsic parameter according to the distance deviation and the angle deviation corresponding to each detected lane line point includes:

[0012] Get the current usage scene corresponding to the target camera;

[0013] Determine a first weight corresponding to the distance deviation and a second weight corresponding to the angle deviation according to the current usage scenario;

[0014] The loss value is determined according to the first weight, the second weight, and the distance deviation and angle deviation corresponding to each detected lane line point.

[0015] Optionally, in another possible implementation manner of the first aspect, performing lane line detection on the image data to determine a detected lane line point set corresponding to each lane line included in the image data includes:

[0016] Input the image data into the lane detection model to determine the detected lane area corresponding to each lane;

[0017] Perform straight line detection on each detected lane line area to determine the detected corner point corresponding to each lane line;

[0018] According to the detected corner points corresponding to each lane line, a detected lane line point set corresponding to each lane line is determined.

[0019] Optionally, in yet another possible implementation of the first aspect, determining a detected lane line point set corresponding to each lane line according to the detected corner point corresponding to each lane line includes:

[0020] The detected corner points corresponding to each lane line are determined as a detected lane line point set corresponding to each lane line.

[0021] Optionally, in yet another possible implementation of the first aspect, before performing straight line detection on each detected lane line area to determine a detected corner point corresponding to each lane line, the method further includes:

[0022] Determine the number of pixels contained in each detected lane line area;

[0023] Remove the detected lane line areas where the number of pixels is less than the threshold.

[0024] Optionally, in another possible implementation manner of the first aspect, reprojecting the map data into a plane coordinate system corresponding to the image data according to the initial external parameters and the initial internal parameters to determine a reference lane line point set corresponding to each lane line includes:

[0025] Perform straight line detection on the map data to determine the reference corner point corresponding to each lane line;

[0026] According to the reference corner points, initial external parameters and initial internal parameters corresponding to each lane line, the map data is reprojected into the plane coordinate system corresponding to the image data to determine the reference lane line point set corresponding to each lane line.

[0027] Optionally, in another possible implementation of the first aspect, the map data is reprojected into a plane coordinate system corresponding to the image data according to the reference corner point, the initial external parameter, and the initial internal parameter corresponding to each lane line to determine a reference lane line point set corresponding to each lane line, including:

[0028] According to the initial external parameters and the initial internal parameters, the reference corner points corresponding to each lane line are reprojected into the plane coordinate system corresponding to the image data to determine the reference lane line point set corresponding to each lane line.

[0029] In a second aspect, an embodiment of the present application provides a camera extrinsic parameter calibration device, comprising: a first acquisition module, used to acquire image data collected by a target camera with initial extrinsic parameters and initial intrinsic parameters and map data corresponding to the field of view of the target camera, wherein the field of view of the target camera includes at least one lane line; a first determination module, used to perform lane line detection on the image data to determine a detected lane line point set corresponding to each lane line contained in the image data; a second determination module, used to reproject the map data into a plane coordinate system corresponding to the image data based on the initial extrinsic parameters and the initial intrinsic parameters to determine a reference lane line point set corresponding to each lane line; and a third determination module, used to iteratively optimize the initial extrinsic parameters based on the difference between the detected lane line point set corresponding to each lane line and the reference lane line point set to determine the target extrinsic parameters of the target camera.

[0030] In a possible implementation manner of the second aspect, the detected lane line point set includes M detected lane line points, the reference lane line point set includes N reference lane line points, and M and N are integers greater than 1; accordingly, the third determination module includes:

[0031] a first determining unit, for determining a j-th reference lane line point that matches the i-th detected lane line point according to distances between the i-th detected lane line point and the M reference lane line points, wherein i is an integer greater than or equal to 1 and less than or equal to M, and j is an integer greater than or equal to 1 and less than or equal to N;

[0032] A second determination unit is used to determine the distance deviation and the angle deviation corresponding to the i-th detected lane line point according to the coordinates of the i-th detected lane line point and the coordinates of the j-th reference lane line point;

[0033] A third determination unit is used to determine the loss value corresponding to the initial external parameter according to the distance deviation and the angle deviation corresponding to each detected lane line point;

[0034] The fourth determination unit is used to iteratively optimize the initial extrinsic parameter according to the loss value to determine the target extrinsic parameter.

[0035] Optionally, in another possible implementation manner of the second aspect, the third determining unit is specifically configured to:

[0036] Get the current usage scene corresponding to the target camera;

[0037] Determine a first weight corresponding to the distance deviation and a second weight corresponding to the angle deviation according to the current usage scenario;

[0038] The loss value is determined according to the first weight, the second weight, and the distance deviation and angle deviation corresponding to each detected lane line point.

[0039] Optionally, in another possible implementation of the second aspect, the first determining module includes:

[0040] A fifth determination unit, configured to input the image data into a lane line detection model to determine a detected lane line area corresponding to each lane line;

[0041] a sixth determination unit, configured to perform straight line detection on each detected lane line area to determine a detected corner point corresponding to each lane line;

[0042] The seventh determination unit is used to determine the detected lane line point set corresponding to each lane line according to the detected corner point corresponding to each lane line.

[0043] Optionally, in yet another possible implementation manner of the second aspect, the seventh determining unit is specifically configured to:

[0044] The detected corner points corresponding to each lane line are determined as a detected lane line point set corresponding to each lane line.

[0045] Optionally, in yet another possible implementation manner of the second aspect, the first determining module further includes:

[0046] an eighth determination unit, used to determine the number of pixels contained in each detected lane line area;

[0047] The first removing unit is used to remove the detected lane line area where the number of pixel points is less than a number threshold.

[0048] Optionally, in another possible implementation of the second aspect, the second determining module includes:

[0049] a ninth determination unit, configured to perform straight line detection on the map data to determine a reference corner point corresponding to each lane line;

[0050] The tenth determination unit is used to reproject the map data into a plane coordinate system corresponding to the image data according to the reference corner point, initial external parameters and initial internal parameters corresponding to each lane line, so as to determine a reference lane line point set corresponding to each lane line.

[0051] Optionally, in yet another possible implementation manner of the second aspect, the tenth determining unit is specifically configured to:

[0052] According to the initial external parameters and the initial internal parameters, the reference corner points corresponding to each lane line are reprojected into the plane coordinate system corresponding to the image data to determine the reference lane line point set corresponding to each lane line.

[0053] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the camera extrinsic parameter calibration method as described above when executing the computer program.

[0054] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the camera extrinsic parameter calibration method as described above.

[0055] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the camera extrinsic parameter calibration method as described above.

[0056] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by performing lane line detection on the image data collected by the target camera to determine the reference lane line point set corresponding to each lane line, and reprojecting the map data corresponding to the field of view of the target camera into the plane coordinate system corresponding to the image data to determine the reference lane line point set corresponding to each lane line as a reference, and then iteratively optimizing the initial extrinsic parameters according to the difference between the detected lane line point set and the reference lane line point set corresponding to each lane line to generate accurate target extrinsic parameters. Thus, by iteratively optimizing the camera extrinsic parameters according to the difference between the lane line collected by the camera and the accurate lane line position in the map data, so that the difference between the lane line collected by the camera and the accurate lane line position in the map data is small enough to obtain accurate camera extrinsic parameters, real-time online automatic optimization of the camera extrinsic parameters is achieved without manual intervention, which not only reduces the labor cost and improves the calibration efficiency, but also improves the accuracy of the camera extrinsic parameter calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 It is a flowchart of a method for calibrating camera extrinsic parameters provided in an embodiment of the present application;

[0059] Figure 2 It is a schematic diagram of converting a point in a rectangular coordinate system to a ρ-θ parameter space (Hough space) provided by an embodiment of the present application;

[0060] Figure 3 It is a schematic diagram of a point in a rectangular coordinate system corresponding to countless straight lines provided by an embodiment of the present application;

[0061] Figure 4 It is a schematic diagram of representing a point in a rectangular coordinate system in a Hough space provided by an embodiment of the present application;

[0062] Figure 5 is a schematic diagram of discretizing a Hough space provided by an embodiment of the present application;

[0063] Figure 6 is a schematic diagram of lane line detection provided by an embodiment of the present application;

[0064] Figure 7 is a schematic diagram of a detected lane line point set and a reference lane line point set provided by an embodiment of the present application;

[0065] Figure 8 is a flowchart of a camera extrinsic parameter calibration method provided by another embodiment of the present application;

[0066] Fig. 9 It is a structural schematic diagram of a camera extrinsic parameter calibration device provided in an embodiment of the present application;

[0067] Fig.10 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0069] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0070] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0071] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0072] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0073] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0074] The camera extrinsic parameter calibration method, apparatus, terminal device, storage medium and computer program provided in the present application are described in detail below with reference to the accompanying drawings.

[0075] Figure 1 A schematic flow chart of a camera extrinsic parameter calibration method provided in an embodiment of the present application is shown.

[0076] like Figure 1 As shown, the calibration method of the camera extrinsic parameters includes the following steps:

[0077] Step 101 , obtaining image data collected by a target camera using initial external parameters and initial internal parameters and map data corresponding to a field of view of the target camera, wherein the field of view of the target camera includes at least one lane line.

[0078] It should be noted that the camera extrinsic parameter calibration method of the embodiment of the present application can be executed by the camera extrinsic parameter calibration device of the embodiment of the present application. The camera extrinsic parameter calibration device of the embodiment of the present application can be configured in any terminal device to execute the camera extrinsic parameter calibration method of the embodiment of the present application. For example, the camera extrinsic parameter calibration device of the embodiment of the present application can be configured in the on-board unit (OBU) of the vehicle to calibrate the extrinsic parameters of the on-board camera, or it can also be configured in the roadside unit corresponding to the roadside camera used for road monitoring to calibrate the extrinsic parameters of the roadside camera.

[0079] The target camera may be a camera that currently needs to be calibrated for external parameters. For example, when the calibration method for camera external parameters in the embodiment of the present application is applied to an autonomous driving vehicle, the target camera may be a camera configured in the autonomous driving vehicle for detecting road conditions; for another example, when the calibration method for camera external parameters in the embodiment of the present application is applied to roadside cameras deployed in tunnels, urban intersections, etc., the target camera may be a roadside camera deployed on the roadside.

[0080] Among them, the initial intrinsic parameter may refer to the intrinsic parameter used by the target camera when acquiring an image before the extrinsic parameter calibration of the target camera is performed; the initial extrinsic parameter may refer to the extrinsic parameter used by the target camera when acquiring an image before the extrinsic parameter calibration of the target camera is performed.

[0081] The map data corresponding to the field of view of the target camera may refer to high-precision map data that can accurately represent the geographic coordinates of buildings, road forms (such as lane slope, curvature, heading and elevation data), road surface markings (such as lane lines) and other objects within the field of view of the target camera. For example, the map data may be high-precision map data of a bird's eye view (BEV) with centimeter-level accuracy.

[0082] In an embodiment of the present application, each time the target camera needs to be calibrated with external parameters, the target camera can be controlled to collect image data with its current initial external parameters and initial internal parameters, and the geographic coordinate range corresponding to its field of view can be determined based on the current positioning data of the target camera (such as longitude and latitude, etc.), and then the high-precision map data within the geographic coordinate range corresponding to its field of view can be extracted from the preset high-precision map data based on the geographic coordinate range corresponding to its field of view, as the map data corresponding to the field of view of the target camera.

[0083] As an example, after obtaining high-precision map data within the geographic coordinate range, the high-precision map data can also be processed using the Mercator projection method to obtain the longitude and latitude data corresponding to the high-precision map data, and the map data after the Mercator projection can be used as the map data corresponding to the field of view of the target camera.

[0084] It should be noted that the timing of calibrating the external parameters of the target camera can be determined according to the actual application scenario, and the embodiment of the present application does not limit this. For example, the target camera can be calibrated when the target camera is used for the first time. For example, when the target camera is installed in an autonomous driving vehicle, the external parameters of the target camera will change due to the ups and downs of the road, the position movement of the vehicle, etc.; or, when the target camera is a roadside camera installed at a tunnel or intersection, the target camera will also cause the position to change over time, so that the camera external parameters are no longer accurate. Therefore, it can also be set to calibrate the external parameters of the target camera with a preset period (such as 1 minute, 1 hour, 1 day, 1 week, 1 month, etc.), so that when the set calibration time arrives, it can be determined that the external parameter calibration request of the target camera is obtained, and the target camera is controlled to collect image data with the initial external parameters and initial internal parameters, and the map data corresponding to the field of view of the target camera is obtained to calibrate the external parameters of the target camera.

[0085] Step 102: perform lane line detection on the image data to determine a detected lane line point set corresponding to each lane line contained in the image data.

[0086] The detected lane line point set corresponding to the lane line may refer to a set consisting of all or part of the pixel points corresponding to the lane line in the image data.

[0087] In an embodiment of the present application, lane line detection can first be performed on the image data captured by the target camera to determine the pixel points corresponding to each lane line, and then the set consisting of all or part of the pixel points corresponding to the lane line can be determined as the detected lane line point set corresponding to the lane line.

[0088] As a possible implementation manner, after the pixel points corresponding to each lane line are determined, a set consisting of all the pixel points corresponding to the lane line can be determined as a detected lane line point set corresponding to the lane line.

[0089] As a possible implementation method, after determining the pixel points corresponding to each lane line, it is also possible to determine the edge pixel points corresponding to each lane line (such as the pixel points corresponding to the corner points, the pixel points corresponding to the edge lines), and determine the edge pixel points corresponding to each lane line as the detected lane line point set corresponding to the lane line.

[0090] As a possible implementation method, due to the limited field of view of the target camera, some lane lines may be obstructed to a certain extent, resulting in incomplete captured lane lines. The lane line data in the high-precision map are complete. Therefore, in order to ensure the accuracy of subsequent matching, it is also possible to judge whether each lane line is obstructed based on the number of pixels corresponding to the lane line; if the number of pixels corresponding to a lane line is less than the number threshold, it can be determined that the lane line is obstructed, which may affect the accuracy of subsequent matching with the high-precision map, and then affect the accuracy of the camera extrinsic calibration. Therefore, the lane line can be removed, that is, the detection lane line point set corresponding to the lane line is not generated, which can not only further improve the accuracy of the camera extrinsic calibration, but also further reduce the data processing complexity of the camera extrinsic calibration, and improve the efficiency of the camera extrinsic calibration.

[0091] Furthermore, the image area corresponding to each lane line can be firstly marked by a pre-trained lane line detection model, and then straight line detection can be performed on each image area to determine the pixel points corresponding to each lane line, so as to further improve the accuracy of lane line detection, and further improve the accuracy of camera extrinsic calibration. That is, in a possible implementation of the embodiment of the present application, the above step 102 can include:

[0092] Input the image data into the lane detection model to determine the detected lane area corresponding to each lane;

[0093] Perform straight line detection on each detected lane line area to determine the detected corner point corresponding to each lane line;

[0094] According to the detected corner points corresponding to each lane line, a detected lane line point set corresponding to each lane line is determined.

[0095] As a possible implementation method, the image data can be input into a pre-trained lane line detection model to detect the lane line area contained in the image data, and mark the image area corresponding to each lane line, so as to determine the detected lane line area corresponding to each lane line. After that, straight line detection can be performed on each detected lane line area to determine the straight lines contained in each lane line area, that is, the edge line of each lane line, and then the detected corner point corresponding to each lane line can be determined according to the edge line of each lane line. Then, the pixel point corresponding to each lane line can be determined according to the coordinates of the detected corner point corresponding to each lane line, and then the detected lane line point set corresponding to each lane line can be generated.

[0096] It should be noted that in actual use, a suitable deep learning model can be selected for training according to actual needs and specific application scenarios to generate a lane detection model. For example, the lane detection model can be trained using a typical U-Net network.

[0097] As a possible implementation method, the Hough transform can be used to perform straight line detection on each detected lane line area, so that the image data can be first converted from the RGB color space to the HSV color space to more easily extract the lane line from the image, and then the image data can be blurred by a fuzzy algorithm (such as Gaussian blur) to smooth the rough edges caused by noise in the image data to further highlight the lane lines in the image data; then the image data can be binarized to remove other objects in the image data and only retain the lane lines in the image data, thereby reducing the impact of other objects in the image data on straight line detection; then, the image data can be edge detected and binarized to obtain image data containing only lane line edges (the pixel value of the lane line edge is 255), and then the pixel coordinates corresponding to each edge in the image data are Hough transformed to convert the pixel coordinates corresponding to each edge to the ρ-θ parameter space (polar coordinate system), where, as Figure 2 As shown in , ρ represents the vertical distance between the straight line where the pixel point is located in the two-dimensional rectangular coordinate system and the origin of the coordinate system, and θ represents the angle between the normal line of the pixel point and the horizontal axis (X axis) of the coordinate system. It can be understood that Figure 3As shown in , since there can be countless straight lines passing through a point (x0, y0) in a two-dimensional rectangular coordinate system, after the coordinates of a pixel point in the image coordinate system are converted to the ρ-θ parameter space, it can correspond to countless pairs of ρ and θ values; that is, after the coordinates of a pixel point in the image coordinate system are converted to the ρ-θ parameter space, the pixel point corresponds to a sine curve in the ρ-θ polar coordinate system. And, since the points located in a straight line in a two-dimensional rectangular coordinate system have the same ρ and θ values, as Figure 3 As shown in , for a straight line A in a two-dimensional rectangular coordinate system, assuming that its corresponding ρ and θ values ​​are (ρ0, θ0), the sine curves corresponding to each point in the straight line will intersect at a point (ρ0, θ0) in the ρ-θ polar coordinate system; Figure 4 As shown in the figure, the sinusoidal curves corresponding to the points (x0, y0) and (x1, y1) intersect at the point (ρ0, θ0) in the ρ-θ parameter space. Therefore, after the pixel coordinates corresponding to each detected edge are subjected to Hough transformation and converted to the ρ-θ parameter space, it can be determined that the pixel points corresponding to each sinusoidal curve intersecting at the same point are located on a straight line, thereby realizing the straight line detection of the image data, that is, detecting the pixel coordinates corresponding to each lane line edge.

[0098] As mentioned above, Hough line detection is to transform the straight line in the image space into a point in the parameter space, and solve the detection problem through statistical characteristics. Specifically, if the pixels in an image form a straight line, then the curves corresponding to the coordinate values ​​(x, y) of these pixels in the parameter space must intersect at a point, so all the pixel points (coordinate values) in the image can be transformed into curves in the parameter space, and the straight line can be determined by detecting the intersection of the curves in the parameter space.

[0099] In theory, a point corresponds to countless straight lines or straight lines in any direction (the slope k or θ represented by the coordinate axis in the parameter space is countless), but in practical applications, the number of straight lines (i.e., a finite number of directions) needs to be limited to perform calculations. Therefore, the direction θ of the straight line can be discretized into a finite number of equally spaced discrete values, and the parameter ρ is also discretized into a finite number of values. As a result, the parameter space is no longer continuous, but is discretized into grid units of equal size, such as Figure 5 After transforming the coordinate value of each pixel point in the image space (rectangular coordinate system) to the parameter space (polar coordinate system), the obtained value will fall within a certain grid, causing the cumulative counter of the grid unit to increase by 1; Figure 5As shown, the value of the accumulation counter corresponding to the grid where (ρ0, θ0) is located is 2, and the values of the accumulation counters corresponding to the other grids passed by the two curves are 1. After all pixels in the image space have undergone the Hough transform, the grid cells are checked, and the ρ and θ values corresponding to the grids with the accumulation counter value greater than the threshold correspond to the straight lines sought in the image space.

[0100] As a possible implementation, after the straight-line detection of the image data is completed, the set of pixel points corresponding to the edges of each lane line can be determined (that is, the pixel set on the same straight line constitutes an edge of a lane line in the image data). Therefore, if a pixel point belongs to both the set of pixel points corresponding to the edge of one lane line and the set of pixel points corresponding to the edge of another lane line, it can be determined that the pixel point is a corner point of the lane line, thereby determining the detected corner points corresponding to each lane line. After determining the detected corner points corresponding to each lane line, the coordinate values of the detected corner points can be used to determine each pixel point inside the lane line, and then some or all of the pixel points corresponding to the lane line can be determined as the detected lane line point set corresponding to the lane line according to actual needs.

[0101] For example, as Figure 6 shown, assume that the coordinates of the four detected corner points A, B, C, and D of the lane line 610 are (x1, y1), (x2, y1), (x1, y2), and (x2, y2) respectively, where x1 < x2 and y1 < y2. Then, all pixel points in the image data with abscissa values greater than or equal to x1 and less than or equal to x2 and ordinate values greater than or equal to y1 and less than or equal to y2 can be determined as all pixel points corresponding to the lane line 610.

[0102] As an example, all pixel points corresponding to the lane line can be determined as the detected lane line point set corresponding to the lane line.

[0103] As an example, all edge pixel points corresponding to the lane line can also be determined as the lane line point set corresponding to the lane line.

[0104] As an example, since when matching the detected lane line point set with the lane lines in the map data, by correcting the difference between the corner point coordinates captured by the camera and the actual corner point coordinates of the lane lines in the map data, the optimization of the camera extrinsic parameters can be achieved, and the optimization through corner point matching is not only efficient but also has high accuracy, thereby further improving the accuracy and efficiency of the camera extrinsic parameter optimization. That is, in a possible implementation manner of the embodiments of the present application, the above-mentioned determining the detected lane line point set corresponding to each lane line according to the detected corner points corresponding to each lane line includes:

[0105] Determining the detected corner points corresponding to each lane line as the detected lane line point set corresponding to each lane line.

[0106] As a possible implementation manner, after the detected corner points corresponding to each lane line are determined, the set consisting of the detected corner points corresponding to each lane line can be respectively determined as the detected lane line point set corresponding to each lane line.

[0107] Furthermore, due to the limited field of view of the target camera, some lane lines may be blocked to a certain extent, resulting in incomplete captured lane lines, while the lane line data in the high-precision map are complete. Therefore, in order to ensure the accuracy of subsequent matching, it is also possible to determine whether each lane line is blocked based on the number of pixels corresponding to the lane line, and remove the blocked lane lines to further improve the accuracy of camera extrinsic calibration, and further reduce the data processing complexity of camera extrinsic calibration, thereby improving the efficiency of camera extrinsic calibration. That is, in a possible implementation method of the present application embodiment, before the above-mentioned straight line detection of each detected lane line area to determine the detected corner point corresponding to each lane line, it also includes:

[0108] Determine the number of pixels contained in each detected lane line area;

[0109] Remove the detected lane line areas where the number of pixels is less than the threshold.

[0110] As a possible implementation method, the sum of the pixels contained in the detected lane line area can be determined as the number of pixels contained in the detected lane line area. If the number of pixels corresponding to a lane line is less than the number threshold, it can be determined that the lane line is blocked, which may affect the accuracy of subsequent matching with the high-precision map, and further affect the accuracy of the camera extrinsic calibration. Therefore, the detected lane line area can be removed, that is, the detected lane line area is not detected in a straight line to generate its corresponding detected lane line point set, which can not only further improve the accuracy of the camera extrinsic calibration, but also further reduce the data processing complexity of the camera extrinsic calibration and improve the efficiency of the camera extrinsic calibration.

[0111] It should be noted that, in actual use, the specific value of the quantity threshold can be determined according to actual needs and specific application scenarios, and the embodiments of the present application do not limit this.

[0112] Step 103 , reprojecting the map data into a plane coordinate system corresponding to the image data according to the initial external parameters and the initial internal parameters, so as to determine a reference lane line point set corresponding to each lane line.

[0113] In an embodiment of the present application, an internal and external parameter matrix can be generated based on the initial external parameters and initial internal parameters of the target camera, and then the longitude and latitude coordinates corresponding to the lane line part in the map data can be transformed according to the internal and external parameter matrix to reproject the lane lines in the map data into the plane coordinate system corresponding to the image data, thereby generating a reference lane line point set corresponding to each lane line.

[0114] As a possible implementation method, if in step 102 the set consisting of all pixel points corresponding to the lane line is determined as the detected lane line point set corresponding to the lane line, then all the longitude and latitude coordinates in each lane line area in the map data can be reprojected into the plane coordinate system corresponding to the image data to generate a reference lane line point set corresponding to each lane line.

[0115] As a possible implementation method, if in step 102 the edge pixel points corresponding to each lane line (such as the pixel points corresponding to the corner points and the pixel points corresponding to the edge lines) are determined as the detected lane line point set corresponding to the lane line, the longitude and latitude coordinates corresponding to the edge of each lane line area in the map data can be reprojected into the plane coordinate system corresponding to the image data to generate a reference lane line point set corresponding to each lane line.

[0116] Furthermore, the map data may be subjected to straight line detection to determine the latitude and longitude coordinates corresponding to each lane line in the map data, so as to further improve the accuracy of lane line detection, and further improve the accuracy of camera extrinsic calibration. That is, in a possible implementation of the embodiment of the present application, the above step 103 may include:

[0117] Perform straight line detection on the map data to determine the reference corner point corresponding to each lane line;

[0118] According to the reference corner points, initial external parameters and initial internal parameters corresponding to each lane line, the map data is reprojected into the plane coordinate system corresponding to the image data to determine the reference lane line point set corresponding to each lane line.

[0119] As a possible implementation method, the map data can be detected by straight line detection through Hough transformation to determine the reference corner points corresponding to each lane line. After the reference corner points corresponding to each lane line in the map data are determined, all the longitude and latitude coordinates corresponding to each lane line in the map data can be determined based on the longitude and latitude coordinates of the reference corner points corresponding to each lane line; and then all the longitude and latitude coordinates corresponding to each lane line in the map data can be transformed according to the initial internal parameters and the initial external parameters to generate the reference lane line point set corresponding to each lane line.

[0120] It should be noted that the specific implementation process and principle of straight line detection on map data are the same as the specific implementation process and principle of straight line detection on image data in the aforementioned steps; the specific implementation process and principle of determining all longitude and latitude coordinates corresponding to the lane line in the map data according to the longitude and latitude coordinates of the reference corner point corresponding to the lane line are the same as the specific implementation process and principle of determining all pixel points corresponding to the lane line according to the detected corner point of the lane line in the aforementioned steps. Please refer to the detailed description of the aforementioned steps and will not be repeated here.

[0121] As an example, if all pixel points corresponding to the lane line are determined as the detected lane line point set corresponding to the lane line in step 102, all latitude and longitude coordinates corresponding to the lane line in the map data can be reprojected to determine the reference lane line point set corresponding to the lane line.

[0122] As an example, if all edge pixel points corresponding to the lane line are determined as the lane line point set corresponding to the lane line in step 102, the latitude and longitude coordinates of the edge line corresponding to the lane line in the map data can be reprojected to determine the reference lane line point set corresponding to the lane line.

[0123] As an example, when matching the detected lane line point set with the lane line in the map data, the camera extrinsic parameters can be optimized by correcting the difference between the corner point coordinates captured by the camera and the actual corner point coordinates of the lane line in the map data, and the optimization by corner point matching is not only efficient but also accurate, thereby further improving the accuracy and efficiency of the camera extrinsic parameter optimization. That is, in a possible implementation of the embodiment of the present application, the above-mentioned reprojection of the map data into the plane coordinate system corresponding to the image data according to the reference corner point, initial extrinsic parameter and initial intrinsic parameter corresponding to each lane line to determine the reference lane line point set corresponding to each lane line may include:

[0124] According to the initial external parameters and the initial internal parameters, the reference corner points corresponding to each lane line are reprojected into the plane coordinate system corresponding to the image data to determine the reference lane line point set corresponding to each lane line.

[0125] As a possible implementation method, if in step 102 the set of detected corner points corresponding to each lane line is determined as the detected lane line point set corresponding to each lane line, the reference corner point corresponding to each lane line can be transformed in coordinates according to the initial external parameters and the initial internal parameters, so as to project the reference corner point corresponding to each lane line into the plane coordinate system corresponding to the image data, thereby generating a reference lane line point set corresponding to each lane line.

[0126] For example, if Figure 6As shown, it is assumed that the image data includes a lane line 610, and the corresponding detected corner points are A, B, C, and D, and the detected corner points A, B, C, and D are used as the detected lane line point set corresponding to the lane line 610; therefore, the reference corner points corresponding to the lane line in the map data can be reprojected into the plane coordinate system corresponding to the image data, thereby generating the reference lane line point set A', B', C', and D' corresponding to the lane line, as shown in FIG. Figure 7 shown.

[0127] Step 104 , iteratively optimize the initial extrinsic parameters according to the difference between the detected lane line point set corresponding to each lane line and the reference lane line point set to determine the target extrinsic parameters of the target camera.

[0128] In the embodiment of the present application, since the reference lane line point set obtained by reprojecting the map data can be used to characterize the real position of the lane line, the greater the difference between the detected lane line point set corresponding to the lane line and its corresponding reference lane line point set, the less accurate the external parameters of the target camera are. Therefore, the initial external parameters can be optimized according to the difference between the detected lane line point set corresponding to each lane line and its corresponding reference lane line point set, and the image data can be recaptured using the updated external parameters and the initial internal parameters, and the map data can be reprojected using the updated external parameters and the initial internal parameters (i.e., repeating the above steps 101-104), and then according to the difference between the newly generated detected lane line point set corresponding to each lane line and the reference lane line point set, the updated external parameters are optimized again, until the updated external parameters can make the difference between the detected lane line point set corresponding to each lane line and the reference lane line point set small enough, then the updated external parameters can be determined as the target external parameters.

[0129] As a possible implementation method, the distance between points can measure the difference between points, that is, the larger the distance between points, the greater the difference between points; and the loss function can also be used to measure the difference between the detected lane line point set and the reference lane line point set, and the camera extrinsic parameters can be iteratively optimized according to the loss function through deep learning to further improve the accuracy of camera extrinsic parameter calibration.

[0130] As an example, for any detected lane line point in the detected lane line point set corresponding to the lane line, the distances between the detected lane line point and each reference lane line point in the reference lane line point set can be first determined, and the reference lane line point closest to the detected lane line point is determined as the reference lane line point matching the detected lane line point, and then the distance between each detected lane line point and the reference lane line point matching it is substituted into the following loss function to determine the loss value corresponding to the initial extrinsic parameter:

[0131]

[0132] Among them, L is the loss value, (u i ,v i ) is the coordinate of the i-th detected lane line point in the detected lane line point set, is the coordinate of the jth reference lane line point in the reference lane line point set, i is the serial number of the detected lane line point, and j is the serial number of the test lane line point.

[0133] After determining the loss value corresponding to the initial external parameter, if the loss value is greater than the loss threshold, the loss value can be optimized by nonlinear optimization to determine the updated external parameter, and then the above steps 101-104 are repeated according to the updated external parameter until the updated external parameter makes the loss value less than or equal to the loss threshold, then the updated external parameter can be determined as the target external parameter.

[0134] For example, if Figure 7 As shown, the detected lane line point set corresponding to the lane line includes detected lane line points A, B, C, and D, and the reference lane line point set corresponding to the lane line includes reference lane line points A', B', C', and D'. Corresponding to the detected lane line point A, the distances between it and the reference lane line points A', B', C', and D' can be determined respectively, and then it can be determined that the distance between the detected lane line point A and the reference lane line point A' is the smallest, so that it can be determined that the detected lane line point A matches the reference lane line point A'; using a corresponding method, it can be determined that the detected lane line point B matches the reference lane line point B', the detected lane line point C matches the reference lane line point C', and the detected lane line point D matches the reference lane line point D'. Then, the distance between the detected lane point A and the reference lane point A', the distance between the detected lane point B and the reference lane point B', the distance between the detected lane point C and the reference lane point C', and the distance between the detected lane point D and the reference lane point D' can be substituted into formula (1) to determine the loss value corresponding to the initial external parameter. If the loss value is greater than the loss threshold, the loss value can be optimized by nonlinear optimization to determine the updated external parameter, and then the above steps 101-104 are repeated according to the updated external parameter until the updated external parameter makes the loss value less than or equal to the loss threshold, and the updated external parameter can be determined as the target external parameter.

[0135] As an example, the steepest descent method can be used to perform nonlinear optimization on the loss value, and the optimization steps can include initial value estimation, gradient descent, and convergence judgment. Setting the correct initial values ​​of internal and external parameters can accelerate convergence and avoid falling into local minima or divergence; along the negative direction of the gradient, the variable value that minimizes the error can be iteratively found, and the gradient can be calculated using discrete numerical methods.

[0136] It should be noted that the above examples are only exemplary and cannot be regarded as limiting the present application. In actual use, the specific method of optimizing the loss value can be determined according to actual needs and specific application scenarios, and the present application embodiment does not limit this.

[0137] The camera extrinsic parameter calibration method provided in the embodiment of the present application performs lane line detection on the image data collected by the target camera to determine the reference lane line point set corresponding to each lane line, and reprojects the map data corresponding to the field of view of the target camera into the plane coordinate system corresponding to the image data to determine the reference lane line point set corresponding to each lane line as a reference, and then iteratively optimizes the initial extrinsic parameters according to the difference between the detected lane line point set corresponding to each lane line and the reference lane line point set to generate accurate target extrinsic parameters. Thus, the camera extrinsic parameters are iteratively optimized according to the difference between the lane line collected by the camera and the accurate lane line position in the map data, so that the difference between the lane line collected by the camera and the accurate lane line position in the map data is small enough to obtain accurate camera extrinsic parameters, thereby realizing real-time online automatic optimization of the camera extrinsic parameters without manual participation, which not only reduces the labor cost and improves the calibration efficiency, but also improves the accuracy of the camera extrinsic parameter calibration.

[0138] In a possible implementation form of the present application, since the difference between the reference lane line point and the detected lane line point includes not only the distance difference but also the angle difference, the loss value can be jointly determined based on the distance deviation and the angle deviation between the detected lane line point and the reference lane line point to further improve the accuracy of the camera extrinsic calibration.

[0139] Combine the following Figure 8 , the calibration method of the camera extrinsic parameters provided in the embodiment of the present application is further explained.

[0140] Figure 8 A schematic flow chart of another camera extrinsic parameter calibration method provided in an embodiment of the present application is shown.

[0141] like Figure 8 As shown, the calibration method of the camera extrinsic parameters includes the following steps:

[0142] Step 801 , obtaining image data collected by a target camera using initial external parameters and initial internal parameters and map data corresponding to a field of view of the target camera, wherein the field of view of the target camera includes at least one lane line.

[0143] Step 802: perform lane line detection on the image data to determine a detected lane line point set corresponding to each lane line contained in the image data.

[0144] Step 803, reprojecting the map data into the plane coordinate system corresponding to the image data according to the initial external parameters and the initial internal parameters to determine the reference lane line point set corresponding to each lane line, wherein the detected lane line point set includes M detected lane line points, and the reference lane line point set includes N reference lane line points, and M and N are integers greater than 1.

[0145] The specific implementation process and principle of the above steps 801-803 can be referred to the detailed description of the above embodiment, which will not be repeated here.

[0146] Step 804, determining the jth reference lane line point that matches the i-th detected lane line point based on the distances between the i-th detected lane line point and the M reference lane line points, wherein i is an integer greater than or equal to 1 and less than or equal to M, and j is an integer greater than or equal to 1 and less than or equal to N.

[0147] As a possible implementation method, for any detected lane line point in the detected lane line point set corresponding to the lane line, the distances between the detected lane line point and each reference lane line point in the reference lane line point set can be first determined, and the reference lane line point closest to the detected lane line point can be determined as the reference lane line point that matches the detected lane line point.

[0148] For example, if Figure 7 As shown, the detected lane line point set corresponding to the lane line includes detected lane line points A, B, C, and D, and the reference lane line point set corresponding to the lane line includes reference lane line points A', B', C', and D'. Corresponding to the detected lane line point A, the distances between it and the reference lane line points A', B', C', and D' can be determined respectively, and then it can be determined that the distance between the detected lane line point A and the reference lane line point A' is the smallest, so that it can be determined that the detected lane line point A matches the reference lane line point A'; using a corresponding method, it can be determined that the detected lane line point B matches the reference lane line point B', the detected lane line point C matches the reference lane line point C', and the detected lane line point D matches the reference lane line point D'.

[0149] Step 805, determining the distance deviation and angle deviation corresponding to the ith detected lane line point according to the coordinates of the ith detected lane line point and the coordinates of the jth reference lane line point.

[0150] In the embodiment of the present application, since the distance difference and angle difference between the detected lane line point and its corresponding reference lane line point can reflect whether the detected lane line point is accurate, and thus reflect the accuracy of the camera extrinsic parameters, after determining the reference lane line point that matches each detected lane line point, the distance deviation and angle deviation corresponding to each detected lane line point can be determined.

[0151] As a possible implementation method, for a detected lane line point, the distance between the detected lane line point and the reference lane line point that matches it can be determined based on the coordinates of the detected lane line point and the coordinates of the reference lane line point that matches it, and the distance is determined as the distance deviation corresponding to the detected lane line point.

[0152] As a possible implementation, for a detected lane line point or a reference lane line point, its corresponding angle may refer to the angle of the normal direction corresponding to the straight line where it is located, so the angle deviation corresponding to the detected lane line point may refer to the difference between the angle of the normal direction corresponding to the straight line where the detected lane line point is located and the angle of the normal direction corresponding to the straight line where the matched reference lane line point is located. As an example, since the ρ-θ parameter space contains the angle information of the point, when the Hough transform is used to perform straight line detection on image data and map data, the θ value corresponding to the straight line where the detected lane line point is located can be determined as the angle corresponding to the detected lane line point, and the θ value corresponding to the straight line where the reference lane line point is located can be determined as the angle corresponding to the reference lane line point, and then the difference between the angle corresponding to the detected lane line and the angle corresponding to the matched reference lane line point is determined as the angle deviation corresponding to the detected lane line point.

[0153] Step 806, determining the loss value corresponding to the initial external parameter according to the distance deviation and angle deviation corresponding to each detected lane line point.

[0154] As a possible implementation method, the difference between the detected lane line point set and the reference lane line point set can be measured through a loss function to iteratively optimize the camera extrinsic parameters. Therefore, the distance deviation and angle deviation corresponding to each detected lane line point set can be substituted into the following loss function to determine the loss value corresponding to the initial extrinsic parameter:

[0155]

[0156] Among them, L is the loss value, (u i ,v i ) is the coordinate of the i-th detected lane line point in the detected lane line point set, is the coordinate of the jth reference lane line point in the reference lane line point set, is the angle corresponding to the i-th detected lane line point, is the angle corresponding to the jth reference lane line point, ω1 is the first weight corresponding to the distance deviation, ω2 is the second weight corresponding to the angle deviation, i is the serial number of the detected lane line point, and j is the serial number of the test lane line point.

[0157] Furthermore, in different application scenarios, the degree of angle deviation and distance deviation caused by inaccurate camera extrinsics may be different. For example, in some scenarios, inaccurate camera extrinsics may cause a large distance deviation of the lane line, while the angle deviation is small. Therefore, the distance deviation and angle deviation in the loss function can also be modified according to the actual application scenario, so that the deviation with a large offset can occupy a larger proportion in the loss function, so as to further improve the accuracy of the camera extrinsic calibration. That is, in a possible implementation method of the embodiment of the present application, the above step 806 may include:

[0158] Get the current usage scene corresponding to the target camera;

[0159] Determine a first weight corresponding to the distance deviation and a second weight corresponding to the angle deviation according to the current usage scenario;

[0160] The loss value is determined according to the first weight, the second weight, and the distance deviation and angle deviation corresponding to each detected lane line point.

[0161] As a possible implementation method, when calibrating the external parameters of the target camera, it can be determined whether the image data collected by the target camera is more likely to produce distance deviation or angle deviation according to the current usage scenario corresponding to the target camera. If the image data collected by the target camera is more likely to produce distance deviation, the first weight corresponding to the distance deviation can be determined as a larger value, and the second weight corresponding to the angle deviation can be determined as a smaller value; if the image data collected by the target camera is more likely to produce angle deviation, the first weight corresponding to the distance deviation can be determined as a smaller value, and the second weight corresponding to the angle deviation can be determined as a larger value. Then, the determined first weight, second weight, and the distance deviation and angle deviation corresponding to each detected lane line point are substituted into formula (2) to determine the loss value corresponding to the initial external parameter.

[0162] Step 807, iteratively optimize the initial extrinsic parameters according to the loss value to determine the target extrinsic parameters.

[0163] In an embodiment of the present application, after determining the loss value corresponding to the initial external parameter, if the loss value is greater than the loss threshold, the loss value can be optimized by nonlinear optimization to determine the updated external parameter, and then the above steps 801-807 are repeated according to the updated external parameter until the updated external parameter makes the loss value less than or equal to the loss threshold, and the updated external parameter can be determined as the target external parameter.

[0164] For other specific implementation processes and principles of the above step 807, please refer to the detailed description of the above embodiment, which will not be repeated here.

[0165] The camera extrinsic calibration method provided in the embodiment of the present application performs lane line detection on the image data collected by the target camera to determine the reference lane line point set corresponding to each lane line, and reprojects the map data corresponding to the field of view of the target camera into the plane coordinate system corresponding to the image data to determine the reference lane line point set corresponding to each lane line as a reference, and then determines the loss value according to the distance deviation and angle deviation between the detected lane line point set and the reference lane line point set corresponding to each lane line, and iteratively optimizes the initial extrinsic parameter according to the loss value to generate accurate target extrinsic parameters. Thus, the camera extrinsic parameters are iteratively optimized according to the distance deviation and angle deviation between the lane line collected by the camera and the accurate lane line position in the map data, and deep learning is combined with the camera extrinsic calibration to make the distance and angle difference between the lane line collected by the camera and the accurate lane line in the map data small enough to obtain accurate camera extrinsic parameters, thereby realizing real-time online automatic optimization of camera extrinsic parameters without manual participation, reducing labor costs, improving calibration efficiency, and further improving the accuracy of camera extrinsic calibration.

[0166] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0167] Corresponding to the calibration method of the camera extrinsic parameters described in the above embodiment, Fig. 9 A schematic diagram of the structure of a camera extrinsic parameter calibration device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0168] Reference Fig. 9 The device 90 comprises:

[0169] A first acquisition module 91 is used to acquire image data collected by a target camera using initial external parameters and initial internal parameters and map data corresponding to a field of view of the target camera, wherein the field of view of the target camera includes at least one lane line;

[0170] A first determination module 92, configured to perform lane line detection on the image data to determine a detected lane line point set corresponding to each lane line contained in the image data;

[0171] A second determination module 93, configured to reproject the map data into a plane coordinate system corresponding to the image data according to the initial external parameters and the initial internal parameters, so as to determine a reference lane line point set corresponding to each lane line;

[0172] The third determination module 94 is used to iteratively optimize the initial extrinsic parameters according to the difference between the detected lane line point set corresponding to each lane line and the reference lane line point set to determine the target extrinsic parameters of the target camera.

[0173] The camera extrinsic parameter calibration device provided in the embodiment of the present application performs lane line detection on the image data collected by the target camera to determine the reference lane line point set corresponding to each lane line, and reprojects the map data corresponding to the field of view of the target camera into the plane coordinate system corresponding to the image data to determine the reference lane line point set corresponding to each lane line as a reference, and then iteratively optimizes the initial extrinsic parameter according to the difference between the detected lane line point set and the reference lane line point set corresponding to each lane line to generate accurate target extrinsic parameters. Thus, by iteratively optimizing the camera extrinsic parameter according to the difference between the lane line collected by the camera and the accurate lane line position in the map data, so that the difference between the lane line collected by the camera and the accurate lane line position in the map data is small enough to obtain accurate camera extrinsic parameters, the real-time online automatic optimization of the camera extrinsic parameter is realized without manual participation, which not only reduces the labor cost and improves the calibration efficiency, but also improves the accuracy of the camera extrinsic parameter calibration.

[0174] In a possible implementation form of the present application, the detected lane line point set includes M detected lane line points, the reference lane line point set includes N reference lane line points, and M and N are integers greater than 1; accordingly, the third determination module 94 includes:

[0175] a first determining unit, for determining a j-th reference lane line point that matches the i-th detected lane line point according to distances between the i-th detected lane line point and the M reference lane line points, wherein i is an integer greater than or equal to 1 and less than or equal to M, and j is an integer greater than or equal to 1 and less than or equal to N;

[0176] A second determination unit is used to determine the distance deviation and the angle deviation corresponding to the i-th detected lane line point according to the coordinates of the i-th detected lane line point and the coordinates of the j-th reference lane line point;

[0177] A third determination unit is used to determine the loss value corresponding to the initial external parameter according to the distance deviation and the angle deviation corresponding to each detected lane line point;

[0178] The fourth determination unit is used to iteratively optimize the initial extrinsic parameter according to the loss value to determine the target extrinsic parameter.

[0179] Furthermore, in another possible implementation form of the present application, the third determining unit is specifically configured to:

[0180] Get the current usage scene corresponding to the target camera;

[0181] Determine a first weight corresponding to the distance deviation and a second weight corresponding to the angle deviation according to the current usage scenario;

[0182] The loss value is determined according to the first weight, the second weight, and the distance deviation and angle deviation corresponding to each detected lane line point.

[0183] Furthermore, in another possible implementation form of the present application, the first determining module 92 includes:

[0184] A fifth determination unit, configured to input the image data into a lane line detection model to determine a detected lane line area corresponding to each lane line;

[0185] a sixth determination unit, configured to perform straight line detection on each detected lane line area to determine a detected corner point corresponding to each lane line;

[0186] The seventh determination unit is used to determine the detected lane line point set corresponding to each lane line according to the detected corner point corresponding to each lane line.

[0187] Furthermore, in another possible implementation form of the present application, the seventh determining unit is specifically configured to:

[0188] The detected corner points corresponding to each lane line are determined as a detected lane line point set corresponding to each lane line.

[0189] Furthermore, in another possible implementation form of the present application, the first determining module 92 further includes:

[0190] an eighth determination unit, used to determine the number of pixels contained in each detected lane line area;

[0191] The first removing unit is used to remove the detected lane line area where the number of pixel points is less than a number threshold.

[0192] Furthermore, in another possible implementation form of the present application, the second determining module 93 includes:

[0193] a ninth determination unit, configured to perform straight line detection on the map data to determine a reference corner point corresponding to each lane line;

[0194] The tenth determination unit is used to reproject the map data into a plane coordinate system corresponding to the image data according to the reference corner point, initial external parameters and initial internal parameters corresponding to each lane line, so as to determine a reference lane line point set corresponding to each lane line.

[0195] Furthermore, in another possible implementation form of the present application, the tenth determining unit is specifically configured to:

[0196] According to the initial external parameters and the initial internal parameters, the reference corner points corresponding to each lane line are reprojected into the plane coordinate system corresponding to the image data to determine the reference lane line point set corresponding to each lane line.

[0197] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0198] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0199] In order to implement the above embodiments, the present application also proposes a terminal device.

[0200] Fig.10 A schematic diagram of the structure of a terminal device according to an embodiment of the present application.

[0201] like Fig.10 As shown, the terminal device 200 includes:

[0202] A memory 210 and at least one processor 220, a bus 230 connecting different components (including the memory 210 and the processor 220), the memory 210 stores a computer program, and when the processor 220 executes the program, the camera extrinsic parameter calibration method described in the embodiment of the present application is implemented.

[0203] Bus 230 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0204] The terminal device 200 typically includes a variety of electronic device readable media, which can be any available media that can be accessed by the terminal device 200, including volatile and non-volatile media, removable and non-removable media.

[0205] The memory 210 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 240 and / or cache memory 250. The terminal device 200 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 260 may be used to read and write non-removable, non-volatile magnetic media ( Fig.10 not shown, usually called a "hard drive"). Although Fig.10 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to bus 230 via one or more data medium interfaces. Memory 210 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0206] A program / utility 280 having a set (at least one) of program modules 270 may be stored, for example, in the memory 210, such program modules 270 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 270 generally perform the functions and / or methods of the embodiments described herein.

[0207] The terminal device 200 can also communicate with one or more external devices 290 (e.g., keyboard, pointing device, display 291, etc.), and can also communicate with one or more devices that enable users to interact with the terminal device 200, and / or communicate with any device that enables the terminal device 200 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be carried out through an input / output (I / O) interface 292. In addition, the terminal device 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through a network adapter 293. As shown in the figure, the network adapter 293 communicates with other modules of the terminal device 200 through a bus 230. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the terminal device 200, including but not limited to: microcode, device driver, redundant processing unit, external disk drive array, RAID system, tape drive, and data backup storage system, etc.

[0208] The processor 220 executes various functional applications and data processing by running the programs stored in the memory 210 .

[0209] It should be noted that the implementation process and technical principles of the terminal device of this embodiment refer to the aforementioned explanation of the calibration method of the camera extrinsic parameters of the embodiment of the present application, and will not be repeated here.

[0210] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0211] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0212] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0213] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0214] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0215] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0216] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0217] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A camera extrinsic parameter calibration method, characterized in that: include: Acquire image data collected by a target camera using initial external parameters and initial internal parameters and map data corresponding to a field of view of the target camera, wherein the field of view of the target camera includes at least one lane line; Performing lane line detection on the image data to determine a detected lane line point set corresponding to each lane line contained in the image data; Reprojecting the map data into a plane coordinate system corresponding to the image data according to the initial external parameters and the initial internal parameters to determine a reference lane line point set corresponding to each lane line; According to the difference between the detected lane line point set and the reference lane line point set corresponding to each lane line, the initial extrinsic parameters are iteratively optimized to determine the target extrinsic parameters of the target camera.

2. The method according to claim 1, characterized in that The detected lane line point set includes M detected lane line points, the reference lane line point set includes N reference lane line points, M and N are integers greater than 1, and the initial extrinsic parameters are iteratively optimized according to the difference between the detected lane line point set and the reference lane line point set corresponding to each lane line to determine the target extrinsic parameters of the target camera, including: Determine, based on the distances between the ith detected lane line point and the M reference lane line points, the jth reference lane line point that matches the ith detected lane line point, where i is an integer greater than or equal to 1 and less than or equal to M, and j is an integer greater than or equal to 1 and less than or equal to N; Determine the distance deviation and angle deviation corresponding to the i-th detected lane line point according to the coordinates of the i-th detected lane line point and the coordinates of the j-th reference lane line point; Determining the loss value corresponding to the initial external parameter according to the distance deviation and the angle deviation corresponding to each of the detected lane line points; The initial extrinsic parameter is iteratively optimized according to the loss value to determine the target extrinsic parameter.

3. The method according to claim 2, characterized in that Determining the loss value corresponding to the initial external parameter according to the distance deviation and the angle deviation corresponding to each of the detected lane line points includes: Obtaining the current usage scenario corresponding to the target camera; Determine a first weight corresponding to the distance deviation and a second weight corresponding to the angle deviation according to the current usage scenario; The loss value is determined according to the first weight, the second weight, and the distance deviation and the angle deviation corresponding to each of the detected lane line points.

4. The method according to claim 1, characterized in that The performing lane line detection on the image data to determine a detected lane line point set corresponding to each lane line contained in the image data includes: Inputting the image data into a lane line detection model to determine a detected lane line area corresponding to each lane line; Performing straight line detection on each of the detected lane line areas to determine a detected corner point corresponding to each of the lane lines; According to the detected corner points corresponding to each lane line, a detected lane line point set corresponding to each lane line is determined.

5. The method according to claim 4, characterized in that The step of determining a detected lane line point set corresponding to each lane line according to the detected corner point corresponding to each lane line comprises: The detected corner points corresponding to each lane line are determined as a detected lane line point set corresponding to each lane line.

6. The method according to claim 4, characterized in that Before performing straight line detection on each of the detected lane line areas to determine the detected corner points corresponding to each of the lane lines, the method further includes: Determine the number of pixels contained in each of the detected lane line areas; The detected lane line area where the number of pixel points is less than the number threshold is removed.

7. The method according to any one of claims 1 to 6, characterized in that: The reprojecting the map data into a plane coordinate system corresponding to the image data according to the initial external parameters and the initial internal parameters to determine a reference lane line point set corresponding to each lane line includes: Performing straight line detection on the map data to determine a reference corner point corresponding to each lane line; The map data is reprojected into a plane coordinate system corresponding to the image data according to the reference corner point corresponding to each lane line, the initial external parameter and the initial internal parameter to determine a reference lane line point set corresponding to each lane line.

8. The method according to claim 7, characterized in that The reprojecting of the map data into a plane coordinate system corresponding to the image data according to the reference corner point corresponding to each lane line, the initial external parameter, and the initial internal parameter to determine a reference lane line point set corresponding to each lane line includes: According to the initial external parameters and the initial internal parameters, the reference corner point corresponding to each lane line is reprojected into the plane coordinate system corresponding to the image data to determine a reference lane line point set corresponding to each lane line.

9. A camera extrinsic parameter calibration device, characterized in that: include: A first acquisition module is used to acquire image data collected by a target camera using initial external parameters and initial internal parameters and map data corresponding to a field of view of the target camera, wherein the field of view of the target camera includes at least one lane line; A first determination module, configured to perform lane line detection on the image data to determine a detected lane line point set corresponding to each lane line contained in the image data; A second determination module, configured to reproject the map data into a plane coordinate system corresponding to the image data according to the initial external parameters and the initial internal parameters, so as to determine a reference lane line point set corresponding to each lane line; The third determination module is used to iteratively optimize the initial extrinsic parameters according to the difference between the detected lane line point set and the reference lane line point set corresponding to each lane line, so as to determine the target extrinsic parameters of the target camera.

10. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images