Method and device for calibrating external parameters of camera

Abstract

The invention provides an external parameter calibration method and device for cameras, and the method comprises the steps: obtaining a plurality of images shot by four cameras of a vehicle when the vehicle is in a straight driving state; determining a first target lane point in a first image shot by the first camera, wherein the first target lane point is located in a common shooting area of the first image and a second image shot by the second camera; and converting the first target lane point to the panoramic plane based on the first external parameter of the first camera to obtain a first projection point. And converting the first projection point based on a second external parameter of the second camera to obtain a second target lane point corresponding to the first projection point in the second image. A first loss function is calculated based on the image information of the first target lane point and the image information of the second target lane point. And iteratively adjusting the second external parameter and recalculating the first loss function until an iteration stop condition is met, thereby obtaining the optimized second external parameter.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of camera technology, in particular to a camera extrinsic parameter calibration method and device. BACKGROUND

[0002] An around view monitor (AVM) is a driving assistance technology based on multi-camera fusion. The AVM system is deployed in front, back, left and right of the vehicle to capture the surrounding environment images. After projection transformation and image stitching, a 360° panoramic view is synthesized. The AVM system can effectively eliminate the driver's blind area and assist the driver to drive safely.

[0003] The core of the AVM system is how to seamlessly stitch the images captured by the four cameras into a complete overhead view. This process relies on the projection conversion of the cameras, and the projection conversion mainly depends on the extrinsic parameters of the cameras. If the extrinsic parameters are not accurately calibrated, the stitched images will be misaligned, distorted or broken, which will affect the usability of the AVM system. Before the vehicle is shipped, a high-precision calibration device (such as a checkerboard or a calibration board) is used for initial calibration to ensure that the projection relationship of the four cameras is correct. However, during the use of the vehicle after it is shipped, due to various reasons such as bumps during driving and replacement of the cameras, the original calibration parameters of the cameras are inaccurate, which leads to poor stitching of the panoramic overhead view. At this time, the cameras need to be recalibrated.

[0004] The recalibration of the cameras is similar to the calibration method before the vehicle is shipped. The calibration is performed by placing a marker at a fixed position and manually selecting a marker point. However, the manual calibration method is low in efficiency, is limited by the site, and can easily affect the accuracy of the calibration. SUMMARY

[0005] Therefore, the present application aims to provide a camera extrinsic parameter calibration method and device to improve the efficiency and accuracy of calibrating the extrinsic parameters of the cameras.

[0006] In a first aspect, the present application provides a camera extrinsic parameter calibration method, which comprises: acquiring a plurality of images captured by four cameras of a vehicle when the vehicle is in a straight driving state;

[0007] determining a first target lane point in a first image captured by a first camera, the first camera representing any one of the four cameras, and the first target lane point belonging to a common shooting area of the first image and a second image captured by a second camera;

[0008] converting the first target lane point to a panoramic plane based on a first extrinsic parameter of the first camera to obtain a first projection point;

[0009] convert the first projection point based on a second extrinsic parameter of the second camera to obtain a second target lane point corresponding to the first projection point in the second image;

[0010] calculate a first loss function based on image information of the first target lane point and image information of the second target lane point;

[0011] iteratively adjust the second extrinsic parameter and recalculate the first loss function until an iteration stop condition is met to obtain an optimized second extrinsic parameter.

[0012] In a possible implementation, the method further includes:

[0013] for an image captured by any camera, convert a target lane point in the image based on an optimized extrinsic parameter of the camera to obtain a world coordinate point corresponding to the target lane point in a world coordinate system;

[0014] for a plurality of target lane points corresponding to different cameras and located on a same lane line, calculate a second loss function based on a positional relationship between a plurality of world coordinate points corresponding to the plurality of target lane points;

[0015] iteratively adjust the optimized extrinsic parameters of the four cameras and recalculate the second loss function until an iteration stop condition is met to obtain further optimized extrinsic parameters.

[0016] In a possible implementation, the calculating a first loss function based on image information of the first target lane point and image information of the second target lane point includes:

[0017] when the four cameras are factory-calibrated, determine four common shooting areas corresponding to adjacent two cameras of the four cameras;

[0018] for any common shooting area, determine first image information and second image information corresponding to the adjacent two cameras associated with the common shooting area, respectively;

[0019] determine an image loss function based on a first image adjustment coefficient and a second image adjustment coefficient corresponding to the adjacent two cameras, respectively, and the first image information and the second image information;

[0020] calculate a first target value corresponding to the first image adjustment coefficient and a second target value corresponding to the second image adjustment coefficient when the image loss function meets a preset condition;

[0021] The first loss function is calculated based on the first target value, the second target value, image information of the first target lane point, and image information of the second target lane point.

[0022] In a possible implementation, the first loss function is calculated based on the image information of the first target lane point and the image information of the second target lane point, including:

[0023] A luminance difference value is determined based on luminance information of the first target lane point and luminance information of the second target lane point.

[0024] A color difference value is determined based on color information of the first target lane point and color information of the second target lane point.

[0025] The first loss function is calculated based on the luminance difference value and the color difference value.

[0026] In a possible implementation, the first image information includes at least one of a luminance mean value or a color mean value corresponding to a common shooting area shot by a third camera, the second image information includes at least one of a luminance mean value or a color mean value corresponding to the common shooting area shot by a fourth camera, the third camera and the fourth camera represent two adjacent cameras, and the first image information corresponds to the second image information.

[0027] In a possible implementation, the second loss function is calculated based on a positional relationship between a plurality of world coordinate points corresponding to a plurality of target lane points, including:

[0028] When a plurality of target lane points corresponding to the front camera, the left camera, and the rear camera are located at a same boundary of a same single solid line, a first lateral coordinate difference between world coordinate points corresponding to the plurality of target lane points is calculated; or

[0029] When a plurality of target lane points corresponding to the front camera, the right camera, and the rear camera are located at a same boundary of a same single solid line, a second lateral coordinate difference between world coordinate points corresponding to the plurality of target lane points is calculated; or

[0030] When a plurality of target lane points corresponding to the front camera, the left camera, and the right camera are located at a same boundary of a same horizontal solid line, a first longitudinal coordinate difference between world coordinate points corresponding to the plurality of target lane points is calculated; or

[0031] When a plurality of target lane points corresponding to the rear camera, the left camera, and the right camera are located at a same boundary of a same horizontal solid line, a second longitudinal coordinate difference between world coordinate points corresponding to the plurality of target lane points is calculated; or

[0032] For any camera, based on the world coordinate points corresponding to the plurality of target lane points located on the same lane line on the image corresponding to the camera, a lane line width is calculated, and a difference between four lane line widths corresponding to four cameras is calculated;

[0033] Based on at least one of the first lateral coordinate difference, the second lateral coordinate difference, the first longitudinal coordinate difference, the second longitudinal coordinate difference, or the difference, the second loss function is determined.

[0034] In a possible implementation, the target lane point in the image is determined, including:

[0035] The target lane line in the image is extracted, and the target lane line includes at least one of a single solid line or a transverse solid line;

[0036] The plurality of boundary lane points located on the boundary of the target lane line are determined by traversing the target lane line;

[0037] The target lane point is determined from the plurality of boundary lane points.

[0038] In a possible implementation, the target lane point is converted based on the optimized extrinsic parameter of the camera to obtain a world coordinate point corresponding to the target lane point in a world coordinate system, including:

[0039] Based on the optimized extrinsic parameter, the intrinsic parameter of the camera, and the depth information corresponding to the target lane point, the image coordinate of the target lane point is converted to a world coordinate system to obtain a world coordinate of the world coordinate point in the world coordinate system.

[0040] In a possible implementation, the first target lane point is converted to a panoramic plane based on the first extrinsic parameter of the first camera to obtain a first projection point, including:

[0041] Based on the first extrinsic parameter, the first intrinsic parameter of the first camera, and the depth information corresponding to the first target lane point, the image coordinate of the first target lane point is converted to a world coordinate system to obtain a world coordinate of a first world coordinate point in the world coordinate system;

[0042] Based on a mapping matrix between the world coordinate system and the panoramic plane, the world coordinate of the first world coordinate point is converted to the panoramic plane to obtain a panoramic coordinate of the first projection point in the panoramic plane.

[0043] In a second aspect, the application provides a camera extrinsic parameter calibration device, including:

[0044] An image acquisition unit is configured to acquire a plurality of images captured by four cameras of a vehicle when the vehicle is in a straight driving state.

[0045] A determination unit is configured to determine a first target lane point in a first image captured by a first camera, the first camera being any one of the four cameras, the first target lane point belonging to a common capturing area of the first image and a second image captured by a second camera;

[0046] A first conversion unit is configured to convert the first target lane point to a panorama plane based on a first extrinsic parameter of the first camera to obtain a first projection point;

[0047] A second conversion unit is configured to convert the first projection point based on a second extrinsic parameter of the second camera to obtain a second target lane point corresponding to the first projection point in the second image;

[0048] A loss function determination unit is configured to calculate a first loss function based on image information of the first target lane point and image information of the second target lane point;

[0049] An iteration unit is configured to iteratively adjust the second extrinsic parameter and recalculate the first loss function until an iteration stop condition is met to obtain an optimized second extrinsic parameter.

[0050] In a third aspect, the present application provides an electronic device, the device comprising: a memory and a processor;

[0051] The memory is configured to store related program codes;

[0052] The processor is configured to call the program codes and execute the camera extrinsic parameter calibration method according to any one of the implementation manners of the first aspect.

[0053] In a fourth aspect, the present application provides a computer readable storage medium for storing a computer program, the computer program being configured to execute the camera extrinsic parameter calibration method according to any one of the implementation manners of the first aspect.

[0054] In a fifth aspect, the present application provides a computer program product comprising computer programs / instructions, the computer programs / instructions being configured to implement the camera extrinsic parameter calibration method according to any one of the implementation manners of the first aspect when executed by a processor.

[0055] In the above implementation of the present application, when the vehicle is in a straight driving state, a plurality of images captured by four cameras of the vehicle are obtained. A first target lane point in a first image captured by a first camera is determined, wherein the first camera represents any one of the four cameras, and the first target lane point is located in a common capturing area of the first image and a second image captured by a second camera. The first target lane point is converted to a panoramic plane based on a first extrinsic parameter of the first camera to obtain a first projection point. The first projection point is converted based on a second extrinsic parameter of the second camera to obtain a second target lane point corresponding to the first projection point in the second image. A first loss function is calculated based on image information of the first target lane point and image information of the second target lane point. The second extrinsic parameter of the second camera is iteratively adjusted with the first extrinsic parameter of the first camera as a reference, and the first loss function is recalculated until an iteration stopping condition is met, thereby obtaining an optimized second extrinsic parameter. By using the method provided in the present application, the image information of the lane points at the same position captured by different cameras can be used to establish a loss function of the extrinsic parameter, and the extrinsic parameter of the camera can be automatically iteratively adjusted to reduce the value of the loss function, thereby obtaining an optimized extrinsic parameter. Compared with the method of manually calibrating the extrinsic parameter, the efficiency and accuracy of the extrinsic parameter calibration can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments provided in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0057] Figure 1 A flowchart of a camera extrinsic parameter calibration method provided by an embodiment of the present application.

[0058] Figure 2 A schematic diagram of a lane line provided by an embodiment of the present application.

[0059] Figure 3 A schematic diagram of traversing a lane line provided by an embodiment of the present application.

[0060] Figure 4 A schematic diagram of a panoramic plane provided by an embodiment of the present application.

[0061] Figure 5a A schematic diagram of capturing the same lane line provided by an embodiment of the present application.

[0062] Figure 5b Another schematic diagram of capturing the same lane line provided by an embodiment of the present application.

[0063] Figure 5cAnother schematic diagram for shooting the same lane line provided by an embodiment of the present application.

[0064] Figure 6 A schematic diagram of an external parameter calibration device of a camera provided by an embodiment of the present application.

[0065] Figure 7 A schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. The described embodiments are only exemplary implementations of the present application, and not all implementations. Those skilled in the art can obtain other embodiments without creative labor by combining the embodiments of the present application, and these embodiments are also within the protection scope of the present application.

[0067] The AVM system is a driving assistance technology based on multi-camera fusion. The AVM system acquires surrounding environment images by deploying cameras in four directions (front, back, left and right) of a vehicle, and composes a 360° panoramic overhead view through projection transformation and image stitching. The AVM system can effectively eliminate the driver's visual dead angle and assist the driver to drive the vehicle safely. The core of the AVM system is how to seamlessly splice the images captured by the four cameras into a complete overhead view. This process relies on the projection conversion of the cameras, and the projection conversion mainly depends on the external parameters of the cameras. Before the vehicle is shipped, high-precision calibration equipment (such as a checkerboard and a calibration board) will be used for initial calibration to ensure that the projection relationship of the four cameras is correct. However, during the use of the vehicle after it is shipped, due to various reasons, such as bumps during driving, replacement of the cameras, etc., the original calibration parameters of the cameras are inaccurate, which leads to poor splicing of the panoramic overhead view. At this time, the cameras need to be recalibrated.

[0068] The recalibration of the cameras is similar to the calibration method before the vehicle is shipped. The calibration equipment is placed at a fixed position, and the calibration is performed manually by selecting the calibration points. However, the manual calibration method is low in efficiency, is limited by the site, and is likely to affect the accuracy of the calibration.

[0069] Based on this, the embodiment of the present application provides a camera extrinsic parameter calibration method to improve the efficiency and accuracy of camera extrinsic parameter calibration. In specific implementation, when the vehicle is in a straight driving state, multiple images captured by four cameras of the vehicle are obtained. A first target lane point in a first image captured by a first camera is determined, wherein the first camera represents any one of the four cameras, and the first target lane point is located in a common shooting area of the first image and a second image captured by a second camera. The first target lane point is converted to a panoramic plane based on a first extrinsic parameter of the first camera to obtain a first projection point. The first projection point is converted based on a second extrinsic parameter of the second camera to obtain a second target lane point corresponding to the first projection point in the second image. A first loss function is calculated based on image information of the first target lane point and image information of the second target lane point. The second extrinsic parameter of the second camera is iteratively adjusted with the first extrinsic parameter of the first camera as a reference, and the first loss function is recalculated until an iteration stopping condition is met, thereby obtaining an optimized second extrinsic parameter. By using the method provided in the embodiment of the present application, the image information of the lane points at the same position captured by different cameras can be used to establish a loss function of the extrinsic parameter, and the extrinsic parameter of the camera can be automatically iteratively adjusted to reduce the value of the loss function, thereby obtaining an optimized extrinsic parameter. Compared with the method of manually calibrating the extrinsic parameter, the efficiency and accuracy of extrinsic parameter calibration can be improved.

[0070] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, the following will be specifically introduced in conjunction with the drawings in the embodiments.

[0071] Referring to Figure 1 As shown in the figure, a flowchart of a camera extrinsic parameter calibration method provided by the embodiment of the present application.

[0072] Optionally, the method can be executed by an image processing device. The method can include the following steps S101-S106:

[0073] S101: When the vehicle is in a straight driving state, multiple images captured by four cameras of the vehicle are obtained.

[0074] The four cameras of the vehicle include a front camera located in front of the vehicle, a rear camera located behind the vehicle, a left camera located on the left side of the vehicle, and a right camera located on the right side of the vehicle. Four images captured by the four cameras at the same time are obtained. Since the position relationship of the lane points on the same lane line captured by different cameras is needed to calculate the loss function in the embodiment of the present application, the extrinsic parameter needs to be optimized when the vehicle is in a straight driving state. That is, the vehicle is parallel to the longitudinal lane and drives.

[0075] Optionally, after obtaining the image captured by the camera, the image can be preprocessed to facilitate subsequent extraction of lane lines, lane points, and the like in the image. For example, the image can be corrected, image filtering, image enhancement, contour detection, and the like.

[0076] In one possible implementation, whether the vehicle is in a straight driving state can be determined by judging the coordinate change of the lane point on the lane line boundary. Referring to FIG. 6, a schematic diagram of a lane line provided by an embodiment of the present application is shown. As shown in FIG. 6, the lane line is a longitudinal lane line, and the vehicle coordinate system can be represented by taking the driving direction of the vehicle as the y-axis and taking the direction perpendicular to the driving direction of the vehicle as the x-axis. When the vehicle is driving, a plurality of images captured by the front camera of the vehicle in succession can be obtained. In each image, the abscissa of the lane point on the same lane line boundary is determined. When the error of the abscissa of the plurality of lane points on the same lane line boundary in the plurality of images in succession is less than a preset value, it can be determined that the vehicle is in a straight driving state. Figure 2 Figure 2 As shown in FIG. 6, the lane line is a longitudinal lane line, and the vehicle coordinate system can be represented by taking the driving direction of the vehicle as the y-axis and taking the direction perpendicular to the driving direction of the vehicle as the x-axis. When the vehicle is driving, a plurality of images captured by the front camera of the vehicle in succession can be obtained. In each image, the abscissa of the lane point on the same lane line boundary is determined. When the error of the abscissa of the plurality of lane points on the same lane line boundary in the plurality of images in succession is less than a preset value, it can be determined that the vehicle is in a straight driving state.

[0077] Generally, a certain marker (or a certain area) in a real scene is captured by two adjacent cameras, and the image information of a certain point of the marker in the images captured by the two cameras should be the same. Further, a certain lane point in a real scene corresponds to two image points in the images captured by the adjacent two cameras, and the image information of the two image points should be the same. Therefore, a loss function can be established based on the image information of the lane point in different images to optimize the extrinsic parameters of the camera.

[0078] S102: Determine a first target lane point in a first image captured by a first camera.

[0079] The first camera represents any one of the four cameras, and the first target lane point belongs to the common shooting area of the first image and the second image. The second image is an image captured by a second camera. That is, after obtaining the images captured by the four cameras, the common shooting area of the adjacent two cameras can be determined, wherein the common shooting area represents an area with the same geographical position. When the first image and the second image have a common shooting area, the first target lane point corresponding to the common shooting area in the first image can be determined.

[0080] ​In a specific implementation, first, a target lane point in the image can be determined, including: extracting a target lane line in the image, where the type of the target lane line can include a single solid line and a transverse solid line. For example, a lane line detection model can be used to extract the lane line in the image, and the lane line detection model can be a pre-trained neural network model for detecting lane lines in the image, including a single solid line, a single dashed line, a double solid line, a transverse solid line, and the like. Then, the lane line extracted by the lane line detection model is screened to obtain at least one of the single solid line and the transverse solid line as the target lane line. It should be noted that the training process of the lane line detection model can refer to an existing method, and will not be specifically introduced in the embodiments of the present application.

[0081] Then, the target lane line is traversed to determine a plurality of boundary lane points located on the boundary of the target lane line, and a target lane point is determined from the plurality of boundary lane points. For details, refer to Figure 3 As shown in FIG. 6, it is a schematic diagram of traversing a lane line provided by an embodiment of the present application. When the target lane line is a longitudinal lane line (single solid line), the target lane line can be traversed horizontally according to a preset distance, and each horizontal traversal can determine a group of boundary lane points, such as a group of boundary lane points H1 and H2. When the target lane line is a transverse solid line, the target lane line can be traversed vertically according to a preset distance, and each vertical traversal can determine a group of boundary lane points, such as a group of boundary lane points Z1 and Z2. The number of traversals can be set according to actual conditions. A plurality of target lane points can be selected from the plurality of determined boundary lane points, and the number of target lane points is at least two.

[0082] After obtaining the plurality of target lane points in the first image, the target lane points located in the common shooting area of the first image and the second image are determined as first target lane points.

[0083] S103: Projecting the first target lane point to the panoramic plane based on the first extrinsic parameter of the first camera to obtain a first projection point.

[0084] The panoramic plane represents a plane on which the panoramic top view is located. For details, refer to Figure 4 As shown in FIG. 7, it is a schematic diagram of a panoramic plane provided by an embodiment of the present application. Wherein P1-P2-P3-P4 represents the ground, Q1-Q2-Q3-Q4 represents the panoramic plane, and in general, the conversion relationship between the panoramic plane and the world coordinate system can be represented by a mapping matrix, which is a known parameter by default.

[0085] In a specific implementation, first, the image coordinates of the first target lane point can be converted to the world coordinate system based on the first extrinsic parameter, the first intrinsic parameter of the first camera, and the depth information corresponding to the first target lane point, to obtain the world coordinates of the first world coordinate point in the world coordinate system. For example, the image coordinates of the first target lane point are represented as P 图 , the depth information corresponding to the first target lane point is represented as d, the first intrinsic parameter of the first camera is represented as N, the first extrinsic parameter is represented as W, and the world coordinates corresponding to the first target lane point are represented as P 世 . The relationship between the image coordinates P 图 and the world coordinates P 世 may be represented as P 图 =(1 / d)NWP 世 . Therefore, the world coordinates can be obtained according to the conversion of the image coordinates, and the first world coordinate point corresponding to the world coordinates in the world coordinate system is obtained.

[0086] Then, the world coordinates of the first world coordinate point are converted to the panoramic plane based on the mapping matrix between the world coordinate system and the panoramic plane, to obtain the panoramic coordinates of the first projection point in the panoramic plane. For example, the product of the mapping matrix and the world coordinates is calculated to obtain the panoramic coordinates.

[0087] S104: The first projection point is converted based on the second extrinsic parameter of the second camera, to obtain the second target lane point corresponding to the first projection point in the second image.

[0088] Since the first target lane point is in the common shooting area of the first camera and the second camera, the position of the first projection point in the second image needs to be found. Therefore, the second extrinsic parameter of the second camera can be used to convert the first projection point in the panoramic plane to the world coordinate system, and then converted from the world coordinate system to the image coordinate system, to obtain the second target lane point corresponding to the first projection point in the second image. That is, similar to the inverse process of converting the first target lane point to obtain the first projection point.

[0089] S105: The first loss function is calculated based on the image information of the first target lane point and the image information of the second target lane point.

[0090] In theory, if the extrinsic parameter is accurate, the second target lane point in the second image searched based on the first target lane point in the first image should correspond to the same point in the real scene, and thus the image information of the first target lane point and the image information of the second target lane point should be the same. Since the first projection point of the panoramic plane is obtained based on the first extrinsic parameter of the first camera, when the image information corresponding to the first target lane point is different from the image information of the second target lane point, it indicates that the second extrinsic parameter of the second camera is not accurate enough. The first loss function can be calculated based on the image information of the first target lane point and the image information of the second target lane point, and is used to represent the accuracy of the second extrinsic parameter. When the first loss function is larger, it indicates that the second extrinsic parameter is less accurate.

[0091] In a possible implementation, the image information can include at least one of luminance information and color information. Specifically, a luminance difference value can be determined based on the luminance information of the first target lane point and the luminance information of the second target lane point. A color difference value can be determined based on the color information of the first target lane point and the color information of the second target lane point. The first loss function can be calculated based on the luminance difference value and the color difference value. For example, the sum of the luminance difference value and the color difference value can be calculated as the first loss function.

[0092] Optionally, the luminance information can be represented by a luminance value, and the color information can be represented by an RGB value. When calculating the color difference value, the difference between the R value of the first target lane point and the R value of the second target lane point, the difference between the G value of the first target lane point and the G value of the second target lane point, and the difference between the B value of the first target lane point and the B value of the second target lane point can be calculated respectively. Then, the sum of the luminance difference value, the R corresponding difference value, the G corresponding difference value, and the B corresponding difference value can be calculated as the first loss function.

[0093] It should be noted that the above embodiments introduce a manner of determining the first loss function based on the luminance information and the color information. When the image information only includes the luminance information or only includes the color information, the first loss function can be determined according to the above principles.

[0094] S106: iteratively adjusting the second extrinsic parameter and recalculating the first loss function until an iteration stop condition is met, to obtain an optimized second extrinsic parameter.

[0095] The first loss function can represent the accuracy of the second extrinsic parameter of the second camera. When the first loss function is larger, it indicates that the second extrinsic parameter is less accurate. Then the second extrinsic parameter can be adjusted, and the first loss function is recalculated. The above process is iteratively performed until the iteration stop condition is met, to obtain the optimized second extrinsic parameter. The iteration stop condition can be that the first loss function is less than a threshold value, or the iteration process reaches a preset number of times.

[0096] In actual application scenarios, considering the influence of the environment, the exposure conditions of the four cameras on the vehicle when shooting may be different, affecting the image information. In order to reduce the influence of the environment as much as possible, the image adjustment coefficients corresponding to each camera can be determined in advance, reducing the influence of the external parameters.

[0097] In a possible implementation, the process can be performed after the external parameter calibration when the camera is shipped. Since the calibrated external parameters before the camera is shipped are accurate, the image adjustment coefficients can be determined based on the accurate external parameters. For two adjacent cameras in the four cameras, for example, the front camera and the left camera, the front camera and the right camera, the rear camera and the left camera, and the rear camera and the right camera, there is a common shooting area in the shooting range of the two adjacent cameras, and the position of the common shooting area in the world coordinate system is the same. Therefore, there are four common shooting areas for the four cameras.

[0098] For any one common shooting area, the first image information and the second image information corresponding to the common shooting area in the two adjacent cameras can be determined. That is, the common shooting area corresponds to an image area in the camera, and the image information of the image area is determined. Based on the first image adjustment coefficient and the second image adjustment coefficient corresponding to the two adjacent cameras, and the first image information and the second image information, the image loss function is determined. For example, the image loss function can be expressed as the difference between the product of the first image adjustment coefficient and the first image information, and the product of the second image adjustment coefficient and the second image information. At this time, the first image adjustment coefficient and the second image adjustment coefficient are unknown numbers, and the values of the first image adjustment coefficient and the second image adjustment coefficient can be adjusted to change the value of the image loss function.

[0099] The first target value corresponding to the first image adjustment coefficient and the second target value corresponding to the second image adjustment coefficient are calculated when the image loss function meets the preset condition. Optionally, since the image loss function can be dynamically adjusted, the preset condition can be that the image loss function reaches the minimum value, and the value of the first image adjustment coefficient corresponding to the minimum value of the image loss function is determined, that is, the first target value, and the value of the second image adjustment coefficient is determined, that is, the second target value.

[0100] Then, based on the first target value, the second target value, the image information of the first target lane point, and the image information of the second target lane point, the first loss function is calculated. Specifically, a product can be obtained by multiplying the image information of the first target lane point by the first target value, another product can be obtained by multiplying the image information of the second target lane point by the second target value, and the difference between the two products is calculated to obtain the first loss function.

[0101] In a possible implementation, two adjacent cameras are represented by the third camera and the fourth camera, the first image information can include at least one of a brightness average or a color average corresponding to a common shooting area shot by the third camera, the second image information can include at least one of a brightness average or a color average corresponding to the common shooting area shot by the fourth camera, and the first image information corresponds to the second image information. The brightness average can represent an average value of brightness corresponding to each pixel of an image formed by shooting the common shooting area, and the color average can include an average value corresponding to RGB respectively.

[0102] According to the above embodiments, the first loss function can be determined based on the sum of the brightness difference value and the color difference value. Based on this, taking the first image information including a first brightness average i1, a first red average r1, a first green average g1, and a first blue average b1 as an example, the second image information including a second brightness average i2, a second red average r2, a second green average g2, and a second blue average b2, the first adjustment coefficient including a first brightness adjustment coefficient j1, a first red adjustment coefficient jr1, a first green adjustment coefficient jg1, and a first blue adjustment coefficient jb1, and the second adjustment coefficient including a second brightness adjustment coefficient j2, a second red adjustment coefficient jr2, a second green adjustment coefficient jg2, and a second blue adjustment coefficient jb2, the first loss function L can be represented as L =‖i 1×j1―i 2×j2‖+‖r1×jr1―r2×jr2‖+‖g1×jg1―g2×jg2‖+‖b1×jb1―b2×jb2‖.

[0103] Through the method, the influence of environmental factors on the external parameter calibration can be reduced as much as possible, and the accuracy of the external parameter calibration can be improved. It should be noted that the first target lane point can be understood as a target lane point, and the first loss function can be calculated based on the method provided in the above embodiments for one target lane point in the common shooting area. When the first target lane point includes multiple target lane points, the first loss function corresponding to each target lane point can be accumulated as the final first loss function.

[0104] Through the method provided in the above embodiments, the image information of the lane points at the same position shot by different cameras is used to establish the loss function of the external parameter, and the external parameter of the camera is automatically iteratively adjusted to reduce the value of the loss function, so that the optimized external parameter is obtained. Compared with the method of manually calibrating the external parameter, the efficiency and accuracy of the external parameter calibration can be improved.

[0105] After obtaining the optimized extrinsic parameters of each camera based on the method provided in steps S101-S106 of the above embodiment, a loss function about the optimized extrinsic parameters can be further established based on the position relationship of the lane points on the lane line in the world coordinate system, and the extrinsic parameters of the camera can be automatically iteratively adjusted to reduce the value of the loss function, so as to obtain further optimized extrinsic parameters and improve the accuracy of the extrinsic parameter calibration.

[0106] The method can include the following steps A1-A3:

[0107] A1: For an image captured by any camera, a target lane point in the image is converted based on the optimized extrinsic parameters of the camera to obtain a corresponding world coordinate point of the target lane point in the world coordinate system.

[0108] The optimized extrinsic parameters of each camera are obtained based on the method embodiments described above. The target lane point in the image can be obtained in the following manner: in a specific implementation, first, a target lane line in the image is extracted, and the type of the target lane line can include a single solid line and a transverse solid line. For example, a lane line detection model can be used to extract the lane line in the image, and the lane line detection model is a pre-trained neural network model that can be used to detect lane lines in the image, including single solid lines, single dashed lines, double solid lines, transverse solid lines, and the like. Then, the lane lines extracted by the lane line detection model are screened to obtain at least one of a single solid line or a transverse solid line as the target lane line. Then, the target lane line is traversed to determine a plurality of boundary lane points located on the boundary of the target lane line, and a target lane point is determined from the plurality of boundary lane points.

[0109] Based on the image captured by the camera, an image coordinate of the target lane point in the image coordinate system can be obtained. Then, the extrinsic parameters of the camera can be used to realize the conversion between the image coordinate system and the world coordinate system to obtain a world coordinate point of the target lane point in the world coordinate system.

[0110] In a possible implementation, the image coordinate of the target lane point can be converted to the world coordinate system based on the extrinsic parameters of the camera, the intrinsic parameters of the camera, and the depth information corresponding to the target lane point to obtain a world coordinate of the world coordinate point in the world coordinate system. For the specific implementation process, reference can be made to the above embodiment, which will not be described here.

[0111] A2: For a plurality of target lane points corresponding to different cameras and located on the same lane line, a second loss function is calculated based on the position relationship between a plurality of world coordinate points corresponding to the plurality of target lane points.

[0112] Since the vehicle is provided with four cameras, there are multiple cameras that can capture the same lane line, for example, the front camera, the rear camera, and the left camera can capture the same lane line on the left side (single solid line), the front camera, the rear camera, and the right camera can capture the same lane line on the right side (single solid line), the front camera, the left camera, and the right camera can capture the same lane line in front (horizontal solid line), and the rear camera, the left camera, and the right camera can capture the same lane line in back (horizontal solid line). The positions of the same lane line in the world coordinate system are the same, so when multiple target lane points captured by different cameras are on the same lane line, the multiple target lane points have a certain positional relationship, and therefore the positional relationship between the world coordinate points corresponding to the multiple target lane points on the same lane line can be used to calculate the second loss function.

[0113] As can be seen from the above embodiments, at least two target lane points on the lane line can be determined in the image captured by each camera. In this embodiment, taking the determination of two target lane points on each lane line as an example, the image captured by each camera is introduced.

[0114] For details, please refer to Figure 5a As shown in the figure, the embodiment of the present application provides a schematic diagram of capturing the same lane line.

[0115] Figure 5a As shown in the figure, the embodiment of the present application provides a schematic diagram of capturing the same lane line.

[0116] When the plurality of target lane points corresponding to the front camera, the left camera and the rear camera respectively are located at the same boundary of the same single solid line, theoretically, the horizontal coordinates of the plurality of world coordinate points corresponding to the plurality of target lane points should be the same. When there is a difference in the horizontal coordinates of the plurality of world coordinate points, it indicates that the extrinsic parameters of the plurality of cameras are possibly inaccurate, resulting in that the conversion of the image coordinates of the target lane points to the world coordinates is not accurate enough, and therefore the extrinsic parameters of the plurality of cameras can be adjusted based on the difference in the horizontal coordinates of the plurality of world coordinate points. Specifically, when the plurality of target lane points corresponding to the front camera, the left camera and the rear camera respectively are located at the same boundary of the same single solid line, a first horizontal coordinate difference between the plurality of world coordinate points corresponding to the plurality of target lane points is calculated. In Figure 5a which F1, L1 and B1 are located at the same boundary of the lane line 1, and therefore the first horizontal coordinate difference between F1, L1 and B1 can be calculated. For example, the horizontal coordinate difference between any two world coordinate points can be calculated, and then the sum of the plurality of horizontal coordinate differences is calculated as the first horizontal coordinate difference. Similarly, F2, L2 and B2 are located at the other boundary of the lane line 1, and the first horizontal coordinate difference between F2, L2 and B2 also needs to be calculated. That is, two first horizontal coordinate differences corresponding to the two boundaries of the same single solid line can be obtained.

[0117] Similarly, when the plurality of target lane points corresponding to the front camera, the right camera and the rear camera respectively are located at the same boundary of the same single solid line, the horizontal coordinates of the plurality of world coordinate points corresponding to the plurality of target lane points should be the same. Specifically, when the plurality of target lane points corresponding to the front camera, the right camera and the rear camera respectively are located at the same boundary of the same single solid line, a second horizontal coordinate difference between the plurality of world coordinate points corresponding to the plurality of target lane points can be calculated. In Figure 5a which F3, R1 and B3 are located at the same boundary of the lane line 2, and therefore the second horizontal coordinate difference between F3, R1 and B3 can be calculated. For example, the horizontal coordinate difference between any two world coordinate points can be calculated, and then the sum of the plurality of horizontal coordinate differences is calculated as the second horizontal coordinate difference. F4, R2 and B4 are located at the other boundary of the lane line 2, and the second horizontal coordinate difference between F4, R2 and B4 also needs to be calculated. That is, two second horizontal coordinate differences corresponding to the two boundaries of the same single solid line can be obtained.

[0118] Referring to Figure 5b , another schematic diagram for photographing the same lane line provided by the embodiment of the present application is shown.

[0119] According to Figure 5bIt can be seen that the same lane line captured by the front camera, left camera, and right camera is a horizontal solid line, denoted as lane line 3. Among them, the world coordinate points corresponding to the two target lane points in lane line 3 extracted by the front camera are denoted as F5 and F6, the two world coordinate points corresponding to lane line 3 by the left camera are denoted as L3 and L4, and the two world coordinate points corresponding to lane line 3 by the right camera are denoted as R3 and R4.

[0120] When multiple target lane points corresponding to the current camera, left camera, and right camera are located on the same boundary of the same horizontal solid line, theoretically, the ordinates of the multiple world coordinate points corresponding to these target lane points should be the same. When there are differences in the ordinates of multiple world coordinate points, it indicates that the extrinsic parameters of the cameras may be inaccurate, leading to inaccurate conversion of the image coordinates of the target lane points to world coordinates. Therefore, the extrinsic parameters of the multiple cameras can be adjusted based on the differences in the ordinates of the multiple world coordinate points. Specifically, when multiple target lane points corresponding to the current camera, left camera, and right camera are located on the same boundary of the same horizontal solid line, the first longitudinal coordinate difference between the world coordinate points corresponding to these multiple target lane points is calculated. Figure 5b In this context, L3, F5, and R3 are located on the same boundary of lane line 3, so the first longitudinal coordinate difference between L3, F5, and R3 can be calculated. For example, the longitudinal coordinate difference between any two world coordinate points can be calculated, and then the sum of multiple longitudinal coordinate differences can be used as the first longitudinal coordinate difference. Similarly, L4, F6, and R4 are located on the other boundary of lane line 3, so the first longitudinal coordinate difference between L4, F6, and R4 also needs to be calculated. In other words, the two first longitudinal coordinate differences corresponding to the two boundaries of the same horizontal solid line can be obtained.

[0121] See Figure 5c The diagram shown is another example of photographing the same lane line provided in this application.

[0122] according to Figure 5c As can be seen, the same lane line captured by the rear camera, left camera, and right camera is a horizontal solid line, denoted as lane line 4. The two world coordinate points corresponding to the rear camera in lane line 4 are denoted as B5 and B6, the two world coordinate points corresponding to the left camera in lane line 4 are denoted as L5 and L6, and the two world coordinate points corresponding to the right camera in lane line 4 are denoted as R5 and R6.

[0123] Similarly, it can be known that when the plurality of target lane points corresponding to the rear camera, the left camera and the right camera are located at the same boundary of the same horizontal solid line, the longitudinal coordinates of the plurality of world coordinate points corresponding to the plurality of target lane points should be the same. Specifically, when the plurality of target lane points corresponding to the rear camera, the left camera and the right camera are located at the same boundary of the same horizontal solid line, the second longitudinal coordinate difference between the world coordinate points corresponding to the plurality of target lane points is calculated. In Figure 5c In the middle, L5, B5, R5 are located at the same boundary of the lane line 4, so the second longitudinal coordinate difference between L5, B5, R5 can be calculated. For example, the longitudinal coordinate difference between any two world coordinate points can be calculated, and then the sum of a plurality of longitudinal coordinate differences is calculated as the second longitudinal coordinate difference. Similarly, L6, B6, R6 are located at the other boundary of the lane line 4, and the second longitudinal coordinate difference between L6, B6, R6 also needs to be calculated. That is, two second longitudinal coordinate differences corresponding to two boundaries of the same horizontal solid line can be obtained.

[0124] In addition, based on the target lane points of the cameras on the two boundaries of the lane line, the lane line width can be calculated, and the lane line width determined by each camera should be the same. Therefore, the extrinsic parameters can be adjusted and optimized by calculating the difference between the lane line widths determined by different cameras. Specifically, based on the world coordinate points corresponding to the plurality of target lane points corresponding to the same lane line of the camera, the lane line width is calculated. The plurality of target lane points include the target lane points on the two boundaries of the lane line. Since the vehicle includes four cameras, four lane line widths corresponding to the four cameras can be calculated. Then, the difference between any two lane line widths of the four lane line widths is calculated to obtain six differences, and the sum of the six differences is calculated as the difference of the lane line width, which is used to represent the accuracy of the camera extrinsic parameters. The larger the sum of the differences, the less accurate the extrinsic parameters of the camera.

[0125] Then the second loss function can be determined based on at least one of the first lateral coordinate difference, the second lateral coordinate difference, the first longitudinal coordinate difference, the second longitudinal coordinate difference, or the difference of the lane line width. For example, the sum of the squares of the lateral coordinate difference, the longitudinal coordinate difference and the difference of the lane line width can be calculated as the second loss function. The lateral coordinate difference can be represented as the sum of each first lateral coordinate difference and each second lateral coordinate difference, and the longitudinal coordinate difference can be represented as the sum of each first longitudinal coordinate difference and each second longitudinal coordinate difference.

[0126] It should be noted that when the second loss function is calculated, the position relationship satisfied by the actual application scenario can be adjusted. For example, when there is no horizontal solid line in the actual application scenario, the vertical coordinate difference cannot be calculated, and the second loss function can be calculated using the horizontal coordinate difference and the difference. Or, when there is only a single solid line on the left side of the vehicle in the actual application scenario, the horizontal coordinate difference only includes the first horizontal coordinate difference. According to the above principle, the calculation method of the second loss function is determined according to the actual application scenario.

[0127] A3: iteratively adjust the extrinsic parameters of the four cameras, and recalculate the second loss function until the iteration stopping condition is met, to obtain the optimized extrinsic parameters.

[0128] The second loss function can represent the accuracy of the extrinsic parameters of the camera. When the second loss function is larger, it indicates that the extrinsic parameters of the camera are less accurate. Then the extrinsic parameters of the four cameras can be adjusted, and the second loss function is recalculated according to the method provided in the above embodiment. The above process is iteratively executed until the iteration stopping condition is met, to obtain the optimized extrinsic parameters of the four cameras. The iteration stopping condition can be that the second loss function is less than a threshold, or the iteration process reaches a preset number of times.

[0129] Through the method provided in the above embodiment, the position relationship of the lane points on the same lane in the world coordinate system can be used to establish a loss function about the optimized extrinsic parameters, and the extrinsic parameters of the camera can be automatically iteratively adjusted to reduce the value of the loss function, thereby obtaining further optimized extrinsic parameters. The accuracy of the extrinsic parameter calibration can be improved.

[0130] Based on the above method embodiment, the embodiment of the present application also provides a camera extrinsic parameter calibration device. Referring to Figure 6 as shown, a schematic diagram of a camera extrinsic parameter calibration device provided by the embodiment of the present application.

[0131] The device 600 comprises:

[0132] The image acquisition unit 601 is configured to acquire a plurality of images captured by four cameras of a vehicle when the vehicle is in a straight driving state.

[0133] The determination unit 602 is configured to determine a first target lane point in a first image captured by a first camera, the first camera representing any camera of the four cameras, and the first target lane point belonging to a common shooting area of the first image and a second image captured by a second camera.

[0134] The first conversion unit 603 is configured to convert the first target lane point to a panoramic plane based on the first extrinsic parameters of the first camera to obtain a first projection point.

[0135] The second conversion unit 604 is configured to convert the first projection point based on the second extrinsic parameter of the second camera, to obtain a second target lane point corresponding to the first projection point in the second image.

[0136] The loss function determination unit 605 is configured to calculate a first loss function based on the image information of the first target lane point and the image information of the second target lane point.

[0137] The iteration unit 606 is configured to iteratively adjust the second extrinsic parameter and recalculate the first loss function until an iteration stop condition is met, to obtain an optimized second extrinsic parameter.

[0138] In a possible implementation, the first conversion unit 603 is further configured to, for an image captured by any camera, convert a target lane point in the image based on the optimized extrinsic parameter of the camera, to obtain a world coordinate point corresponding to the target lane point in a world coordinate system.

[0139] The loss function determination unit 605 is further configured to, for a plurality of target lane points corresponding to different cameras and located on a same lane line, calculate a second loss function based on a positional relationship between a plurality of world coordinate points corresponding to the plurality of target lane points.

[0140] The iteration unit 606 is further configured to iteratively adjust the optimized extrinsic parameters of the four cameras and recalculate the second loss function until an iteration stop condition is met, to obtain further optimized extrinsic parameters.

[0141] In a possible implementation, the loss function determination unit 605 is configured to, when the four cameras are factory-calibrated, determine four common shooting areas corresponding to any two adjacent cameras of the four cameras; for any common shooting area, determine first image information and second image information corresponding to the two adjacent cameras associated with the common shooting area; determine an image loss function based on a first image adjustment coefficient and a second image adjustment coefficient corresponding to the two adjacent cameras, and the first image information and the second image information; calculate a first target value corresponding to the first image adjustment coefficient and a second target value corresponding to the second image adjustment coefficient when the image loss function meets a preset condition; and calculate the first loss function based on the first target value, the second target value, the image information of the first target lane point, and the image information of the second target lane point.

[0142] In a possible implementation, the loss function determination unit 605 is configured to determine a luminance difference value based on the luminance information of the first target lane point and the luminance information of the second target lane point; determine a color difference value based on the color information of the first target lane point and the color information of the second target lane point; and calculate the first loss function based on the luminance difference value and the color difference value.

[0143] In a possible implementation, the first image information includes at least one of a luminance mean value or a color mean value corresponding to a common shooting area shot by a third camera, and the second image information includes at least one of a luminance mean value or a color mean value corresponding to a common shooting area shot by a fourth camera, the third camera and the fourth camera represent two adjacent cameras, and the first image information corresponds to the second image information.

[0144] In a possible implementation, the loss function determination unit 605 is further configured to, when a plurality of target lane points corresponding to a current camera, a left camera and a rear camera are located on a same boundary of a same single solid line, calculate a first lateral coordinate difference between world coordinate points corresponding to the plurality of target lane points; or when a plurality of target lane points corresponding to the current camera, a right camera and the rear camera are located on a same boundary of a same single solid line, calculate a second lateral coordinate difference between world coordinate points corresponding to the plurality of target lane points; or when a plurality of target lane points corresponding to the current camera, the left camera and the right camera are located on a same boundary of a same horizontal solid line, calculate a first longitudinal coordinate difference between world coordinate points corresponding to the plurality of target lane points; or when a plurality of target lane points corresponding to the rear camera, the left camera and the right camera are located on a same boundary of a same horizontal solid line, calculate a second longitudinal coordinate difference between world coordinate points corresponding to the plurality of target lane points; or for any camera, based on world coordinate points corresponding to a plurality of target lane points located on a same lane line on an image corresponding to the camera, calculate a lane line width, and calculate a difference value between four lane line widths corresponding to four cameras; and determine the first loss function based on at least one of the first lateral coordinate difference, the second lateral coordinate difference, the first longitudinal coordinate difference, the second longitudinal coordinate difference or the difference value.

[0145] In a possible implementation, the determination unit 602 is configured to extract a target lane line in the image, the target lane line including at least one of a single solid line or a horizontal solid line; traverse the target lane line to determine a plurality of boundary lane points located on a boundary of the target lane line; and determine the target lane point from the plurality of boundary lane points.

[0146] In a possible implementation, the first conversion unit 603 is further configured to convert the image coordinates of the target lane point to a world coordinate system based on the optimized extrinsic parameter, the intrinsic parameter of the camera, and depth information corresponding to the target lane point, to obtain a world coordinate of the world coordinate point in the world coordinate system.

[0147] In a possible implementation, the first conversion unit 603 is configured to convert the image coordinates of the first target lane point to a world coordinate system based on the first extrinsic parameter, the first intrinsic parameter of the first camera, and depth information corresponding to the first target lane point, to obtain a world coordinate of the first world coordinate point in the world coordinate system; and convert the world coordinate of the first world coordinate point to a panoramic plane based on a mapping matrix between the world coordinate system and the panoramic plane, to obtain a panoramic coordinate of the first projection point in the panoramic plane.

[0148] Based on the above method embodiments and device embodiments, an electronic device is further provided in the embodiments of the present application. The embodiments will be described below with reference to the accompanying drawings.

[0149] Referring to Figure 7 , Figure 7 FIG. 1 is a schematic diagram of an electronic device provided in the embodiments of the present application.

[0150] The device 700 includes a memory 701 and a processor 702.

[0151] The memory 701 is configured to store related program codes.

[0152] The processor 702 is configured to invoke the program codes and perform the extrinsic parameter calibration method of the camera described in the above method embodiments.

[0153] In addition, the embodiments of the present application further provide a computer readable storage medium configured to store a computer program, and the computer program is configured to perform the extrinsic parameter calibration method of the camera described in the above method embodiments.

[0154] The embodiments of the present application further provide a computer program product, which includes computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the extrinsic parameter calibration method of the camera described in the above method embodiments.

[0155] It should be noted that the computer-readable medium in the above embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0156] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0157] It should be noted that the embodiments of the present application are described in progressive manner in the present specification, and each embodiment focuses on the difference from other embodiments, and the same or similar parts among the embodiments can be mutually referred to. Especially, the system or device embodiments are described more simply, and the relevant part can be referred to the part of the method embodiment. The device embodiment described above is only schematic, and the units or modules described as separate components can or can not be physically separate, and the components shown as units or modules can or can not be physical modules, i.e. can be located in one place, or can be distributed on multiple network units, and part or all of the units or modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement it without creative labor.

[0158] The flow diagrams and block diagrams in the drawings are illustrations of architectures, functions, and operations that can be implemented in methods, apparatus, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code that comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0159] It should be understood that, in this application, "at least one", "one or more", "multiple", "two or more" means one or more than one. "And / or" is used to describe the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0160] It should also be noted that in this application, relationship terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0161] The steps of the methods or algorithms described in conjunction with the embodiments disclosed in this application can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0162] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for calibrating extrinsic parameters of a camera, characterized in that, The method comprises: When the vehicle is in a straight driving state, a plurality of images captured by four cameras of the vehicle are acquired; A first target lane point in a first image captured by a first camera is determined, the first camera representing any one of the four cameras, the first target lane point belonging to a common capturing area of the first image and a second image captured by a second camera; The first target lane point is converted to a panorama plane based on a first extrinsic parameter of the first camera to obtain a first projection point; The first projection point is converted based on a second extrinsic parameter of the second camera to obtain a second target lane point corresponding to the first projection point in the second image; A first loss function is calculated based on image information of the first target lane point and image information of the second target lane point; The second extrinsic parameter is iteratively adjusted, and the first loss function is recalculated until an iteration stop condition is met to obtain an optimized second extrinsic parameter.

2. The method of claim 1, wherein, The method further comprises: For an image captured by any camera, a target lane point in the image is converted based on an optimized extrinsic parameter of the camera to obtain a world coordinate point corresponding to the target lane point in a world coordinate system; For a plurality of target lane points corresponding to different cameras and located on a same lane line, a second loss function is calculated based on a positional relationship between a plurality of world coordinate points corresponding to the plurality of target lane points; The optimized extrinsic parameters of the four cameras are iteratively adjusted, and the second loss function is recalculated until an iteration stop condition is met to obtain further optimized extrinsic parameters.

3. The method of claim 1, wherein, The first loss function is calculated based on the image information of the first target lane point and the image information of the second target lane point, comprising: After the four cameras are calibrated for extrinsic parameters when they are shipped, four common capturing areas corresponding to adjacent two cameras of the four cameras are determined; For any common capturing area, first image information and second image information corresponding to the adjacent two cameras associated with the common capturing area are determined; Based on first image adjustment coefficients and second image adjustment coefficients corresponding to the adjacent two cameras, and the first image information and the second image information, an image loss function is determined; When the image loss function meets a preset condition, a first target value corresponding to the first image adjustment coefficient and a second target value corresponding to the second image adjustment coefficient are calculated; Based on the first target value, the second target value, the image information of the first target lane point, and the image information of the second target lane point, the first loss function is calculated.

4. The method of claim 1, wherein, The first loss function is calculated based on the image information of the first target lane point and the image information of the second target lane point, comprising: Based on brightness information of the first target lane point and brightness information of the second target lane point, a brightness difference value is determined; Based on color information of the first target lane point and color information of the second target lane point, a color difference value is determined; Based on the brightness difference value and the color difference value, the first loss function is calculated.

5. The method of claim 3, wherein, The first image information includes at least one of a brightness average value or a color average value corresponding to a common shooting area shot by a third camera, and the second image information includes at least one of a brightness average value or a color average value corresponding to a common shooting area shot by a fourth camera, the third camera and the fourth camera represent two adjacent cameras, and the first image information corresponds to the second image information.

6. The method of claim 2, wherein, The second loss function is calculated based on a positional relationship between a plurality of world coordinate points corresponding to a plurality of target lane points. When a plurality of target lane points corresponding to a current camera, a left camera and a rear camera are located at a same boundary of a same single solid line, a first lateral coordinate difference between world coordinate points corresponding to the plurality of target lane points is calculated; or, When a plurality of target lane points corresponding to a current camera, a right camera and a rear camera are located at a same boundary of a same single solid line, a second lateral coordinate difference between world coordinate points corresponding to the plurality of target lane points is calculated; or, When a plurality of target lane points corresponding to a current camera, a left camera and a right camera are located at a same boundary of a same horizontal solid line, a first longitudinal coordinate difference between world coordinate points corresponding to the plurality of target lane points is calculated; or, When a plurality of target lane points corresponding to a rear camera, a left camera and a right camera are located at a same boundary of a same horizontal solid line, a second longitudinal coordinate difference between world coordinate points corresponding to the plurality of target lane points is calculated; or, For any camera, a lane width is calculated based on world coordinate points corresponding to a plurality of target lane points located on a same lane line on an image corresponding to the camera, and a difference value between four lane widths corresponding to four cameras is calculated; The second loss function is determined based on at least one of the first lateral coordinate difference, the second lateral coordinate difference, the first longitudinal coordinate difference, the second longitudinal coordinate difference or the difference value.

7. The method of claim 2, wherein, The target lane point in the image is determined, including: A target lane line in the image is extracted, the target lane line including at least one of a single solid line or a horizontal solid line; A plurality of boundary lane points located on a boundary of the target lane line are determined by traversing the target lane line; The target lane point is determined from the plurality of boundary lane points.

8. The method of claim 2, wherein, The target lane point is converted based on the optimized extrinsic parameter of the camera to obtain a world coordinate point corresponding to the target lane point in a world coordinate system, including: Based on the optimized extrinsic parameter, an intrinsic parameter of the camera and depth information corresponding to the target lane point, an image coordinate of the target lane point is converted to a world coordinate system to obtain a world coordinate of the world coordinate point in the world coordinate system.

9. The method according to any one of claims 1 to 8, characterized in that, The first target lane point is converted to a panoramic plane based on the first extrinsic parameter of the first camera to obtain a first projection point, including: The image coordinate of the first target lane point is converted to a world coordinate system based on the first extrinsic parameter, a first intrinsic parameter of the first camera, and depth information corresponding to the first target lane point, to obtain a world coordinate of a first world coordinate point in the world coordinate system; The world coordinate of the first world coordinate point is converted to a panoramic plane based on a mapping matrix between the world coordinate system and the panoramic plane, to obtain a panoramic coordinate of the first projection point in the panoramic plane.

10. An external parameter calibration device of a camera, characterized in that, The apparatus includes: An image acquisition unit configured to acquire a plurality of images captured by four cameras of a vehicle when the vehicle is in a straight driving state; A determination unit configured to determine a first target lane point in a first image captured by a first camera, the first camera representing any one of the four cameras, the first target lane point belonging to a common capturing region of the first image and a second image captured by a second camera; A first conversion unit configured to convert the first target lane point to a panoramic plane based on a first extrinsic parameter of the first camera, to obtain a first projection point; A second conversion unit configured to convert the first projection point based on a second extrinsic parameter of the second camera, to obtain a second target lane point corresponding to the first projection point in the second image; A loss function determination unit configured to calculate a first loss function based on image information of the first target lane point and image information of the second target lane point; An iteration unit configured to iteratively adjust the second extrinsic parameter and recalculate the first loss function until an iteration stop condition is met, to obtain an optimized second extrinsic parameter.

Images