Camera extrinsic parameter calibration method, device, electronic equipment and medium

Abstract

The present invention discloses a camera extrinsic parameter calibration method, device, electronic device and storage medium. The method includes: determining N bird's-eye view images corresponding to the N cameras based on the intrinsic parameters and current extrinsic parameters of each camera in N cameras installed on a target vehicle, wherein the target vehicle is located in a linear calibration site and N is a positive integer; identifying multiple target straight lines in each bird's-eye view, and constructing an objective function with the extrinsic parameters of the N cameras as variables based on multiple target straight lines parallel to each other in each bird's-eye view and multiple target straight lines overlapping with each other in adjacent bird's-eye views with a common area; optimizing the current extrinsic parameters of each camera based on the objective function, and using the optimized extrinsic parameters of each camera as the final extrinsic parameters of the corresponding camera. This solution can effectively improve the accuracy of camera extrinsic parameter calibration.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a camera extrinsic parameter calibration method, device, electronic equipment, and medium. Background Art

[0002] With the rapid development of vehicle driver assistance technology, in-vehicle panoramic imaging systems are widely used in various vehicles. In-vehicle panoramic imaging systems usually use multiple fisheye cameras to capture images, perform distortion correction, perspective projection, and stitching on the captured images to form a panoramic stitched image (a bird's-eye view image) around the object.

[0003] In the existing technology, in the process of generating panoramic stitching images, it is necessary to use the camera's extrinsic parameters and intrinsic parameters to convert between world coordinates, camera coordinates and pixel coordinates, so as to generate the bird's-eye view corresponding to each camera; however, if there are errors in the calibrated camera extrinsic parameters, the final panoramic stitching image will be misaligned and deviated. Summary of the Invention

[0004] Embodiments of the present invention provide a camera extrinsic parameter calibration method, device, electronic device, and medium, which effectively improve the accuracy of camera extrinsic parameter calibration.

[0005] A first aspect of an embodiment of the present invention provides a camera extrinsic parameter calibration method, comprising:

[0006] Determine N bird's-eye views corresponding to each of N cameras installed on a target vehicle based on the intrinsic parameters of each camera and the current extrinsic parameters, wherein the target vehicle is located in a linear calibration site, and N is a positive integer;

[0007] Identify multiple target lines in each bird's-eye view, and construct an objective function with extrinsic parameters of the N cameras as variables based on the multiple parallel target lines in each bird's-eye view and the multiple overlapping target lines in adjacent bird's-eye views that have a common area;

[0008] Based on the objective function, the current extrinsic parameters of each camera are optimized, and the optimized extrinsic parameters of each camera are used as the final extrinsic parameters of the corresponding camera.

[0009] Optionally, determining N bird's-eye views corresponding to N cameras arranged on the target vehicle based on the intrinsic parameters of each camera and the current extrinsic parameters includes:

[0010] For each of the N cameras, based on a current extrinsic parameter of the camera, world coordinates are converted into camera coordinates; based on an intrinsic parameter of the camera, the camera coordinates are converted into pixel coordinates; and based on the pixel coordinates, a bird's-eye view corresponding to the camera is generated.

[0011] Optionally, constructing an objective function with the extrinsic parameters of the N cameras as variables based on the multiple parallel target lines in each bird's-eye view and the multiple overlapping target lines in adjacent bird's-eye views having a common area includes:

[0012] Based on multiple mutually parallel target straight lines in each bird's-eye view, construct a first sub-objective function corresponding to each bird's-eye view, and obtain a total of N first sub-objective functions;

[0013] Based on each group of M groups of adjacent bird's-eye views with a common area, based on multiple overlapping target straight lines in the group of adjacent bird's-eye views, a second sub-objective function corresponding to the group of adjacent bird's-eye views is constructed, and a total of M second sub-objective functions are obtained, where M is a positive integer;

[0014] The objective function is constructed based on the N first sub-objective functions and the M second sub-objective functions.

[0015] Optionally, constructing a first sub-objective function corresponding to each bird's-eye view based on a plurality of mutually parallel target straight lines in each bird's-eye view includes:

[0016] For each of the bird's-eye view images, the plurality of target straight lines in the bird's-eye view image are divided into K groups of parallel straight lines, wherein each group of parallel straight lines includes a first straight line and a second straight line that are parallel to each other, and K is a positive integer;

[0017] For each set of parallel straight lines, determining the distances between any two points on the first straight line and the second straight line, and the theoretical distance between the first straight line and the second straight line;

[0018] For each group of adjacent bird's-eye view images, the first sub-objective function of the bird's-eye view image is constructed based on the distances from any two points on the first straight line to the second straight line in each group of parallel straight lines in the bird's-eye view image, and the theoretical distance between the first straight line and the second straight line in each group of parallel straight lines.

[0019] Optionally, each group of adjacent bird's-eye views includes a first bird's-eye view and a second bird's-eye view, and constructing a second sub-objective function corresponding to each group of adjacent bird's-eye views in the M groups of adjacent bird's-eye views with a common area based on a plurality of mutually overlapping target straight lines in the group of adjacent bird's-eye views includes:

[0020] For each set of adjacent bird's-eye views, S sets of coincident straight lines are determined from the set of adjacent bird's-eye views, where each set of coincident straight lines includes a third straight line and a fourth straight line that coincide with each other, S is a positive integer, the third straight line is the target straight line in the first bird's-eye view of the set of adjacent bird's-eye views, and the fourth straight line is the target straight line in the second bird's-eye view of the set of adjacent bird's-eye views;

[0021] For each set of coincident straight lines, calculate the distance between any two points on the third straight line and the fourth straight line in the set of coincident straight lines;

[0022] For each group of adjacent bird's-eye views, a second sub-objective function of the group of adjacent bird's-eye views is constructed based on the distances from any two points on the third straight line to the fourth straight line in each group of coincident straight lines in the adjacent bird's-eye views.

[0023] Optionally, optimizing the current extrinsic parameters of each camera based on the objective function, and using the optimized extrinsic parameters of each camera as the final extrinsic parameters of the corresponding camera, includes:

[0024] Each of the cameras is used as a target camera, and the following iterative steps are performed: optimizing the current extrinsic parameters of the target camera using a preset optimization algorithm to obtain optimized target extrinsic parameters; transforming each target straight line in the bird's-eye view corresponding to the target camera into a target bird's-eye view based on the current extrinsic parameters of the target camera to obtain a straight line equation for each target straight line in the target bird's-eye view, wherein the target bird's-eye view corresponds to the target extrinsic parameters; and recalculating the function value of the objective function based on the straight line equations of the target straight lines in the N target bird's-eye views corresponding to the N cameras;

[0025] If the function value of the recalculated objective function is less than the threshold, the iteration is stopped, and the target extrinsic parameter after this round of iteration is used as the final extrinsic parameter of the corresponding camera;

[0026] If the function value of the recalculated objective function is greater than or equal to the threshold, the target external parameter after this round of iteration is used as the current external parameter in the next round of iteration, and the iterative step is performed again until the function value of the objective function is less than the threshold.

[0027] Optionally, the transforming each target straight line in the bird's-eye view corresponding to the target camera into a target bird's-eye view based on the current extrinsic parameter of the target camera and the target extrinsic parameter to obtain a straight line equation of each target straight line in the target bird's-eye view includes:

[0028] For each target straight line in the bird's-eye view corresponding to the target camera, determine the coordinates of two target points on the target straight line, the two target points being different;

[0029] Based on the current extrinsic parameter and the target extrinsic parameter, mapping the coordinates of the two target points into the target bird's-eye view to obtain the mapping coordinates of the two target points in the target bird's-eye view;

[0030] The equation of a straight line formed by the mapping coordinates of the two target points is determined as the equation of a straight line transformed from the target straight line to the target bird's-eye view.

[0031] A second aspect of an embodiment of the present invention further provides a camera extrinsic parameter calibration device, comprising:

[0032] a bird's-eye view determination module, configured to determine N bird's-eye view images corresponding to N cameras disposed on a target vehicle based on an intrinsic parameter of each camera and a current extrinsic parameter, wherein the target vehicle is located in a linear calibration site, and N is a positive integer;

[0033] an objective function construction module for identifying multiple target lines in each bird's-eye view and constructing an objective function with extrinsic parameters of the N cameras as variables based on the multiple parallel target lines in each bird's-eye view and the multiple overlapping target lines in adjacent bird's-eye views that have a common area;

[0034] The optimization module is used to optimize the current extrinsic parameters of each camera based on the objective function, and use the optimized extrinsic parameters of each camera as the final extrinsic parameters of the corresponding camera.

[0035] A third aspect of an embodiment of the present invention provides an electronic device, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors to execute operating instructions corresponding to the camera extrinsic parameter calibration method provided in the first aspect.

[0036] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the external parameter calibration method provided in the first aspect.

[0037] The above one or at least one technical solution in the embodiments of the present application has at least the following technical effects:

[0038] The camera extrinsic parameter calibration method in the embodiment of this specification determines N bird's-eye view images corresponding to the N cameras one by one based on the intrinsic parameters and current extrinsic parameters of each camera in the N cameras installed on the target vehicle, wherein the target vehicle is located in a linear calibration field and N is a positive integer; identifies multiple target straight lines in each bird's-eye view, and constructs an objective function with the extrinsic parameters of the N cameras as variables based on multiple target straight lines that are parallel to each other in the bird's-eye view and multiple target straight lines that overlap with each other in adjacent bird's-eye views with a common area; optimizes the current extrinsic parameters of each camera based on the objective function, and uses the optimized extrinsic parameters of each camera as the final extrinsic parameters of the corresponding camera. In the above scheme, since the objective function is constructed using the target straight lines that overlap with each other in adjacent bird's-eye views with a common area, the optimized camera extrinsic parameters can minimize the overall misalignment error of the stitched panoramic image, thereby obtaining a panoramic image with a better overall effect and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a camera extrinsic calibration method provided in an embodiment of this specification;

[0040] Figure 2 A schematic diagram of a linear calibration site provided in an embodiment of this specification;

[0041] Figure 3 A schematic diagram of four original fisheye images provided in the embodiments of this specification;

[0042] Figure 4 A schematic diagram of four bird's-eye views corresponding to four original fisheye images provided in an embodiment of this specification;

[0043] Figure 5 A schematic diagram of the straight line recognition results corresponding to the four-way bird's-eye view provided in the embodiment of this specification;

[0044] Figure 6 A schematic diagram of a camera extrinsic calibration device provided in an embodiment of this specification;

[0045] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0046] The embodiments of the present application provide a camera extrinsic parameter calibration method, device, electronic device and storage medium.

[0047] The overall idea of ​​the technical solution of the embodiment of the present application is as follows: based on the intrinsic parameters and current extrinsic parameters of each camera among the N cameras set on the target vehicle, N bird's-eye views corresponding to the N cameras are determined, wherein the target vehicle is located in a linear calibration site and N is a positive integer; multiple target straight lines in each bird's-eye view are identified, and based on multiple target straight lines parallel to each other in each bird's-eye view and multiple target straight lines overlapping with each other in adjacent bird's-eye views with a common area, an objective function with the extrinsic parameters of the N cameras as variables is constructed; based on the objective function, the current extrinsic parameters of each camera are optimized, and the optimized extrinsic parameters of each camera are used as the final extrinsic parameters of the corresponding camera.

[0048] The solution in this application uses overlapping target straight lines in adjacent bird's-eye views with common areas to construct the objective function. The optimized camera extrinsics can minimize the overall misalignment error of the stitched panoramic image, thereby obtaining a panoramic image with better overall effect and improving the user experience.

[0049] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0050] First, the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the associated objects.

[0051] like Figure 1 As shown in FIG, a camera extrinsic parameter calibration method provided in an embodiment of this specification includes the following steps:

[0052] Step S101: Based on the intrinsic parameters of each camera and the current extrinsic parameters of N cameras installed on the target vehicle, N bird's-eye views corresponding to the N cameras are determined, wherein the target vehicle is located in a linear calibration site, and N is a positive integer;

[0053] Step S102: Identify multiple target lines in each bird's-eye view, and construct an objective function with the extrinsic parameters of the N cameras as variables based on the multiple parallel target lines in each bird's-eye view and the multiple overlapping target lines in adjacent bird's-eye views that have a common area.

[0054] Step S103: Based on the objective function, the current extrinsic parameters of each camera are optimized, and the optimized extrinsic parameters of each camera are used as the final extrinsic parameters of the corresponding camera.

[0055] The method provided in the embodiments of this specification can be applied to a vehicle-mounted terminal, to a server that can communicate with a vehicle, or to a system consisting of a vehicle-mounted terminal and a server, without limitation here.

[0056] In step S101, the target vehicle can be an electric vehicle, a fuel vehicle, a hybrid vehicle, etc., which is not limited here. The target vehicle is equipped with N cameras, wherein the camera can be a wide-angle camera, a fisheye camera, etc., and N can be set according to actual needs, for example, N is 4, 5, 6, etc. In the embodiment of this specification, taking N as 4 as an example, the four cameras are respectively set at the front, rear, left, and right of the target vehicle. In one embodiment, the four cameras are all fisheye cameras, respectively set at the front, rear, left rearview mirror, and right rearview mirror of the target vehicle.

[0057] Each camera has its own intrinsic and extrinsic parameters. The camera's intrinsic parameters can be pre-calibrated, while the camera's current extrinsic parameters can be the theoretical camera installation position provided by the user, or they can be intermediate values ​​obtained during the optimization process. It should be noted that since the camera's current extrinsic parameters may differ from the actual camera extrinsic parameters, the methods provided in the embodiments of this specification enable accurate calibration of the camera's extrinsic parameters.

[0058] During the specific implementation process, for each camera, the bird's-eye view corresponding to the camera is determined through the intrinsic parameters and current extrinsic parameters of each camera. The specific acquisition process of the bird's-eye view can be achieved through the following steps: for each camera among the N cameras, based on the current extrinsic parameters of the camera, the world coordinates are converted into camera coordinates; based on the intrinsic parameters of the camera, the camera coordinates are converted into pixel coordinates; based on the pixel coordinates, a bird's-eye view corresponding to the camera is generated.

[0059] Specifically, taking the four cameras set in front, behind, left and right of the target vehicle as an example, the camera set in front of the target vehicle is the first camera, the camera set on the right side of the target vehicle is the second camera, the camera set behind the target vehicle is the third camera, and the camera set on the left side of the target vehicle is the fourth camera.

[0060] When generating the bird's-eye view corresponding to each camera, first, you can use the Z coordinate system of the world coordinate system to calculate the w =0 plane defines the area for generating the bird's-eye view, and sets the width and height of the bird's-eye view, that is, the number of pixels in the width and height directions, respectively. Then, the distance in the width direction corresponding to the world coordinate system and the distance in the height direction corresponding to the world coordinate system are defined, thereby determining the change relationship between the bird's-eye view pixel coordinate system and the world coordinate system. Next, taking the first camera as an example, the world coordinates are converted to camera coordinates based on the current extrinsic parameters of the first camera; then, based on the intrinsic parameters of the first camera, the camera coordinates are converted to pixel coordinates in the image; finally, the pixel values ​​of the pixels in the bird's-eye view corresponding to the first camera can be obtained through bilinear interpolation. It can be seen that by performing an IPM (Inverse Projection Mapping) operation on the original image captured by the first camera based on the intrinsic parameters of the first camera, the current extrinsic parameters, and the camera forming principle, the front view bird's-eye view corresponding to the first camera can be generated. Based on the same method, the right view bird's-eye view corresponding to the second camera, the rear view bird's-eye view corresponding to the third camera, and the left view bird's-eye view corresponding to the fourth camera can be obtained.

[0061] In the examples of this specification, a linear calibration field can be used to calibrate the extrinsic parameters of the target vehicle's N cameras. This field is more compact than a common checkerboard or zigzag field, takes up less space, and can be easily modified to accommodate different vehicle models.

[0062] like Figure 2 As shown in the figure, it is a schematic diagram of a linear calibration site provided by an embodiment of this specification. When the target vehicle is located at Figure 2 When the linear calibration site is shown in the figure, and all four cameras are fisheye cameras, the original four-way fisheye images are as follows: Figure 3 As shown, the four fisheye images are transformed to obtain the bird's-eye views in four directions. Figure 4 shown.

[0063] It should be noted that the specific form of the linear calibration site can be changed according to actual needs. For example, the position and number of the straight lines can be changed, which is not limited here.

[0064] In step S102, after obtaining N bird's-eye views from N cameras, multiple target straight lines contained in each bird's-eye view are identified, where the multiple target straight lines are the straight lines used for extrinsic parameter calibration in the calibration site captured by the camera.

[0065] by Figure 2 As an example of the linear calibration site, Figure 4 The four-way bird's-eye view is used to identify the target straight line, and the recognition results are as follows Figure 5 Please refer to Figure 5 For the front bird's-eye view, 12 target lines are identified, and these 12 target lines can be marked as 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, and 11, respectively. For the right bird's-eye view, 8 target lines are identified, and these 6 target lines can be marked as 0, 1, 2, 3, 4, 5, 6, and 7, respectively. For the rear bird's-eye view, 12 target lines are identified, and these 12 target lines can be marked as 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, and 11, respectively. For the left bird's-eye view, 8 target lines are identified, and these 6 target lines can be marked as 0, 1, 2, 3, 4, 5, 6, and 7, respectively.

[0066] Furthermore, after obtaining the line recognition structure of each bird's-eye view, an objective function is constructed according to the relationship between target lines in each bird's-eye view and the relationship between target lines between adjacent bird's-eye views with common areas.

[0067] In one embodiment, since each bird's-eye view image contains a plurality of target straight lines that are parallel to each other, the parallel relationship of the target straight lines in each bird's-eye view image can be used as a factor in constructing the objective function.

[0068] In addition, considering that there are common areas in some adjacent bird's-eye views, for example Figure 5 For example, the common areas include the following four cases: the common area between the front bird's-eye view and the right bird's-eye view, the common area between the front bird's-eye view and the left bird's-eye view, the common area between the rear bird's-eye view and the right bird's-eye view, and the common area between the rear bird's-eye view and the left bird's-eye view. For adjacent views with common areas, there is a coincidence relationship between the target lines. Taking the front bird's-eye view and the right bird's-eye view as an example, the target line 4 in the front bird's-eye view and the target line 0 in the right bird's-eye view are actually the same line. During the stitching process, the target line 4 in the front bird's-eye view and the target line 0 in the right bird's-eye view need to coincide. Therefore, when constructing the objective function, the coincidence relationship between the lines in the common areas of adjacent views can also be taken into consideration.

[0069] In the embodiment of this specification, the objective function is to reduce the overall misalignment error of the stitched image. Based on the parallel relationship of the straight lines in each bird's-eye view and the coincidence of the straight lines in the common area of ​​adjacent views, an objective function f(x) for the joint optimization of all camera extrinsic parameters is established. Where x is the extrinsic parameter of N cameras. Still taking the four cameras located in front, behind, left, and right of the target vehicle as an example, x is specifically:

[0070] x=[x0,x1,x2,x3]

[0071] x0=[yaw cam0 ,pitch cam0 ,roll cam0 ,x cam0 ,y cam0 ,z cam0 ]

[0072] x1=[yaw cam1 ,pitch cam1 ,roll cam1 ,x cam1 ,y cam1 ,z cam1 ]

[0073] x2=[yaw cam2 ,pitch cam2 ,roll cam2 ,x cam2 ,y cam2 ,z cam2 ]

[0074] x3=[yawcam3 ,pitch cam3 ,roll cam3 ,x cam3 ,y cam3 ,z cam3 ]

[0075] Among them, cam0 represents the camera located in front of the target vehicle, cam1 represents the camera located on the right side of the target vehicle, cam2 represents the camera located behind the target vehicle, and cam3 represents the camera located on the left side of the target vehicle. x0 represents the camera extrinsic parameter of cam0, x1 represents the camera extrinsic parameter of cam1, x2 represents the camera extrinsic parameter of cam2, and x3 represents the camera extrinsic parameter of cam3. i (i=0、1、2、3), yaw cami Represents cam i Yaw angle, pitch cami Represents cam i Pitch angle, roll cami Representative cam i Roll angle; x cami 、y cami 、z cami Represents cam i The coordinates of the target vehicle in the vehicle coordinate system, cam i Installation location on the target vehicle. It should be noted that the vehicle coordinate system of the target vehicle can be constructed according to actual needs and is not limited here. For example, the vehicle coordinate system can be a coordinate system constructed with the intersection of the horizontal axis and the vertical axis of the target vehicle as the origin.

[0076] During the specific implementation process, the objective function can be constructed through the following steps: based on multiple target straight lines parallel to each other in each bird's-eye view, construct a first sub-objective function corresponding to each bird's-eye view, and obtain a total of N first sub-objective functions; based on each group of M groups of adjacent bird's-eye views with common areas, based on multiple target straight lines overlapping with each other in the group of adjacent bird's-eye views, construct a second sub-objective function corresponding to the group of adjacent bird's-eye views, and obtain a total of M second sub-objective functions, where M is a positive integer; construct the objective function based on the N first sub-objective functions and the M second sub-objective functions.

[0077] Specifically, in the process of constructing the objective function, on the one hand, it is necessary to consider the parallel relationship of each straight line in each bird's-eye view, and on the other hand, it is necessary to consider the coincidence relationship of straight lines in the common area of ​​adjacent views. Therefore, for each bird's-eye view, based on the parallel relationship of multiple target straight lines in each bird's-eye view, a respective first sub-objective function is constructed. For adjacent views with common areas, since there are coincident straight lines in adjacent bird's-eye views, a corresponding second sub-objective function can be constructed for each common area. It should be noted that the number M of adjacent bird's-eye views with common areas can be determined according to actual conditions, and the above method can be used. Figure 5 In the common area shown in , the value of M is 4.

[0078] The construction of the first sub-objective function and the second sub-objective function are explained below.

[0079] 1. Construct the first sub-objective function corresponding to each bird's-eye view.

[0080] In the specific implementation process, the construction process of the first sub-objective function can be: for each of the bird's-eye view images, the multiple target straight lines in the bird's-eye view image are divided into K groups of parallel straight lines, wherein each group of parallel straight lines contains a first straight line and a second straight line parallel to each other, and K is a positive integer; for the first straight line and the second straight line in each group of parallel straight lines, the distance from any two points on the first straight line in the group of parallel straight lines to the second straight line, as well as the theoretical distance between the first straight line and the second straight line in the group of parallel straight lines are determined; for each group of adjacent bird's-eye views, the first sub-objective function of the bird's-eye view is constructed based on the distance from any two points on the first straight line in each group of parallel straight lines in the bird's-eye view to the second straight line, as well as the theoretical distance between the first straight line and the second straight line in each group of parallel straight lines.

[0081] For the sake of illustration, take the bird's-eye view as an example. Figure 5 , the target straight lines 0, 1, 2, and 3 contained in the forward bird's-eye view are parallel to each other, and the target straight lines 4, 5, 6, and 7 are parallel to each other. Further, the multiple target straight lines are divided into K groups of parallel straight lines. K can be selected according to actual needs and is not limited here. In the embodiment of this specification, the eight target straight lines 0 to 7 in the forward bird's-eye view can be divided into 6 groups of parallel straight lines (that is, K is 6), each group includes two straight lines, and the 6 groups of target straight lines can be: 0 and 1, 1 and 2, 2 and 3, 4 and 5, 5 and 6, and 6 and 7. For each group of parallel straight lines, the first straight line of the group can be any one of the two target straight lines in the group.

[0082] For each set of parallel lines, determine the theoretical distance between two target lines in each set. Specifically, determine the theoretical distance d0 between target lines 0 and 1, the theoretical distance d1 between target lines 1 and 2, the theoretical distance d2 between target lines 2 and 3, the theoretical distance d4 between target lines 4 and 5, the theoretical distance d5 between target lines 5 and 6, and the theoretical distance d6 between target lines 6 and 7. It should be noted that the theoretical distances between target lines can be obtained through actual measurements.

[0083] For the first and second lines in each set of parallel lines, the distances from any two points on the first line to the second line are the same, which is the theoretical distance between the first and second lines. Taking target lines 0 and 1 as an example, the distances from any two points on target line 0 to target line 1 are all d0. The conversion of other sets of parallel lines can be deduced in the same way, and the first sub-objective function f corresponding to the forward bird's-eye view can be obtained. cam0 (x0), f cam0 The specific calculation method of (x0) is as follows:

[0084]

[0085] in, Indicates the distance from point P0 on the line li to the line li+1 in the forward bird's-eye view. Indicates the distance from point P1 on line li to line li+1 in the forward bird's-eye view.

[0086] Similarly, for the right-view bird's-eye view, if Figure 5 As shown, target lines 0, 1, 2, and 3 are parallel to each other. These four target lines can be divided into three groups of parallel lines: 0 and 1, 1 and 2, and 2 and 3. For each group of parallel lines, the theoretical distance between two target lines in each group is determined. Among them, the theoretical distance between target lines 0 and 1 is d4, the theoretical distance between target lines 1 and 2 is d5, and the theoretical distance between target lines 2 and 3 is d6. Furthermore, the first sub-objective function f corresponding to the right-view bird's-eye view is constructed. cam1 (x1), its calculation formula is similar to f cam0 (x0), I will not go into details here.

[0087] For rear-view bird's-eye view, such as Figure 5As shown, target lines 0, 1, 2, and 3 are parallel to each other, and target lines 4, 5, 6, and 7 are parallel to each other. The eight target lines 0 to 7 can be divided into 6 groups of parallel lines, each group includes two lines, and the 6 groups of target lines can be: 0 and 1, 1 and 2, 2 and 3, 4 and 5, 5 and 6, and 6 and 7. For each group of parallel lines, determine the theoretical distance between the two target lines in each group of parallel lines. Among them, the theoretical distance between target lines 0 and 1 is d0, the theoretical distance between target lines 1 and 2 is d1, the theoretical distance between target lines 2 and 3 is d2, the theoretical distance between target lines 4 and 5 is d4, the theoretical distance between target lines 5 and 6 is d5, and the theoretical distance between target lines 6 and 7 is d6. Further, construct the first sub-objective function f corresponding to the rearview bird's-eye view. cam2 (x2), which is calculated similarly to f cam0 (x0), I will not go into details here.

[0088] For the left bird's-eye view, if Figure 5 As shown, target lines 0, 1, 2, and 3 are parallel to each other. These four target lines can be divided into three groups of parallel lines: 0 and 1, 1 and 2, and 2 and 3. For each group of parallel lines, the theoretical distance between two target lines in each group is determined. The theoretical distance between target lines 0 and 1 is d0, the theoretical distance between target lines 1 and 2 is d1, and the theoretical distance between target lines 2 and 3 is d2. Furthermore, the first sub-objective function f corresponding to the right-view bird's-eye view is constructed. cam3 (x3), its calculation formula is similar to f cam0 (x0), I will not go into details here.

[0089] 2. Construct the second sub-objective function corresponding to the common area of ​​adjacent views.

[0090] In the specific implementation process, each group of adjacent bird's-eye views includes two bird's-eye views, namely the first bird's-eye view and the second bird's-eye view. The construction process of the second sub-objective function can be: for each group of adjacent bird's-eye views, S groups of coincident straight lines are determined from the group of adjacent bird's-eye views, wherein each group of coincident straight lines includes a third straight line and a fourth straight line that coincide with each other, S is a positive integer, the third straight line is the target straight line in the first bird's-eye view in the group of adjacent bird's-eye views, and the fourth straight line is the target straight line in the second bird's-eye view in the group of adjacent bird's-eye views; for each group of coincident straight lines, the distance from any two points on the third straight line in the group of coincident straight lines to the fourth straight line is calculated; for each group of adjacent bird's-eye views, based on the distance from any two points on the third straight line in each group of coincident straight lines in the adjacent bird's-eye views to the fourth straight line, the second sub-objective function of the group of adjacent bird's-eye views is constructed.

[0091] It should be noted that for each group of adjacent bird's-eye views, the first bird's-eye view can be any one of the two bird's-eye views included in the group. For ease of explanation, taking the common area of ​​the front view bird's-eye view and the right view bird's-eye view as an example, if the first bird's-eye view can be the front view bird's-eye view, then the second bird's-eye view can be the right view bird's-eye view; if the first bird's-eye view is the right view bird's-eye view, then the second bird's-eye view can be the front view bird's-eye view.

[0092] For the common areas of the front bird's-eye view and the right bird's-eye view, such as Figure 5 As shown, the target straight lines 0, 1, 2, 3, 4, and 5 in the right bird's-eye view overlap with the target straight lines 4, 5, 6, 7, 10, and 11 in the front bird's-eye view, respectively. In the specific implementation process, S groups of coincident straight lines can be selected according to actual needs. In the embodiment of this specification, all corresponding coincident target straight lines in the front bird's-eye view and the right bird's-eye view can be used as S groups of coincident straight lines. Then, the group of adjacent bird's-eye views formed by the front bird's-eye view and the right bird's-eye view contains 6 groups of coincident straight lines, that is, S is 6. Of course, the selection of coincident straight lines can be set according to actual needs and is not limited here. For each group of coincident straight lines, the third straight line can be any one of the two target straight lines in the group.

[0093] For each set of coincident lines, the theoretical distance between the two target lines in the set should be 0. For example, the target line 0 in the right bird's-eye view overlaps with the target line 4 in the front bird's-eye view. This coincidence relationship can be converted to the distance between two different points on the target line 0 and the target line 4 being 0. The conversion of other sets of coincident lines can be deduced in the same way, and the second sub-objective function f corresponding to the common area of ​​the front bird's-eye view and the right bird's-eye view can be obtained. cam1,cam0 (x1,x0), its calculation formula is similar to f cam0 (x0), I will not go into details here.

[0094] Similarly, for the common areas of the right-view bird's-eye view and the rear-view bird's-eye view, such as Figure 5 As shown, the target lines 0, 1, 2, 3, 6, and 7 in the right-view bird's-eye view coincide with the lines 4, 5, 6, 7, 10, and 11 in the rear-view bird's-eye view. The second sub-objective function corresponding to the common area of ​​the right-view bird's-eye view and the rear-view bird's-eye view is f cam1,cam2 (x1,x2), its calculation formula is similar to f cam0 (x0), I will not go into details here.

[0095] For the common areas of the front bird's eye view and the left bird's eye view, such as Figure 5 As shown, the target lines 0, 1, 2, 3, 4, and 5 in the left bird's-eye view coincide with 0, 1, 2, 3, 8, and 9 in the front bird's-eye view. The second sub-objective function corresponding to the common area of ​​the front bird's-eye view and the left bird's-eye view is f cam3,cam0(x3,x0), the calculation formula is similar to f cam0 (x0), which is calculated similarly to f cam0 (x0), I will not go into details here.

[0096] For the common areas of the left-view bird's-eye view and the rear-view bird's-eye view, such as Figure 5 As shown, the target lines 0, 1, 2, 3, 6, and 7 in the left-view bird's-eye view coincide with the lines 0, 1, 2, 3, 8, and 9 in the rear-view bird's-eye view. The second sub-objective function corresponding to the common area of ​​the left-view bird's-eye view and the rear-view bird's-eye view is f cam3,cam2 (x3,x2), the calculation formula is similar to f cam0 (x0), I will not go into details here.

[0097] Furthermore, after obtaining the above N first sub-objective functions and M second sub-objective functions, the joint optimization objective function f(x0, x1, x2, x3) is obtained, which is specifically in the following form:

[0098] f(x0,x1,x2,x3)=f cam0 (x0)+f cam1 (x1)+f cam2 (x2)+f cam3 (x3)+f cam1,cam0 (x1,x0)+f cam1,cam2 (x1,x2)+f cam3,cam0 (x3,x0)+f cam3,cam2 (x3,x2)

[0099] In step S103, after obtaining the objective function, the current external parameters of the camera are optimized based on the objective function. The optimization method can be set according to actual needs and is not limited here. For example, a nonlinear optimization algorithm can be used for optimization.

[0100] In the embodiments of this specification, the Levenberg-Marquardt optimization algorithm can be used to solve the camera extrinsic parameters. This algorithm combines the Gauss-Newton method with the steepest descent method: when the damping factor is small, it is closer to the Gauss-Newton method; when the damping factor is large, it is closer to the steepest descent method. This algorithm can, to a certain extent, avoid non-singularity and ill-conditioning issues in the coefficient matrix of the linear system, providing a more stable and accurate increment Δx.

[0101] In its implementation, the Levenberg-Marquardt algorithm adds a trust region to the increment Δx. The range of the trust region is determined by the difference between the approximation model and the actual function. If the difference is small, the approximation is good, and the range of the approximation is expanded; if the difference is large, the range of the approximation is narrowed. The quality of the approximation model can be characterized by ρ, where the numerator is the value of the decrease in the actual function and the denominator is the value of the decrease in the approximation model. If ρ is close to 1, the approximation is good. If ρ is too small, the actual decrease is much smaller than the decrease in the approximation model, and the approximation is considered poor, requiring a narrowing of the approximation range. If ρ is large, the actual decrease is greater than the approximation model, and the approximation range needs to be expanded. The specific steps of the Levenberg-Marquardt algorithm can be referred to existing methods and will not be detailed here.

[0102] Generally speaking, when optimizing the current extrinsic parameters of a camera, it is first necessary to generate a corresponding bird's-eye view based on the current extrinsic parameters, and then perform line detection on the bird's-eye view to identify the target straight line in each bird's-eye view, thereby obtaining the objective function. After a round of optimization of the current extrinsic parameters using an optimization algorithm, new extrinsic parameters will be obtained. A new bird's-eye view can be generated based on the new extrinsic parameters, and then line detection can be performed on the new bird's-eye view to obtain a new objective function. However, if the current extrinsic parameters are iteratively updated in this way, each time a new extrinsic parameter is obtained, the bird's-eye view must be recalculated and line detection must be performed, which will result in the consumption of larger computing resources and a longer time. At the same time, if the line detection fails, the optimization process will fail.

[0103] Based on this, when optimizing camera extrinsics, the embodiments of this specification can be implemented in the following manner: each camera is used as a target camera, and the following iterative steps are performed: the current extrinsics of the target camera are optimized by a preset optimization algorithm to obtain optimized target extrinsics; based on the current extrinsics of the target camera and the target extrinsics, each target straight line in the bird's-eye view corresponding to the target camera is transformed into a target bird's-eye view to obtain the linear equation of each target straight line in the target bird's-eye view, wherein the target bird's-eye view corresponds to the target extrinsics; based on the linear equations of the target straight lines in the N target bird's-eye views corresponding to the N cameras, the function value of the target function is recalculated. If the function value of the recalculated target function is less than a threshold, the iteration is stopped, and the target extrinsics after this round of iteration is used as the final extrinsics of the corresponding camera; if the function value of the recalculated target function is greater than or equal to the threshold, the target extrinsics after this round of iteration is used as the current extrinsics in the next round of iteration, and the iterative steps are performed again until the function value of the target function is less than the threshold.

[0104] Specifically, the preset optimization algorithm can be set based on actual needs. In the embodiments of this specification, the preset optimization algorithm can be the Levenberg-Marquardt algorithm. For ease of illustration, let's take each camera's initial theoretical extrinsic parameters as an example. The initial theoretical extrinsic parameters can be the theoretical camera installation position provided by the user. Then, for each camera's initial theoretical extrinsic parameters, a corresponding bird's-eye view can be generated, and the objective function can be obtained by performing line detection on the bird's-eye view.

[0105] Furthermore, the initial theoretical extrinsic parameters are optimized for multiple rounds of iterations. Taking the first round of iterations as an example, the initial theoretical extrinsic parameters of each camera are optimized for the first round of extrinsic parameters. Specifically, taking one of the cameras as an example, the camera is used as the target camera, and the initial theoretical extrinsic parameters of the target camera are optimized by a preset optimization algorithm to obtain the target extrinsic parameters after the first round of optimization. Then, based on the bird's-eye view generated by the initial theoretical extrinsic parameters, the linear equation of each target straight line in the bird's-eye view is transformed into a new bird's-eye view (i.e., the target bird's-eye view corresponding to the target extrinsic parameter) to obtain the linear equation of the target straight line in the new bird's-eye view. Based on the linear equations in the new bird's-eye view corresponding to all cameras, the function value of the objective function corresponding to the target extrinsic parameter can be calculated.

[0106] It can be seen that in this way, there is no need to regenerate the bird's-eye view or re-detect the straight line. Instead, the target straight line in the original bird's-eye view is directly changed to the new bird's-eye view corresponding to the target external parameter through coordinate changes, and the straight line equation of the target straight line in the new bird's-eye view is obtained. In this way, the function value of the objective function can be calculated based on the straight line equation in the new bird's-eye view.

[0107] In the specific implementation process, the process of transforming the target straight line from the original bird's-eye view to the new bird's-eye view can be achieved in the following way: for each target straight line in the bird's-eye view corresponding to the target camera, the coordinates of two target points on the target straight line are determined, and the two target points are different; based on the current extrinsic parameters and the target extrinsic parameters, the coordinates of the two target points are mapped to the target bird's-eye view to obtain the mapping coordinates of the two target points in the target bird's-eye view; and the equation of the straight line formed by the mapping coordinates of the two target points is determined as the equation of the straight line transformed from the target straight line to the target bird's-eye view.

[0108] For example, the above Figure 5Take the forward bird's-eye view map in as an example. The forward bird's-eye view map includes 12 target lines 0 to 11. In order to obtain the straight line equations of these 12 target lines in the new bird's-eye view map, for each target line, two target points can be randomly selected on the target line. Taking target line 0 as an example, two different target points are selected from target line 0, and the coordinates of these two target points are transformed into the new bird's-eye view map through the current extrinsic parameter and the target extrinsic parameter, and the mapping coordinates of the two target points on target line 0 in the new bird's-eye view are obtained. Based on the mapping coordinates of these two target points, the corresponding straight line equation can be determined.

[0109] Next, the transformation process of one of the target points is described.

[0110] Taking the current extrinsic parameters as the initial theoretical extrinsic parameters as an example, after one round of optimization of the initial theoretical extrinsic parameters, we obtain the target extrinsic parameters after one round of optimization. Assume that there is a point P1 on a target line in the bird's-eye view image corresponding to the initial theoretical extrinsic parameters, and the corresponding point of P1 in the new bird's-eye view image (i.e., the target bird's-eye view image corresponding to the target extrinsic parameters) is P2. Then, P1 and P2 both correspond to the same point P in the original image captured by the camera.

[0111] Specifically, first transform P1 from the pixel coordinate system to the world coordinate system; then according to the initial theoretical external parameter x initial , get the transformation matrix from the world coordinate system to the initial camera coordinate system, and convert P 1,w (World coordinates corresponding to P1) are transformed from the world coordinate system to the initial camera coordinate system to obtain P 1,c1 ; Calculate the initial camera coordinate system origin and P 1,c1 The determined straight line L c1 . According to the target parameter x optimal , get the transformation matrix from the new camera coordinate system to the world coordinate system, and transform the straight line L c2 (L c2 With L c1 The equation of the straight line is the same as that of the new camera coordinate system) and the world coordinate system is transformed to obtain L w , then P 2,w (World coordinates corresponding to P2) is L w With Z w =0 plane intersection. Finally, P 2,w Transform from the world coordinate system to the pixel coordinate system to obtain the pixel coordinates of P2 in the new bird's-eye view image.

[0112] By using the same method, the target points selected on multiple target lines in the bird's-eye view corresponding to the initial theoretical external parameters are transformed into the new bird's-eye view, thereby obtaining the equation of the target line in the new bird's-eye view.

[0113] In the embodiments of this specification, after each round of iterative optimization, the objective function's value is recalculated based on the target line equation in the new bird's-eye view. If the function value is less than a threshold, the target extrinsic parameters after this round of iteration are used as the final extrinsic parameters for the corresponding camera. The threshold can be set based on actual needs and is not limited here.

[0114] If the function value of the recalculated objective function is greater than or equal to the threshold after each round of iterative optimization, the next round of iteration is entered, and the equation of the straight line in the new bird's-eye view is determined again based on the initial theoretical extrinsic parameters (or the extrinsic parameters of the previous round) and the target extrinsic parameters after this round of iteration, and the function value of the objective function is calculated again until the function value is less than the threshold, and the iterative process is stopped.

[0115] In addition, in the embodiment of this specification, after the number of iterations reaches a preset number, the optimized target extrinsic parameter finally obtained may be used as the final extrinsic parameter of the corresponding camera.

[0116] In the embodiment of this specification, after the final extrinsic parameters of each camera are determined, a corresponding bird's-eye view can be obtained based on the final extrinsic parameters of each camera, and then the bird's-eye view images can be spliced ​​to obtain a surround bird's-eye view image.

[0117] In summary, the solutions provided in the embodiments of this specification have at least the following technical effects:

[0118] First, the embodiments of this specification use a linear calibration field for camera extrinsic calibration. This is more compact and occupies less space than conventional checkerboard or U-shaped fields, saving space in the vehicle factory. Furthermore, the solutions described in this specification are applicable to linear calibration fields of varying sizes and for surround view calibration of different vehicle models.

[0119] Secondly, the objective function constructed in the embodiments of this specification not only takes into account the misalignment errors of each perspective, but also incorporates the misalignment errors of the common areas of adjacent cameras into the objective function to be optimized, and jointly optimizes the external parameters of all cameras, so that the overall misalignment error of the final stitched image is minimized.

[0120] Again, in the embodiments of this specification, after obtaining the external parameters after each round of iteration, there is no need to recalculate the bird's-eye view and call line detection. The lines detected in the original bird's-eye view can be directly transformed into the new bird's-eye view, which not only reduces the time consumption of the algorithm, but also improves the robustness of the algorithm.

[0121] A camera extrinsic parameter calibration method is provided in accordance with the above embodiment. This application also provides a camera extrinsic parameter calibration device. Figure 6 , the device comprises:

[0122] A bird's-eye view determination module 601 is configured to determine N bird's-eye views corresponding to N cameras installed on a target vehicle based on the intrinsic parameters of each camera and the current extrinsic parameters, wherein the target vehicle is located in a linear calibration site, and N is a positive integer;

[0123] An objective function construction module 602 is configured to identify multiple target lines in each bird's-eye view and construct an objective function using the extrinsic parameters of the N cameras as variables based on the multiple parallel target lines in each bird's-eye view and the multiple overlapping target lines in adjacent bird's-eye views that have a common area.

[0124] The optimization module 603 is configured to optimize the current extrinsic parameters of each camera based on the objective function, and use the optimized extrinsic parameters of each camera as the final extrinsic parameters of the corresponding camera.

[0125] Optionally, the bird's-eye view determination module 601 is configured to:

[0126] For each of the N cameras, based on a current extrinsic parameter of the camera, world coordinates are converted into camera coordinates; based on an intrinsic parameter of the camera, the camera coordinates are converted into pixel coordinates; and based on the pixel coordinates, a bird's-eye view corresponding to the camera is generated.

[0127] Optionally, the objective function construction module 602 is used to:

[0128] Based on multiple mutually parallel target straight lines in each bird's-eye view, construct a first sub-objective function corresponding to each bird's-eye view, and obtain a total of N first sub-objective functions;

[0129] Based on each group of M groups of adjacent bird's-eye views with a common area, based on multiple overlapping target straight lines in the group of adjacent bird's-eye views, a second sub-objective function corresponding to the group of adjacent bird's-eye views is constructed, and a total of M second sub-objective functions are obtained, where M is a positive integer;

[0130] The objective function is constructed based on the N first sub-objective functions and the M second sub-objective functions.

[0131] Optionally, the objective function construction module 602 is used to:

[0132] For each of the bird's-eye view images, the plurality of target straight lines in the bird's-eye view image are divided into K groups of parallel straight lines, wherein each group of parallel straight lines includes a first straight line and a second straight line that are parallel to each other, and K is a positive integer;

[0133] For each set of parallel straight lines, determining the distances between any two points on the first straight line and the second straight line, and the theoretical distance between the first straight line and the second straight line;

[0134] For each group of adjacent bird's-eye view images, the first sub-objective function of the bird's-eye view image is constructed based on the distances from any two points on the first straight line to the second straight line in each group of parallel straight lines in the bird's-eye view image, and the theoretical distance between the first straight line and the second straight line in each group of parallel straight lines.

[0135] Optionally, each group of adjacent bird's-eye views includes a first bird's-eye view and a second bird's-eye view. Optionally, the objective function construction module 602 is configured to:

[0136] For each set of adjacent bird's-eye views, S sets of coincident straight lines are determined from the set of adjacent bird's-eye views, where each set of coincident straight lines includes a third straight line and a fourth straight line that coincide with each other, S is a positive integer, the third straight line is the target straight line in the first bird's-eye view of the set of adjacent bird's-eye views, and the fourth straight line is the target straight line in the second bird's-eye view of the set of adjacent bird's-eye views;

[0137] For each set of coincident straight lines, calculate the distance between any two points on the third straight line and the fourth straight line in the set of coincident straight lines;

[0138] For each group of adjacent bird's-eye views, a second sub-objective function of the group of adjacent bird's-eye views is constructed based on the distances from any two points on the third straight line to the fourth straight line in each group of coincident straight lines in the adjacent bird's-eye views.

[0139] Optionally, the optimization module 603 is configured to:

[0140] Each of the cameras is used as a target camera, and the following iterative steps are performed: optimizing the current extrinsic parameters of the target camera using a preset optimization algorithm to obtain optimized target extrinsic parameters; transforming each target straight line in the bird's-eye view corresponding to the target camera into a target bird's-eye view based on the current extrinsic parameters of the target camera to obtain a straight line equation for each target straight line in the target bird's-eye view, wherein the target bird's-eye view corresponds to the target extrinsic parameters; and recalculating the function value of the objective function based on the straight line equations of the target straight lines in the N target bird's-eye views corresponding to the N cameras;

[0141] If the function value of the recalculated objective function is less than the threshold, the iteration is stopped, and the target extrinsic parameter after this round of iteration is used as the final extrinsic parameter of the corresponding camera;

[0142] If the function value of the recalculated objective function is greater than or equal to the threshold, the target external parameter after this round of iteration is used as the current external parameter in the next round of iteration, and the iterative step is performed again until the function value of the objective function is less than the threshold.

[0143] Optionally, the optimization module 603 is configured to:

[0144] For each target straight line in the bird's-eye view corresponding to the target camera, determine the coordinates of two target points on the target straight line, the two target points being different;

[0145] Based on the current extrinsic parameter and the target extrinsic parameter, mapping the coordinates of the two target points into the target bird's-eye view to obtain the mapping coordinates of the two target points in the target bird's-eye view;

[0146] The equation of a straight line formed by the mapping coordinates of the two target points is determined as the equation of a straight line transformed from the target straight line to the target bird's-eye view.

[0147] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0148] Figure 7 FIG1 is a block diagram of an electronic device 800 for use in a camera extrinsic calibration method according to an exemplary embodiment. For example, the electronic device 800 may be an in-vehicle terminal, a server, a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0149] Reference Figure 7 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / display (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0150] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0151] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0152] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0153] The multimedia component 808 includes a screen that provides a presentation interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0154] The audio component 810 is configured to present and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for presenting audio signals.

[0155] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0156] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor assembly 814 can also detect changes in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and temperature changes of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0157] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0158] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0159] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0160] In addition, it should be noted that: the embodiment of the present application also provides a computer program product or computer program, which may include computer instructions, which may be stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor may execute the computer instructions, so that the computer device performs the above Figure 1 The description of the image color adjustment method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated here. For technical details not disclosed in the computer program product or computer program embodiments involved in this application, please refer to the description of the method embodiments of this application.

[0161] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

[0162] It should be understood that the present invention is not limited to the exact construction described above and shown in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0163] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A camera extrinsic calibration method, characterized in that: include: Determine N bird's-eye views corresponding to each of N cameras installed on a target vehicle based on the intrinsic parameters of each camera and the current extrinsic parameters, wherein the target vehicle is located in a linear calibration site, and N is a positive integer; Identify multiple target lines in each bird's-eye view, and construct an objective function with the extrinsic parameters of the N cameras as variables based on the multiple target lines that are parallel to each other in each bird's-eye view and the multiple target lines that overlap with each other in adjacent bird's-eye views with a common area, including: constructing a first sub-objective function corresponding to each bird's-eye view based on the multiple target lines that are parallel to each other in each bird's-eye view, obtaining a total of N first sub-objective functions; constructing a second sub-objective function corresponding to each group of M groups of adjacent bird's-eye views with a common area based on the multiple target lines that overlap with each other in the group of adjacent bird's-eye views, obtaining a total of M second sub-objective functions, where M is a positive integer; and constructing the objective function based on the N first sub-objective functions and the M second sub-objective functions; Based on the objective function, the current extrinsic parameters of each camera are optimized, and the optimized extrinsic parameters of each camera are used as the final extrinsic parameters of the corresponding camera; Wherein, the constructing of the first sub-objective function corresponding to each bird's-eye view includes: for each bird's-eye view, dividing the multiple target straight lines in the bird's-eye view into K groups of parallel straight lines, wherein each group of parallel straight lines includes a first straight line and a second straight line parallel to each other, and K is a positive integer; for the first straight line and the second straight line in each group of parallel straight lines, determining the distance between any two points on the first straight line in the group of parallel straight lines to the second straight line, and the theoretical distance between the first straight line and the second straight line in the group of parallel straight lines; for each group of adjacent bird's-eye views, constructing the first sub-objective function of the bird's-eye view based on the distance between any two points on the first straight line in each group of parallel straight lines in the bird's-eye view to the second straight line, and the theoretical distance between the first straight line and the second straight line in each group of parallel straight lines; Each group of adjacent bird's-eye views includes a first bird's-eye view and a second bird's-eye view. For each group of adjacent bird's-eye views in the M groups of adjacent bird's-eye views with a common area, based on a plurality of target straight lines that overlap with each other in the group of adjacent bird's-eye views, a second sub-objective function corresponding to the group of adjacent bird's-eye views is constructed, including: for each group of adjacent bird's-eye views, S groups of overlapping straight lines are determined from the group of adjacent bird's-eye views, wherein each group of overlapping straight lines includes a third straight line and a fourth straight line that overlap with each other, S is a positive integer, the third straight line is the target straight line in the first bird's-eye view in the group of adjacent bird's-eye views, and the fourth straight line is the target straight line in the second bird's-eye view in the group of adjacent bird's-eye views; for each group of overlapping straight lines, the distance from any two points on the third straight line in the group of overlapping straight lines to the fourth straight line is calculated; for each group of adjacent bird's-eye views, the second sub-objective function of the group of adjacent bird's-eye views is constructed based on the distance from any two points on the third straight line in each group of overlapping straight lines in the adjacent bird's-eye views to the fourth straight line.

2. The method according to claim 1, wherein The method of determining N bird's-eye views corresponding to the N cameras on the target vehicle based on the intrinsic parameters of each camera and the current extrinsic parameters includes: For each of the N cameras, based on a current extrinsic parameter of the camera, world coordinates are converted into camera coordinates; based on an intrinsic parameter of the camera, the camera coordinates are converted into pixel coordinates; and based on the pixel coordinates, a bird's-eye view corresponding to the camera is generated.

3. The method according to claim 1, wherein The optimizing the current extrinsic parameters of each camera based on the objective function and using the optimized extrinsic parameters of each camera as the final extrinsic parameters of the corresponding camera includes: Each of the cameras is used as a target camera, and the following iterative steps are performed: optimizing the current extrinsic parameters of the target camera using a preset optimization algorithm to obtain optimized target extrinsic parameters; transforming each target straight line in the bird's-eye view corresponding to the target camera into a target bird's-eye view based on the current extrinsic parameters of the target camera to obtain a straight line equation for each target straight line in the target bird's-eye view, wherein the target bird's-eye view corresponds to the target extrinsic parameters; and recalculating the function value of the objective function based on the straight line equations of the target straight lines in the N target bird's-eye views corresponding to the N cameras; If the function value of the recalculated objective function is less than the threshold, the iteration is stopped, and the target extrinsic parameter after this round of iteration is used as the final extrinsic parameter of the corresponding camera; If the function value of the recalculated objective function is greater than or equal to the threshold, the target external parameter after this round of iteration is used as the current external parameter in the next round of iteration, and the iterative step is performed again until the function value of the objective function is less than the threshold.

4. The method according to claim 3, wherein The step of transforming each target straight line in the bird's-eye view corresponding to the target camera into the target bird's-eye view based on the current extrinsic parameter of the target camera and the target extrinsic parameter to obtain a straight line equation of each target straight line in the target bird's-eye view includes: For each target straight line in the bird's-eye view corresponding to the target camera, determine the coordinates of two target points on the target straight line, the two target points being different; Based on the current extrinsic parameter and the target extrinsic parameter, mapping the coordinates of the two target points into the target bird's-eye view to obtain the mapping coordinates of the two target points in the target bird's-eye view; The equation of a straight line formed by the mapping coordinates of the two target points is determined as the equation of a straight line transformed from the target straight line to the target bird's-eye view.

5. A camera extrinsic parameter calibration device, characterized in that: include: a bird's-eye view determination module, configured to determine N bird's-eye view images corresponding to N cameras disposed on a target vehicle based on an intrinsic parameter of each camera and a current extrinsic parameter, wherein the target vehicle is located in a linear calibration site, and N is a positive integer; an objective function construction module for identifying multiple target lines in each bird's-eye view and constructing an objective function with extrinsic parameters of the N cameras as variables based on the multiple parallel target lines in each bird's-eye view and the multiple overlapping target lines in adjacent bird's-eye views that have a common area; An optimization module, configured to optimize the current extrinsic parameters of each camera based on the objective function, and use the optimized extrinsic parameters of each camera as the final extrinsic parameters of the corresponding camera; The objective function construction module is configured to construct a first sub-objective function corresponding to each bird's-eye view based on multiple mutually parallel target straight lines in each bird's-eye view, obtaining a total of N first sub-objective functions; based on each group of M groups of adjacent bird's-eye views with a common area, construct a second sub-objective function corresponding to the group of adjacent bird's-eye views based on multiple mutually overlapping target straight lines in the group of adjacent bird's-eye views, obtaining a total of M second sub-objective functions, where M is a positive integer; and construct the objective function based on the N first sub-objective functions and the M second sub-objective functions; The objective function construction module is further configured to, for each of the bird's-eye view images, divide the plurality of target straight lines in the bird's-eye view image into K groups of parallel straight lines, wherein each group of parallel straight lines includes a first straight line and a second straight line parallel to each other, and K is a positive integer; for the first straight line and the second straight line in each group of parallel straight lines, determine the distance between any two points on the first straight line in the group of parallel straight lines and the second straight line, as well as the theoretical distance between the first straight line and the second straight line in the group of parallel straight lines; for each group of adjacent bird's-eye view images, construct a first sub-objective function for the bird's-eye view based on the distance between any two points on the first straight line in each group of parallel straight lines in the bird's-eye view and the second straight line, as well as the theoretical distance between the first straight line and the second straight line in each group of parallel straight lines; Each group of adjacent bird's-eye views includes a first bird's-eye view and a second bird's-eye view. The objective function construction module is further used to: for each group of adjacent bird's-eye views, determine S groups of coincident straight lines from the group of adjacent bird's-eye views, wherein each group of coincident straight lines includes a third straight line and a fourth straight line that coincide with each other, S is a positive integer, the third straight line is the target straight line in the first bird's-eye view in the group of adjacent bird's-eye views, and the fourth straight line is the target straight line in the second bird's-eye view in the group of adjacent bird's-eye views; for each group of coincident straight lines, calculate the distance from any two points on the third straight line in the group of coincident straight lines to the fourth straight line; for each group of adjacent bird's-eye views, construct a second sub-objective function for the group of adjacent bird's-eye views based on the distance from any two points on the third straight line in each group of coincident straight lines in the adjacent bird's-eye views to the fourth straight line.

6. An electronic device, characterized in that: The invention comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors to execute the operating instructions corresponding to the method according to any one of claims 1 to 4 contained in the one or more programs.

7. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images