Camera external parameter determination method and device, vehicle and computer readable storage medium

Abstract

The invention discloses a method for determining external parameters of a camera, and the method comprises the steps: obtaining an image collected by the camera, determining a plumb line in the image, and determining the external parameters of the camera according to the determined plumb line, the image shot by the camera and predetermined internal parameters of the camera. Therefore, according to the method, the camera external reference is determined based on the plumb line, the camera internal reference and the image shot by the camera, so that the camera external reference can be determined through a relatively reliable marker, namely the plumb line, the reliability of the camera external reference can be ensured to a certain extent, and a reliable basis is provided for subsequent aerial view generation; and the credibility of the aerial view is ensured. Meanwhile, the plumb line can be determined through objects such as a building outline and a telegraph pole in the image, so that the determination efficiency of the external parameters of the camera can be improved or guaranteed to a certain extent.

Description

Technical Field

[0001] The present application relates to the field of image technology, and more specifically, to a method for determining camera extrinsic parameters, a device for determining camera extrinsic parameters, a vehicle, and a computer-readable storage medium. Background Art

[0002] In order to enable users to conveniently observe the vehicle's surroundings while driving, current vehicles usually have the function of generating and displaying a bird's-eye view. The bird's-eye view is usually generated by mapping the images taken by the four cameras on the front, back, left, and right sides of the vehicle on the same plane based on the determined camera intrinsics and extrinsics to obtain the bird's-eye view.

[0003] However, the camera intrinsic parameters depend on the physical properties of the camera itself and are less affected by the outside world. Therefore, the determination of camera extrinsic parameters is extremely important for bird's-eye view generation. Summary of the invention

[0004] The present application provides a method for determining camera extrinsic parameters, an apparatus for determining camera extrinsic parameters, a vehicle, and a computer-readable storage medium.

[0005] The present application embodiment provides a method for determining a bird's-eye view, including:

[0006] Get the image captured by the camera;

[0007] determining a plumb line in the image;

[0008] The camera extrinsic parameters are determined according to the plumb line, the image and predetermined camera intrinsic parameters.

[0009] In the method for determining the bird's-eye view provided in the embodiment of the present application, the vehicle can determine the plumb line in the image after acquiring the image captured by the camera, and determine the camera extrinsic parameters based on the determined plumb line, the image captured by the camera, and the predetermined camera intrinsic parameters, thereby completing the camera calibration.

[0010] In this way, the embodiment of the present application determines the camera extrinsic parameters based on the plumb line, the camera intrinsic parameters and the image captured by the camera, so that the camera extrinsic parameters can be determined by a relatively reliable marker, that is, the plumb line, so that the reliability of the camera extrinsic parameters can be guaranteed to a certain extent, and then, a reliable basis is provided for the subsequent generation of the bird's-eye view, and the credibility of the bird's-eye view is guaranteed. At the same time, because the plumb line can be determined by objects such as building outlines and electric poles in the image, in other words, it is relatively simple to obtain the plumb line, and then, the efficiency of determining the camera extrinsic parameters can be improved or guaranteed to a certain extent.

[0011] In some embodiments, the plumb line includes a plurality of plumb lines; and determining the camera extrinsic parameters according to the plumb line, the image, and a predetermined camera intrinsic parameter includes:

[0012] Determining a reference extrinsic parameter of the camera according to the camera intrinsic parameter and the image;

[0013] Determine a reference bird's-eye view according to the image, the camera intrinsic parameters and the reference extrinsic parameters;

[0014] Determining the projection line of each of the plumb lines in the reference bird's-eye view;

[0015] The camera extrinsic parameters are determined according to the intersection point of each of the projection lines and the projection point of the camera optical center in the reference bird's-eye view.

[0016] In this way, the embodiment of the present application determines the camera extrinsic parameters based on the plumb line in the bird's-eye view and the optical center of the camera, which ensures the stable determination of the camera extrinsic parameters to a certain extent.

[0017] In some embodiments, determining the camera extrinsic parameters according to the intersection of each of the projection lines and the projection point of the camera optical center in the reference bird's-eye view includes:

[0018] Determine a candidate extrinsic parameter according to the intersection point and the projection point;

[0019] When the candidate extrinsic parameter satisfies a preset condition, the candidate extrinsic parameter is determined as the camera extrinsic parameter.

[0020] In this way, the implementation manner of the present application is based on determining the external parameters that meet the preset conditions as the camera external parameters, so that the accuracy or credibility of the camera external parameters finally determined can be guaranteed to a certain extent.

[0021] In some embodiments, determining the reference bird's-eye view according to the image, the camera intrinsic parameter and the reference extrinsic parameter further includes:

[0022] When the candidate external parameter does not meet the preset condition, the candidate external parameter is determined as the new reference external parameter, and the reference bird's-eye view is determined according to the image, the camera intrinsic parameter and the new reference external parameter.

[0023] In this way, the implementation mode of the present application determines the candidate extrinsic parameter as the reference extrinsic parameter based on the fact that the candidate extrinsic parameter does not meet the preset conditions, so that a new bird's-eye view determination, a new projection line and projection point determination, and a new candidate extrinsic parameter determination can be performed based on the newly determined reference extrinsic parameter, thereby realizing the iterative update of the external parameter to a certain extent, and ensuring the reliability and accuracy of the camera extrinsic parameter finally determined.

[0024] In some embodiments, determining the camera extrinsic parameters according to the intersection of each of the projection lines and the projection point of the camera optical center in the reference bird's-eye view includes:

[0025] Determining an objective function according to the image and the reference extrinsic parameter;

[0026] Determining a first constraint according to a first distance between the intersection point and the projection point;

[0027] The camera extrinsic parameters are determined according to the first constraint and the objective function.

[0028] In this way, after the objective function is constructed by the image and reference extrinsic parameters in the implementation mode of the present application, a first constraint is formed based on the first distance between the intersection of the projection lines of multiple plumb lines and the projection point of the camera optical center. Therefore, the objective function can be solved by the first constraint to obtain the corresponding function solution, that is, the camera extrinsic parameters are obtained, which ensures the effective determination of the camera extrinsic parameters to a certain extent.

[0029] In some implementations, determining the first constraint according to the first distance between the intersection point and the projection point includes:

[0030] The first constraint is determined with the goal of the first distance being equal to a preset value.

[0031] In this way, the implementation method of the present application enables the objective function to be solved with the first distance being equal to the preset value as the goal during the solving process, thereby ensuring the credibility of the camera extrinsic parameters to a certain extent.

[0032] In some embodiments, the image includes a plurality of images; and determining the objective function according to the image and the reference extrinsic parameter includes:

[0033] Determine an image feature point in each of the images, and a first coordinate of the image feature point in each of the images;

[0034] Determine, among all the image feature points, image feature points that exist in at least two of the images as target feature points;

[0035] Determine a plurality of second coordinates of the target feature point in the reference bird's-eye view according to the first coordinates of the target feature point in each of the images;

[0036] The objective function is determined according to the reference extrinsic parameter and the plurality of second coordinates of the target feature point.

[0037] In this way, the implementation method of the present application determines the second coordinate of the target feature point in the bird's-eye view based on the first coordinate of the target feature point in the images captured by different cameras, so as to construct a corresponding objective function, so that the objective function can describe the relationship between pixel points in different images reflected by the camera extrinsic parameters to a certain extent. Therefore, the credibility of the camera extrinsic parameters obtained by solving the objective function can be guaranteed to a certain extent.

[0038] In some embodiments, determining the objective function according to the reference extrinsic parameter and the plurality of second coordinates of the target feature point includes:

[0039] Determining a second constraint according to a second distance between any two of the plurality of second coordinates of the target feature point;

[0040] The camera extrinsic parameters are determined according to the first constraint, the second constraint and the objective function.

[0041] In this way, the implementation method of the present application enables the solution of the objective function to be completed based on the first constraint corresponding to the plumb line and the second constraint corresponding to the feature point. In other words, the solution of the camera extrinsic parameters can be solved based on the constraints corresponding to the line features and the constraints corresponding to the point features. Furthermore, the credibility of the camera extrinsic parameters can be guaranteed to a certain extent.

[0042] In certain embodiments of the present application, the method further comprises:

[0043] When a target image is acquired based on the camera, a bird's-eye view is determined according to the camera extrinsic parameters, the camera intrinsic parameters and the target image.

[0044] In this way, the embodiment of the present application enables the camera to determine or convert the target image into a bird's-eye view according to the previously determined camera extrinsics and camera intrinsics when capturing the target image for the bird's-eye view, thereby realizing the generation of the bird's-eye view. In addition, because the extrinsics used in the embodiment of the present application when generating the bird's-eye view are calibrated by the plumb line in the image or scene, that is, the calibration of the camera extrinsics is based on reliable markers, the reliability of the camera extrinsics is guaranteed, and thus, the credibility of the generated bird's-eye view can also be guaranteed to a certain extent.

[0045] The present application provides a device for determining camera extrinsic parameters, including:

[0046] An acquisition module is used to acquire images captured by a camera;

[0047] A plumb line determination module, used to determine a plumb line in the image;

[0048] The extrinsic parameter determination module is used to determine the camera extrinsic parameters according to the plumb line, the image and the predetermined camera intrinsic parameters.

[0049] An embodiment of the present application provides a vehicle, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the above-mentioned method for determining camera extrinsic parameters is implemented.

[0050] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by one or more processors, the above-mentioned method for determining camera extrinsic parameters is implemented.

[0051] The camera extrinsic parameter determination device, vehicle and computer-readable storage medium provided in the embodiments of the present application determine the camera extrinsic parameters based on the plumb line, the camera intrinsic parameters and the image captured by the camera, so that the camera extrinsic parameters can be determined by a relatively reliable marker, that is, the plumb line, thereby ensuring the reliability of the camera extrinsic parameters to a certain extent, and further providing a reliable basis for the subsequent generation of a bird's-eye view, thereby ensuring the credibility of the bird's-eye view. At the same time, because the plumb line can be determined by objects such as building outlines and electric poles in the image, in other words, it is relatively simple to obtain the plumb line, and further, the efficiency of determining the camera extrinsic parameters can be improved or guaranteed to a certain extent.

[0052] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0054] Figure 1 A schematic diagram for generating a bird's eye view;

[0055] Figure 2 A schematic diagram of a flow chart of a method for determining camera extrinsic parameters in certain implementation modes of the present application;

[0056] Figure 3 A schematic diagram of a device for determining camera extrinsic parameters in certain embodiments of the present application;

[0057] Figure 4 A schematic diagram of application scenarios of certain embodiments of the present application;

[0058] Figure 5 A schematic diagram of a flow chart of a method for determining camera extrinsic parameters in certain implementation modes of the present application;

[0059] Figure 6 A schematic diagram of application scenarios of certain embodiments of the present application;

[0060] Figure 7 A schematic diagram of application scenarios of certain embodiments of the present application;

[0061] Figure 8 A schematic diagram of application scenarios of certain embodiments of the present application;

[0062] Fig. 9 A schematic diagram of a flow chart of a method for determining camera extrinsic parameters in certain implementation modes of the present application;

[0063] Fig.10 A schematic diagram of a flow chart of a method for determining camera extrinsic parameters in certain implementation modes of the present application;

[0064] Fig.11 A schematic diagram of a flow chart of a method for determining camera extrinsic parameters in certain implementation modes of the present application;

[0065] Fig.12 Schematic diagram of a flow chart of a method for determining camera extrinsic parameters in certain embodiments of the present application. DETAILED DESCRIPTION

[0066] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.

[0067] In scenarios such as reversing, in order to facilitate users or drivers to observe the surrounding conditions of the vehicle, most vehicles can use the installed heads-up camera or surround-view camera to capture images from all directions outside the vehicle (such as directly behind the vehicle or on the left and right sides of the vehicle) and display the images on the display screen inside the vehicle, allowing users to clearly understand the surrounding conditions of the vehicle, such as whether there are any obstacles.

[0068] Furthermore, in order to allow users to know the surrounding conditions of the vehicle more accurately and intuitively, some vehicles may also be equipped with an Around View Monitor (AVM) system, which uses four fisheye cameras in the AVM system to capture images of the four directions of the vehicle. Figure 1 , Figure 1 Schematic diagram of the generation of a bird's eye view. Figure 1 The four pictures above the middle arrow represent the images of the vehicle in the four directions of front, back, left and right.

[0069] After the images of the four directions of the vehicle are mapped to the same plane, a corresponding top view, also known as a bird's eye view (BEV), can be formed. For an example, please refer to Figure 1 , Figure 1 The picture below the middle arrow represents the bird's-eye view of the vehicle after the images of the front, back, left, and right directions are mapped onto the same plane. As a result, users can intuitively understand the situation around the vehicle based on the bird's-eye view presented on the in-car display.

[0070] Among them, the generation of the bird's-eye view is usually based on the calibration of the intrinsic and extrinsic parameters of each camera, and the images taken by each fisheye camera are mapped to a preset plane. Therefore, the accuracy of the bird's-eye view is directly related to the calibration accuracy of the internal and external parameters of the fisheye camera. At the same time, because the camera's intrinsic parameters (such as the position of the principal point and the focal length) are determined by the physical properties of the fisheye camera itself and are less affected by the outside world, the calibration accuracy of the camera's extrinsic parameters can be highly correlated with the accuracy of the bird's-eye view.

[0071] In the common vehicle camera calibration process, the vehicle is usually moved to a site with multiple markers so that when different fisheye cameras capture the same marker, for example, when both the front fisheye camera and the left fisheye camera capture the same marker, the camera extrinsic parameters of the front fisheye camera and the left fisheye camera are determined based on the position of the marker in the image captured by the front fisheye camera and the position of the marker in the image captured by the left fisheye camera.

[0072] However, the above-mentioned camera extrinsic parameter determination scheme depends on the marker within the camera's shooting range (or field of view). If the marker is close to the camera, the calibrated camera extrinsic parameters can guarantee the image accuracy within a short distance to a certain extent. If the marker is far from the camera, the calibrated camera extrinsic parameters can guarantee the image accuracy within a long distance to a certain extent.

[0073] However, if the marker needs to be farther away from the camera, there are requirements for the scope of the site, that is, the scope of the site where the vehicle is located is large enough to install the corresponding marker at a distance from the camera. Therefore, if the camera external parameters need to ensure the image accuracy at a long distance, the calibration cost (that is, the cost of using and setting up the site used in the calibration process) is high. In addition, the function of the surround fisheye is mainly to shoot the ground, and the pitch angle is low, so it is usually necessary to obtain the markers on the ground to complete the calibration of the fisheye camera.

[0074] For the above problems that may be encountered, please refer to Figure 2 , the embodiment of the present application provides a method for determining a bird's-eye view, including:

[0075] 01: Get the image captured by the camera;

[0076] 02: Determine the plumb line in the image;

[0077] 03: Determine the camera extrinsic parameters based on the plumb line, image and pre-determined camera intrinsic parameters.

[0078] See also Figure 3, the embodiment of the present application also provides a device 200 for generating a bird's-eye view. The method for determining the camera extrinsic parameters of the embodiment of the present application can be implemented by the device 200 for generating a bird's-eye view of the embodiment of the present application. Specifically, the generating device 200 includes an acquisition module 210, a plumb line determination module 220 and an extrinsic parameter determination model 230. Among them, the acquisition module 210 is used to acquire an image captured by a camera; the plumb line determination module 220 is used to determine the plumb line in the image; the extrinsic parameter determination model 230 is used to determine the camera extrinsic parameters according to the plumb line, the image and the predetermined camera intrinsic parameters.

[0079] The embodiment of the present application also provides a vehicle, which includes a memory and a processor. The method for determining the camera extrinsic parameters of the embodiment of the present application can be implemented by the vehicle of the embodiment of the present application. Specifically, a computer program is stored in the memory, and the processor is used to obtain an image captured by the camera; determine a plumb line in the image; and determine the camera extrinsic parameters based on the plumb line, the image, and the predetermined camera intrinsic parameters.

[0080] That is, the vehicle (or the controller in the vehicle) of the embodiment of the present application controls the camera to perform a shooting action during the camera calibration process to obtain the image captured by the camera; when the image captured by the camera is obtained, the plumb line in the image is detected to determine each plumb line in the image. When the plumb line is determined, the camera's extrinsic parameters are solved according to the determined plumb line, the image captured by the camera, and the predetermined (or pre-read) camera intrinsic parameters to obtain the camera extrinsic parameters. Furthermore, if the vehicle needs to construct a bird's-eye view through multiple images captured by the camera, and the image used to construct the bird's-eye view has been captured by the camera, the vehicle combines the camera extrinsic parameters that have been obtained and the camera intrinsic parameters that are known in advance to map the multiple images captured by the camera to the same plane to obtain a bird's-eye view.

[0081] It is understood that the camera in the embodiments of the present application can be understood as a shooting device installed on the vehicle for shooting images outside the vehicle (or pictures outside the vehicle), and the images shot can be used to generate a bird's-eye view. In some embodiments, the camera in the embodiments of the present application can be understood as a camera in an AVM system, that is, a fisheye camera.

[0082] It can also be understood that the plumb line captured by the head-up camera can be understood as a line perpendicular to the horizontal plane and pointing to the center of the earth, while in the image captured by the fisheye camera, the plumb line may be deformed to a certain extent. Figure 4 , Figure 4 This is a schematic diagram of application scenarios of certain implementation methods of this application. It should be noted that: Figure 4 It is based on the images taken by the fisheye camera. Figure 4As shown, the embodiments of the present application can detect line segments in the image such as the first plumb line 301 and the second plumb line 302 .

[0083] Furthermore, it is understood that the method of determining the plumb line can be set according to actual conditions. For example, in some embodiments, the building outline in the image can be detected, and the downward line in the detected building outline can be determined as the plumb line. Alternatively, it can be detected whether there are vertical objects such as electric poles in the image, and then the line segment corresponding to the pole shaft can be used as the plumb line.

[0084] Furthermore, it is understandable that although Figure 4 The first plumb line 301 and the second plumb line 302 shown are difficult to reflect, but the points on the same plumb line in the real scene all fall on the same straight line. Therefore, the implementation method of the present application can map the points on the plumb line in the image captured by the fisheye camera to a preset plane, that is, the BEV plane, and use the points on the plumb line in the BEV plane to construct corresponding constraints, so that the relationship between the points on the plumb line (falling on the same straight line together) can be considered when solving the external parameters.

[0085] Also, it is understandable that when the plumb line is used as a marker in the camera calibration process, the plumb line is easy to find and can be determined in a simple way, so the efficiency of determining the external parameters based on the plumb line can be guaranteed. At the same time, it is also understandable that when the plumb line and the feature points (such as corner points) in the image are used to determine the camera external parameters at the same time, if the number of feature points is insufficient, the plumb line can also be used to supplement the features, or in other words, the line features corresponding to the plumb line can be used to compensate for the insufficient number of point features, thereby ensuring the effective determination of the camera external parameters.

[0086] In addition, it should be understood that the process of determining the camera extrinsic parameters based on the plumb line, the image and the camera intrinsic parameters is a content that can be set according to the actual situation. As in the above example, the corresponding constraints can be constructed based on the characteristic that all points on the plumb line fall on the same straight line. Furthermore, after constructing the corresponding objective function for solving the camera extrinsic parameters using the camera intrinsic parameters and the image captured by the camera, the constraints corresponding to the plumb line can be used to assist in solving the objective function.

[0087] It can be understood that the camera intrinsic parameters of the implementation mode of the present application are predetermined, including focal length, principal point position and distortion parameters.

[0088] In certain embodiments, the camera intrinsic parameters of the embodiments of the present application are determined based on the Kannala-Brandt model.

[0089] In some embodiments, the camera manufacturer will provide the internal parameters of each camera. Then, after the internal parameters provided by the camera manufacturer are stored in the vehicle's memory, the vehicle of the embodiment of the present application can obtain the camera internal parameters by reading the data in the memory.

[0090] In this way, the embodiment of the present application determines the camera extrinsic parameters based on the plumb line, the camera intrinsic parameters and the image captured by the camera, so that the camera extrinsic parameters can be determined by a relatively reliable marker, that is, the plumb line, so that the reliability of the camera extrinsic parameters can be guaranteed to a certain extent, and then, a reliable basis is provided for the subsequent generation of the bird's-eye view, and the credibility of the bird's-eye view is guaranteed. At the same time, because the plumb line can be determined by objects such as building outlines and electric poles in the image, in other words, it is relatively simple to obtain the plumb line, and then, the efficiency of determining the camera extrinsic parameters can be improved or guaranteed to a certain extent.

[0091] In addition, the implementation manner of the present application also determines the camera extrinsic parameters based on the plumb line, so that points actually on the plumb line in the image, as well as points on the extension line of the plumb line that do not exist in the image, can also be used to determine the extrinsic parameters. In this way, when the vehicle is in a small space, the extrinsic parameters can be determined by the points on the plumb line and the points on the extension line of the plumb line. In other words, the site limitations and shooting angle limitations of the fisheye camera can be avoided to a certain extent.

[0092] In certain embodiments of this application, please refer to Figure 5 , the plumb line includes multiple ones. Step 03 in the control method includes:

[0093] 030: Determine the reference extrinsic parameters of the camera based on the camera intrinsic parameters and the image;

[0094] 031: Determine the reference bird's-eye view based on the image, camera intrinsic parameters and reference extrinsic parameters;

[0095] 032: Determine the projection line of each plumb line in the reference bird's-eye view;

[0096] 033: Determine the camera extrinsic parameters based on the intersection of each projection line and the projection point of the camera optical center in the reference bird's-eye view.

[0097] The external parameter determination module of the implementation mode of the present application is also used to determine the reference external parameters of the camera based on the camera internal parameters and the image; determine the reference bird's-eye view based on the image, the camera internal parameters and the reference external parameters; determine the projection line of each plumb line in the reference bird's-eye view; determine the camera external parameters based on the intersection of each projection line and the projection point of the camera optical center in the reference bird's-eye view.

[0098] The processor of the embodiment of the present application is also used to determine the reference extrinsic parameters of the camera based on the camera intrinsic parameters and the image; determine the reference bird's-eye view based on the image, the camera intrinsic parameters and the reference extrinsic parameters; determine the projection line of each plumb line in the reference bird's-eye view; determine the camera extrinsic parameters based on the intersection of each projection line and the projection point of the camera optical center in the reference bird's-eye view.

[0099] It should be noted that in a bird's-eye view (or, BEV plane), multiple plumb lines captured by the same camera can intersect at the vertical projection point of the camera optical center in the bird's-eye view. Figure 6 , Figure 6 Schematic diagram of application scenarios of some embodiments of the present application, that is, the third plumb line 303 and the fourth plumb line 304 intersect at the projection point of the optical center in the reference bird's-eye view (this point is not in Figure 6 ).

[0100] To further illustrate the implementation of this application, please refer to Figure 7 , Figure 7 Schematic diagram of application scenarios of certain embodiments of the present application. z The dotted line represents any point on the vertical line from the optical center to the optical center ground point. The optical center ground point refers to the projection point of the optical center vertically projected onto the BEV plane. P1 represents a point on the plumb line in the BEV plane.

[0101] Furthermore, let P z It can be described by the following formula:

[0102]

[0103] Using s1 to P z Scale to convert P in the camera coordinate system z Mapped to the coordinate system (which can be the world coordinate system) where the BEV plane (or perspective projection to the BEV plane) is located, that is:

[0104] s1P z =K[R|-RC]P z =K(r1c x +r2c y +r3z-(r1c x +r2c y +r3z))

[0105] Among them, K can be understood as the intrinsic parameter matrix or upper triangular matrix, R can be understood as the rotation matrix, and RC can be understood as the position of the origin of the world coordinate system in the camera coordinate system.

[0106] Therefore, P z It can also be expressed as:

[0107]

[0108] Then, the vector r3 consisting of the elements in the third column of R is expressed as follows:

[0109]

[0110] Then P z It can also be expressed as:

[0111]

[0112] Further, to meet the following conditions, namely:

[0113]

[0114] Then it is necessary to satisfy s1=(zc z )v3, at the same time, z in z <c z The above situation also holds true when the direction changes arbitrarily. Where c0 represents a constant.

[0115] Similarly, any point P1 on a plumb line in space can be expressed as:

[0116]

[0117] Similarly, if we scale P1, we get:

[0118] s2P z =K[R|-RC]P1=K[(p 1x -c x )r1+(p 1y -c y )r2+(p 1z -c z )r3)]

[0119] It is understandable that when p 1z As it approaches infinity, (p 1x -c x )r1 and (p 1y -c y )r2 can be ignored. And, similar to the argument of s1 above, it can be known that each point on the vertical line in space can be projected to the same image point, and when projected to the BEV plane, it can intersect with the projection point of the camera optical center on the BEV plane.

[0120] Based on the above demonstration, after obtaining the image taken by the camera, the vehicle of the embodiment of the present application can determine the corresponding external parameters as reference external parameters in combination with the predetermined camera internal parameters.

[0121] It should be understood that the reference extrinsic parameters in the implementation mode of the present application can be understood as camera extrinsic parameters that are not determined by a plumb line, and can also be understood as camera extrinsic parameters whose accuracy has not yet reached the expected standard.

[0122] It should also be understood that the method for determining the reference external parameters is something that can be set according to actual conditions. For example, in some embodiments, the vehicle of the present application embodiment can complete the determination of the reference external parameters based on the SFM (Structure From Motion) algorithm.

[0123] After obtaining the reference extrinsic parameters, the embodiments of the present application can use the reference extrinsic parameters and the camera intrinsic parameters to project or reproject the image captured by the camera to generate a bird's-eye view, or a reference bird's-eye view.

[0124] After the generation of the reference bird's-eye view is completed, the plumb lines can be projected into the reference bird's-eye view according to the determined plumb lines, thereby determining the projection lines corresponding to each plumb line. In some embodiments, the vehicle can discretize the determined plumb lines into multiple points, and project the multiple points corresponding to the plumb lines into the reference bird's-eye view to obtain corresponding projection coordinates, and then construct a straight line equation according to all the projection coordinates, so that the projection lines corresponding to the plumb lines can be characterized by the straight line equations.

[0125] Furthermore, the process of projecting the point of the plumb line onto the reference bird's-eye view, i.e., the BEV plane, may include: constructing a world coordinate system based on the BEV plane, or setting the BEV plane as a plane of z=0 in the world coordinate system, setting a point X of the plumb line oc The coordinates of (x oc ,y oc ), based on the focal length f in the camera intrinsic parameter, we can know that X in the camera coordinate system oc The coordinates can be represented as (x oc ,y oc , f). Combined with the rotation matrix R in the reference extrinsic parameter wc and the translation matrix t wc , X oc Mapped to the world coordinate system, we have:

[0126]

[0127] Among them, X ow Represents X oc The coordinates in the world coordinate system.

[0128] Furthermore, since the coordinates of the camera optical center C in the world coordinate system are known, the ray connecting the camera optical center to the image point can be calculated. The ray is calculated according to the two-point equation, and then:

[0129]

[0130]

[0131] Among them, x og It represents the horizontal coordinate of the point on the phase plane (that is, the point on the plumb line) when projected to the world coordinate system, y og It represents the ordinate of the point on the phase plane when projected to the world coordinate system, x o represents the horizontal coordinate of a point on the phase plane, x o represents the ordinate of a point on the phase plane, x c represents the horizontal coordinate of the light point on the phase plane, y c In order to more clearly explain the implementation of this application, please refer to Figure 8 , Figure 8 This is a schematic diagram of application scenarios of certain embodiments of the present application. That is, the optical center c and the point o on the image plane form a ray, and the ray intersects the BEV plane at point g.

[0132] It can be understood that since the BEV plane is set as the plane with z=0 in the world coordinate system, z og is 0, and then the horizontal and vertical coordinates x of the point on the image plane (that is, the point on the plumb line) on the BEV plane can be calculated. og and og .

[0133] Therefore, according to the horizontal and vertical coordinates of each point on the plumb line on the BEV plane, the corresponding straight line equation is constructed to obtain the projection line of the plumb line in the reference bird's-eye view.

[0134] Finally, the implementation method of the present application can determine the camera extrinsic parameters based on the intersection points of each projection line and the projection point (or vertical projection point) of the camera optical center in the reference bird's-eye view, or construct corresponding constraints to solve the extrinsic parameters. For example, based on the prior knowledge that the plumb lines of the same camera shooting path intersect with the camera optical center, corresponding constraints are constructed to solve the camera extrinsic parameters.

[0135] In this way, the embodiment of the present application determines the camera extrinsic parameters based on the plumb line in the bird's-eye view and the optical center of the camera, which ensures the stable determination of the camera extrinsic parameters to a certain extent.

[0136] In certain embodiments of this application, please refer to Fig. 9 , step 033 comprises:

[0137] 0330: Determine candidate external parameters based on intersection points and projection points;

[0138] 0331: When the candidate external parameter meets the preset condition, the candidate external parameter is determined as the camera external parameter.

[0139] The extrinsic parameter determination module of the implementation manner of the present application is also used to determine candidate extrinsic parameters based on the intersection points and the projection points; when the candidate extrinsic parameters meet preset conditions, the candidate extrinsic parameters are determined as camera extrinsic parameters.

[0140] The processor of the implementation mode of the present application is also used to determine a candidate external parameter based on the intersection point and the projection point; when the candidate external parameter meets a preset condition, the candidate external parameter is determined as the camera external parameter.

[0141] That is, after obtaining the extrinsic parameters through the plumb line, the implementation mode of the present application will also identify the determined values ​​of the external parameters to determine whether the external parameters meet the expected goals. That is, based on the intersection points of multiple plumb lines in the BEV plane and the projection points of the camera optical center in the BEV plane, after determining the candidate extrinsic parameters of the camera, the implementation mode of the present application can perform preset tests on the candidate extrinsic parameters to determine whether the candidate extrinsic parameters meet the preset conditions. If so, the determined candidate extrinsic parameters will be regarded as valid extrinsic parameters, or in other words, camera extrinsic parameters that can be used to construct a bird's-eye view.

[0142] It should be understood that the embodiments of the present application utilize reference external parameters and reference bird's-eye view in the process of determining camera external parameters. In other words, external parameters and bird's-eye view with low accuracy or accuracy that does not reach the expected level are used to determine new external parameters, i.e., candidate external parameters. However, the accuracy and credibility of the determined external parameters may be difficult to guarantee. Therefore, the embodiments of the present application may identify or verify the determined external parameters to determine whether the external parameters or candidate external parameters meet the preset conditions.

[0143] It is understandable that the specific setting method of the preset condition is the content that can be set according to the actual situation. For example, in some embodiments, the embodiments of the present application can check whether the difference (or difference) or gradient between the candidate external parameter determined this time and the candidate external parameter determined last time is within the preset range to determine whether the preset condition is met. Exemplarily, if the difference or gradient between the candidate external parameter determined this time and the candidate external parameter determined last time is within the preset range, it is determined that the candidate external parameter determined this time meets the preset condition. It should be noted that in the case of the candidate external parameter determined for the first time, it can be based on the difference or gradient between the reference external parameter (referring to the external parameter determined not through the plumb line, but through the image captured by the camera and the camera internal parameter) and the candidate external parameter (determined for the first time) to check whether the candidate external parameter meets the preset condition.

[0144] In this way, the implementation manner of the present application is based on determining the external parameters that meet the preset conditions as the camera external parameters, so that the accuracy or credibility of the camera external parameters finally determined can be guaranteed to a certain extent.

[0145] In some implementations of the present application, step 031 further includes:

[0146] When the candidate external parameter does not meet the preset conditions, the candidate external parameter is determined as a new reference external parameter, and a reference bird's-eye view is determined according to the image, the camera intrinsic parameter and the new reference external parameter.

[0147] The external parameter determination module of the implementation mode of the present application is also used to determine the candidate external parameter as a new reference external parameter when the candidate external parameter does not meet the preset conditions, and determine the reference bird's-eye view based on the image, camera intrinsic parameters and the new reference external parameter.

[0148] The processor of the embodiment of the present application is also used to determine the candidate external parameter as a new reference external parameter when the candidate external parameter does not meet the preset conditions, and determine the reference bird's-eye view based on the image, camera intrinsic parameters and the new reference external parameter.

[0149] That is, the implementation mode of the present application can re-execute the external parameter determination process when the candidate external parameter does not meet the preset conditions, or in other words, when the candidate external parameter does not meet the preset conditions, use the candidate external parameter as a new reference external parameter, and then generate a new reference bird's-eye view based on the image previously captured by the camera, the known camera intrinsic parameters and the newly set reference external parameters.

[0150] Then, the projection lines of the plurality of plumb lines in the new reference bird's-eye view are determined, and the projection points of the camera optical center in the new reference bird's-eye view are determined, and new candidate external parameters are determined according to the projection lines and the projection points.

[0151] Then, determine whether the new candidate external parameters meet the preset conditions. If they do, the iteration is terminated and the candidate external parameters determined this time are used as the final camera external parameters. If they do not meet the conditions, the next round of iteration is started, that is, the candidate external parameters determined this time are used as the latest reference external parameters, so as to determine the new bird's-eye view, new projection points and projection lines, and new candidate external parameters.

[0152] In this way, the implementation mode of the present application determines the candidate extrinsic parameter as the reference extrinsic parameter based on the fact that the candidate extrinsic parameter does not meet the preset conditions, so that a new bird's-eye view determination, a new projection line and projection point determination, and a new candidate extrinsic parameter determination can be performed based on the newly determined reference extrinsic parameter, thereby realizing the iterative update of the external parameter to a certain extent, and ensuring the reliability and accuracy of the camera extrinsic parameter finally determined.

[0153] In certain embodiments of this application, please refer to Fig.10 , step 033 comprises:

[0154] 0332: Determine the objective function based on the image and reference external parameters;

[0155] 0333: Determine the first constraint according to the first distance between the intersection point and the projection point;

[0156] 0334: Determine the camera extrinsic parameters based on the first constraint and the objective function.

[0157] The external parameter determination module of the implementation manner of the present application is also used to determine the objective function based on the image and the reference external parameters; determine the first constraint based on the first distance between the intersection point and the projection point; and determine the camera external parameters based on the first constraint and the objective function.

[0158] The processor of the implementation mode of the present application is also used to determine the objective function according to the image and the reference external parameters; determine the first constraint according to the first distance between the intersection point and the projection point; and determine the camera external parameters according to the first constraint and the objective function.

[0159] That is, the embodiments of the present application can construct a functional relationship between each point in the image based on the image previously taken by the camera and the reference extrinsic parameters previously determined for reference, that is, generate an objective function that can describe the camera extrinsic parameters. In some embodiments, the objective function is constructed based on the bundle adjustment method.

[0160] Then, after determining the intersection of the projection line of the plumb line on the BEV plane and the vertical projection point of the camera optical center on the BEV plane, the Euclidean distance between the intersection of the plumb lines and the projection point of the optical center, i.e., the first distance, is calculated, so that the constraint conditions of the objective function can be constructed based on the first distance, or the constraint conditions that the camera extrinsic parameters can satisfy. It can be understood that, according to the content discussed above, it can be known that the intersection of the plumb lines coincides with the projection point of the optical center, or in other words, the intersection of the plumb lines is the projection point of the optical center. Therefore, the implementation method of the present application can construct the above-mentioned first constraint based on this prior knowledge.

[0161] Finally, based on the determined first constraint, the objective function is solved to obtain the camera extrinsic parameters. It can be understood that the camera extrinsic parameters obtained by solving the objective function can also be understood as the above-mentioned candidate extrinsic parameters.

[0162] In this way, after the objective function is constructed by the image and reference extrinsic parameters in the implementation mode of the present application, a first constraint is formed based on the first distance between the intersection of the projection lines of multiple plumb lines and the projection point of the camera optical center. Therefore, the objective function can be solved by the first constraint to obtain the corresponding function solution, that is, the camera extrinsic parameters are obtained, which ensures the effective determination of the camera extrinsic parameters to a certain extent.

[0163] In some embodiments, step 0333 includes:

[0164] The first constraint is determined with the first distance being equal to a preset value as a goal.

[0165] The external parameter determination module in the implementation manner of the present application is also used to determine the first constraint with the first distance being equal to a preset value as a goal.

[0166] The processor of the embodiment of the present application is also used to determine the first constraint with the first distance being equal to a preset value as a goal.

[0167] That is, the implementation of the present application can complete the construction of the first constraint based on the prior knowledge that the intersection point of the plumb line on the BEV plane coincides with the projection point of the camera optical center on the BEV (or the vertical projection point).

[0168] In some embodiments, the embodiments of the present application will set the first distance as 0 to complete the setting of the first constraint.

[0169] In other implementations, due to calculation errors and camera accuracy limitations, the implementations of the present application will target the first distance as a smaller value (such as 5 unit distance) to complete the setting of the first constraint.

[0170] In this way, the implementation method of the present application enables the objective function to be solved with the first distance being equal to the preset value as the goal during the solving process, thereby ensuring the credibility of the camera extrinsic parameters to a certain extent.

[0171] In certain embodiments of this application, please refer to Fig.11 , the image includes multiple images. Step 0332 includes:

[0172] 03320: Determine the image feature points in each image and the first coordinates of the image feature points in each image;

[0173] 03321: Determine the image feature points that exist in at least two images among all the image feature points as target feature points;

[0174] 03322: Determine multiple second coordinates of the target feature point in the reference bird's-eye view image according to the first coordinate of the target feature point in each image;

[0175] 03323: Determine the objective function based on the reference extrinsic parameters and multiple second coordinates of the target feature points.

[0176] The external parameter determination module of the implementation mode of the present application is also used to determine the image feature points in each image and the first coordinates of the image feature points in each image; determine the image feature points that exist in at least two images among all the image feature points as target feature points; determine multiple second coordinates of the target feature points in the reference bird's-eye view based on the first coordinates of the target feature points in each image; determine the target function based on the reference external parameters and the multiple second coordinates of the target feature points.

[0177] The processor of the embodiment of the present application is also used to determine the image feature points in each image and the first coordinates of the image feature points in each image; determine the image feature points that exist in at least two images among all the image feature points as target feature points; determine multiple second coordinates of the target feature points in the reference bird's-eye view based on the first coordinates of the target feature points in each image; determine the target function based on the reference external parameters and the multiple second coordinates of the target feature points.

[0178] That is, in the embodiment of the present application, when different cameras observe the same feature point, after mapping the images taken by different cameras into a bird's-eye view, the extrinsic parameters of each camera can be solved according to the position difference of the same feature point observed by different cameras in the bird's-eye view, thereby obtaining the extrinsic parameters of each camera.

[0179] It is understandable that due to the different external and internal parameters of the camera, the same feature point has certain position differences in the images captured by different cameras. For example, if the front camera and the left camera of the vehicle both capture the same corner point, in an ideal situation, when the image captured by the front camera and the image captured by the left camera are mapped into a bird's-eye view, in the part of the bird's-eye view mapped by the image captured by the front camera and the image captured by the left camera, the above-mentioned "same corner point should only be one. However, due to the different external and internal parameters of the camera, as well as the differences in shooting angles, the same feature point captured by different cameras may appear repeatedly in the bird's-eye view.

[0180] Based on this, the implementation method of the present application first determines the feature points in the images taken by each camera. Among them, the feature points are contents that can be set according to actual conditions. For example, in some implementation methods, the present application completes the determination of extrinsic parameters based on SIFT (Scale-invariant feature transform) feature points and ORB (Oriented fast and Rotated Brief) feature points in the image. In addition, it can be understood that the number of images in the implementation method of the present application is the same as the number of cameras. If there are 4 cameras on board the vehicle, the implementation method of the present application should obtain 4 or multiples of 4 images through 4 cameras to complete the determination of the camera extrinsic parameters when performing a determination of the camera extrinsic parameters.

[0181] Then, after determining the image feature points in different images, the embodiment of the present application can regard the image feature points included in at least two images among all the image feature points as target feature points. For example, assuming that the image captured by the left-view camera includes image feature points fp1 and fp2, and the image captured by the front-view camera includes image feature points fp2 and fp3, since both the image captured by the front-view camera and the image captured by the left-view camera include image feature point fp2, fp2 can be used as the target feature point of the embodiment of the present application.

[0182] After determining the target feature point, the embodiment of the present application can map the target feature point to the reference bird's-eye view (or the world coordinate system) according to the first coordinate of the target feature point to obtain the second coordinate of the target feature point. It should be noted that since the target feature point is included in at least two images, the target feature point corresponds to a first coordinate relative to each image, and then corresponds to the same number of second coordinates.

[0183] Taking the above fp2 as an example, since the image taken by the front camera and the image taken by the left camera both include the image feature point fp2, fp2 corresponds to the first coordinate of the image taken by the front camera and the first coordinate of the image taken by the left camera. After mapping the two coordinates corresponding to fp2 to the reference bird's-eye view, two second coordinates will be obtained.

[0184] Optionally, in certain embodiments, the method for determining the second coordinate of the target feature point is consistent with the aforementioned method for determining the projection point of the point on the plumb line on the BEV plane, that is, the first coordinate of the target feature point is mapped to the world coordinate system based on the rotation torque and translation matrix in the reference extrinsic parameter, and then the ray equation of the target feature point and the optical center in the world coordinate system is constructed according to the coordinate of the optical center of the camera in the world coordinate system. The ray equation corresponding to the target feature point is used to determine the intersection of the ray between the target feature point and the optical center and the BEV plane, thereby obtaining the projection of the target feature point on the BEV plane and the second coordinate of the target feature point.

[0185] Finally, the embodiments of the present application may construct a corresponding objective function by using the reference external parameters and the target feature points at multiple second coordinates of the reference bird's-eye view. In some embodiments, the objective function may be as shown in the following formula, namely:

[0186]

[0187] Where f(x) represents the objective function, x represents the n-dimensional matrix constructed by the reference extrinsic parameter and the second coordinates of the target feature point, and y represents the m-dimensional matrix constructed by the first coordinates of the target feature point.

[0188] In this way, the implementation method of the present application determines the second coordinate of the target feature point in the bird's-eye view based on the first coordinate of the target feature point in the images captured by different cameras, so as to construct a corresponding objective function, so that the objective function can describe the relationship between pixel points in different images reflected by the camera extrinsic parameters to a certain extent. Therefore, the credibility of the camera extrinsic parameters obtained by solving the objective function can be guaranteed to a certain extent.

[0189] In certain embodiments of this application, please refer to Fig.12 , step 03323 includes:

[0190] 033230: Determine a second constraint according to a second distance between any two of a plurality of second coordinates of the target feature point;

[0191] 033231: Determine the camera extrinsic parameters based on the first constraint, the second constraint and the objective function.

[0192] The external parameter determination module of the implementation manner of the present application is also used to determine the second constraint based on the second distance between any two of the multiple second coordinates of the target feature point; and determine the camera external parameter based on the first constraint, the second constraint and the objective function.

[0193] The processor of the implementation manner of the present application is also used to determine the second constraint based on the second distance between any two of the multiple second coordinates of the target feature point; and determine the camera extrinsic parameters based on the first constraint, the second constraint and the objective function.

[0194] That is, the implementation manner of the present application can calculate the corresponding Euclidean distance as the second distance based on any two of the second coordinates corresponding to the target feature points, and construct the corresponding constraint as the second constraint according to the second distance.

[0195] Therefore, according to the first constraint that the plumb lines captured by the same camera intersect at the projection point of the camera optical center on the BEV plane in the BEV plane, and then according to the second constraint that when a target feature point is captured by different cameras and the images captured by different cameras are mapped to the same plane (corresponding to the BEV plane), a target feature point should correspond to a coordinate (corresponding to the second coordinate), the objective function constructed previously is solved to obtain the camera extrinsic parameters.

[0196] In this way, the implementation method of the present application enables the solution of the objective function to be completed based on the first constraint corresponding to the plumb line and the second constraint corresponding to the feature point. In other words, the solution of the camera extrinsic parameters can be solved based on the constraints corresponding to the line features and the constraints corresponding to the point features. Furthermore, the credibility of the camera extrinsic parameters can be guaranteed to a certain extent.

[0197] In some embodiments of the present application, the method for determining camera extrinsic parameters further includes:

[0198] When a target image is acquired based on the camera, a bird's-eye view is determined according to the camera extrinsic parameters, the camera intrinsic parameters and the target image.

[0199] The bird's-eye view generation device of the implementation mode of the present application also includes a bird's-eye view determination module, which is used to determine the bird's-eye view based on camera extrinsic parameters, camera intrinsic parameters and the target image when the target image is acquired based on the camera.

[0200] The processor of the embodiment of the present application is also used to determine a bird's-eye view based on camera extrinsic parameters, camera intrinsic parameters and the target image when the target image is acquired based on the camera.

[0201] That is, the vehicle of the embodiment of the present application can complete the determination or calibration of the camera extrinsic parameters, and when the camera captures the target image or target scene used to generate a bird's-eye view, use the previously determined camera extrinsic parameters (i.e., the camera extrinsic parameters determined by steps 01-03) and the camera intrinsic parameters to map the target image captured by the camera onto the BEV plane, thereby obtaining a bird's-eye view.

[0202] For example, when a vehicle is equipped with four fisheye cameras, the calibration of the four fisheye cameras is completed through the aforementioned steps 01-03, and when the four fisheye cameras all capture an image (i.e., the target image) at a certain moment or within a certain period of time, the vehicle of the implementation mode of the present application can map the four images captured by the four fisheye cameras onto a preset plane based on the camera intrinsic parameters and the camera extrinsic parameters, that is, after mapping onto the preset, aforementioned BEV plane, an aerial view corresponding to the four images captured by the four fisheye cameras can be obtained.

[0203] In this way, the embodiment of the present application enables the camera to determine or convert the target image into a bird's-eye view according to the previously determined camera extrinsics and camera intrinsics when capturing the target image for the bird's-eye view, thereby realizing the generation of the bird's-eye view. In addition, because the extrinsics used in the embodiment of the present application when generating the bird's-eye view are calibrated by the plumb line in the image or scene, that is, the calibration of the camera extrinsics is based on reliable markers, the reliability of the camera extrinsics is guaranteed, and thus, the credibility of the generated bird's-eye view can also be guaranteed to a certain extent.

[0204] In certain embodiments of the present application, the process of constructing the objective function and the process of solving the objective function may include: assuming that in a process of determining an external parameter at a certain time, the vehicle controls the camera on the vehicle to capture m images (for example, the vehicle controls the front view camera, the left view camera, the rear view camera, and the right view camera to capture once to obtain 4 images respectively), and the external parameter of each camera is vectorized to obtain a vector a j .

[0205] Perform feature point detection and target feature point determination on m images. If n target feature points are determined, all first coordinates of the n target feature points are vector b. j .

[0206] Assume that any target feature point exists in m images, x ij represents the first coordinate of the i-th target feature point on the j-th image, and defines Q(a j , b i ) is a function that reprojects the first coordinate of the i-th target feature point to the second coordinate on the j-th image. Then, the objective function is constructed as:

[0207]

[0208] Where d(,) represents the Euclidean distance between the two parameters in the brackets.

[0209] The process of solving the above objective function using Levenverg-Marquardt (LM) iterative optimization is as follows:

[0210] The above objective function is regarded as follows:

[0211]

[0212] Where f(x) represents the objective function, x represents the n-dimensional matrix constructed by the reference extrinsic parameter and the second coordinates of the target feature point, and y represents the m-dimensional matrix constructed by the first coordinates of the target feature point.

[0213] Furthermore, let the initial estimate of x be x0 and the observed value of y be The covariance of is Σ (Σ is a symmetric positive definite matrix), and the optimized value x is calculated iteratively (i+1) =x i +δ i .

[0214] Substitute the above objective function into the equation:

[0215]

[0216] Then we get:

[0217]

[0218] Given a function:

[0219]

[0220] Then the function value F(x) satisfies the following relative increment of the independent variable x:

[0221]

[0222] Where θ is the angle between vector δ and F'(x).

[0223] When θ=π, the relative increment is the largest, and δ=F'(x). i When the value is far from the optimal value x*, the optimization speed of the function is stable. i When it is closer to the optimal value x*, the function is optimized more slowly.

[0224] On the other hand, according to Taylor expansion:

[0225] f(x i+1 )=f(x i +δ)=f(x i )+J(x i )δ+O(||δ 2 ||)

[0226] Where J(x) is the Jacobian matrix of f(x).

[0227] And then:

[0228]

[0229] Use a linear model about f(x) to approximate the change of f in a certain neighborhood, that is:

[0230]

[0231] Therefore, the objective function can be changed to:

[0232]

[0233] At this time, the problem of solving the objective function is converted into a least squares problem, so δ i becomes the solution of the following normal equation, namely:

[0234]

[0235] In order to avoid jumping out of the local optimum or global optimum during the iterative solution process, resulting in non-convergence, an augmented normal linear equation is added in each iteration, as shown below:

[0236]

[0237] Therefore, μ will be added to the diagonal elements of the original normal equation.

[0238] At the same time, when the value of μ is large, we have:

[0239]

[0240] At this time, the execution effect of the iterative incremental method is similar to that of the steepest descent method.

[0241] When the value of μ is small, the execution effect of the iterative incremental method is similar to that of the Gauss-Newton iterative method.

[0242] When the value of δ is close to the above optimal value x*, the optimization speed is close to the speed of quadratic convergence.

[0243] In addition, the initial value of μ at the beginning of the iteration can be calculated according to the normal equation matrix N0=J(x0) T ∑ -1 The elements in J(x0) are determined. So we have:

[0244]

[0245] Therefore, when the value of μ is small, the execution effect of the iterative incremental method is similar to that of the Gauss-Newton iterative method. When the value of δ is close to the above optimal value x*, the optimization speed is similar to the speed of quadratic convergence.

[0246] In addition, when the value of x0 is close to the optimal value x*, τ can take a smaller value such as 10 -6 On the contrary, τ can take a larger value such as 10 -3 or 1.

[0247] It is understandable that the value of μ will change in each iteration. The value of μ can be set according to the similarity between the approximate linear model and the true cost function. Then, the approximate linear model is defined as:

[0248]

[0249] The similarity between the approximate linear model and the true cost function is defined as:

[0250]

[0251] When ρ takes a larger value, it means that the approximate linear model is a better approximation of the true cost function, and then μ can be increased to adopt the effect of the Gauss-Newton iteration method in the next iteration. When ρ takes a smaller value, it means that the approximate linear model is a poor approximation of the true cost function, and then μ can be reduced to make the next iteration closer to the effect of the steepest descent method.

[0252] And, it should be noted that in the process of solving the objective function, solving (N i +μI)δ=b i This is the most time-consuming part of the calculation. Solving (Ni +μI)δ=b i The time complexity is O(n 3 ). To ensure the efficiency of the solution, the sparsity of the matrix can be used to achieve a fast solution.

[0253] Specifically, assuming there are p images and q target feature points, and each target feature point appears in each image, then:

[0254] x=(c1 T ,c2 T ,…,c p T ,p1 T ,p2 T ,…,p q T ) T

[0255] y=(y 11 T ,y 12 T ,…,y 1p T ,y 21 T ,y 22 T ,…,y 2p T ,…,y q1 T ,y q2 T ,…,y qp T ) T

[0256] In the formula, c i represents the external parameter of the i-th image, p j Represents the second coordinate of the jth target feature point, y ji Represents the first coordinate of the j-th point of the same name on the i-th image.

[0257] The corresponding Jacobian matrix is:

[0258]

[0259] Assuming that the covariance matrix Σ is a diagonal block matrix, we have:

[0260]

[0261] Furthermore, the normal equation J T Σ -1 J+μI can be changed to:

[0262]

[0263]

[0264] at the same time, It can be expressed as:

[0265]

[0266]

[0267] Therefore, the normal linear equations to be solved can be expressed as:

[0268]

[0269] Multiply both sides of the equation by the same matrix:

[0270]

[0271] Then we have:

[0272]

[0273] δ c Solving it, we get:

[0274] (U-WV -1 W T )δ c =e c -WV -1 e p

[0275] Substitute back to solve for δ p , then:

[0276] δ p =V -1 (e p -W T δ c )

[0277] Among them, because V is a diagonal block matrix, so solve V -1 The time complexity is O(q), or, V -1 It can be solved in O(q) time. Furthermore, the time complexity of solving the entire normal linear equation system can be O(p 3 At the same time, since the number of target feature points is usually much greater than the number of captured images, the above-mentioned solution method provided in the embodiment of the present application can effectively reduce the time required for calculation.

[0278] In addition, in the process of solving the objective function, that is, in solving the cost vector function y=(y 11T ,y 12 T ,…,y 1p T ,y 21 T ,y 22 T ,…,y 2p T ,…,y q1 T ,y q2 T ,…,y qp T ) T In the process of adding specific cost function items and adding the derivative of the corresponding cost vector to the camera extrinsic parameter in the Jacobian matrix, the addition of the plumb line constraint (that is, the first constraint) can be achieved.

[0279] It is understandable that in a practical environment, if the analytical expression of the Jacobian matrix is ​​difficult to obtain, the derivative can be obtained by using finite differences in the general BA optimization library. Therefore, the cost function term of BA optimization can be directly added to the cost vector function without manually constructing the Jacobian matrix.

[0280] It can also be understood that the cost function term of the implementation manner of the present application may include the above-mentioned first constraint and may also include the above-mentioned second constraint.

[0281] The present application also provides a computer-readable storage medium containing a computer program. When the computer program is executed by one or more processors, the one or more processors execute the method for determining camera extrinsic parameters provided in the embodiments of the present application.

[0282] In the description of this specification, the descriptions with reference to the terms "certain embodiments", "in an example", "exemplarily", etc., mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0283] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0284] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for determining camera extrinsic parameters, characterized in that: include: Get the image captured by the camera; determining a plumb line in the image; The camera extrinsic parameters are determined according to the plumb line, the image and predetermined camera intrinsic parameters.

2. The method according to claim 1, characterized in that The plumb lines include a plurality of plumb lines; and determining the camera extrinsic parameters according to the plumb lines, the image, and the predetermined camera intrinsic parameters, including: Determining a reference extrinsic parameter of the camera according to the camera intrinsic parameter and the image; Determine a reference bird's-eye view according to the image, the camera intrinsic parameters and the reference extrinsic parameters; Determining the projection line of each of the plumb lines in the reference bird's-eye view; The camera extrinsic parameters are determined according to the intersection point of each of the projection lines and the projection point of the camera optical center in the reference bird's-eye view.

3. The method according to claim 2, characterized in that Determining the camera extrinsic parameters according to the intersection of each of the projection lines and the projection point of the camera optical center in the reference bird's-eye view includes: Determine a candidate extrinsic parameter according to the intersection point and the projection point; When the candidate extrinsic parameter satisfies a preset condition, the candidate extrinsic parameter is determined as the camera extrinsic parameter.

4. The method according to claim 3, characterized in that The determining of the reference bird's-eye view according to the image, the camera intrinsic parameter and the reference extrinsic parameter further includes: When the candidate external parameter does not meet the preset condition, the candidate external parameter is determined as the new reference external parameter, and the reference bird's-eye view is determined according to the image, the camera intrinsic parameter and the new reference external parameter.

5. The method according to claim 2, characterized in that: Determining the camera extrinsic parameters according to the intersection of each of the projection lines and the projection point of the camera optical center in the reference bird's-eye view includes: Determining an objective function according to the image and the reference extrinsic parameter; Determining a first constraint according to a first distance between the intersection point and the projection point; The camera extrinsic parameters are determined according to the first constraint and the objective function.

6. The method according to claim 5, characterized in that The determining the first constraint according to the first distance between the intersection point and the projection point includes: The first constraint is determined with the goal of the first distance being equal to a preset value.

7. The method according to claim 5, characterized in that The image includes a plurality of images; and determining the objective function according to the image and the reference external parameter includes: Determine an image feature point in each of the images, and a first coordinate of the image feature point in each of the images; Determine, among all the image feature points, image feature points that exist in at least two of the images as target feature points; Determine a plurality of second coordinates of the target feature point in the reference bird's-eye view according to the first coordinates of the target feature point in each of the images; The objective function is determined according to the reference extrinsic parameter and the plurality of second coordinates of the target feature point.

8. The method according to claim 7, characterized in that The determining the objective function according to the reference extrinsic parameter and the plurality of second coordinates of the target feature point comprises: Determining a second constraint according to a second distance between any two of the plurality of second coordinates of the target feature point; The camera extrinsic parameters are determined according to the first constraint, the second constraint and the objective function.

9. The method according to claim 1, characterized in that: The method further comprises: When a target image is acquired based on the camera, a bird's-eye view is determined according to the camera extrinsic parameters, the camera intrinsic parameters and the target image.

10. A device for determining camera extrinsic parameters, characterized in that: include: An acquisition module is used to acquire images captured by a camera; A plumb line determination module, used to determine a plumb line in the image; The extrinsic parameter determination module is used to determine the camera extrinsic parameters according to the plumb line, the image and the predetermined camera intrinsic parameters.

11. A vehicle, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method according to any one of claims 1 to 9 is implemented.

Images