Camera calibration method and device, electronic device and storage medium

Abstract

The present disclosure relates to a camera calibration method and device, electronic device and storage medium. The method comprises: obtaining the intrinsic parameters of a camera and the height of the camera relative to a target plane; determining the three-dimensional coordinates of a plurality of spatial points in the camera coordinate system of the camera according to the image captured by the camera; determining the normal vector of the target plane according to the three-dimensional coordinates of the plurality of spatial points in the camera coordinate system; determining the attitude angle of the camera relative to the target plane according to the normal vector of the target plane; determining the homography matrix from the camera to the target plane according to the intrinsic parameters, the height and the attitude angle.

Description

Technical Field

[0001] The present disclosure relates to the field of camera technology, and in particular to a camera calibration method and device, an electronic device, and a storage medium. Background Art

[0002] In the application scenario of autonomous driving, a monocular camera is usually needed to perceive the environmental information around the vehicle, such as lane lines, obstacles, etc. Since a monocular camera cannot directly restore the three-dimensional scale of objects (such as objects or people) around the vehicle, it is generally assumed that the vehicle is traveling on a local road plane, and the three-dimensional scale in the mapped road plane coordinate system is obtained through the pre-calibrated homography matrix from the camera to the road plane. It can be seen that an accurate homography matrix is ​​of great significance for application scenarios such as autonomous driving. Summary of the invention

[0003] The present disclosure provides a camera calibration technical solution.

[0004] According to one aspect of the present disclosure, a camera calibration method is provided, comprising:

[0005] Obtaining the intrinsic parameters of the camera and the height of the camera relative to the target plane;

[0006] Determining three-dimensional coordinates of a plurality of spatial points in a camera coordinate system of the camera according to the image captured by the camera;

[0007] Determining a normal vector of the target plane according to the three-dimensional coordinates of the multiple spatial points in the camera coordinate system;

[0008] Determining the attitude angle of the camera relative to the target plane according to the normal vector of the target plane;

[0009] A homography matrix from the camera to the target plane is determined according to the intrinsic parameter, the height, and the attitude angle.

[0010] By obtaining the intrinsic parameters of the camera and the height of the camera relative to the target plane, according to the image captured by the camera, the three-dimensional coordinates of multiple spatial points in the camera coordinate system of the camera are determined, according to the three-dimensional coordinates of the multiple spatial points in the camera coordinate system, the normal vector of the target plane is determined, according to the normal vector of the target plane, the attitude angle of the camera relative to the target plane is determined, and according to the intrinsic parameters, the height and the attitude angle, the homography matrix from the camera to the target plane is determined, thereby only needing to obtain the intrinsic parameters of the camera and the height of the camera relative to the target plane, the calibration of the homography matrix from the camera to the target plane can be achieved, so that the camera calibration result is more adaptable to the changes in the scene and the accuracy during online use is improved. In addition, by adopting the camera calibration method provided by the embodiment of the present disclosure, the problem of manual dependence in the calibration process can be solved, the calibration process can be automated, and no manual participation is required, thereby reducing the probability of errors. Furthermore, the problem of dependence on artificial reference objects can be solved, no longer dependent on special calibration scenes, and can be applicable to a wider range of application scenes.

[0011] In a possible implementation manner, determining the normal vector of the target plane according to the three-dimensional coordinates of the multiple spatial points in the camera coordinate system includes:

[0012] Extracting a plurality of candidate planes according to the three-dimensional coordinates of the plurality of spatial points in the camera coordinate system;

[0013] The normal vector of the target plane is determined according to the normal vectors of the multiple candidate planes.

[0014] In this implementation, multiple candidate planes are extracted based on the three-dimensional coordinates of the multiple spatial points in the camera coordinate system, and the normal vector of the target plane is determined based on the normal vectors of the multiple candidate planes, thereby enabling the normal vector of the target plane to be determined effectively and accurately.

[0015] In a possible implementation manner, determining the normal vector of the target plane according to the normal vectors of the multiple candidate planes includes:

[0016] Filtering a potential plane of the target plane from the multiple candidate planes according to the movement direction of the camera and the normal vectors of the multiple candidate planes;

[0017] The normal vector of the target plane is determined according to the normal vector of the potential plane.

[0018] In this implementation, by screening out potential planes of the target plane from the multiple candidate planes based on the movement direction of the camera and the normal vectors of the multiple candidate planes, and determining the normal vector of the target plane based on the normal vector of the potential plane, the accuracy of the determined normal vector of the target plane can be further improved, and it helps to increase the speed of determining the normal vector of the target plane.

[0019] In a possible implementation, the step of selecting a potential plane of the target plane from the multiple candidate planes according to the movement direction of the camera and the normal vectors of the multiple candidate planes includes:

[0020] For any candidate plane among the plurality of candidate planes, in response to a normal vector of the candidate plane being substantially perpendicular to a forward motion direction vector of the camera, the candidate plane is determined as a potential plane of the target plane.

[0021] In this implementation, for any candidate plane among the multiple candidate planes, in response to the normal vector of the candidate plane being approximately perpendicular to the forward motion direction vector of the camera, the candidate plane is determined as a potential plane of the target plane, thereby being able to quickly and accurately screen out the potential plane of the target plane from a large number of candidate planes.

[0022] In a possible implementation manner, determining the normal vector of the target plane according to the normal vector of the potential plane includes:

[0023] In the case where there are multiple potential planes, for any of the multiple potential planes, determining the support number of the potential plane according to the normal vectors of the multiple potential planes, wherein the support number of the potential plane represents the number of potential planes in the multiple potential planes that are substantially parallel to the potential plane;

[0024] A normal vector of the target plane is determined according to the support numbers of the multiple potential planes.

[0025] In this implementation, in this example, when there are multiple potential planes, for any potential plane among the multiple potential planes, the support number of the potential plane is determined based on the normal vectors of the multiple potential planes, and the normal vector of the target plane is determined based on the support number of the multiple potential planes, thereby improving the accuracy of the normal vector of the determined target plane.

[0026] In a possible implementation manner, determining the normal vector of the target plane according to the support numbers of the multiple potential planes includes:

[0027] The normal vector of the potential plane with the largest support number among the multiple potential planes is determined as the normal vector of the target plane.

[0028] In this implementation, by determining the normal vector of the potential plane with the largest support number among multiple potential planes as the normal vector of the target plane, the accuracy of the normal vector of the determined target plane can be improved, that is, the accuracy of the determined target plane can be improved, thereby improving the accuracy of the camera calibration result.

[0029] In a possible implementation, determining the three-dimensional coordinates of a plurality of spatial points in a camera coordinate system of the camera according to the image captured by the camera includes:

[0030] Acquire a first image and a second image captured by the camera in a moving state;

[0031] Determine a first rotation matrix and a translation vector of the camera motion according to the first image and the second image;

[0032] The three-dimensional coordinates of a plurality of spatial points in the camera coordinate system of the camera are determined according to the first rotation matrix and the translation vector.

[0033] In this implementation, by acquiring the first image and the second image captured by the camera in a moving state, the first rotation matrix and the translation vector of the camera movement are determined according to the first image and the second image, and the three-dimensional coordinates of multiple spatial points in the camera coordinate system of the camera are determined according to the first rotation matrix and the translation vector. In this way, the three-dimensional coordinates of multiple spatial points in the camera coordinate system of the camera can be quickly and accurately determined based on the images captured by the camera.

[0034] In a possible implementation manner, determining a first rotation matrix and a translation vector of the camera motion according to the first image and the second image includes:

[0035] extracting feature points of the first image and feature points of the second image;

[0036] determining an essential matrix or a basic matrix from the first image to the second image according to a matching relationship between feature points of the first image and feature points of the second image;

[0037] A first rotation matrix and a translation vector of the camera motion are determined according to the essential matrix or the basic matrix.

[0038] In this implementation, by extracting feature points of the first image and feature points of the second image, the essential matrix or basic matrix from the first image to the second image is determined based on the matching relationship between the feature points of the first image and the feature points of the second image, and the first rotation matrix and translation vector of the camera motion are determined based on the essential matrix or the basic matrix, thereby accurately determining the first rotation matrix and translation vector of the camera motion, thereby improving the accuracy of the three-dimensional coordinates of the determined spatial point in the camera coordinate system of the camera.

[0039] In a possible implementation, determining the homography matrix from the camera to the target plane according to the intrinsic parameter, the height, and the attitude angle includes:

[0040] Determine, according to the attitude angle, a second rotation matrix for rotating the camera from a front-view state to a top-view state;

[0041] Converting the third image captured by the camera in the forward-looking state into a fourth image, wherein the fourth image is an image of looking down at the target plane;

[0042] Acquire image coordinates of a first image point in the third image and image coordinates of a second image point in the fourth image, wherein the first image point and the second image point are corresponding image points;

[0043] A homography matrix from the camera to the target plane is determined according to the image coordinates of the first image point, the image coordinates of the second image point, the second rotation matrix, the intrinsic parameter, and the height.

[0044] In this implementation, a second rotation matrix of the camera rotating from a forward-looking state to a downward-looking state is determined according to the attitude angle, a third image captured by the camera in the forward-looking state is converted into a fourth image, the image coordinates of the first image point in the third image and the image coordinates of the second image point in the fourth image are obtained, and the homography matrix from the camera to the target plane is determined according to the image coordinates of the first image point, the image coordinates of the second image point, the second rotation matrix, the intrinsic parameters and the height, thereby achieving accurate camera calibration.

[0045] In a possible implementation manner, after determining the homography matrix from the camera to the target plane, the method further includes:

[0046] Acquire image coordinates of a target object from a fifth image captured by the camera;

[0047] According to the homography matrix, the image coordinates of the target object are converted to the target plane to obtain the coordinates of the target object on the target plane;

[0048] The verification result of the homography matrix is ​​determined according to preset attribute information of the target object and the coordinates of the target object on the target plane, wherein the preset attribute information includes preset shape information and / or preset size information of the target object.

[0049] In this implementation, the image coordinates of the target object are obtained from the fifth image captured by the camera, and the image coordinates of the target object are converted to the target plane according to the homography matrix to obtain the coordinates of the target object in the target plane. According to preset attribute information including preset shape information and / or preset size information of the target object, and the coordinates of the target object in the target plane, the verification result of the homography matrix is ​​determined. In this way, the calibration result of the camera can be automatically and effectively verified, and it can be determined whether the current camera calibration result is reliable. It is more reliable than manual subjective verification, and there is no need to repeat the calibration process after manually discovering problems, thereby reducing manual participation and improving efficiency.

[0050] In a possible implementation, determining the verification result of the homography matrix according to the preset attribute information of the target object and the coordinates of the target object on the target plane includes:

[0051] Determining shape information and size information of the target object according to the coordinates of the target object on the target plane;

[0052] In response to the shape information of the target object being consistent with the preset shape information, and the size information of the target object being consistent with the preset size information, it is determined that the homography matrix verification is successful; or, in response to the shape information of the target object being inconsistent with the preset shape information, or the size information of the target object being inconsistent with the preset size information, it is determined that the homography matrix verification has failed.

[0053] In this implementation, the shape information and size information of the target object are determined according to the coordinates of the target object on the target plane, and in response to the shape information of the target object being consistent with the preset shape information, and the size information of the target object being consistent with the preset size information, it is determined that the homography matrix verification is successful; or, in response to the shape information of the target object being inconsistent with the preset shape information, or the size information of the target object being inconsistent with the preset size information, it is determined that the homography matrix verification has failed, thereby improving the accuracy of the verification result of the homography matrix.

[0054] In a possible implementation, the camera is installed in a vehicle cabin, and the target plane is a road plane.

[0055] According to this implementation, it is only necessary to obtain the intrinsic parameters of the camera in the cabin and the height of the camera relative to the road plane to realize the calibration of the homography matrix from the camera to the road plane, so that the camera calibration results are more adaptable to changes in the driving environment and improve the accuracy during online use. In addition, by adopting this implementation, the problem of manual dependence in the cabin camera calibration process can be solved, and the calibration process can be automated without manual participation, thereby reducing the probability of errors. Furthermore, it can solve the problem of dependence on artificial reference objects, no longer rely on special calibration scenarios, and can be applied to more diverse driving environments.

[0056] According to one aspect of the present disclosure, a camera calibration device is provided, comprising:

[0057] A first acquisition module, used to acquire the intrinsic parameters of the camera and the height of the camera relative to the target plane;

[0058] A first determination module, used to determine the three-dimensional coordinates of a plurality of spatial points in a camera coordinate system of the camera according to the image captured by the camera;

[0059] A second determination module, configured to determine a normal vector of the target plane according to the three-dimensional coordinates of the plurality of spatial points in the camera coordinate system;

[0060] A third determination module, used to determine the attitude angle of the camera relative to the target plane according to the normal vector of the target plane;

[0061] The fourth determination module is used to determine the homography matrix from the camera to the target plane according to the intrinsic parameter, the height and the attitude angle.

[0062] In a possible implementation manner, the second determining module is used to:

[0063] Extracting a plurality of candidate planes according to the three-dimensional coordinates of the plurality of spatial points in the camera coordinate system;

[0064] The normal vector of the target plane is determined according to the normal vectors of the multiple candidate planes.

[0065] In a possible implementation manner, the second determining module is used to:

[0066] Filtering a potential plane of the target plane from the multiple candidate planes according to the movement direction of the camera and the normal vectors of the multiple candidate planes;

[0067] The normal vector of the target plane is determined according to the normal vector of the potential plane.

[0068] In a possible implementation manner, the second determining module is used to:

[0069] For any candidate plane among the plurality of candidate planes, in response to a normal vector of the candidate plane being substantially perpendicular to a forward motion direction vector of the camera, the candidate plane is determined as a potential plane of the target plane.

[0070] In a possible implementation manner, the second determining module is used to:

[0071] In the case where there are multiple potential planes, for any of the multiple potential planes, determining the support number of the potential plane according to the normal vectors of the multiple potential planes, wherein the support number of the potential plane represents the number of potential planes in the multiple potential planes that are substantially parallel to the potential plane;

[0072] A normal vector of the target plane is determined according to the support numbers of the multiple potential planes.

[0073] In a possible implementation manner, the second determining module is used to:

[0074] The normal vector of the potential plane with the largest support number among the multiple potential planes is determined as the normal vector of the target plane.

[0075] In a possible implementation manner, the first determining module is used to:

[0076] Acquire a first image and a second image captured by the camera in a moving state;

[0077] Determine a first rotation matrix and a translation vector of the camera motion according to the first image and the second image;

[0078] The three-dimensional coordinates of a plurality of spatial points in the camera coordinate system of the camera are determined according to the first rotation matrix and the translation vector.

[0079] In a possible implementation manner, the first determining module is used to:

[0080] extracting feature points of the first image and feature points of the second image;

[0081] determining an essential matrix or a basic matrix from the first image to the second image according to a matching relationship between feature points of the first image and feature points of the second image;

[0082] A first rotation matrix and a translation vector of the camera motion are determined according to the essential matrix or the basic matrix.

[0083] In a possible implementation manner, the fourth determining module is used to:

[0084] Determine, according to the attitude angle, a second rotation matrix for rotating the camera from a front-view state to a top-view state;

[0085] Converting the third image captured by the camera in the forward-looking state into a fourth image, wherein the fourth image is an image of looking down at the target plane;

[0086] Acquire image coordinates of a first image point in the third image and image coordinates of a second image point in the fourth image, wherein the first image point and the second image point are corresponding image points;

[0087] A homography matrix from the camera to the target plane is determined according to the image coordinates of the first image point, the image coordinates of the second image point, the second rotation matrix, the intrinsic parameter, and the height.

[0088] In a possible implementation manner, the device further includes:

[0089] A second acquisition module, used to acquire the image coordinates of the target object from the fifth image captured by the camera;

[0090] A conversion module, used for converting the image coordinates of the target object to the target plane according to the homography matrix to obtain the coordinates of the target object on the target plane;

[0091] The fifth determination module is used to determine the verification result of the homography matrix according to the preset attribute information of the target object and the coordinates of the target object on the target plane, wherein the preset attribute information includes preset shape information and / or preset size information of the target object.

[0092] In a possible implementation manner, the fifth determining module is used to:

[0093] Determining shape information and size information of the target object according to the coordinates of the target object on the target plane;

[0094] In response to the shape information of the target object being consistent with the preset shape information, and the size information of the target object being consistent with the preset size information, it is determined that the homography matrix verification is successful; or, in response to the shape information of the target object being inconsistent with the preset shape information, or the size information of the target object being inconsistent with the preset size information, it is determined that the homography matrix verification has failed.

[0095] In a possible implementation, the camera is installed in a vehicle cabin, and the target plane is a road plane.

[0096] According to one aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above method.

[0097] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0098] In the embodiment of the present disclosure, by obtaining the intrinsic parameters of the camera and the height of the camera relative to the target plane, according to the image captured by the camera, the three-dimensional coordinates of multiple spatial points in the camera coordinate system of the camera are determined, according to the three-dimensional coordinates of the multiple spatial points in the camera coordinate system, the normal vector of the target plane is determined, according to the normal vector of the target plane, the attitude angle of the camera relative to the target plane is determined, and according to the intrinsic parameters, the height and the attitude angle, the homography matrix from the camera to the target plane is determined, thereby only needing to obtain the intrinsic parameters of the camera and the height of the camera relative to the target plane, the calibration of the homography matrix from the camera to the target plane can be achieved, so that the camera calibration result is more adaptable to the changes in the scene and the accuracy during online use is improved. In addition, by adopting the camera calibration method provided in the embodiment of the present disclosure, the problem of manual dependence in the calibration process can be solved, the calibration process can be automated, and no manual participation is required, thereby reducing the probability of errors. Furthermore, the problem of dependence on artificial reference objects can be solved, no longer dependent on special calibration scenes, and can be applied to a wider range of application scenes.

[0099] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0100] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0102] Figure 1 A flowchart of a camera calibration method provided by an embodiment of the present disclosure is shown.

[0103] Figure 2 A schematic diagram of a potential plane in a camera calibration method provided by an embodiment of the present disclosure is shown.

[0104] Figure 3A schematic diagram showing a camera in a forward-looking state and a camera in a downward-looking state in a camera calibration method provided by an embodiment of the present disclosure is shown.

[0105] Figure 4 A schematic diagram showing an image captured by a camera in a camera calibration method provided by an embodiment of the present disclosure.

[0106] Figure 5 A schematic diagram showing the coordinates of a target object on a target plane in a camera calibration method provided by an embodiment of the present disclosure.

[0107] Figure 6 A schematic diagram showing an application scenario of the camera calibration method provided by an embodiment of the present disclosure.

[0108] Figure 7 A block diagram of a camera calibration device provided by an embodiment of the present disclosure is shown.

[0109] Figure 8 A block diagram of an electronic device 800 provided by an embodiment of the present disclosure is shown.

[0110] Fig. 9 A block diagram of an electronic device 1900 provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0111] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0112] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0113] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0114] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0115] In the related art, in the application scenario of autonomous driving, the monocular camera is usually calibrated in a specific offline scenario, and the homography matrix is ​​no longer adjusted regardless of whether the relative relationship between the camera plane and the road plane changes. When performing camera calibration, the related art usually selects several reference point pairs on the image plane and the road plane, and then calculates the mapping relationship between the reference point pairs to obtain the calibration result of the homography matrix.

[0116] Since the related art adopts an offline camera calibration method, even if the homography matrix from the camera to the road plane in a specific scene can be fitted, it cannot cope with the change in the relative relationship between the camera and the road plane caused by vehicle bumps, etc., resulting in low or even wrong perception accuracy of the environmental information around the vehicle in the application scenario of autonomous driving. In addition, the camera calibration method in the related art is a highly manual method that requires manual selection of control point pairs. For each control point on the image, it is necessary to manually find its precise corresponding point from the road plane coordinate system. Since the selection of control points on the road plane is carried out in the real physical coordinate system, it is difficult to ensure that the coordinates of the measured control points in the road plane coordinate system are accurate, which seriously affects the accuracy of camera calibration. Furthermore, since it is necessary to set artificial reference objects, such as QR codes or cone barrels, when manually selecting ground control points, it is necessary to perform camera calibration in special scenes, and it is impossible to perform calibration in more general scenes, which limits the scope of application.

[0117] The embodiment of the present disclosure provides a camera calibration method, by obtaining the intrinsic parameters of the camera and the height of the camera relative to the target plane, according to the image captured by the camera, determining the three-dimensional coordinates of multiple spatial points in the camera coordinate system of the camera, according to the three-dimensional coordinates of the multiple spatial points in the camera coordinate system, determining the normal vector of the target plane, according to the normal vector of the target plane, determining the attitude angle of the camera relative to the target plane, and determining the homography matrix from the camera to the target plane according to the intrinsic parameters, the height and the attitude angle, thereby only needing to obtain the intrinsic parameters of the camera and the height of the camera relative to the target plane, the calibration of the homography matrix from the camera to the target plane can be achieved, so that the camera calibration result is more adaptable to the changes in the scene and the accuracy when used online is improved. In addition, by adopting the camera calibration method provided by the embodiment of the present disclosure, the problem of manual dependence in the calibration process can be solved, the calibration process can be automated, and no manual participation is required, thereby reducing the probability of errors. Furthermore, the problem of dependence on artificial reference objects can be solved, no longer dependent on special calibration scenes, and can be applied to a wider range of application scenes.

[0118] The camera calibration method provided by the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0119] Figure 1 A flowchart of a camera calibration method provided by an embodiment of the present disclosure is shown. In one possible implementation, the camera calibration method may be executed by a terminal device or a server or other processing device. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a camera, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device or a wearable device, etc. In some possible implementations, the camera calibration method may be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1 As shown, the camera calibration method includes steps S11 to S15.

[0120] In step S11, the intrinsic parameters of the camera and the height of the camera relative to the target plane are obtained.

[0121] In step S12, the three-dimensional coordinates of a plurality of spatial points in the camera coordinate system of the camera are determined according to the image captured by the camera.

[0122] In step S13, the normal vector of the target plane is determined according to the three-dimensional coordinates of the multiple spatial points in the camera coordinate system.

[0123] In step S14, the attitude angle of the camera relative to the target plane is determined according to the normal vector of the target plane.

[0124] In step S15, a homography matrix from the camera to the target plane is determined according to the intrinsic parameters, the height and the attitude angle.

[0125] In the embodiment of the present disclosure, the pre-calibrated parameters include the intrinsic parameters of the camera and the height of the camera relative to the target plane.

[0126] In one possible implementation, the camera is installed in the cabin, and the target plane is the road plane. According to this implementation, it is only necessary to obtain the intrinsic parameters of the camera in the cabin and the height of the camera relative to the road plane to realize the calibration of the homography matrix from the camera to the road plane, so that the camera calibration result can be more adaptable to changes in the driving environment and improve the accuracy during online use. In addition, by adopting this implementation, the problem of manual dependence in the cabin camera calibration process can be solved, and the calibration process can be automated without manual participation, thereby reducing the probability of errors. Furthermore, it can solve the problem of dependence on artificial reference objects, no longer rely on special calibration scenarios, and can be applicable to more diverse driving environments.

[0127] Of course, in other application scenarios, the camera may be installed at other locations, and the target plane may be other planes, which are not limited here.

[0128] In the disclosed embodiment, based on multiple frames of images captured by a camera in motion, a method such as SFM (Structure From Motion) can be used to obtain the three-dimensional coordinates of multiple spatial points in the camera coordinate system of the camera. The spatial points can represent static object points in the three-dimensional space. The coordinates of these spatial points in the camera coordinate system can be three-dimensional coordinates without absolute scale.

[0129] In a possible implementation, the determining of the three-dimensional coordinates of multiple spatial points in the camera coordinate system of the camera according to the images captured by the camera includes: acquiring a first image and a second image captured by the camera in motion; determining a first rotation matrix and a translation vector of the camera motion according to the first image and the second image; and determining the three-dimensional coordinates of multiple spatial points in the camera coordinate system of the camera according to the first rotation matrix and the translation vector. In this implementation, the first image and the second image captured by the camera in motion at different times can be acquired. That is, the first image and the second image are images captured by the camera at different positions. According to the correlation between the image information of the first image and the image information of the second image, the first rotation matrix and the translation vector of the camera motion can be obtained. Among them, the first rotation matrix represents the rotation matrix of the camera motion. Based on the first rotation matrix and the translation vector of the camera motion, the three-dimensional coordinates of multiple spatial points in the camera coordinate system of the camera can be reconstructed. In this implementation, by acquiring the first image and the second image captured by the camera in a moving state, the first rotation matrix and the translation vector of the camera movement are determined according to the first image and the second image, and the three-dimensional coordinates of multiple spatial points in the camera coordinate system of the camera are determined according to the first rotation matrix and the translation vector. In this way, the three-dimensional coordinates of multiple spatial points in the camera coordinate system of the camera can be quickly and accurately determined based on the images captured by the camera.

[0130] As an example of this implementation, the first rotation matrix and translation vector of the camera motion are determined according to the first image and the second image, including: extracting feature points of the first image and the second image; determining the essential matrix or basic matrix from the first image to the second image according to the matching relationship between the feature points of the first image and the feature points of the second image; determining the first rotation matrix and translation vector of the camera motion according to the essential matrix or the basic matrix. In this example, the feature points extracted from the first image and the second image may be FAST (Features from Accelerated Segment Test) feature points, SIFT (Scale-Invariant Feature Transform) feature points or SURF (Speed-Up Robust Features) feature points, etc., which are not limited here. In this example, sparse optical flow tracking and other methods can be used to determine the matching relationship between the feature points of the first image and the feature points of the second image.

[0131] In this example, when the number of matching point pairs between the first image and the second image is greater than or equal to a first preset threshold, an essential matrix or a basic matrix from the first image to the second image may be determined. When the number of matching point pairs between the first image and the second image is less than the first preset threshold, the image may be recaptured. For example, the first preset threshold may be 5 or 10, etc.

[0132] In one example, the essential matrix (E matrix) from the first image to the second image can be determined based on the matching relationship between the feature points of the first image and the feature points of the second image, and the first rotation matrix and translation vector of the camera motion can be determined based on the essential matrix. In another example, the fundamental matrix (F matrix) from the first image to the second image can be determined based on the matching relationship between the feature points of the first image and the feature points of the second image, and the first rotation matrix and translation vector of the camera motion can be determined based on the fundamental matrix.

[0133] In this example, by extracting feature points of the first image and feature points of the second image, the essential matrix or basic matrix from the first image to the second image is determined based on the matching relationship between the feature points of the first image and the feature points of the second image, and the first rotation matrix and translation vector of the camera motion are determined based on the essential matrix or the basic matrix, thereby accurately determining the first rotation matrix and translation vector of the camera motion, thereby improving the accuracy of the three-dimensional coordinates of the determined spatial point in the camera coordinate system of the camera.

[0134] In a possible implementation, determining the normal vector of the target plane according to the three-dimensional coordinates of the multiple spatial points in the camera coordinate system includes: extracting multiple candidate planes according to the three-dimensional coordinates of the multiple spatial points in the camera coordinate system; and determining the normal vector of the target plane according to the normal vectors of the multiple candidate planes. In this implementation, the candidate plane may represent any plane extracted from the first image or the second image. Considering that at least three non-collinear points are required to represent a plane, in this implementation, a triangulation method may be used to extract multiple candidate planes from the first image or the second image according to the three-dimensional coordinates of the multiple spatial points in the camera coordinate system. For example, adjacent triangles may be divided on the image plane of the first image or the second image based on the Delaunay triangulation method, and these triangles may have corner points in the first image or the second image as vertices, wherein each corner point has a three-dimensional coordinate of the spatial point reconstructed based on it in the camera coordinate system. For any candidate plane among the multiple candidate planes, the normal vector of the candidate plane may be obtained according to the three-dimensional coordinates of the spatial points corresponding to the three vertices of the candidate plane in the camera coordinate system. In this implementation, the normal vector of the candidate plane may represent the normal vector of the candidate plane in the camera coordinate system, and the normal vector of the target plane may represent the normal vector of the target plane in the camera coordinate system. In this implementation, multiple candidate planes are extracted according to the three-dimensional coordinates of the multiple spatial points in the camera coordinate system, and the normal vector of the target plane is determined according to the normal vectors of the multiple candidate planes, thereby being able to effectively and accurately determine the normal vector of the target plane.

[0135] As an example of this implementation, the determining the normal vector of the target plane according to the normal vectors of the multiple candidate planes includes: screening out the potential plane of the target plane from the multiple candidate planes according to the direction of motion of the camera and the normal vectors of the multiple candidate planes; and determining the normal vector of the target plane according to the normal vector of the potential plane. In this example, the potential plane may represent a plane with a high probability of belonging to the target plane among the candidate planes. The normal vector of the potential plane may represent the normal vector of the potential plane in the camera coordinate system. According to the relationship between the target plane and the direction of motion of the camera in different application scenarios, the potential plane of the target plane may be screened out from the multiple candidate planes, thereby eliminating most of the candidate planes. By screening out the potential plane of the target plane from the multiple candidate planes according to the direction of motion of the camera and the normal vectors of the multiple candidate planes, and determining the normal vector of the target plane according to the normal vector of the potential plane, the accuracy of the normal vector of the determined target plane can be further improved, and it helps to increase the speed of determining the normal vector of the target plane.

[0136] In one example, the method of filtering out a potential plane of the target plane from the multiple candidate planes according to the direction of motion of the camera and the normal vectors of the multiple candidate planes includes: for any candidate plane among the multiple candidate planes, in response to the normal vector of the candidate plane being approximately perpendicular to the forward motion direction vector of the camera, determining the candidate plane as a potential plane of the target plane. In application scenarios such as camera calibration for autonomous driving, the normal vector of the target plane is approximately perpendicular to the direction of forward motion of the camera. Therefore, the potential plane can be filtered out from the candidate planes by judging whether the normal vector of the candidate plane is approximately perpendicular to the forward motion direction vector of the camera. The normal vector of the candidate plane is approximately perpendicular to the forward motion direction vector of the camera, which means that the angle between the normal vector of the candidate plane and the forward motion direction vector of the camera is less than or equal to a second preset threshold. For example, the second preset threshold can be 10°, 8°, 15°, etc. In this example, for any candidate plane among the multiple candidate planes, if the normal vector of the candidate plane is approximately perpendicular to the forward motion direction vector of the camera, the candidate plane can be determined as a potential plane of the target plane; if the normal vector of the candidate plane is not approximately perpendicular to the forward motion direction vector of the camera (for example, the angle between the normal vector of the candidate plane and the forward motion direction vector of the camera is greater than a second preset threshold), the candidate plane may not be determined as a potential plane of the target plane, that is, the candidate plane can be eliminated. In this example, by determining any candidate plane among the multiple candidate planes as a potential plane of the target plane in response to the normal vector of the candidate plane being approximately perpendicular to the forward motion direction vector of the camera, the potential plane of the target plane can be quickly and accurately screened out from a large number of candidate planes.

[0137] In one example, determining the normal vector of the target plane according to the normal vector of the potential plane includes: in the case where there are multiple potential planes, for any of the multiple potential planes, determining the support number of the potential plane according to the normal vector of the multiple potential planes, wherein the support number of the potential plane represents the number of potential planes in the multiple potential planes that are substantially parallel to the potential plane; and determining the normal vector of the target plane according to the support number of the multiple potential planes. For example, there are 20 potential planes, namely potential plane 1 to potential plane 20. For potential plane 1, if there are 6 potential planes substantially parallel to potential plane 1 among potential planes 2 to potential plane 20, the support number of potential plane 1 can be determined to be 6. For potential plane 12, if there are 2 potential planes substantially parallel to potential plane 12 among potential planes 1 to potential plane 11 and potential planes 13 to potential plane 20, the support number of potential plane 12 can be determined to be 2. In which, two potential planes are substantially parallel, which can be that the angle between the normal vectors of the two potential planes is less than or equal to the third preset threshold. That is, if the angle between the normal vectors of two potential planes is less than or equal to the third preset threshold, it can be determined that the two potential planes are approximately parallel; if the angle between the normal vectors of the two potential planes is greater than the third preset threshold, it can be determined that the two potential planes are not approximately parallel. For example, the third preset threshold may be 2°, 3°, 5°, and so on. In this example, when there are multiple potential planes, for any potential plane among the multiple potential planes, the support number of the potential plane is determined according to the normal vectors of the multiple potential planes, and the normal vector of the target plane is determined according to the support number of the multiple potential planes, thereby improving the accuracy of the normal vector of the determined target plane.

[0138] For example, determining the normal vector of the target plane according to the support number of the multiple potential planes includes: determining the normal vector of the potential plane with the largest support number among the multiple potential planes as the normal vector of the target plane. In this example, in order to obtain the normal vector of the target plane, it can be assumed that the planes belonging to the target plane occupy the majority among the potential planes. By determining the normal vector of the potential plane with the largest support number among the multiple potential planes as the normal vector of the target plane, the accuracy of the normal vector of the determined target plane can be improved, that is, the accuracy of the determined target plane can be improved, thereby improving the accuracy of the camera calibration result.

[0139] Figure 2 A schematic diagram of a potential plane in a camera calibration method provided by an embodiment of the present disclosure is shown. Figure 2In the example shown, among the multiple potential planes, potential plane 5's supporters include potential planes 0 to 4, potential plane 6, and potential plane 7. Potential plane 5 is the potential plane with the largest number of supporters among the multiple potential planes. Therefore, the normal vector of potential plane 5 can be determined as the normal vector of the target plane. Figure 2 In the example shown, the target plane is the road plane, which is Figure 2 It can be seen that potential planes 1 to 7 are all on the real road plane.

[0140] For another example, determining the normal vector of the target plane according to the support numbers of the multiple potential planes includes: determining the normal vector of any supporter of the potential plane with the largest support number among the multiple potential planes as the normal vector of the target plane. The supporter of any potential plane may represent a potential plane that is substantially parallel to the potential plane.

[0141] In another example, determining the normal vector of the target plane according to the normal vector of the potential plane includes: if there are multiple potential planes, determining a first average vector of the normal vectors of the multiple potential planes; and determining the normal vector of the multiple potential planes that has the smallest angle with the first average vector as the normal vector of the target plane. The first average vector represents the average vector of the normal vectors of the multiple potential planes.

[0142] In another example, determining the normal vector of the target plane based on the normal vector of the potential plane includes: in the case where there are multiple potential planes, determining a first average vector of the normal vectors of the multiple potential planes, and determining the first average vector as the normal vector of the target plane.

[0143] As another example of this implementation, determining the normal vector of the target plane according to the normal vectors of the multiple candidate planes includes: determining a second average vector of the normal vectors of the multiple candidate planes; and determining the normal vector of the multiple candidate planes that has the smallest angle with the second average vector as the normal vector of the target plane. The second average vector represents the average vector of the normal vectors of the multiple candidate planes.

[0144] As another example of this implementation, determining the normal vector of the target plane based on the normal vectors of the multiple candidate planes includes: determining a second average vector of the normal vectors of the multiple candidate planes, and determining the second average vector as the normal vector of the target plane.

[0145] In the disclosed embodiment, after obtaining the normal vector of the target plane, the attitude angle of the camera relative to the target plane can be determined according to the normal vector of the target plane. The attitude angle can be used to represent the horizontal attitude of the camera relative to the target plane. For example, the attitude angle may include the pitch angle and roll angle of the camera relative to the target plane. The roll angle of the camera relative to the target plane may represent the angle of rotation around the z-axis.

[0146] In the disclosed embodiment, the camera's intrinsic parameters, the camera's height relative to the target plane, and the attitude angle may be used to determine the camera-to-target plane homography matrix, wherein the camera-to-target plane homography matrix may represent a homography matrix from the image coordinate system corresponding to the camera to the world coordinate system corresponding to the target plane.

[0147] In a possible implementation, determining the homography matrix from the camera to the target plane based on the intrinsic parameter, the height and the attitude angle includes: determining a second rotation matrix for rotating the camera from a forward-looking state to a downward-looking state based on the attitude angle; converting a third image captured by the camera in the forward-looking state into a fourth image, wherein the fourth image is an image of the target plane viewed from above; acquiring image coordinates of a first image point in the third image and image coordinates of a second image point in the fourth image, wherein the first image point and the second image point are corresponding image points; and determining the homography matrix from the camera to the target plane based on the image coordinates of the first image point, the image coordinates of the second image point, the second rotation matrix, the intrinsic parameter and the height. Figure 3 A schematic diagram showing a camera in a forward-looking state and a camera in a downward-looking state in a camera calibration method provided by an embodiment of the present disclosure is shown. Figure 3 In , the origin of the camera coordinate system of the camera is o, n represents the normal vector of the target plane, h represents the height of the camera relative to the target plane (that is, the distance between the origin of the camera coordinate system and the target plane), Figure 3 The y-axis and z-axis of the camera coordinate system are also shown, R DF represents the second rotation matrix, wherein the second rotation matrix represents the rotation matrix of the camera from the front-view state to the top-view state. Figure 3 As shown, after obtaining the attitude angle, it can be assumed that the camera is rotated from the forward-looking state to the downward-looking state. After the rotation, the z-axis of the camera coordinate system of the camera is parallel to the normal vector of the target plane, and the optical center position of the camera remains unchanged.

[0148] In this implementation, the first image point represents a point in the third image, the second image point represents a point in the fourth image, the image coordinates of the first image point represent the coordinates of the first image point in the image coordinate system, and the image coordinates of the second image point represent the coordinates of the second image point in the image coordinate system. D The image coordinates x of the second image point F The corresponding relationship between them can be expressed by formula 1:

[0149] K -1 x D =λR DF K -1 x F Formula 1,

[0150] Wherein, K represents the intrinsic parameter of the camera, R DF represents the second rotation matrix, and λ represents the scale factor. λ is caused by the unknown depth of the camera coordinate system of the camera in the forward-looking state and the camera coordinate system of the camera in the downward-looking state. λ can be eliminated by aligning the secondary coordinates and normalizing them.

[0151] By adding x F Divide by K -1 Then multiply it by h to get the coordinates of the second image point normalized to h in the camera coordinate system. The homography matrix H from the camera to the target plane can be expressed as formula 2:

[0152] H=R DF K -1 Formula 2.

[0153] In the above implementation, the second rotation matrix of the camera rotating from the forward-looking state to the downward-looking state is determined according to the attitude angle, the third image captured by the camera in the forward-looking state is converted into a fourth image, the image coordinates of the first image point in the third image and the image coordinates of the second image point in the fourth image are obtained, and the homography matrix from the camera to the target plane is determined according to the image coordinates of the first image point, the image coordinates of the second image point, the second rotation matrix, the intrinsic parameters and the height, thereby achieving accurate camera calibration.

[0154] In a possible implementation, after determining the homography matrix from the camera to the target plane, the method further includes: obtaining the image coordinates of the target object from the fifth image captured by the camera; converting the image coordinates of the target object to the target plane according to the homography matrix to obtain the coordinates of the target object in the target plane; determining the verification result of the homography matrix according to the preset attribute information of the target object and the coordinates of the target object in the target plane, wherein the preset attribute information includes the preset shape information and / or preset size information of the target object. In this implementation, the target object can represent any object used to verify the homography matrix. For example, in the application scenario of autonomous driving, the target object can be a road sign, etc. In this implementation, a pre-trained neural network can be used to identify the target object in the fifth image to obtain the image coordinates of the target object in the fifth image. Alternatively, the fifth image can be compared with a preset image template of the target object to determine the image coordinates of the target object in the fifth image.

[0155] In this implementation, according to the homography matrix, the image coordinates of the target object can be converted to the target plane to obtain the coordinates of the target object in the target plane. Among them, the coordinates of the target object in the target plane can represent the coordinates of the target object in the world coordinate system corresponding to the target plane. Among them, the X-axis and Y-axis of the world coordinate system corresponding to the target plane can be on the target plane, and the Z-axis can be perpendicular to the target plane. As an example of this implementation, the coordinates of the target object in the target plane can include only the X-coordinate and the Y-coordinate. As another example of this implementation, the coordinates of the target object in the target plane can include the X-coordinate, the Y-coordinate and the Z-coordinate.

[0156] In this implementation, the preset attribute information may represent preset attribute information of the target object. As an example of this implementation, the preset attribute information includes preset shape information of the target object, wherein the preset shape information may represent preset shape information of the target object. As another example of this implementation, the preset attribute information includes preset size information of the target object, wherein the preset size information may represent preset size information of the target object. As another example of this implementation, the preset attribute information includes preset shape information and preset size information of the target object. Of course, the preset attribute information may also include other appearance attribute information of the preset target object, which is not limited here.

[0157] In this implementation, according to the preset shape information of the target object and the coordinates of the target object in the target plane, it can be determined whether the shape information of the target object is consistent with the preset shape information; and / or, according to the preset size information of the target object and the coordinates of the target object in the target plane, it can be determined whether the size information of the target object is consistent with the preset size information. In this implementation, by obtaining the image coordinates of the target object from the fifth image captured by the camera, according to the homography matrix, the image coordinates of the target object are converted to the target plane to obtain the coordinates of the target object in the target plane, and according to the preset attribute information including the preset shape information and / or preset size information of the target object, and the coordinates of the target object in the target plane, the verification result of the homography matrix is ​​determined, thereby the calibration result of the camera can be automatically and effectively verified, and it can be determined whether the current camera calibration result is reliable, which is more reliable than manual subjective verification, and there is no need to repeat the calibration process after manually discovering problems, thereby reducing manual participation and improving efficiency.

[0158] As an example of this implementation, the verification result of the homography matrix is ​​determined according to the preset attribute information of the target object and the coordinates of the target object on the target plane, including: determining the shape information and size information of the target object according to the coordinates of the target object on the target plane; in response to the shape information of the target object being consistent with the preset shape information, and the size information of the target object being consistent with the preset size information, determining that the homography matrix verification is successful; or, in response to the shape information of the target object not being consistent with the preset shape information, or the size information of the target object not being consistent with the preset size information, determining that the homography matrix verification fails. In this example, the success of the homography matrix verification needs to meet two conditions, namely, "the shape information of the target object is consistent with the preset shape information" and "the size information of the target object is consistent with the preset size information". If any of these two conditions is not met, it is determined that the homography matrix verification fails. In this example, the shape information and size information of the target object are determined according to the coordinates of the target object on the target plane, and in response to the shape information of the target object being consistent with the preset shape information, and the size information of the target object being consistent with the preset size information, it is determined that the homography matrix verification is successful; or, in response to the shape information of the target object being inconsistent with the preset shape information, or the size information of the target object being inconsistent with the preset size information, it is determined that the homography matrix verification has failed, thereby improving the accuracy of the verification result of the homography matrix.

[0159] Figure 4 A schematic diagram showing an image captured by a camera in a camera calibration method provided by an embodiment of the present disclosure is shown. Figure 4In the example shown, the target plane is the road plane. Figure 4 As shown in the figure, since the projection of the image is a perspective projection, parallel lines in the real world will intersect at the vanishing point in the image. If the homography matrix is ​​accurate, the parallel lines should continue to remain parallel after the points in the image are transformed to the road plane through the homography matrix. In one example, the target object can be a dashed line segment in the lane line. Figure 5 A schematic diagram showing the coordinates of the target object on the target plane in the camera calibration method provided by an embodiment of the present disclosure. Figure 5 As shown in the figure, straight lines can be extracted from the image. After dense sampling of the straight lines, the dashed line segments can be mapped to the world coordinate system corresponding to the road plane to obtain the X coordinate and Y coordinate of the dashed line segment on the road plane. According to the X coordinate and Y coordinate of the endpoint of the dashed line segment on the road plane, it can be detected whether the left and right line segments of the dashed line segment are parallel. In addition, it can also be verified whether the size of the dashed line segment is correct. For example, the distance between two parallel lines should be about 15 cm.

[0160] As another example of this implementation, determining the verification result of the homography matrix based on preset attribute information of the target object and the coordinates of the target object in the target plane includes: determining the shape information of the target object based on the coordinates of the target object in the target plane; in response to the shape information of the target object being consistent with the preset shape information, determining that the homography matrix verification is successful; or, in response to the shape information of the target object being inconsistent with the preset shape information, determining that the homography matrix verification fails.

[0161] As another example of this implementation, determining the verification result of the homography matrix based on preset attribute information of the target object and the coordinates of the target object in the target plane includes: determining the size information of the target object based on the coordinates of the target object in the target plane; in response to the size information of the target object being consistent with the preset size information, determining that the homography matrix verification is successful; or, in response to the size information of the target object being inconsistent with the preset size information, determining that the homography matrix verification fails.

[0162] In one possible implementation, in response to a verification result of the homography matrix being a verification failure, online calibration may be re-performed, thereby enabling real-time updating of the current calibration result, making the homography matrix from the camera to the target plane more adaptable to changes in the current scene and improving accuracy during online use.

[0163] The camera calibration method provided by the embodiment of the present disclosure can be applied to technical fields such as unmanned driving, monocular camera, automatic calibration, etc. For example, the camera calibration method provided by the embodiment of the present disclosure can be applied to the offline calibration of the homography matrix from the camera to the ground in the unmanned driving scene, and can also be applied to the online calibration of the homography matrix from the camera to the ground in the unmanned driving scene.

[0164] The camera calibration method provided by the embodiment of the present disclosure is described below through a specific application scenario. The application scenario may be an application scenario of autonomous driving. The camera may be a monocular camera installed in a vehicle cabin, and the target plane may be a road plane. Figure 6 A schematic diagram of an application scenario of the camera calibration method provided by an embodiment of the present disclosure is shown. In this application scenario, the pre-calibrated parameters include the camera's intrinsic parameter K and the height h of the camera relative to the road plane.

[0165] The images I0 and I1 captured by the camera can be obtained, and feature points of the images I0 and I1 can be extracted respectively to obtain feature points of the image I0 and feature points of the image I1. The sparse optical flow tracking method can be used to determine the matching relationship between the feature points of the image I0 and the image I1. According to the matching relationship between the feature points of the image I0 and the image I1, the essential matrix or basic matrix from the image I0 to the image I1 can be determined. According to the essential matrix or the basic matrix, the first rotation matrix and the translation vector of the camera motion can be determined. According to the first rotation matrix and the translation vector, 3D reconstruction can be performed to obtain the three-dimensional coordinates of multiple spatial points in the camera coordinate system of the camera.

[0166] According to the three-dimensional coordinates of the multiple spatial points in the camera coordinate system, multiple candidate planes can be extracted based on the Delaunay triangulation method. For any candidate plane among the multiple candidate planes, the candidate plane can be determined as a potential plane of the road plane in response to the normal vector of the candidate plane being approximately perpendicular to the forward motion direction vector of the camera. In the case of multiple potential planes, the support number of each potential plane can be determined separately, and the normal vector of the potential plane with the largest support number among the multiple potential planes can be determined as the normal vector of the road plane.

[0167] After obtaining the normal vector of the road plane, the pitch angle and roll angle of the camera relative to the road plane can be determined according to the normal vector of the road plane. According to the pitch angle and roll angle of the camera relative to the road plane, a second rotation matrix of the camera rotating from the front view state to the top view state can be obtained. The homography matrix from the camera to the road plane can be obtained by using the above formula 1 and formula 2.

[0168] After determining the homography matrix from the camera to the road plane, the verification result of the homography matrix can be obtained according to the shape and size of the dotted line segments in the lane lines in the image captured by the camera, as well as the preset shape information and preset size information of the dotted line segments.

[0169] According to this application scenario, it is possible to realize online automatic calibration of the homography matrix from the camera to the road plane in the application scenario of autonomous driving, and to update the current calibration result in real time, so that it can better adapt to changes in the autonomous driving scene, perceive changes in the ground, and improve the accuracy during online use.

[0170] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0171] In addition, the present disclosure also provides a camera calibration device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any camera calibration method provided by the present disclosure. The corresponding technical solutions and technical effects can be found in the corresponding records in the method part and will not be repeated here.

[0172] Figure 7 FIG. 1 is a block diagram of a camera calibration device provided by an embodiment of the present disclosure. Figure 7 As shown, the camera calibration device includes:

[0173] A first acquisition module 71, used to acquire the intrinsic parameters of the camera and the height of the camera relative to the target plane;

[0174] A first determining module 72, configured to determine the three-dimensional coordinates of a plurality of spatial points in a camera coordinate system of the camera according to the image captured by the camera;

[0175] A second determining module 73, configured to determine a normal vector of the target plane according to the three-dimensional coordinates of the plurality of spatial points in the camera coordinate system;

[0176] A third determination module 74 is used to determine the attitude angle of the camera relative to the target plane according to the normal vector of the target plane;

[0177] The fourth determination module 75 is used to determine the homography matrix from the camera to the target plane according to the intrinsic parameter, the height and the attitude angle.

[0178] In a possible implementation, the second determining module 73 is used to:

[0179] Extracting a plurality of candidate planes according to the three-dimensional coordinates of the plurality of spatial points in the camera coordinate system;

[0180] The normal vector of the target plane is determined according to the normal vectors of the multiple candidate planes.

[0181] In a possible implementation, the second determining module 73 is used to:

[0182] Filtering a potential plane of the target plane from the multiple candidate planes according to the movement direction of the camera and the normal vectors of the multiple candidate planes;

[0183] The normal vector of the target plane is determined according to the normal vector of the potential plane.

[0184] In a possible implementation, the second determining module 73 is used to:

[0185] For any candidate plane among the plurality of candidate planes, in response to a normal vector of the candidate plane being substantially perpendicular to a forward motion direction vector of the camera, the candidate plane is determined as a potential plane of the target plane.

[0186] In a possible implementation, the second determining module 73 is used to:

[0187] In the case where there are multiple potential planes, for any of the multiple potential planes, determining the support number of the potential plane according to the normal vectors of the multiple potential planes, wherein the support number of the potential plane represents the number of potential planes in the multiple potential planes that are substantially parallel to the potential plane;

[0188] A normal vector of the target plane is determined according to the support numbers of the multiple potential planes.

[0189] In a possible implementation, the second determining module 73 is used to:

[0190] The normal vector of the potential plane with the largest support number among the multiple potential planes is determined as the normal vector of the target plane.

[0191] In a possible implementation, the first determining module 72 is configured to:

[0192] Acquire a first image and a second image captured by the camera in a moving state;

[0193] Determine a first rotation matrix and a translation vector of the camera motion according to the first image and the second image;

[0194] The three-dimensional coordinates of a plurality of spatial points in the camera coordinate system of the camera are determined according to the first rotation matrix and the translation vector.

[0195] In a possible implementation, the first determining module 72 is configured to:

[0196] extracting feature points of the first image and feature points of the second image;

[0197] determining an essential matrix or a basic matrix from the first image to the second image according to a matching relationship between feature points of the first image and feature points of the second image;

[0198] A first rotation matrix and a translation vector of the camera motion are determined according to the essential matrix or the basic matrix.

[0199] In a possible implementation, the fourth determining module 75 is used to:

[0200] Determine, according to the attitude angle, a second rotation matrix for rotating the camera from a front-view state to a top-view state;

[0201] Converting the third image captured by the camera in the forward-looking state into a fourth image, wherein the fourth image is an image of looking down at the target plane;

[0202] Acquire image coordinates of a first image point in the third image and image coordinates of a second image point in the fourth image, wherein the first image point and the second image point are corresponding image points;

[0203] A homography matrix from the camera to the target plane is determined according to the image coordinates of the first image point, the image coordinates of the second image point, the second rotation matrix, the intrinsic parameter, and the height.

[0204] In a possible implementation manner, the device further includes:

[0205] A second acquisition module, used to acquire the image coordinates of the target object from the fifth image captured by the camera;

[0206] A conversion module, used for converting the image coordinates of the target object to the target plane according to the homography matrix to obtain the coordinates of the target object on the target plane;

[0207] The fifth determination module is used to determine the verification result of the homography matrix according to the preset attribute information of the target object and the coordinates of the target object on the target plane, wherein the preset attribute information includes preset shape information and / or preset size information of the target object.

[0208] In a possible implementation manner, the fifth determining module is used to:

[0209] Determining shape information and size information of the target object according to the coordinates of the target object on the target plane;

[0210] In response to the shape information of the target object being consistent with the preset shape information, and the size information of the target object being consistent with the preset size information, it is determined that the homography matrix verification is successful; or, in response to the shape information of the target object being inconsistent with the preset shape information, or the size information of the target object being inconsistent with the preset size information, it is determined that the homography matrix verification has failed.

[0211] In a possible implementation, the camera is installed in a vehicle cabin, and the target plane is a road plane.

[0212] In the embodiment of the present disclosure, by obtaining the intrinsic parameters of the camera and the height of the camera relative to the target plane, the three-dimensional coordinates of multiple spatial points in the camera coordinate system of the camera are determined according to the image captured by the camera, the normal vector of the target plane is determined according to the three-dimensional coordinates of the multiple spatial points in the camera coordinate system, the attitude angle of the camera relative to the target plane is determined according to the normal vector of the target plane, and the homography matrix from the camera to the target plane is determined according to the intrinsic parameters, the height and the attitude angle, thereby only needing to obtain the intrinsic parameters of the camera and the height of the camera relative to the target plane, the calibration of the homography matrix from the camera to the target plane can be achieved, so that the camera calibration result is more adaptable to the changes in the scene and the accuracy during online use is improved. In addition, by adopting the camera calibration device provided in the embodiment of the present disclosure, the problem of manual dependence in the calibration process can be solved, the calibration process can be automated, and no manual participation is required, thereby reducing the probability of errors. Furthermore, the problem of dependence on artificial reference objects can be solved, and it is no longer dependent on special calibration scenes, and can be applicable to a wider range of application scenes.

[0213] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. Its specific implementation and technical effects can refer to the description of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0214] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium, or may be a volatile computer-readable storage medium.

[0215] The embodiment of the present disclosure further provides a computer program, including a computer-readable code. When the computer-readable code is executed in an electronic device, a processor in the electronic device executes the above method.

[0216] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes the above method.

[0217] An embodiment of the present disclosure also provides an electronic device, comprising: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above method.

[0218] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0219] Figure 8 A block diagram of an electronic device 800 provided by an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0220] Reference Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0221] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0222] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0223] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0224] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0225] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0226] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0227] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0228] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (Wi-Fi), a second generation mobile communication technology (2G), a third generation mobile communication technology (3G), a fourth generation mobile communication technology (4G), a long-term evolution (LTE) of a universal mobile communication technology, a fifth generation mobile communication technology (5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0229] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0230] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0231] Fig. 9 FIG. 1 is a block diagram of an electronic device 1900 provided by an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. Fig. 9 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0232] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0233] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0234] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0235] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0236] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0237] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0238] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0239] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0240] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0241] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0242] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0243] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A camera calibration method, characterized in that: include: Obtaining the intrinsic parameters of the camera and the height of the camera relative to the target plane; Determining three-dimensional coordinates of a plurality of spatial points in a camera coordinate system of the camera according to the image captured by the camera; Determining a normal vector of the target plane according to the three-dimensional coordinates of the multiple spatial points in the camera coordinate system; Determining the attitude angle of the camera relative to the target plane according to the normal vector of the target plane; Determine, according to the attitude angle, a second rotation matrix for rotating the camera from a front-view state to a top-view state; Converting the third image captured by the camera in the forward-looking state into a fourth image, wherein the fourth image is an image of looking down at the target plane; Acquire image coordinates of a first image point in the third image and image coordinates of a second image point in the fourth image, wherein the first image point represents a point in the third image, the second image point represents a point in the fourth image, the first image point and the second image point are corresponding image points, the image coordinates of the first image point represent coordinates of the first image point in an image coordinate system, and the image coordinates of the second image point represent coordinates of the second image point in an image coordinate system; Determine a homography matrix from the camera to the target plane, wherein the homography matrix is ​​obtained by multiplying the second rotation matrix by an inverse matrix of the intrinsic parameter.

2. The method according to claim 1, characterized in that The step of determining the normal vector of the target plane according to the three-dimensional coordinates of the plurality of spatial points in the camera coordinate system comprises: Extracting a plurality of candidate planes according to the three-dimensional coordinates of the plurality of spatial points in the camera coordinate system; The normal vector of the target plane is determined according to the normal vectors of the multiple candidate planes.

3. The method according to claim 2, characterized in that The step of determining the normal vector of the target plane according to the normal vectors of the plurality of candidate planes comprises: Filtering a potential plane of the target plane from the multiple candidate planes according to the movement direction of the camera and the normal vectors of the multiple candidate planes; The normal vector of the target plane is determined according to the normal vector of the potential plane.

4. The method according to claim 3, characterized in that The step of selecting a potential plane of the target plane from the multiple candidate planes according to the movement direction of the camera and the normal vectors of the multiple candidate planes includes: For any candidate plane among the plurality of candidate planes, in response to a normal vector of the candidate plane being substantially perpendicular to a forward motion direction vector of the camera, the candidate plane is determined as a potential plane of the target plane.

5. The method according to claim 3, characterized in that: The step of determining the normal vector of the target plane according to the normal vector of the potential plane comprises: In the case where there are multiple potential planes, for any of the multiple potential planes, determining the support number of the potential plane according to the normal vectors of the multiple potential planes, wherein the support number of the potential plane represents the number of potential planes in the multiple potential planes that are substantially parallel to the potential plane; A normal vector of the target plane is determined according to the support numbers of the multiple potential planes.

6. The method according to claim 5, characterized in that The step of determining the normal vector of the target plane according to the support numbers of the multiple potential planes includes: The normal vector of the potential plane with the largest support number among the multiple potential planes is determined as the normal vector of the target plane.

7. The method according to any one of claims 1 to 6, characterized in that: The step of determining the three-dimensional coordinates of a plurality of spatial points in a camera coordinate system of the camera according to the image captured by the camera comprises: Acquire a first image and a second image captured by the camera in a moving state; Determine a first rotation matrix and a translation vector of the camera motion according to the first image and the second image; The three-dimensional coordinates of a plurality of spatial points in the camera coordinate system of the camera are determined according to the first rotation matrix and the translation vector.

8. The method according to claim 7, characterized in that Determining a first rotation matrix and a translation vector of the camera motion according to the first image and the second image includes: extracting feature points of the first image and feature points of the second image; determining an essential matrix or a basic matrix from the first image to the second image according to a matching relationship between feature points of the first image and feature points of the second image; A first rotation matrix and a translation vector of the camera motion are determined according to the essential matrix or the basic matrix.

9. The method according to any one of claims 1 to 6, characterized in that: After determining the homography matrix from the camera to the target plane, the method further includes: Acquire image coordinates of a target object from a fifth image captured by the camera; According to the homography matrix, the image coordinates of the target object are converted to the target plane to obtain the coordinates of the target object on the target plane; The verification result of the homography matrix is ​​determined according to preset attribute information of the target object and the coordinates of the target object on the target plane, wherein the preset attribute information includes preset shape information and / or preset size information of the target object.

10. The method according to claim 9, characterized in that Determining the verification result of the homography matrix according to the preset attribute information of the target object and the coordinates of the target object on the target plane includes: Determining shape information and size information of the target object according to the coordinates of the target object on the target plane; In response to the shape information of the target object being consistent with the preset shape information, and the size information of the target object being consistent with the preset size information, it is determined that the homography matrix verification is successful; or, in response to the shape information of the target object being inconsistent with the preset shape information, or the size information of the target object being inconsistent with the preset size information, it is determined that the homography matrix verification has failed.

11. The method according to any one of claims 1 to 6, characterized in that: The camera is installed in the vehicle cabin, and the target plane is the road plane.

12. A camera calibration device, characterized in that: include: A first acquisition module, used to acquire the intrinsic parameters of the camera and the height of the camera relative to the target plane; A first determination module, used to determine the three-dimensional coordinates of a plurality of spatial points in a camera coordinate system of the camera according to the image captured by the camera; A second determination module, configured to determine a normal vector of the target plane according to the three-dimensional coordinates of the plurality of spatial points in the camera coordinate system; A third determination module, used to determine the attitude angle of the camera relative to the target plane according to the normal vector of the target plane; A fourth determination module is used to determine, based on the attitude angle, a second rotation matrix of the camera rotating from a forward-looking state to a downward-looking state; convert a third image captured by the camera in the forward-looking state into a fourth image, wherein the fourth image is an image of the target plane viewed from the downward direction; obtain image coordinates of a first image point in the third image and image coordinates of a second image point in the fourth image, wherein the first image point represents a point in the third image, the second image point represents a point in the fourth image, the first image point and the second image point are corresponding image points, the image coordinates of the first image point represent the coordinates of the first image point in the image coordinate system, and the image coordinates of the second image point represent the coordinates of the second image point in the image coordinate system; determine a homography matrix from the camera to the target plane, wherein the homography matrix is ​​obtained by multiplying the second rotation matrix by the inverse matrix of the intrinsic parameter.

13. An electronic device, characterized in that: include: one or more processors; a memory for storing executable instructions; The one or more processors are configured to call the executable instructions stored in the memory to execute the method described in any one of claims 1 to 11.

14. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images