Camera extrinsic parameter calibration method, device, equipment, storage medium and system

Abstract

The present application provides a camera extrinsic calibration method, apparatus, device, storage medium, and system. In one example, the method includes: for any calibration site, using the onboard surround-view camera of the calibration vehicle to capture a reference frame image of the calibration site, and generating reference frame information corresponding to the reference frame image based on detected annotation instructions; for any vehicle to be calibrated, based on the current calibration site, obtaining reference frame information of the same calibration site, and performing extrinsic calibration on the onboard surround-view camera of the vehicle to be calibrated based on the obtained reference frame information. This method can improve the versatility of the calibration solution while ensuring calibration accuracy.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a camera extrinsic parameter calibration method, device, equipment, storage medium and system. Background Art

[0002] External calibration of automotive surround view cameras is typically performed after a vehicle rolls off the production line at the factory. This is often referred to as end-of-line calibration. Once the camera's intrinsic parameters and the physical dimensions of the target on the calibration site are known, corner detection technology is used to locate the target's corners in the image, completing the camera's external calibration and supporting intelligent driving.

[0003] However, due to the large number of suppliers of surround view extrinsic parameter calibration algorithms on the market, the calibration site patterns built by different car manufacturers are diverse. Therefore, when supporting mass production projects of different models, new calibration site patterns will always be encountered.

[0004] How to respond to the calibration needs of different sites while ensuring high precision of external parameter calibration has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the present application provides a camera extrinsic parameter calibration method, apparatus, device, storage medium and system.

[0006] Specifically, this application is implemented through the following technical solutions:

[0007] According to a first aspect of an embodiment of the present application, a camera extrinsic parameter calibration method is provided, comprising:

[0008] For any calibration site, if it is determined that no reference frame information associated with the calibration site is stored, reference frame images of the calibration site are collected using the on-board surround-view camera of the calibration vehicle; wherein the calibration site is characterized by a target laying scheme, the target laying scheme includes a target pattern and the number of targets, the same target laying scheme uses the same target pattern, and the number of targets in the same view of the on-board surround-view camera is the same;

[0009] Generate reference frame information corresponding to the reference frame image based on the detected annotation instruction, and associate the calibration site with the reference frame information and store it; wherein the reference frame information includes:

[0010] Identification information of designated target corner points in the reference frame image;

[0011] The physical coordinates of the designated target corner points; and

[0012] Position information of the designated target corner points in the reference frame image; wherein the number of the designated target corner points is greater than or equal to 4, and all of the designated target corner points are located in the common viewing area of ​​the adjacent on-board surround view cameras of the calibrated vehicle in the reference frame image and are not collinear;

[0013] For any vehicle to be calibrated, the reference frame information of the same calibration site is obtained based on the current calibration site, and the extrinsic parameter calibration of the surround view camera on the vehicle to be calibrated is performed based on the obtained reference frame information.

[0014] According to a second aspect of an embodiment of the present application, a camera extrinsic parameter calibration device is provided, comprising:

[0015] A generating unit is configured to, for any calibration site, if it is determined that no reference frame information associated with the calibration site is stored, acquire a reference frame image of the calibration site through the on-board surround-view camera of the calibration vehicle, generate reference frame information corresponding to the reference frame image based on the detected labeling instruction, and store the calibration site in association with the reference frame information; wherein the calibration site is characterized by a target laying scheme, the target laying scheme includes a target pattern and the number of targets, the same target laying scheme uses the same target pattern, and the number of targets in the same view of the on-board surround-view camera is the same; the reference frame information includes identification information of designated target corner points in the reference frame image, the physical coordinates of the designated target corner points, and position information of the designated target corner points in the reference frame image; the number of the designated target corner points is greater than or equal to 4, all of which are located in the common viewing area of ​​the adjacent on-board surround-view cameras of the calibration vehicle in the reference frame image and are not collinear;

[0016] The processing unit is used to obtain the reference frame information of the same calibration site for any vehicle to be calibrated based on the current calibration site, and perform external parameter calibration on the surround view camera on the vehicle to be calibrated based on the obtained reference frame information.

[0017] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor is used to execute the machine-executable instructions to implement the method provided in the first aspect.

[0018] According to a fourth aspect of an embodiment of the present application, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0019] According to a fifth aspect of an embodiment of the present application, a camera extrinsic calibration system is provided, comprising: a processor, a memory, and a display; wherein:

[0020] The memory is used to store the association relationship between the calibration site and the reference frame information;

[0021] The processor is configured to, for any calibration site, if it is determined that no reference frame information associated with the calibration site is stored in the memory, acquire a reference frame image of the calibration site using a surround-view camera on a calibration vehicle; wherein the calibration site is characterized by a target placement scheme, the target placement scheme including a target pattern and a number of targets, the same target placement scheme using the same target pattern, and the number of targets in images of the same view of the surround-view camera on the vehicle being calibrated;

[0022] The display is used to display the reference frame image;

[0023] The processor is further configured to generate reference frame information corresponding to the reference frame image based on the detected annotation instruction for the reference frame image displayed on the display; wherein the reference frame information includes:

[0024] Identification information of designated target corner points in the reference frame image;

[0025] The physical coordinates of the designated target corner points; and

[0026] Position information of the designated target corner points in the reference frame image; wherein the number of the designated target corner points is greater than or equal to 4, and all of the designated target corner points are located in the common viewing area of ​​the adjacent on-board surround view cameras of the calibrated vehicle in the reference frame image and are not collinear;

[0027] The memory is further used to associate and store the calibration site with the reference frame information;

[0028] The processor is further configured to obtain, for any vehicle to be calibrated, reference frame information of the same calibration site from the memory based on the current calibration site, and perform external parameter calibration on the surround view camera on the vehicle to be calibrated based on the obtained reference frame information.

[0029] The technical solution provided by this application can at least bring the following beneficial effects:

[0030] By collecting reference frame images for the calibration site and generating reference frame information corresponding to the reference frame images based on the detected annotation instructions, during the process of extrinsic parameter calibration of the camera of the vehicle to be calibrated, the reference frame information of the same calibration site can be obtained based on the current calibration site, and the extrinsic parameter calibration of the surround-view camera on the vehicle to be calibrated can be performed based on the obtained reference frame information. By setting the reference frame information for the calibration site, when there is reference frame information of the same calibration site, there is no need to re-upgrade the calibration algorithm for changes in physical environment, camera model, vehicle model and other parameters, which can improve the versatility of the calibration scheme while ensuring the accuracy of calibration. In addition, when the calibration site changes, only the reference frame information needs to be updated to support the camera extrinsic parameter calibration of the new calibration site, which improves the scalability of the scheme and effectively reduces the development cost of expanding the calibration site. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 1 is a flow chart of a camera extrinsic calibration method shown in an exemplary embodiment of the present application;

[0032] Figure 2A is a schematic diagram of a vehicle with four onboard surround-view cameras, shown in an exemplary embodiment of the present application;

[0033] Figure 2B is a schematic diagram of a vehicle with six onboard surround-view cameras, shown in an exemplary embodiment of the present application;

[0034] Figure 3 1 is a flow chart of a camera extrinsic calibration method shown in an exemplary embodiment of the present application;

[0035] Figure 4 is a schematic diagram of a calibration system device shown in an exemplary embodiment of the present application;

[0036] Figure 5 This is a schematic diagram of a specific implementation process of camera extrinsic calibration shown in an exemplary embodiment of the present application;

[0037] Figure 6 is a schematic diagram of an image to be calibrated shown in an exemplary embodiment of the present application;

[0038] Figure 7 is a schematic diagram of a reference frame image shown in an exemplary embodiment of the present application;

[0039] Figure 8 is a schematic diagram of matching between an image to be calibrated and a reference frame image shown in an exemplary embodiment of the present application;

[0040] Figure 9 is a schematic diagram of matching between an image to be calibrated and a reference frame image shown in an exemplary embodiment of the present application;

[0041] Figure 10 1 is a schematic diagram of matching between an image to be calibrated and a reference frame image after feature point screening, as shown in an exemplary embodiment of the present application;

[0042] Figure 11 1 is a schematic structural diagram of a camera extrinsic calibration device shown in an exemplary embodiment of the present application;

[0043] Figure 12 1 is a schematic structural diagram of a camera extrinsic calibration device shown in an exemplary embodiment of the present application;

[0044] Figure 13 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0046] It should be noted that the serial numbers of the steps in the embodiments of the present application do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0047] See Figure 1 , is a flow chart of a camera extrinsic parameter calibration method provided in an embodiment of the present application, such as Figure 1 As shown, the camera extrinsic parameter calibration method may include the following steps:

[0048] Step S100: For any calibration site, if it is determined that no reference frame information associated with the calibration site is stored, reference frame images of the calibration site are collected through the on-board surround-view camera of the calibration vehicle; wherein the calibration site is characterized by a target laying scheme, and the target laying scheme includes a target pattern and the number of targets. The same target laying scheme uses the same target pattern, and the number of targets in the same view of the on-board surround-view camera is the same.

[0049] Step S110: Generate reference frame information corresponding to the reference frame image based on the detected annotation instruction, and associate the calibration site with the reference frame information and store it; wherein the reference frame information includes identification information of designated target corner points in the reference frame image, physical coordinates of designated target corner points, and position information of designated target corner points in the reference frame image; the number of designated target corner points is greater than or equal to 4, all of which are located in the common viewing area of ​​adjacent on-board surround-view cameras of the calibration vehicle in the reference frame image and are not collinear.

[0050] Step S120: For any vehicle to be calibrated, obtain reference frame information of the same calibration site based on the current calibration site, and perform extrinsic calibration on the surround view camera on the vehicle to be calibrated based on the obtained reference frame information.

[0051] In an embodiment of the present application, the vehicle-mounted surround view camera includes a vehicle-mounted camera installed on the vehicle body for obtaining the surrounding environment of the vehicle body to achieve 360° surround view.

[0052] For example, for any vehicle equipped with onboard surround view cameras, there are usually multiple onboard surround view cameras.

[0053] For example, the number of onboard surround view cameras may be 4 or 6.

[0054] Taking the number of on-board surround view cameras as 4 as an example, the on-board surround view cameras generally include on-board cameras deployed in front of the vehicle, behind the vehicle, on the left side of the vehicle, and on the right side of the vehicle (which can be respectively referred to as the front on-board camera, the rear on-board camera, the left on-board camera, and the right on-board camera). The schematic diagram can be as follows: Figure 2A shown.

[0055] Taking the number of on-board surround-view cameras as 6 as an example, the on-board surround-view cameras generally include on-board cameras deployed in front of the vehicle, behind the vehicle, on the left front side of the vehicle, on the left rear side of the vehicle, on the right front side of the vehicle, and on the right rear side of the vehicle (which can be respectively referred to as the front on-board camera, the rear on-board camera, the left front on-board camera, the left rear on-board camera, the right front on-board camera, and the right rear on-board camera). The schematic diagram can be shown as follows: Figure 2B shown.

[0056] In an embodiment of the present application, in order to improve the versatility of the camera extrinsic parameter calibration scheme, reference frame information can be set for different calibration sites. Thus, in the process of performing extrinsic parameter calibration on the on-board surround-view camera of the vehicle to be calibrated (which can be referred to as the on-board surround-view camera to be calibrated), the corresponding reference frame information can be determined based on the actual calibration site, and the on-board surround-view camera to be calibrated can be extrinsically calibrated based on the reference frame information.

[0057] Accordingly, for any calibration site, the association relationship between the stored calibration site and the reference frame information can be queried to determine whether the reference frame information associated with the calibration site is stored. If it is determined that the reference frame information associated with the calibration site is not stored, the associated reference frame information is generated for the calibration site.

[0058] For example, if a factory or 4S store needs to add a new vehicle surround view camera external parameter calibration service, the relevant staff can submit a request for a new calibration site based on the actual target layout plan.

[0059] When receiving the request to add a new calibration site, the system can determine whether reference frame information associated with the calibration site is stored, and if it is determined that the reference frame information associated with the calibration site is not stored, generate associated reference frame information for the calibration site.

[0060] For example, the calibration site can be characterized by a target laying scheme, where the same target laying scheme corresponds to the same calibration site, and different target laying schemes correspond to different calibration sites.

[0061] That is, in the embodiment of the present application, for different physical environments, different camera models, and different car models, the same reference frame information can be used to calibrate the camera extrinsic parameters when the target laying scheme is the same.

[0062] For example, the target placement scheme may include a target pattern and a target quantity. The same target placement scheme uses the same target pattern, and the same number of targets is used in images of the same view of the onboard surround view camera.

[0063] Exemplarily, images of the same view of the vehicle-mounted surround-view cameras refer to images collected by vehicle-mounted surround-view cameras installed in the same orientation.

[0064] For example, if there are four onboard surround-view cameras, the views of the onboard surround-view cameras include a front view, a rear view, a left view, and a right view.

[0065] In the case where the number of the onboard surround view cameras is 6, the onboard surround view camera views include a front view, a rear view, a left front view, a left rear view, a right front view, and a right rear view.

[0066] Taking the example of four on-board surround-view cameras, assuming that both target layout plan 1 and target layout plan 2 use checkerboard targets, and that the number of targets in the front view / rear view / left view / right view images of the on-board surround-view cameras under target layout plan 1 and target layout plan 2 is the same, then target layout plan 1 and target layout plan 2 are the same.

[0067] In one example, for the images to be calibrated of the first calibration site acquired by the surround-view camera on the vehicle to be calibrated, and the reference frame images of the second calibration site, when the target pattern and the number of targets in each image to be calibrated are the same as the target pattern and the number of targets in the reference frame images of the same view, the first calibration site and the second calibration site are determined to be the same calibration site.

[0068] In an embodiment of the present application, the reference frame information may include identification information (such as a serial number) of a specified target corner point in the reference frame image, and position information (such as image coordinates) of the specified target corner point in the reference frame image.

[0069] The designated target corner points are target corner points marked in the reference frame image.

[0070] Exemplarily, for any calibration site, the image of the calibration site (i.e., the reference frame image) can be collected by the on-board surround-view camera of any vehicle (which can be called the annotation vehicle), and according to the detected annotation instructions, the target corner points of the reference frame image of each view are annotated respectively (the annotated target corner points are recorded as the above-mentioned designated target corner points), the reference frame information of the calibration site is generated, and the calibration site and the reference frame information are associated and stored.

[0071] For example, the marked designated target corner points may be selected to be within a common viewing area of ​​adjacent on-board surround view cameras of the calibrated vehicle in the reference frame image.

[0072] For example, taking the number of on-board surround-view cameras as 4, the front on-board camera can have a common viewing area with the left on-board camera and the right on-board camera respectively. For the reference frame image captured by the front on-board camera, the specified target corner points can select the common viewing area of ​​the front on-board camera and the left on-board camera, and / or the target corner points in the common viewing area of ​​the front on-board camera and the right on-board camera.

[0073] Exemplarily, the marking of the target corner point may include marking an identifier of the designated target corner point, the physical coordinates of the designated target corner point, and the position information of the designated target corner point.

[0074] The physical coordinates of the target corner points may include the physical coordinates of the target corner points in the target world coordinate system.

[0075] Exemplarily, the target world coordinate system (also referred to as the calibration site world coordinate system) may include a world coordinate system established with the center of the calibration site as the coordinate origin.

[0076] For example, the X / Y axes of the world coordinate system are usually parallel to the main direction of the field (such as the edge of the wall or the direction of the target rows and columns), and the Z axis is perpendicular to the ground.

[0077] It should be noted that when a large target (such as an extra-large chessboard) is used in the calibration site, the center of the target can be used as the center of the calibration site.

[0078] When the calibration site uses a plurality of dispersed small targets, the center point of the calibration site may be the geometric center of these targets.

[0079] In addition, the center of the calibration site can also be manually marked by setting a fixed marking point in the center of the site using high-precision measurement equipment (such as a total station or laser tracker).

[0080] Exemplarily, for any reference frame image, the number of marked designated target corner points is greater than or equal to 4, and the marked designated target corner points are not collinear (ie, there is no straight line passing through all designated target corner points in the image plane).

[0081] In an embodiment of the present application, for any vehicle to be calibrated, reference frame information of the same calibration site can be obtained based on the current calibration site, and external parameter calibration can be performed on the surround view camera on the vehicle to be calibrated based on the obtained reference frame information.

[0082] For example, in the process of calibrating the camera extrinsic parameters of the vehicle to be calibrated, on the one hand, the on-board surround-view camera of the vehicle to be calibrated (which can be referred to as the on-board surround-view camera to be calibrated) can be used to capture images of the current calibration scene to obtain images to be calibrated; on the other hand, based on the current calibration scene, reference frame information of the same calibration site can be obtained, and based on the obtained images to be calibrated and the reference frame information of the same calibration site, the on-board surround-view camera to be calibrated can be calibrated for extrinsic parameters to determine the extrinsic parameters of the on-board surround-view camera to be calibrated.

[0083] It should be noted that in an embodiment of the present application, during the process of calibrating the camera extrinsic parameters of any vehicle to be calibrated, when it is determined that there is no reference frame information of the same calibration site based on the current calibration site, the reference frame information of the current calibration site can also be generated through real-time labeling. Then, based on the reference frame information of the current calibration site, the extrinsic parameters of the on-board surround view camera of the vehicle to be calibrated can be calibrated.

[0084] It can be seen that in Figure 1 In the illustrated method flow, by capturing reference frame images for the calibration site and generating reference frame information corresponding to the reference frame images based on the detected annotation instructions, during the process of extrinsic calibration of the camera of the vehicle to be calibrated, the reference frame information of the same calibration site can be obtained based on the current calibration site, and the extrinsic calibration of the surround-view camera on the vehicle to be calibrated can be performed based on the obtained reference frame information. By setting the reference frame information for the calibration site, if there is reference frame information for the same calibration site, there is no need to re-upgrade the calibration algorithm for changes in other parameters such as the physical environment, camera model, and vehicle model. This can improve the versatility of the calibration scheme while ensuring calibration accuracy. In addition, if the calibration site changes, only the reference frame information needs to be updated to support the camera extrinsic calibration of the new calibration site, which improves the scalability of the scheme and effectively reduces the development cost of expanding the calibration site.

[0085] The following describes the implementation of extrinsic calibration of the on-board surround view camera to be calibrated based on the reference frame information.

[0086] See Figure 3, is a flow chart of a camera extrinsic parameter calibration method provided in an embodiment of the present application, such as Figure 3 As shown, the camera extrinsic parameter calibration method may include the following steps:

[0087] Step S300: Acquire the image to be calibrated; wherein, the image to be calibrated is obtained by collecting images of the calibration site by the surround-view camera on the vehicle to be calibrated.

[0088] In an embodiment of the present application, for any vehicle to be calibrated, during the process of calibrating the camera extrinsic parameters of the vehicle to be calibrated, the surround-view camera on the vehicle to be calibrated can be used to capture images of the calibration site to obtain images to be calibrated.

[0089] Step S310: Determine the designated target corner point information in the image to be calibrated based on the reference frame information of the same calibration site; wherein the designated target corner point information includes the identification information of the designated target corner point in the image to be calibrated, the physical coordinates of the designated target corner point, and the position information of the designated target corner point in the image to be calibrated.

[0090] In an embodiment of the present application, when the image to be calibrated is obtained, the reference frame information of the same calibration site can be obtained based on the calibration site where the image to be calibrated is collected, and the designated target corner point information in the image to be calibrated can be determined based on the obtained reference frame information.

[0091] Exemplarily, for images to be calibrated of different views, the reference frame information of the same calibration site obtained contains reference frame information of the corresponding view.

[0092] The target corner information of the image to be calibrated of the view can be determined based on the reference frame information of the same view.

[0093] It should be noted that in the embodiment of the present application, for any calibration site, when it is determined that there is currently no reference frame information for the same calibration site, the reference frame information of the calibration site can also be generated through real-time annotation. Then, based on the reference frame information of the calibration site, the external parameters of the on-board surround view camera of the vehicle to be calibrated at the calibration site are calibrated.

[0094] Step S320: Determine the extrinsic parameters of the vehicle-mounted surround view camera to be calibrated based on the designated target corner point information in the image to be calibrated and the intrinsic parameters of the vehicle-mounted surround view camera to be calibrated.

[0095] In this embodiment of the present application, after determining the designated target corner points in the image to be calibrated as described above, the physical coordinates of the designated target corner points are also annotated during the reference frame calibration process, meaning that the physical coordinates of the designated target corner points are known. Therefore, the extrinsic parameters of the onboard camera to be calibrated can be determined based on the physical coordinates of each designated target corner point, the positional information of the designated target corner point in the image to be calibrated, and the intrinsic parameters of the onboard surround view camera to be calibrated.

[0096] It should be noted that in the embodiment of the present application, since the reference frame information is marked in the target world coordinate system, the external parameters of the vehicle-mounted camera to be calibrated are directly determined based on the physical coordinates of each designated target corner point, the position information of the designated target corner point in the image to be calibrated, and the internal parameters of the vehicle-mounted surround-view camera to be calibrated. They are the external parameters in the target world coordinate system, and can be further converted into external parameters in the vehicle body world coordinate system.

[0097] For example, when the coordinate origin of the vehicle body world coordinate system coincides with the coordinate origin of the target world coordinate system (a tolerable deviation may be allowed), for example, in a factory calibration environment, since the factory has a centering device, the vehicle is generally parked to ensure that the center of the vehicle body coincides with the center of the target world coordinate system. In this case, the external parameters under the target world coordinate system are usually consistent with the external parameters under the vehicle body world coordinate system.

[0098] When the coordinate origin of the vehicle body world coordinate system does not coincide with the coordinate origin of the target world coordinate system, the offset between the center of the vehicle body and the center of the target world coordinate system can be obtained. For example, the offset between the center of the vehicle body and the center of the target world coordinate system can be obtained by measurement. Then, based on the offset between the center of the vehicle body and the center of the target world coordinate system, the external parameters in the target world coordinate system can be converted into external parameters in the vehicle body world coordinate system.

[0099] It can be seen that in Figure 3In the illustrated method flow, by pre-setting corresponding reference frame information for different calibration sites, during the extrinsic parameter calibration of the on-board surround-view camera, for the image to be calibrated obtained by capturing the calibration site through the on-board surround-view camera to be calibrated, the designated target corner point information in the image to be calibrated can be determined based on the reference frame information of the same calibration site. Furthermore, the extrinsic parameters of the on-board surround-view camera to be calibrated can be determined based on the designated target corner point information in the image to be calibrated and the intrinsic parameters of the on-board surround-view camera to be calibrated. By setting the reference frame information for the calibration site, if the reference frame information of the same calibration site exists, the calibration algorithm does not need to be re-updated for changes in other parameters such as the physical environment, camera model, and vehicle model. This can improve the versatility of the calibration scheme while ensuring calibration accuracy. In addition, if the calibration site changes, only the reference frame information needs to be updated to support the camera extrinsic parameter calibration of the new calibration site, thereby improving the scalability of the scheme and effectively reducing the development cost of expanding the calibration site.

[0100] In some embodiments, the determining of the designated target corner point information in the image to be calibrated based on the reference frame information of the same calibration site may include:

[0101] For any image to be calibrated, corner point matching is performed between the image to be calibrated and the reference frame image according to the reference frame image of the corresponding view in the same calibration site to determine the designated target corner point information in the image to be calibrated.

[0102] For example, the designated target corner points in the image to be calibrated can be determined based on the designated target corner points marked in the reference frame image by performing corner point matching between the image to be calibrated and the reference frame image in the same calibration field.

[0103] Accordingly, for any image to be calibrated, the image to be calibrated and the reference frame image can be matched with each other based on the reference frame image of the corresponding view in the same calibration site to determine the designated target corner points in the image to be calibrated, and then the designated target corner point information in the image to be calibrated can be determined, that is, the identification information of the designated target corner points in the image to be calibrated and the position information of the designated target corner points in the image to be calibrated can be determined.

[0104] For example, taking the number of on-board surround-view cameras as 4, for the front view image to be calibrated (i.e., the image to be calibrated captured by the front on-board camera), the front view reference frame image in the same calibration site (the image captured by the front on-board camera and used to mark the designated target corner points during the reference frame calibration process) can be used to perform corner matching on the front view image to be calibrated, determine the designated target corner points in the front view image to be calibrated, and then determine the designated target corner point information in the front view image to be calibrated.

[0105] Assuming that the front view reference frame image includes designated target corner points 1 to 4, the position information of the designated target corner points 1 to 4 in the front view to be calibrated image can be determined respectively.

[0106] In some embodiments, performing corner point matching on the image to be calibrated and the reference frame image based on the reference frame image of the corresponding view in the same calibration site to determine the designated target corner point information in the image to be calibrated may include:

[0107] Extracting feature points from the image to be calibrated and the reference frame image respectively to obtain a first feature point set in the image to be calibrated and a second feature point set in the reference frame image;

[0108] Performing feature point matching on the first feature point set and the second feature point set to obtain mutually matching first feature points and second feature points;

[0109] The target first feature point matched with the target second feature point is determined as the designated target corner point in the image to be calibrated; wherein the target second feature point is the second feature point in the reference frame image that is closest to the designated target corner point;

[0110] The designated target corner point information is determined according to the designated target corner point in the image to be calibrated.

[0111] Exemplarily, for any image to be calibrated, feature point extraction can be performed on the image to be calibrated and the reference frame image of the same view in the same calibration scene, respectively, to obtain a set of feature points in the image to be calibrated (which can be called a first feature point set) and a set of feature points in the reference frame image (which can be called a second feature point set).

[0112] Exemplarily, the feature point extraction algorithm may include, but is not limited to, the traditional ORB (Oriented FAST and Rotated BRIEF) algorithm, SIFT (Scale-Invariant Feature Transform) algorithm, SURF (Speeded-Up Robust Features) algorithm, or a deep learning corner point extraction algorithm (such as SuperPoint).

[0113] For the first feature point set and the second feature point set, feature point matching may be performed on the first feature point set and the second feature point set to determine mutually matching first feature points and second feature points.

[0114] Exemplarily, the feature point matching method may include, but is not limited to, a traditional RANSAC (Random Sample Consensus) matching algorithm, a brute force matching algorithm, or deep learning algorithms such as SuperGlue and LightGlue (an image matching algorithm based on deep learning).

[0115] In one example, a deep learning-based feature point extraction and matching technology can be used to extract and match feature points of the image to be calibrated and the reference frame image.

[0116] That is, a feature point extraction method based on deep learning can be used to extract feature points from the image to be calibrated and the reference frame image respectively; and

[0117] A feature point matching method based on deep learning is used to perform feature point matching on the first feature point set and the second feature point set.

[0118] Exemplarily, the feature point extraction network supports the matching of the center position of a circular area, the feature point extraction network outputs the position and descriptor of the feature point, and the feature matching network outputs the position coordinates of the matching point.

[0119] In this example, the feature points may include not only checkerboard "X"-shaped corner points, square "L"-shaped corner points, and U-shaped "L"-shaped corner points, but also circular "O"-shaped corner points, etc.

[0120] That is, in the embodiment of the present application, the target pattern is no longer limited to a checkerboard target, a square target, or a U-shaped target, but can also be a circular target (the corner point of the circular target is the center of the circle).

[0121] Exemplarily, when the first feature point and the second feature point that match each other are determined, the second feature point in the reference frame image that is closest to the designated target corner point (which can be called the target second feature point) can be used as the designated target corner point in the reference frame image, and the first feature point in the image to be calibrated that matches the target feature point (which can be called the target first feature point) can be determined as the designated target corner point in the image to be calibrated.

[0122] In some embodiments, the above-mentioned determination of the external parameters of the vehicle-mounted surround view camera to be calibrated based on the designated target corner point information in the image to be calibrated and the internal parameters of the vehicle-mounted surround view camera to be calibrated includes:

[0123] For any on-board surround-view camera to be calibrated, determine the homography matrix corresponding to the on-board surround-view camera to be calibrated based on the intrinsic parameters of the on-board surround-view camera to be calibrated, the position information of the target designated target corner points in the image to be calibrated captured by the on-board surround-view camera to be calibrated, and the physical coordinates of the target designated target corner points; wherein the target designated target corner points are the designated target corner points in the image to be calibrated captured by the on-board surround-view camera to be calibrated;

[0124] According to the homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated and the physical coordinates of the target corner point, the target corner point is projected onto the image to be calibrated captured by the vehicle-mounted surround-view camera to obtain the projection position of the target corner point in the image to be calibrated;

[0125] Determining a reprojection error of the target designated target corner point based on a projection position of the target designated target corner point in the image to be calibrated and position information of the target designated target corner point in the image to be calibrated;

[0126] When it is determined that the calibration of the vehicle-mounted surround view camera to be calibrated is successful based on the reprojection error of the target corner point, the extrinsic parameters of the vehicle-mounted surround view camera to be calibrated are determined based on the homography matrix corresponding to the vehicle-mounted surround view camera to be calibrated.

[0127] For example, when the designated target corner point information in the image to be calibrated is determined in the above manner, for any vehicle-mounted surround-view camera, the physical coordinates of the target designated target corner point can be determined based on the identification information of the designated target corner point (which can be referred to as the target designated target corner point) in the image to be calibrated captured by the vehicle-mounted surround-view camera to be calibrated.

[0128] Among them, since the physical coordinates of the designated target corner points of different identification information are also marked during the reference frame calibration process, that is, they are known conditions. Therefore, the homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated can be determined based on the internal parameters of the vehicle-mounted surround-view camera to be calibrated, the position information of the target designated target corner points in the image to be calibrated captured by the vehicle-mounted surround-view camera to be calibrated, and the physical coordinates of the target designated target corner points. Its specific implementation can be explained in conjunction with specific examples below.

[0129] When the homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated is determined, the target-specified target corner points can be projected onto the image to be calibrated captured by the vehicle-mounted surround-view camera to be calibrated based on the homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated and the physical coordinates of the target-specified target corner points to obtain the projection position of the target-specified target corner points in the image to be calibrated. Based on the projection position of the target-specified target corner points in the image to be calibrated and the position information of the target-specified target corner points in the image to be calibrated, the reprojection error of the target-specified target corner points can be determined. The specific implementation of this can be explained below in conjunction with specific examples.

[0130] For example, whether the calibration of the vehicle-mounted surround view camera to be calibrated is successful may be determined based on the reprojection error of the target corner point.

[0131] In one example, determining that the calibration of the vehicle-mounted surround view camera to be calibrated is successful based on the reprojection error of the target corner point may include:

[0132] When the reprojection error of each target designated target corner point is less than the first error threshold, and / or when the average reprojection error of the target designated target corner points is less than the second error threshold, it is determined that the calibration of the vehicle-mounted surround view camera to be calibrated is successful.

[0133] Exemplarily, whether the on-board surround view camera to be calibrated is successfully calibrated may be determined based on the reprojection error of each target designated corner point and / or the average reprojection error of the target designated corner points.

[0134] As an example, if the reprojection errors of the designated target corner points of each target are less than a preset error threshold (which can be called the first error threshold), it can be determined that the calibration of the vehicle-mounted surround view camera to be calibrated is successful; otherwise, it can be determined that the calibration of the vehicle-mounted surround view camera to be calibrated has failed.

[0135] As another example, if the average reprojection error of the target corner points is less than a preset error threshold (which can be called a second error threshold), it can be determined that the calibration of the vehicle-mounted surround view camera to be calibrated is successful; otherwise, it can be determined that the calibration of the vehicle-mounted surround view camera to be calibrated has failed.

[0136] As another example, if the reprojection error of each target designated target corner point is less than the first error threshold, and the average reprojection error of the target designated target corner point is less than the second error threshold, it can be determined that the calibration of the vehicle-mounted surround-view camera to be calibrated is successful; otherwise, it is determined that the calibration of the vehicle-mounted surround-view camera to be calibrated has failed.

[0137] Exemplarily, when it is determined that the calibration of the vehicle-mounted surround view camera to be calibrated is successful based on the reprojection error of the target corner point, the extrinsic parameters of the vehicle-mounted surround view camera to be calibrated are determined based on the homography matrix corresponding to the vehicle-mounted surround view camera to be calibrated.

[0138] For example, based on the homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated and the intrinsic parameters of the vehicle-mounted surround-view camera to be calibrated, the camera's extrinsic parameters, such as the camera position (the camera's coordinates in the vehicle body's world coordinate system) and installation angle, can be obtained using SVD (Singular Value Decomposition) decomposition.

[0139] In one example, determining the homography matrix corresponding to the vehicle-mounted surround view camera to be calibrated based on the intrinsic parameters of the vehicle-mounted surround view camera to be calibrated, the position information of the target corner points in the image to be calibrated captured by the vehicle-mounted surround view camera to be calibrated, and the physical coordinates of the target corner points may include:

[0140] Determine an initial homography matrix corresponding to the vehicle-mounted surround view camera to be calibrated based on the intrinsic parameters of the vehicle-mounted surround view camera to be calibrated, the position information of the target corner points in the image to be calibrated captured by the vehicle-mounted surround view camera to be calibrated, and the physical coordinates of the target corner points;

[0141] Using the pre-optimization algorithm, the initial homography matrix is ​​optimized to obtain the final homography matrix corresponding to the vehicle-mounted surround view camera to be calibrated;

[0142] The above-mentioned projecting the target designated target corner point onto the image to be calibrated captured by the vehicle-mounted surround view camera to be calibrated based on the homography matrix corresponding to the vehicle-mounted surround view camera to be calibrated and the physical coordinates of the target designated target corner point to obtain the projection position of the target designated target corner point in the image to be calibrated may include:

[0143] According to the final homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated and the physical coordinates of the target corner points, the target corner points are projected onto the image to be calibrated captured by the vehicle-mounted surround-view camera to obtain the projection position of the target corner points in the image to be calibrated.

[0144] For example, in order to improve the calibration accuracy, after the initial homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated is determined in the above manner, the preset optimization algorithm can be used to optimize the initial homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated to obtain the final homography matrix.

[0145] For example, when the initial homography matrix of the vehicle-mounted surround view camera to be calibrated is determined in the above manner, the initial homography matrix of the vehicle-mounted surround view camera to be calibrated can be optimized using the LM (Levenberg-Marquardt) algorithm to obtain the final homography matrix.

[0146] After obtaining the final homography matrix of the vehicle-mounted surround view camera to be calibrated, the reprojection error of the target corner points can be determined based on the final homography matrix, and the reprojection error of the target corner points can be used to determine whether the vehicle-mounted surround view camera to be calibrated is successful.

[0147] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below with reference to specific examples.

[0148] This embodiment provides a universal camera extrinsic calibration solution. By setting reference frame information, it assists in obtaining corner point information in the image to be calibrated (the above-mentioned specified target corner point information), thereby completing the vehicle camera extrinsic calibration. It is compatible with different calibration sites, different vehicle models, and different application scenarios.

[0149] For example, the PNP algorithm can be used to complete the extrinsic calibration of each camera's on-board surround view camera by detecting the ordered corner points of the target on the image and combining it with known target physical size information (target length and width, distance from the center of the vehicle body, etc.).

[0150] For example, reference frame information and image matching technology can be used to complete the detection and sorting of target corner points on the image to be calibrated (taking the identification information of the index target corner point as the corner point serial number as an example, the specified target corner point can be identified by the serial number).

[0151] The calibration system device is first described below.

[0152] In this embodiment, the schematic diagram of the calibration system device can be found in Figure 4 ,like Figure 4 As shown, the calibration system device may include: a camera (i.e., a vehicle-mounted surround view camera), a controller, a memory, and a display device.

[0153] The camera is used to capture the image to be calibrated; the controller is used to perform corresponding control processing according to the detected control instructions; the memory is used to store reference frame information; and the display device is used to display the calibration operation interface.

[0154] For example, based on Figure 4 For the calibration system shown in the figure, the camera extrinsic calibration process can be as follows:

[0155] S1. Select an operation instruction according to the detected calibration site and determine the corresponding calibration site pattern.

[0156] For example, the calibration operation interface may provide a "calibration site selection" function button. When a selection instruction of the function button is detected, candidate calibration site patterns may be displayed in the calibration operation interface. Then, the selected calibration site pattern may be determined based on the selection instruction for the calibration site pattern.

[0157] S2. Select an instruction based on the detected reference frame information to determine the corresponding reference frame information.

[0158] For example, when the selected calibration scene pattern is determined, the reference frame information corresponding to the calibration scene pattern can be displayed in the display interface. When a selection instruction for the reference frame information is detected, it can be determined to use the selected reference frame information for camera extrinsic calibration.

[0159] S3. When a calibration start instruction is detected, the camera extrinsic parameters of the vehicle to be calibrated are calibrated according to the determined reference frame information.

[0160] For example, when it is determined that the calibration of the vehicle to be calibrated is successful, a bird's-eye view effect of the calibration image may be displayed.

[0161] The specific implementation process of camera extrinsic calibration is described below.

[0162] In this embodiment, the specific implementation process of camera extrinsic calibration can be found in Figure 5 ,like Figure 5 As shown, the specific implementation process of the camera extrinsic calibration may include:

[0163] 1) Obtain the image to be calibrated.

[0164] For example, taking the number of onboard surround view cameras as 4, the schematic diagram of the image to be calibrated can be found in Figure 6 ,like Figure 6 As shown, Figure 6 From left to right and from top to bottom in the figure, they are the front view image to be calibrated (which can be referred to as the front view to be calibrated), the rear view image to be calibrated (which can be referred to as the rear view to be calibrated), the left view image to be calibrated (which can be referred to as the left view to be calibrated), and the right view image to be calibrated (which can be referred to as the right view to be calibrated).

[0165] 2) Obtain reference frame information under the same calibration site.

[0166] Exemplarily, the reference frame information may include the serial number and position coordinates of a designated target corner point in the reference frame image.

[0167] For example, a schematic diagram of a reference frame image can be found in Figure 7 ,like Figure 7 As shown, Figure 7 From left to right and from top to bottom in the figure, they are the front view reference frame image (which may be referred to as the reference frame front view), the rear view reference frame image (which may be referred to as the reference frame rear view), the left view reference frame image (which may be referred to as the reference frame left view), and the right view reference frame image (which may be referred to as the reference frame right view).

[0168] Exemplarily, the reference frame image may be a fisheye original image captured by a vehicle-mounted surround view camera during the calibration process, or a dedistorted image obtained by performing dedistortion processing on the fisheye original image.

[0169] It should be noted that, when the reference frame image is a fisheye original image, the image to be calibrated also uses the fisheye original image; when the reference frame image is a dedistorted image, the image to be calibrated also uses the dedistorted image.

[0170] Exemplarily, the number of reference frames in each reference frame image is greater than or equal to 4 ( Figure 7 Taking 8 as an example), they are not collinear and are within the common viewing area of ​​adjacent vehicle-mounted surround view cameras.

[0171] For example, the reference frame image and the image to be calibrated may be acquired in the same or different vehicle type, acquisition environment, and camera resolution.

[0172] For example, the reference frame image and the image to be calibrated may be images captured by a surround-view camera mounted on a vehicle of the same model, or may be images captured by surround-view cameras mounted on vehicles of different models.

[0173] The reference frame image (or the image to be calibrated) can be collected in a factory calibration board environment or in an outdoor environment of a 4S shop by laying a calibration cloth. The resolution of the acquisition camera can be 1280*720, 1280*960, or other resolutions.

[0174] For example, for any calibration site, in the process of obtaining the reference frame image of the calibration site, a set of fisheye original images of the calibration site (including front, rear, left and right views) can be obtained through the calibration vehicle, and the serial number and position information of the specified target corner point can be determined based on the detected labeling instructions.

[0175] 3) Based on the reference frame information, determine the designated target corner point information in the image to be calibrated.

[0176] For example, image matching technology may be used to perform image matching on each image to be calibrated and a reference frame image of a corresponding view to obtain a corner point matching relationship between the two images.

[0177] Exemplarily, the above-mentioned image matching technology may adopt image feature point matching technology, or adopt image dense matching technology.

[0178] In one example, in order to reduce the computational time of image matching and improve processing efficiency, the image matching technology may adopt image feature point matching technology.

[0179] Exemplarily, the feature point matching technology may include feature point extraction and feature point matching.

[0180] Exemplarily, the feature point extraction algorithm may include, but is not limited to, a traditional ORB algorithm, a SIFT algorithm, a SURF algorithm, or a deep learning corner point extraction algorithm (such as SuperPoint).

[0181] Exemplarily, the feature point matching method may include but is not limited to a traditional RANSAC matching algorithm, a brute force matching algorithm, or a deep learning algorithm such as SuperGlue and LightGlue.

[0182] In one example, in order to obtain a more robust matching effect, a feature point extraction and matching technology based on deep learning can be used to extract and match feature points of the image to be calibrated and the reference frame image.

[0183] Exemplarily, the feature point extraction network supports the matching of the center position of a circular area, the feature point extraction network outputs the position and descriptor of the feature point, and the feature matching network outputs the position coordinates of the matching point.

[0184] For example, the feature points may include not only checkerboard "X"-shaped corner points, square "L"-shaped corner points, and circular "L"-shaped corner points, but also circular "O"-shaped corner points (the corner point of the circular pattern is the center of the circle), etc.

[0185] For example, through the above feature point matching technology, the images to be calibrated and the reference frame images collected by vehicles of different lengths in the same calibration site can be correctly matched to the center position of the circular target. The schematic diagram can be seen in Figure 8 .like Figure 8 As shown, in order from left to right and from top to bottom, there are the front view matching result schematic diagram, the rear view matching result schematic diagram, the left view matching result schematic diagram, and the right view matching result schematic diagram.

[0186] In addition, in the same calibration site, different target materials, and different calibration environments, the image can correctly match the corner points of the checkerboard target. The schematic diagram can be seen in Figure 9 .like Figure 9 As shown, in order from left to right and from top to bottom, there are the front view matching result schematic diagram, the rear view matching result schematic diagram, the left view matching result schematic diagram, and the right view matching result schematic diagram.

[0187] Exemplarily, based on the serial number and coordinate information of the designated target corner point marked in the reference frame image, the image matching information is used to filter out the corner point closest to the designated target corner point in the image to be calibrated, and the serial number of the corner point marked in the reference frame image is assigned to the designated target corner point in the image to be calibrated.

[0188] Taking the front view as an example, after filtering, only the corner points corresponding to the marked corner points on the reference frame image are retained, which can eliminate abnormal results in the image matching results. The schematic diagram can be seen in Figure 10 The same applies to other view filters.

[0189] Among them, such as Figure 10 As shown in the figure, the left side is the front view of the reference frame, and the right side is the target corner points selected from the image to be calibrated.

[0190] 4) Based on the known physical coordinates of the specified target corner points and the camera intrinsic parameters, use PNP to calculate the homography matrix.

[0191] For example, based on the image coordinates, corner point sequence number, target size information, etc. of the specified target corner point in the image to be calibrated, the known physical coordinates corresponding to the specified target corner point (which can be the X and Y coordinates with the center of the vehicle body as the origin) can be obtained, and the PNP is used to calculate the homography matrix of each view.

[0192] The following is an example of using DLT direct linear transformation to solve the homography matrix:

[0193]

[0194] Where X and Y represent the positions in the world coordinate system, u and v represent the pixel coordinates of the target corner points on the dedistorted image, and h0 to h8 are the values ​​of each element in the 3*3 homography matrix.

[0195] Eliminate s with the last line and get two constraints:

[0196]

[0197] Listed as a system of linear equations:

[0198]

[0199] When the number of matching points is greater than or equal to 4, the above equation has a solution. That is, the solution of the linear equation system AH=0 is A T The eigenvector corresponding to the smallest eigenvalue of A.

[0200] For example, the eigenvalues ​​and eigenvectors may be solved by the Jacobi iteration method.

[0201] In addition, other methods for solving linear equations can be used: SVD decomposition, QR decomposition, LU decomposition, Cholesky decomposition, etc.

[0202] 5) Based on the current homography matrix, determine the reprojection error of the specified target corner point in the image to be calibrated. If the calibration is successful based on the reprojection error, the camera extrinsic parameters corresponding to the current homography matrix are determined as the extrinsic parameters of the on-board surround view camera to be calibrated.

[0203] For example, when the homography matrix of the vehicle-mounted surround view camera to be calibrated is obtained, the reprojected pixel point of the world coordinate of the specified target corner point on the image can be obtained through the relationship between the world coordinate point and the homography matrix.

[0204] For example, after obtaining the initial homography matrix of the on-board surround view camera to be calibrated, LM optimization can be used to optimize the homography matrix by optimizing the sum of the reprojection errors (RMS) of the corner points in the common view area to obtain the final homography matrix, thereby achieving higher calibration accuracy.

[0205] When the reprojection error is obtained to determine that the on-board surround view camera calibration is successful, the homography matrix and the camera intrinsic parameters can be combined to obtain the camera's extrinsic parameters (including camera position and installation angle) using SVD decomposition.

[0206] For example, for the determined homography matrix, a threshold can be set to determine whether the RMS of this calibration meets the requirements, thereby outputting whether the calibration is successful or failed.

[0207]

[0208]

[0209] Where (u, v) represents the world coordinates of the specified target corner point projected onto the pixel coordinates in the image, and (x, y) represents the pixel coordinates of the specified target corner point extracted from the image to be calibrated. N represents the number of specified target corner points in the image.

[0210] For example, the calibration may be determined to be successful when the average image reprojection error of all designated target corner points is less than a threshold (eg, 0.5 pix).

[0211] It can be seen that the camera extrinsic calibration solution provided in the embodiment of the present application obtains the target corner information in the image to be calibrated by introducing a reference frame, thereby completing the camera extrinsic calibration, which is compatible with different car models and various business application scenarios.

[0212] 1) If the calibration site changes, only the reference frame information needs to be re-annotated, and the algorithm does not need to be upgraded or adapted to support the project.

[0213] 2) Even if the application scenario, vehicle model, or camera model (including resolution) changes, the project can be supported without changing the reference frame information or upgrading the algorithm.

[0214] 3) If the number of surround view cameras on the vehicle changes, there is no need to update if the calibration site remains unchanged. If the calibration site changes, simply update the reference frame information.

[0215] The calibration site refers to the pattern and number of targets laid out. For example, assuming the target layout remains unchanged but the number of onboard surround-view cameras changes, the reference frames can be reused, requiring only the selection of the corresponding reference frames for each onboard surround-view camera. This means that even if the calibration site changes, only the reference frame information needs to be updated, without the need for algorithm upgrades, to support camera extrinsic calibration. Furthermore, if the camera model or vehicle type changes, there's no need to re-upgrade the calibration algorithm, allowing for rapid support of project solutions with high ease of use and calibration accuracy, effectively addressing industry pain points.

[0216] In addition, the reference frame information is used to obtain the target corner point information in the image to be calibrated, and then PNP is used for calibration. The calibration accuracy is high, and it does not rely on the external parameter information of the reference frame. It also has a certain tolerance for the labeling accuracy of the target corner points in the reference frame, and the algorithm is highly robust.

[0217] The above describes the method provided by this application. The following describes the device provided by this application:

[0218] See Figure 11 , is a structural diagram of a camera extrinsic parameter calibration device provided in an embodiment of the present application, such as Figure 11 As shown, the camera extrinsic parameter calibration device may include:

[0219] a generating unit configured to, for any calibration site, if it is determined that no reference frame information associated with the calibration site is stored, acquire a reference frame image of the calibration site using a surround-view camera on a calibration vehicle; wherein the calibration site is characterized by a target placement scheme, the target placement scheme including a target pattern and a number of targets, the same target placement scheme using the same target pattern, and the number of targets in images of the same view of the on-board surround-view camera being the same;

[0220] Generate reference frame information corresponding to the reference frame image based on the detected annotation instruction; wherein the reference frame information includes identification information of designated target corner points in the reference frame image, physical coordinates of the designated target corner points, and position information of the designated target corner points in the reference frame image; the number of designated target corner points is greater than or equal to 4, and all are located in a common viewing area of ​​adjacent on-board surround view cameras of the calibrated vehicle in the reference frame image and are not collinear;

[0221] The processing unit is used to obtain the reference frame information of the same calibration site for any vehicle to be calibrated based on the current calibration site, and perform external parameter calibration on the surround view camera on the vehicle to be calibrated based on the obtained reference frame information.

[0222] For example, the specific implementation process of the generation unit and the processing unit to implement the camera extrinsic parameter calibration can be found in the relevant description in the above embodiment, and the embodiments of the present application will not be repeated here.

[0223] See Figure 12 , is a structural diagram of a camera extrinsic parameter calibration device provided in an embodiment of the present application, such as Figure 12 As shown, the camera extrinsic parameter calibration device may include:

[0224] An acquisition unit, configured to acquire an image to be calibrated; wherein the image to be calibrated is obtained by capturing an image of the calibration site using a surround-view camera mounted on the vehicle to be calibrated;

[0225] A determination unit is used to determine the designated target corner point information in the image to be calibrated based on the reference frame information of the same calibration site; wherein the designated target corner point information includes the identification information of the designated target corner point in the image to be calibrated, the physical coordinates of the designated target corner point, and the position information of the designated target corner point in the image to be calibrated; the reference frame information includes the identification information of the designated target corner point in the reference frame image, the physical coordinates of the designated target corner point, and the position information of the designated target corner point in the reference frame image; during the calibration process, the reference frame image is obtained by respectively capturing images of the calibration site by each on-board surround-view camera of the calibration vehicle; the number of the designated target corner points is greater than or equal to 4, and all of them are located in the common viewing area of ​​the adjacent on-board surround-view cameras of the calibration vehicle in the reference frame image and are not collinear;

[0226] The processing unit is configured to determine the external parameters of the vehicle-mounted surround view camera to be calibrated based on the designated target corner point information in the image to be calibrated and the internal parameters of the vehicle-mounted surround view camera to be calibrated.

[0227] For example, the specific implementation process of the acquisition unit, the determination unit and the processing unit to implement the camera extrinsic parameter calibration can be referred to the relevant description in the above embodiment, and the embodiments of the present application will not be repeated here.

[0228] An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor is used to execute the machine-executable instructions to implement the camera extrinsic parameter calibration method described above.

[0229] See Figure 13, is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 1301 and a memory 1302 storing machine-executable instructions. The processor 1301 and the memory 1302 may communicate via a system bus 1303. Furthermore, by reading and executing the machine-executable instructions corresponding to the camera extrinsic calibration logic in the memory 1302, the processor 1301 may perform the camera extrinsic calibration method described above.

[0230] The memory 1302 mentioned herein can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, and the like. For example, the machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0231] In some embodiments, a machine-readable storage medium is also provided. Figure 13 The memory 1302 in the machine-readable storage medium stores machine-executable instructions. When executed by the processor, the machine-executable instructions implement the camera extrinsic calibration method described above. For example, the storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0232] The embodiment of the present application further provides a camera extrinsic parameter calibration system, comprising: a processor, a memory, and a display; wherein:

[0233] The memory is used to store the association relationship between the calibration site and the reference frame information;

[0234] The processor is configured to, for any calibration site, if it is determined that no reference frame information associated with the calibration site is stored in the memory, acquire a reference frame image of the calibration site using a surround-view camera on a calibration vehicle; wherein the calibration site is characterized by a target placement scheme, the target placement scheme including a target pattern and a number of targets, the same target placement scheme using the same target pattern, and the number of targets in images of the same view of the surround-view camera on the vehicle being calibrated;

[0235] The display is used to display the reference frame image;

[0236] The processor is further configured to generate reference frame information corresponding to the reference frame image based on the detected annotation instruction for the reference frame image displayed on the display; wherein the reference frame information includes:

[0237] Identification information of designated target corner points in the reference frame image;

[0238] The physical coordinates of the designated target corner points; and

[0239] Position information of the designated target corner points in the reference frame image; wherein the number of the designated target corner points is greater than or equal to 4, and all of the designated target corner points are located in the common viewing area of ​​the adjacent on-board surround view cameras of the calibrated vehicle in the reference frame image and are not collinear;

[0240] The memory is further used to associate and store the calibration site with the reference frame information;

[0241] The processor is further configured to obtain, for any vehicle to be calibrated, reference frame information of the same calibration site from the memory based on the current calibration site, and perform external parameter calibration on the surround view camera on the vehicle to be calibrated based on the obtained reference frame information.

[0242] For example, the specific implementation process of the processor implementing camera extrinsic parameter calibration can be found in the relevant description in the above embodiment, and the embodiments of the present application will not be described in detail here.

Claims

1. A camera extrinsic calibration method, characterized in that: include: For any calibration site, if it is determined that no reference frame information associated with the calibration site is stored, reference frame images of the calibration site are collected using the on-board surround-view camera of the calibration vehicle; wherein the calibration site is characterized by a target laying scheme, the target laying scheme includes a target pattern and the number of targets, the same target laying scheme uses the same target pattern, and the number of targets in the same view of the on-board surround-view camera is the same; Generate reference frame information corresponding to the reference frame image based on the detected annotation instruction, and associate the calibration site with the reference frame information and store it; wherein the reference frame information includes: Identification information of designated target corner points in the reference frame image; The physical coordinates of the designated target corner points; and Position information of the designated target corner points in the reference frame image; wherein the number of the designated target corner points is greater than or equal to 4, and all of the designated target corner points are located in the common viewing area of ​​the adjacent on-board surround view cameras of the calibrated vehicle in the reference frame image and are not collinear; For any vehicle to be calibrated, obtain the reference frame information of the same calibration site based on the current calibration site, and perform external parameter calibration on the surround view camera on the vehicle to be calibrated based on the obtained reference frame information; Wherein, for the reference frame image of the first calibration site and the image to be calibrated of the second calibration site acquired by the surround-view camera on the vehicle to be calibrated, if the target pattern and the number of targets in each image to be calibrated are the same as the target pattern and the number of targets in the reference frame image of the same view, the first calibration site and the second calibration site are determined to be the same calibration site; The extrinsic parameter calibration of the vehicle-mounted surround view camera to be calibrated based on the acquired reference frame information includes: For any image to be calibrated, corner point matching is performed between the image to be calibrated and the reference frame image based on the reference frame image of the corresponding view in the same calibration site, so as to determine the designated target corner point information in the image to be calibrated; wherein, the image to be calibrated is obtained by capturing images of the calibration site by the surround-view camera on the vehicle to be calibrated; The external parameters of the vehicle-mounted surround view camera to be calibrated are determined according to the designated target corner point information in the image to be calibrated and the internal parameters of the vehicle-mounted surround view camera to be calibrated.

2. The method according to claim 1, characterized in that The target pattern includes a checkerboard target, a square target, a circular target, or a circular target.

3. The method according to claim 1, characterized in that The method of obtaining reference frame information of the same calibration site based on the current calibration site and performing external parameter calibration on the vehicle-mounted surround view camera to be calibrated based on the obtained reference frame information includes: Determine the current calibration site based on the detected calibration site selection instruction; Determining a target reference frame image of the current calibration site according to the detected reference frame image selection instruction for the current calibration site; Determining reference frame information of a current calibration site based on the target reference frame image; The extrinsic parameters of the surround view camera to be calibrated are calibrated based on the reference frame information of the current calibration site.

4. The method according to claim 1, wherein After the extrinsic parameters of the vehicle-mounted surround view camera to be calibrated are calibrated based on the acquired reference frame information, the method further includes: When the surround view camera on the vehicle to be calibrated is calibrated successfully, the bird's-eye view effect of the vehicle to be calibrated is displayed.

5. The method according to claim 1, wherein The step of performing corner point matching on the image to be calibrated and the reference frame image based on a reference frame image of a corresponding view in the same calibration site to determine the designated target corner point information in the image to be calibrated includes: Extracting feature points from the image to be calibrated and the reference frame image respectively to obtain a first feature point set in the image to be calibrated and a second feature point set in the reference frame image; Performing feature point matching on the first feature point set and the second feature point set to obtain mutually matching first feature points and second feature points; Determine the target first feature point that matches the target second feature point as the designated target corner point in the image to be calibrated; wherein the target second feature point is the second feature point in the reference frame image that is closest to the designated target corner point; The designated target corner point information is determined according to the designated target corner point in the image to be calibrated.

6. A camera extrinsic parameter calibration device, characterized in that: include: A generating unit is configured to, for any calibration site, if it is determined that no reference frame information associated with the calibration site is stored, collect reference frame images of the calibration site through the on-board surround-view camera of the calibration vehicle, and generate reference frame information corresponding to the reference frame image based on the detected labeling instruction; wherein the calibration site is characterized by a target laying scheme, the target laying scheme includes a target pattern and the number of targets, the same target laying scheme uses the same target pattern, and the number of targets in the same view of the on-board surround-view camera is the same; the reference frame information includes identification information of designated target corner points in the reference frame image, the physical coordinates of the designated target corner points, and position information of the designated target corner points in the reference frame image; the number of the designated target corner points is greater than or equal to 4, all of which are located in the common viewing area of ​​the adjacent on-board surround-view cameras of the calibration vehicle in the reference frame image and are not collinear; A processing unit is configured to obtain, for any vehicle to be calibrated, reference frame information of the same calibration site based on the current calibration site, and perform external parameter calibration on the surround view camera on the vehicle to be calibrated based on the obtained reference frame information; Wherein, for the reference frame image of the first calibration site and the image to be calibrated of the second calibration site acquired by the surround-view camera on the vehicle to be calibrated, if the target pattern and the number of targets in each image to be calibrated are the same as the target pattern and the number of targets in the reference frame image of the same view, the first calibration site and the second calibration site are determined to be the same calibration site; The processing unit performs external parameter calibration on the vehicle-mounted surround view camera to be calibrated based on the acquired reference frame information, including: For any image to be calibrated, corner point matching is performed between the image to be calibrated and the reference frame image based on the reference frame image of the corresponding view in the same calibration site, so as to determine the designated target corner point information in the image to be calibrated; wherein, the image to be calibrated is obtained by capturing images of the calibration site by the surround-view camera on the vehicle to be calibrated; The external parameters of the vehicle-mounted surround view camera to be calibrated are determined according to the designated target corner point information in the image to be calibrated and the internal parameters of the vehicle-mounted surround view camera to be calibrated.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor is configured to execute the machine-executable instructions to implement the method according to any one of claims 1 to 5.

8. A machine-readable storage medium, characterized in that The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. A camera extrinsic calibration system, characterized in that: include: A processor, a memory, and a display; wherein: The memory is used to store the association relationship between the calibration site and the reference frame information; The processor is configured to, for any calibration site, if it is determined that no reference frame information associated with the calibration site is stored in the memory, acquire a reference frame image of the calibration site using a surround-view camera on a calibration vehicle; wherein the calibration site is characterized by a target placement scheme, the target placement scheme including a target pattern and a number of targets, the same target placement scheme using the same target pattern, and the number of targets in images of the same view of the surround-view camera on the vehicle being calibrated; The display is used to display the reference frame image; The processor is further configured to generate reference frame information corresponding to the reference frame image based on the detected annotation instruction for the reference frame image displayed on the display; wherein the reference frame information includes: Identification information of designated target corner points in the reference frame image; The physical coordinates of the designated target corner points; and Position information of the designated target corner points in the reference frame image; wherein the number of the designated target corner points is greater than or equal to 4, and all of the designated target corner points are located in the common viewing area of ​​the adjacent on-board surround view cameras of the calibrated vehicle in the reference frame image and are not collinear; The memory is further used to associate and store the calibration site with the reference frame information; The processor is further configured to, for any vehicle to be calibrated, obtain reference frame information of the same calibration site from the memory based on the current calibration site, and perform external parameter calibration on the surround view camera on the vehicle to be calibrated based on the obtained reference frame information; Wherein, for the reference frame image of the first calibration site and the image to be calibrated of the second calibration site acquired by the surround-view camera on the vehicle to be calibrated, if the target pattern and the number of targets in each image to be calibrated are the same as the target pattern and the number of targets in the reference frame image of the same view, the first calibration site and the second calibration site are determined to be the same calibration site; The processor performs external parameter calibration on the vehicle-mounted surround view camera to be calibrated based on the acquired reference frame information, including: For any image to be calibrated, corner point matching is performed between the image to be calibrated and the reference frame image based on the reference frame image of the corresponding view in the same calibration site, so as to determine the designated target corner point information in the image to be calibrated; wherein, the image to be calibrated is obtained by capturing images of the calibration site by the surround-view camera on the vehicle to be calibrated; The external parameters of the vehicle-mounted surround view camera to be calibrated are determined according to the designated target corner point information in the image to be calibrated and the internal parameters of the vehicle-mounted surround view camera to be calibrated.

Images