Camera external parameter calibration method, device and equipment and storage medium

Abstract

The invention provides a camera external parameter calibration method and device, equipment and a storage medium. In one example, the method comprises the following steps: acquiring an image to be calibrated; according to the reference frame information of the same calibration site, determining the angular point information of a specified target in the image to be calibrated; and determining external parameters of the to-be-calibrated vehicle-mounted around-view camera according to the specified target angular point information in the to-be-calibrated image and the internal parameters of the to-be-calibrated vehicle-mounted around-view camera. According to the method, the universality of the calibration scheme can be improved under the condition that the calibration accuracy is ensured.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular, to a method, device, equipment and storage medium for calibrating external parameters of a camera. Background Art

[0002] The external parameter calibration of on-vehicle surround cameras is usually carried out after the vehicle rolls off the production line in the factory, which is usually also called the off-line calibration of the vehicle. After knowing the internal parameters of the camera and the physical size information of the target on the calibration site, the corner detection technology is used to locate the target corners in the image, so as to complete the external parameter calibration of the camera and provide support for the intelligent driving of the vehicle.

[0003] However, due to the large number of suppliers of surround external parameter calibration algorithms on the market, resulting in a variety of calibration site patterns built by different vehicle manufacturers. Therefore, when supporting mass production projects of different vehicle models, new calibration site patterns are always encountered.

[0004] How to respond to the calibration requirements of different sites while ensuring high-precision external parameter calibration has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, this application provides a method, device, equipment and storage medium for calibrating external parameters of a camera.

[0006] Specifically, this application is implemented through the following technical solutions: According to the first aspect of the embodiments of this application, a method for calibrating external parameters of a camera is provided, including: Obtain an image to be calibrated; wherein, the image to be calibrated is obtained by collecting an image of a calibration site through a vehicle-mounted surround camera to be calibrated; Determine the specified target corner point information in the image to be calibrated according to the reference frame information of the same calibration site; wherein, the specified target corner point information includes: The identification information of the specified target corner point in the image to be calibrated; The physical coordinates of the specified target corner point; and, The position information of the specified target corner point in the image to be calibrated; The reference frame information includes: The identification information of the specified target corner point in the reference frame image; The physical coordinates of the specified target corner point; and, The position information of the specified target corner point in the reference frame image; wherein, during the calibration process, the reference frame image is obtained by collecting images of the calibration site through each vehicle-mounted surround camera of the calibration vehicle; the number of the specified target corner points is greater than or equal to 4, and they are all located in the co-viewing area of the adjacent vehicle-mounted surround cameras of the calibration vehicle in the reference frame image and are not collinear; Determine the external parameters of the to-be-calibrated vehicle surround-view camera based on the specified target corner point information in the to-be-calibrated image and the internal parameters of the to-be-calibrated vehicle surround-view camera.

[0007] According to a second aspect of the embodiments of the present application, there is provided a camera external parameter calibration device, including: An acquisition unit, configured to acquire a to-be-calibrated image; wherein, the to-be-calibrated image is obtained by the to-be-calibrated vehicle surround-view camera collecting an image of a calibration site; A determination unit, configured to determine the specified target corner point information in the to-be-calibrated image according to the reference frame information of the same calibration site; wherein, the specified target corner point information includes the identification information of the specified target corner point in the to-be-calibrated image, the physical coordinates of the specified target corner point, and the position information of the specified target corner point in the to-be-calibrated image; the reference frame information includes the identification information of the specified target corner point in the reference frame image, the physical coordinates of the specified target corner point, and the position information of the specified target corner point in the reference frame image; during the calibration process, the reference frame image is obtained by each vehicle surround-view camera of the calibration vehicle collecting an image of the calibration site; the number of the specified target corner points is greater than or equal to 4, and all are located in the co-viewing area of the adjacent vehicle surround-view cameras of the calibration vehicle in the reference frame image and are not collinear; A processing unit, configured to determine the external parameters of the to-be-calibrated vehicle surround-view camera based on the specified target corner point information in the to-be-calibrated image and the internal parameters of the to-be-calibrated vehicle surround-view camera.

[0008] According to a third aspect of the embodiments of the present application, there is provided an electronic device, including a processor and a memory, the memory stores machine-executable instructions that can be executed by the processor, and the processor is configured to execute the machine-executable instructions to implement the method provided in the first aspect.

[0009] According to a fourth aspect of the embodiments of the present application, there is provided a machine-readable storage medium, the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0010] The technical solution provided by the present application can at least bring the following beneficial effects: By pre-setting corresponding reference frame information for different calibration sites, during the process of calibrating the external parameters of an in-vehicle surround-view camera, for the to-be-calibrated images obtained by collecting images of the calibration site through the to-be-calibrated in-vehicle surround-view camera, the information of the specified target corner points in the to-be-calibrated images can be determined based on the reference frame information of the same calibration site. Furthermore, based on the information of the specified target corner points in the to-be-calibrated images and the internal parameters of the to-be-calibrated in-vehicle surround-view camera, the external parameters of the to-be-calibrated in-vehicle surround-view camera can be determined. By setting the reference frame information for the calibration site, in the case of the existence of the reference frame information of the same calibration site, for changes in other parameters such as the physical environment, camera model, vehicle type, etc., there is no need to re-upgrade the calibration algorithm, and the versatility of the calibration scheme can be improved while ensuring the calibration accuracy. In addition, in the case of a change in the calibration site, only the reference frame information needs to be updated to support the calibration of the external parameters of the camera for the new calibration site, which improves the scalability of the scheme and effectively reduces the development cost of expanding the calibration site. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a schematic flowchart of a method for calibrating the external parameters of a camera shown in an exemplary embodiment of the present application; Figure 2A is a schematic diagram of a vehicle with 4 in-vehicle surround-view cameras shown in an exemplary embodiment of the present application; Figure 2B is a schematic diagram of a vehicle with 6 in-vehicle surround-view cameras shown in an exemplary embodiment of the present application; Figure 3 is a schematic flowchart of a method for calibrating the external parameters of a camera shown in an exemplary embodiment of the present application; Figure 4 is a schematic diagram of a calibration system device shown in an exemplary embodiment of the present application; Figure 5 is a schematic flowchart of a specific implementation process for calibrating the external parameters of a camera shown in an exemplary embodiment of the present application; Figure 6 is a schematic diagram of a to-be-calibrated image shown in an exemplary embodiment of the present application; Figure 7 is a schematic diagram of a reference frame image shown in an exemplary embodiment of the present application; Figure 8 is a schematic diagram of the matching between a to-be-calibrated image and a reference frame image shown in an exemplary embodiment of the present application; Figure 9 is a schematic diagram of the matching between a to-be-calibrated image and a reference frame image shown in an exemplary embodiment of the present application; Figure 10 is a schematic diagram of the matching between a to-be-calibrated image and a reference frame image after feature point screening shown in an exemplary embodiment of the present application; Figure 11 It is a schematic structural diagram of an external parameter calibration device for a camera shown in an exemplary embodiment of the present application; Figure 12 It is a schematic structural diagram of an external parameter calibration device for a camera shown in an exemplary embodiment of the present application; Figure 13 It is a schematic hardware structure diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0012] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned objects, features, and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0013] It should be noted that the sequence numbers of the steps in the embodiments of the present application do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0014] Please refer to Figure 1 , which is a schematic flowchart of a camera external parameter calibration method provided by an embodiment of the present application. As Figure 1 shown, the camera external parameter calibration method may include the following steps: Step S100: For any calibration site, use the on-vehicle surround camera of the calibration vehicle to collect reference frame images of the calibration site, and generate reference frame information corresponding to the reference frame images according to the detected annotation instructions; wherein, the reference frame information includes the identification information of the specified target corner points in the reference frame image, the physical coordinates of the specified target corner points, and the position information of the specified target corner points in the reference frame image; the number of the specified target corner points is greater than or equal to 4, and they are all located in the common viewing area of adjacent on-vehicle surround cameras of the calibration vehicle in the reference frame image and are not collinear.

[0015] Step S110: For any vehicle to be calibrated, obtain the reference frame information of the same calibration site according to the current calibration site, and perform external parameter calibration on the on-vehicle surround camera to be calibrated according to the obtained reference frame information.

[0016] In the embodiments of the present application, the on-vehicle surround camera includes on-vehicle cameras installed on the vehicle body for obtaining the surrounding environment of the vehicle body to achieve 360° surround view.

[0017] Exemplarily, for any vehicle equipped with an on-vehicle surround camera, the number of on-vehicle surround cameras is usually multiple.

[0018] For example, the number of on-vehicle surround cameras can be 4 or 6.

[0019] Taking the number of vehicle surround cameras as 4 as an example, vehicle surround cameras usually include vehicle cameras deployed in front of the vehicle, behind the vehicle, on the left side of the vehicle, and on the right side of the vehicle (which can be respectively called the front vehicle camera, the rear vehicle camera, the left vehicle camera, and the right vehicle camera), and its schematic diagram can be as Figure 2A shown.

[0020] Taking the number of vehicle surround cameras as 6 as an example, vehicle surround cameras usually include vehicle cameras deployed in front of the vehicle, behind the vehicle, in the front left side of the vehicle, in the rear left side of the vehicle, in the front right side of the vehicle, and in the rear right side of the vehicle (which can be respectively called the front vehicle camera, the rear vehicle camera, the front left vehicle camera, the rear left vehicle camera, the front right vehicle camera, and the rear right vehicle camera), and its schematic diagram can be as Figure 2B shown.

[0021] In the embodiments of the present application, in order to improve the generality of the camera extrinsic parameter calibration scheme, reference frame information can be set for different calibration sites. Thus, in the process of calibrating the extrinsic parameters of the vehicle surround cameras of the vehicle to be calibrated (which can be called the vehicle surround cameras to be calibrated), the corresponding reference frame information can be determined according to the actual calibration site, and the extrinsic parameters of the vehicle surround cameras to be calibrated can be calibrated according to the reference frame information.

[0022] Exemplarily, the calibration site can be characterized by a target laying scheme. The same target laying scheme corresponds to the same calibration site, and different target laying schemes correspond to different calibration sites.

[0023] That is, in the embodiments of the present application, for different physical environments, different camera models, and different vehicle models, the same reference frame information can be used for camera extrinsic parameter calibration when the target laying schemes are the same.

[0024] Exemplarily, the target laying scheme can include a target pattern and the number of targets. The same target laying scheme uses the same target pattern, and the number of targets in the images of the same view of the vehicle surround cameras is the same.

[0025] Exemplarily, the images of the same view of the vehicle surround cameras refer to the images collected by the vehicle surround cameras installed in the same orientation.

[0026] For example, for the case where the number of vehicle surround cameras is 4, the views of the vehicle surround cameras include a front view, a rear view, a left view, and a right view.

[0027] For the case where the number of vehicle surround cameras is 6, the views of the vehicle surround cameras include a front view, a rear view, a front left view, a rear left view, a front right view, and a rear right view.

[0028] Taking the number of on-vehicle surround cameras as 4 as an example, assuming that checkerboard targets are used in both target laying scheme 1 and target laying scheme 2, and the number of targets in the images of the front view / rear view / left view / right view of the on-vehicle surround cameras is the same in target laying scheme 1 and target laying scheme 2, then target laying scheme 1 and target laying scheme 2 are the same.

[0029] In one example, for the to-be-calibrated images of the first calibration site obtained by the to-be-calibrated on-vehicle surround camera, and the reference frame images of the second calibration site, when the target patterns and the number of targets in each to-be-calibrated image are the same as those in the reference frame images of the same view, it is determined that the first calibration site and the second calibration site are the same calibration site.

[0030] In the embodiments of the present application, the reference frame information may include the identification information (such as serial numbers) of the specified target corner points in the reference frame image, and the position information (such as image coordinates) of the specified target corner points in the reference frame image.

[0031] Among them, the specified target corner points are the target corner points marked in the reference frame image.

[0032] Exemplarily, for any calibration site, the images of this calibration site (i.e., reference frame images) can be collected by the on-vehicle surround camera of any vehicle (which can be called a marking vehicle), and according to the detected marking instructions, the target corner points of the reference frame images of each view are marked respectively (the marked target corner points are recorded as the above-mentioned specified target corner points).

[0033] Exemplarily, the marked specified target corner points can be selected within the co-visible area of the adjacent on-vehicle surround cameras of the calibration vehicle in the reference frame image.

[0034] For example, taking the number of on-vehicle surround cameras as 4 as an example, the front on-vehicle camera can have co-visible areas with the left on-vehicle camera and the right on-vehicle camera respectively. For the reference frame image collected by the front on-vehicle camera, the specified target corner points can be selected from the co-visible area of the front on-vehicle camera and the left on-vehicle camera, and / or the target corner points within the co-visible area of the front on-vehicle camera and the right on-vehicle camera.

[0035] Exemplarily, the marking of the target corner points may include marking the identification of the specified target corner points, the physical coordinates of the specified target corner points, and the position information of the specified target corner points.

[0036] Among them, the physical coordinates of the target corner points may include the physical coordinates of the target corner points in the target world coordinate system.

[0037] Exemplarily, the target world coordinate system (which can also be called the calibration site world coordinate system) may include a world coordinate system established with the center of the calibration site as the coordinate origin.

[0038] Exemplarily, the X / Y axes of the world coordinate system are generally parallel to the main direction of the site (such as the edge of the wall or the direction of the target rows and columns), and the Z axis is perpendicular to the ground.

[0039] It should be noted that when using a large target (such as an extra-large checkerboard) in the calibration site, the center of the target can be used as the center of the calibration site.

[0040] In the case of using multiple scattered small targets in the calibration site, the center point of the calibration site can be the geometric center of these targets.

[0041] In addition, the center of the calibration site can also be manually marked by using high-precision measuring equipment (such as total station, laser tracker) to set fixed marking points at the center of the site.

[0042] Exemplarily, for any reference frame image, the number of designated target corner points marked is greater than or equal to 4, and the marked designated target corner points are not collinear (that is, there is no straight line passing through all the designated target corner points in the image plane).

[0043] In the embodiments of the present application, for any vehicle to be calibrated, the reference frame information of the same calibration site can be obtained according to the current calibration site, and the external parameters of the on-vehicle surround-view camera of the vehicle to be calibrated can be calibrated according to the obtained reference frame information.

[0044] Exemplarily, in the process of calibrating the external parameters of the camera of the vehicle to be calibrated, on the one hand, the on-vehicle surround-view camera of the vehicle to be calibrated (which can be called the on-vehicle surround-view camera to be calibrated) can be used to collect images of the current calibration scene to obtain the images to be calibrated; on the other hand, according to the current calibration scene, the reference frame information of the same calibration site can be obtained, and according to the obtained images to be calibrated and the reference frame information of the same calibration site, the external parameters of the on-vehicle surround-view camera to be calibrated can be calibrated to determine the external parameters of the on-vehicle surround-view camera to be calibrated.

[0045] It should be noted that in the embodiments of the present application, in the process of calibrating the external parameters of the camera of any vehicle to be calibrated, in the case that it is determined according to the current calibration site that there is no reference frame information of the same calibration site, the reference frame information of the current calibration site can also be generated by means of real-time marking, and then, according to the reference frame information of the current calibration site, the external parameters of the on-vehicle surround-view camera of the vehicle to be calibrated can be calibrated.

[0046] It can be seen that in Figure 1In the method flow shown, by collecting reference frame images for the calibration site and generating reference frame information corresponding to the reference frame images according to the detected annotation instructions, during the process of calibrating the external parameters of the camera of the vehicle to be calibrated, the reference frame information of the same calibration site can be obtained according to the current calibration site, and the external parameters of the on-vehicle surround-view camera to be calibrated can be calibrated according to the obtained reference frame information. By setting the reference frame information for the calibration site, in the case where there is reference frame information for the same calibration site, for changes in other parameters such as the physical environment, camera model, and vehicle model, there is no need to re-upgrade the calibration algorithm, and the versatility of the calibration scheme can be improved while ensuring the calibration accuracy. The implementation of calibrating the external parameters of the on-vehicle surround-view camera according to the reference frame map information will be described below; in addition, in the case where the calibration site changes, only the reference frame information needs to be updated to support the calibration of the external parameters of the camera for the new calibration site, which improves the scalability of the scheme and effectively reduces the development cost of expanding the calibration site.

[0047] Please refer to Figure 3 , which is a schematic flow diagram of a method for calibrating the external parameters of a camera provided by an embodiment of the present application. As Figure 3 shown, the method for calibrating the external parameters of the camera may include the following steps: Step S300, obtaining an image to be calibrated; wherein, the image to be calibrated is obtained by the on-vehicle surround-view camera to be calibrated collecting images of the calibration site.

[0048] In an embodiment of the present application, for any vehicle to be calibrated, during the process of calibrating the external parameters of the camera of the vehicle to be calibrated, images of the calibration site can be collected by the on-vehicle surround-view camera to be calibrated to obtain an image to be calibrated.

[0049] Step S310, determining the information of the specified target corner points in the image to be calibrated according to the reference frame information of the same calibration site; wherein, the information of the specified target corner points includes the identification information of the specified target corner points in the image to be calibrated, the physical coordinates of the specified target corner points, and the position information of the specified target corner points in the image to be calibrated.

[0050] In an embodiment of the present application, in the case where an image to be calibrated is obtained, the reference frame information of the same calibration site can be obtained according to the calibration site where the image to be calibrated is collected, and the information of the specified target corner points in the image to be calibrated can be determined according to the obtained reference frame information.

[0051] Exemplarily, for the images to be calibrated with different views, in the obtained reference frame information of the same calibration site, there is reference frame information corresponding to each view.

[0052] The information of the specified target corner points in the image to be calibrated with this view can be determined according to the reference frame information of the same view.

[0053] It should be noted that in the embodiments of the present application, for any calibration site, when it is determined that there is no reference frame information of the same calibration site currently, the reference frame information of the calibration site can also be generated by real-time annotation. Furthermore, based on the reference frame information of the calibration site, the extrinsic calibration of the on-vehicle surround-view camera of the vehicle to be calibrated at the calibration site can be performed.

[0054] Step S320: Determine the extrinsic parameters of the on-vehicle surround-view camera to be calibrated based on the specified target corner point information in the image to be calibrated and the intrinsic parameters of the on-vehicle surround-view camera to be calibrated.

[0055] In the embodiments of the present application, when the specified target corner point information in the image to be calibrated is determined as above, during the reference frame calibration process, the physical coordinates of the specified target corner points are also marked, that is, the physical coordinates of the specified target corner points are known conditions. Therefore, the extrinsic parameters of the on-vehicle camera to be calibrated can be determined based on the physical coordinates of each specified target corner point, the position information of the specified target corner point in the image to be calibrated, and the intrinsic parameters of the on-vehicle surround-view camera to be calibrated.

[0056] It should be noted that in the embodiments of the present application, since the reference frame information is marked in the target world coordinate system, therefore, the extrinsic parameters of the on-vehicle camera to be calibrated directly determined based on the physical coordinates of each specified target corner point, the position information of the specified target corner point in the image to be calibrated, and the intrinsic parameters of the on-vehicle surround-view camera to be calibrated are the extrinsic parameters in the target world coordinate system, and they can be further converted into the extrinsic parameters in the vehicle body world coordinate system.

[0057] Exemplarily, when the origin of the vehicle body world coordinate system coincides with the origin of the target world coordinate system (a tolerable deviation is allowed), for example, for the factory calibration environment, since there is a centering device in the factory, generally, the vehicle parking will ensure that the center of the vehicle body coincides with the center of the target world coordinate system. In this case, the extrinsic parameters in the target world coordinate system are usually the same as the extrinsic parameters in the vehicle body world coordinate system.

[0058] When the origin of the vehicle body world coordinate system does not coincide with the origin of the target world coordinate system, the offset between the center of the vehicle body and the center of the target world coordinate system can be obtained. For example, the offset between the center of the vehicle body and the center of the target world coordinate system can be obtained by measurement. Furthermore, based on the offset between the center of the vehicle body and the center of the target world coordinate system, the extrinsic parameters in the target world coordinate system can be converted into the extrinsic parameters in the vehicle body world coordinate system.

[0059] It can be seen that in Figure 3In the method flow shown, by pre-setting corresponding reference frame information for different calibration sites, during the process of calibrating the external parameters of the vehicle-mounted surround-view camera, for the to-be-calibrated image obtained by collecting images of the calibration site through the to-be-calibrated vehicle-mounted surround-view camera, the specified target corner point information in the to-be-calibrated image can be determined based on the reference frame information of the same calibration site. Furthermore, based on the specified target corner point information in the to-be-calibrated image and the internal parameters of the to-be-calibrated vehicle-mounted surround-view camera, the external parameters of the to-be-calibrated vehicle-mounted surround-view camera can be determined. By setting the reference frame information for the calibration site, in the case where there is reference frame information of the same calibration site, for changes in other parameters such as the physical environment, camera model, vehicle model, etc., there is no need to re-upgrade the calibration algorithm, and the versatility of the calibration scheme can be improved while ensuring calibration accuracy; in addition, in the case where the calibration site changes, only the reference frame information needs to be updated to support the calibration of the external parameters of the camera for the new calibration site, which improves the scalability of the scheme and effectively reduces the development cost of expanding the calibration site.

[0060] In some embodiments, the above-mentioned determining the specified target corner point information in the to-be-calibrated image based on the reference frame information of the same calibration site may include: For any to-be-calibrated image, based on the reference frame image of the corresponding view under the same calibration site, perform corner point matching on the to-be-calibrated image and the reference frame image to determine the specified target corner point information in the to-be-calibrated image.

[0061] Exemplarily, by performing corner point matching on the to-be-calibrated image and the reference frame image under the same calibration site, based on the specified target corner points marked in the reference frame image, the specified target corner points in the to-be-calibrated image can be determined.

[0062] Correspondingly, for any to-be-calibrated image, based on the reference frame image of the corresponding view under the same calibration site, perform corner point matching on the to-be-calibrated image and the reference frame image to determine the specified target corner points in the to-be-calibrated image. Furthermore, determine the specified target corner point information in the to-be-calibrated image, that is, determine the identification information of the specified target corner points in the to-be-calibrated image and the position information of the specified target corner points in the to-be-calibrated image.

[0063] For example, taking the number of vehicle-mounted surround-view cameras as 4, for the to-be-calibrated front-view image (i.e., the to-be-calibrated image collected by the front vehicle camera), based on the front-view reference frame image under the same calibration site (the image collected by the front vehicle camera and used for marking the specified target corner points during the reference frame calibration), perform corner point matching on the to-be-calibrated front-view image and the front-view reference frame image to determine the specified target corner points in the to-be-calibrated front-view image. Furthermore, determine the specified target corner point information in the to-be-calibrated front-view image.

[0064] Among them, assuming that the front view reference frame image includes designated target corner points 1 to 4, the position information of the designated target corner points 1 to 4 in the front view image to be calibrated can be determined respectively.

[0065] In some embodiments, the corner point matching of the image to be calibrated and the reference frame image based on the reference frame images of corresponding views under the same calibration site, and determining the designated target corner point information in the image to be calibrated may include: Extract feature points from the image to be calibrated and the reference frame image respectively, to obtain a first set of feature points in the image to be calibrated and a second set of feature points in the reference frame image; Perform feature point matching on the first set of feature points and the second set of feature points to obtain mutually matching first feature points and second feature points; Determine the target first feature point that matches the target second feature point as the designated target corner point in the image to be calibrated; where the target second feature point is the second feature point in the reference frame image that is closest to the designated target corner point; Determine the designated target corner point information based on the designated target corner points in the image to be calibrated.

[0066] Exemplarily, for any image to be calibrated, feature points can be extracted from the image to be calibrated and the reference frame image of the same view under the same calibration site respectively, to obtain a set of feature points in the image to be calibrated (which can be called the first set of feature points) and a set of feature points in the reference frame image (which can be called the second set of feature points).

[0067] Exemplarily, the feature point extraction algorithm may include but is not limited to traditional ORB (Oriented FAST and Rotated BRIEF) algorithm, SIFT (Scale-Invariant Feature Transform) algorithm, SURF (Speeded-Up Robust Features) algorithm, or deep learning corner point extraction algorithms (such as SuperPoint), etc.

[0068] For the above first set of feature points and the second set of feature points, feature point matching can be performed on the first set of feature points and the second set of feature points to determine mutually matching first feature points and second feature points.

[0069] Exemplarily, the feature point matching method may include but is not limited to traditional RANSAC (Random Sample Consensus) matching algorithm, brute-force matching algorithm, or deep learning algorithms such as SuperGlue and LightGlue (image matching algorithms based on deep learning), etc.

[0070] In one example, a deep learning-based feature point extraction and matching technique can be used to extract and match feature points of the image to be calibrated and the reference frame image.

[0071] That is, a deep learning-based feature point extraction method can be used to extract feature points from the image to be calibrated and the reference frame image respectively; and, A deep learning-based feature point matching method is used to match feature points between the first set of feature points and the second set of feature points.

[0072] Exemplarily, the feature point extraction network supports the matching of the center position of the circular region. The feature point extraction network outputs the positions and descriptors of the feature points, and the feature matching network outputs the position coordinates of the matching points.

[0073] In this example, the feature points can include not only the "X"-type corner points of the checkerboard, the "L"-type corner points of the square, the "L"-type corner points of the double-square, but also the "O"-type corner points of the circle, etc.

[0074] That is, in the embodiments of the present application, the target pattern is no longer limited to the checkerboard target, the square target, and the double-square target, but can also be a circular target (the corner points of the circular target are the centers).

[0075] Exemplarily, when the mutually matching first feature point and second feature point are determined, the second feature point closest to the specified target corner point in the reference frame image (which can be called the target second feature point) can be used as the specified target corner point in the reference frame image, and the first feature point matching the target feature point in the image to be calibrated (which can be called the target first feature point) is determined as the specified target corner point in the image to be calibrated.

[0076] In some embodiments, determining the external parameters of the vehicle-mounted surround-view camera to be calibrated based on the information of the specified target corner points in the image to be calibrated and the internal parameters of the vehicle-mounted surround-view camera to be calibrated includes: For any vehicle-mounted surround-view camera to be calibrated, based on the internal parameters of the vehicle-mounted surround-view camera to be calibrated, the position information of the target specified target corner point in the image to be calibrated collected by the vehicle-mounted surround-view camera to be calibrated, and the physical coordinates of the target specified target corner point, the homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated is determined; wherein, the target specified target corner point is the specified target corner point in the image to be calibrated collected by the vehicle-mounted surround-view camera to be calibrated; Based on the homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated and the physical coordinates of the target specified target corner point, the target specified target corner point is projected onto the image to be calibrated collected by the vehicle-mounted surround-view camera to be calibrated, and the projected position of the target specified target corner point in the image to be calibrated is obtained; Determine the reprojection error of the target-specified target corner point based on the projection position of the target-specified target corner point in the image to be calibrated and the position information of the target-specified target corner point in the image to be calibrated. When it is determined that the calibration of the vehicle-mounted surround-view camera to be calibrated is successful based on the reprojection error of the target-specified target corner point, determine the external parameters of the vehicle-mounted surround-view camera to be calibrated according to the homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated.

[0077] Exemplarily, when the information of the specified target corner point in the image to be calibrated is determined in the above manner, for any vehicle-mounted surround-view camera, the physical coordinates of the target-specified target corner point can be determined based on the identification information of the specified target corner point (which can be called the target-specified target corner point) in the image to be calibrated collected by the vehicle-mounted surround-view camera to be calibrated.

[0078] Among them, during the calibration process of the reference frame, the physical coordinates of the specified target corner points with different identification information are also marked, that is, they are known conditions. Therefore, the homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated can be determined based on the internal parameters of the vehicle-mounted surround-view camera to be calibrated, the position information of the target-specified target corner point in the image to be calibrated collected by the vehicle-mounted surround-view camera to be calibrated, and the physical coordinates of the target-specified target corner point. Its specific implementation will be described in the following in combination with specific examples.

[0079] When the homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated is determined, the target-specified target corner point can be projected onto the image to be calibrated collected by the vehicle-mounted surround-view camera to be calibrated according to the homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated and the physical coordinates of the target-specified target corner point, to obtain the projection position of the target-specified target corner point in the image to be calibrated, and determine the reprojection error of the target-specified target corner point based on the projection position of the target-specified target corner point in the image to be calibrated and the position information of the target-specified target corner point in the image to be calibrated. Its specific implementation will be described in the following in combination with specific examples.

[0080] Exemplarily, it is possible to determine whether the calibration of the vehicle-mounted surround-view camera to be calibrated is successful based on the reprojection error of the target-specified target corner point.

[0081] In one example, determining that the calibration of the vehicle-mounted surround-view camera to be calibrated is successful based on the reprojection error of the target-specified target corner point may include: Determine that the calibration of the vehicle-mounted surround-view camera to be calibrated is successful when the reprojection error of each target-specified target corner point is less than the first error threshold, and / or when the average reprojection error of the target-specified target corner point is less than the second error threshold.

[0082] Exemplarily, it is possible to determine whether the on-vehicle surround-view camera to be calibrated is successfully calibrated based on the reprojection error of each target-specified target corner point and / or the average reprojection error of the target-specified target corner points.

[0083] As an example, it is possible to determine that the on-vehicle surround-view camera to be calibrated is successfully calibrated when the reprojection error of each target-specified target corner point is less than a preset error threshold (which can be referred to as the first error threshold); otherwise, it is determined that the on-vehicle surround-view camera to be calibrated fails to be calibrated.

[0084] As another example, it is possible to determine that the on-vehicle surround-view camera to be calibrated is successfully calibrated when the average reprojection error of the target-specified target corner points is less than a preset error threshold (which can be referred to as the second error threshold); otherwise, it is determined that the on-vehicle surround-view camera to be calibrated fails to be calibrated.

[0085] As yet another example, it is possible to determine that the on-vehicle surround-view camera to be calibrated is successfully calibrated when the reprojection error of each target-specified target corner point is less than the first error threshold and the average reprojection error of the target-specified target corner points is less than the second error threshold; otherwise, it is determined that the on-vehicle surround-view camera to be calibrated fails to be calibrated.

[0086] Exemplarily, when it is determined that the on-vehicle surround-view camera to be calibrated is successfully calibrated based on the reprojection error of the target-specified target corner points, the external parameters of the on-vehicle surround-view camera to be calibrated are determined according to the homography matrix corresponding to the on-vehicle surround-view camera to be calibrated.

[0087] For example, according to the homography matrix corresponding to the on-vehicle surround-view camera to be calibrated and the internal parameters of the on-vehicle surround-view camera to be calibrated, the external parameters of the camera can be obtained by using SVD (Singular Value Decomposition), such as the camera position (the coordinates of the camera in the vehicle body world coordinate system) and the installation angle.

[0088] In one example, the above determination of the homography matrix corresponding to the on-vehicle surround-view camera to be calibrated based on the internal parameters of the on-vehicle surround-view camera to be calibrated, the position information of the target-specified target corner points in the calibration image collected by the on-vehicle surround-view camera to be calibrated, and the physical coordinates of the target-specified target corner points may include: Determining an initial homography matrix corresponding to the on-vehicle surround-view camera to be calibrated based on the internal parameters of the on-vehicle surround-view camera to be calibrated, the position information of the target-specified target corner points in the calibration image collected by the on-vehicle surround-view camera to be calibrated, and the physical coordinates of the target-specified target corner points; Optimizing the initial homography matrix by using a pre-optimization algorithm to obtain the final homography matrix corresponding to the on-vehicle surround-view camera to be calibrated; Based on the homography matrix corresponding to the in-vehicle surround-view camera to be calibrated, and the physical coordinates of the target calibration target corner points, projecting the target calibration target corner points onto the image to be calibrated collected by the in-vehicle surround-view camera to be calibrated, obtaining the projection positions of the target calibration target corner points in the image to be calibrated, may include: Based on the final homography matrix corresponding to the in-vehicle surround-view camera to be calibrated, and the physical coordinates of the target calibration target corner points, projecting the target calibration target corner points onto the image to be calibrated collected by the in-vehicle surround-view camera to be calibrated, obtaining the projection positions of the target calibration target corner points in the image to be calibrated.

[0089] Exemplarily, in order to improve the calibration accuracy, when the initial homography matrix corresponding to the in-vehicle surround-view camera to be calibrated is determined in the above manner, a preset optimization algorithm can be used to optimize the initial homography matrix corresponding to the in-vehicle surround-view camera to be calibrated to obtain the final homography matrix.

[0090] For example, when the initial homography matrix corresponding to the in-vehicle surround-view camera to be calibrated is determined in the above manner, the LM (Levenberg-Marquardt) algorithm can be used to optimize the initial homography matrix corresponding to the in-vehicle surround-view camera to be calibrated to obtain the final homography matrix.

[0091] When the final homography matrix of the in-vehicle surround-view camera to be calibrated is obtained, the reprojection error of the target calibration target corner points can be determined based on the final homography matrix, and whether the in-vehicle surround-view camera to be calibrated is successfully calibrated can be determined based on the reprojection error of the target calibration target corner points.

[0092] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of the present application, the technical solutions provided in the embodiments of the present application will be described below in conjunction with specific examples.

[0093] This embodiment provides a general camera extrinsic parameter calibration scheme. By setting reference frame information to assist in obtaining the corner point information (the above-mentioned calibration target corner point information) in the image to be calibrated, the camera extrinsic parameter calibration of the vehicle can be completed, which can be compatible with different calibration sites, different vehicle models, and different application scenarios.

[0094] Exemplarily, the extrinsic parameters of the in-vehicle surround-view cameras of each camera can be calibrated by detecting the ordered corner points of the calibration target on the image and combining the known physical size information of the calibration target (such as the length and width of the calibration target, the distance from the center of the vehicle body, etc.) and using the PNP algorithm.

[0095] Exemplarily, the reference frame information and image matching technology can be used to complete the detection and sorting of the calibration target corner points on the image to be calibrated (taking the identification information of the calibration target corner points as the corner point serial numbers as an example, the specified calibration target corner points can be identified by the serial numbers).

[0096] First, the calibration system device will be described below.

[0097] In this embodiment, the schematic diagram of the calibration system device can be referred to Figure 4 , such as Figure 4 shown. The calibration system device may include: a camera (i.e., a vehicle surround-view camera), a controller, a memory, and a display device. Among them: The camera is used to collect the images to be calibrated; the controller is used to execute corresponding control processing according to the detected control instructions; the memory is used to store reference frame information; the display device is used to display the calibration operation interface.

[0098] Exemplarily, based on Figure 4 the calibration system device shown, the external parameter calibration process of the camera can be as follows: S1. According to the detected calibration site selection operation instruction, determine the corresponding calibration site pattern.

[0099] Exemplarily, the calibration operation interface may provide a "calibration site selection" function button. When the selection instruction of this function button is detected, candidate calibration site patterns can be displayed in the calibration operation interface. Furthermore, the selected calibration site pattern can be determined according to the selection instruction for the calibration site pattern.

[0100] S2. According to the detected reference frame information selection instruction, determine the corresponding reference frame information.

[0101] Exemplarily, when the selected calibration site pattern is determined, the reference frame information corresponding to the calibration scene pattern can be displayed in the display interface. When the selection instruction for this reference frame information is detected, it can be determined to use the selected reference frame information for the external parameter calibration of the camera.

[0102] S3. When the start calibration operation instruction is detected, perform the external parameter calibration of the camera on the vehicle to be calibrated according to the determined reference frame information.

[0103] Exemplarily, when it is determined that the calibration of the vehicle to be calibrated is successful, the bird's-eye view effect of the calibration image can be displayed.

[0104] Next, the specific implementation process of the external parameter calibration of the camera will be described.

[0105] In this embodiment, the specific implementation process of the external parameter calibration of the camera can be referred to Figure 5 , such as Figure 5 shown. The specific implementation process of the external parameter calibration of the camera may include: 1) Obtain the images to be calibrated.

[0106] Exemplarily, taking the number of in-vehicle surround-view cameras as 4 as an example, a schematic diagram of the image to be calibrated can be seen in Figure 6 , such as Figure 6 . As shown in Figure 6 , from left to right and from top to bottom in are the front-view image to be calibrated (which can be simply referred to as the to-be-calibrated front view), the rear-view image to be calibrated (which can be simply referred to as the to-be-calibrated rear view), the left-view image to be calibrated (which can be simply referred to as the to-be-calibrated left view), and the right-view image to be calibrated (which can be simply referred to as the to-be-calibrated right view).

[0107] 2) Obtain the reference frame information under the same calibration site.

[0108] Exemplarily, the reference frame information may include the serial numbers and position coordinates of the specified target corner points in the reference frame image.

[0109] Exemplarily, a schematic diagram of the reference frame image can be seen in Figure 7 , such as Figure 7 . As shown in Figure 7 , from left to right and from top to bottom in are the front-view reference frame image (which can be simply referred to as the reference frame front view), the rear-view reference frame image (which can be simply referred to as the reference frame rear view), the left-view reference frame image (which can be simply referred to as the reference frame left view), and the right-view reference frame image (which can be simply referred to as the reference frame right view).

[0110] Exemplarily, the reference frame image may be the original fisheye image collected by the in-vehicle surround-view camera during the calibration process, or the undistorted image obtained by performing undistortion processing on the original fisheye image.

[0111] It should be noted that in the case where the reference frame image is the original fisheye image, the image to be calibrated also uses the original fisheye image; in the case where the reference frame image is the undistorted image, the image to be calibrated also uses the undistorted image.

[0112] Exemplarily, the number of reference frames in each reference frame image is greater than or equal to 4 ( Figure 7 taking 8 as an example), non-collinear, and within the common view area of adjacent in-vehicle surround-view cameras.

[0113] Exemplarily, the vehicle model, acquisition environment, and camera resolution of the reference frame image and the image to be calibrated can be the same or different.

[0114] For example, the reference frame image and the image to be calibrated can be collected by the in-vehicle surround-view cameras of the same vehicle model, or can be images collected by the in-vehicle surround-view cameras of different vehicle models.

[0115] The reference frame image (or the image to be calibrated) can be collected in the factory calibration board environment or by laying a calibration cloth in the outdoor environment of a 4S store; the resolution of the camera for collection can be 1280*720 or other resolutions such as 1280*960.

[0116] For example, for any calibration site, during the process of obtaining the reference frame image of the calibration site, a set of fisheye original images (including front, rear, left, and right views) of the calibration site can be obtained through a calibration vehicle, and according to the detected annotation instructions, the serial numbers and position information of the specified target corner points can be determined.

[0117] 3) Determine the specified target corner point information in the image to be calibrated based on the reference frame information.

[0118] Exemplarily, image matching technology can be used to perform image matching for each image to be calibrated and the reference frame image of the corresponding view to obtain the corner point matching relationship between the two images.

[0119] Exemplarily, the above image matching technology can adopt image feature point matching technology or image dense matching technology.

[0120] In one example, in order to reduce the computational time consumption of image matching and improve the processing efficiency, the image matching technology can adopt image feature point matching technology.

[0121] Exemplarily, the feature point matching technology can include feature point extraction and feature point matching.

[0122] Exemplarily, the feature point extraction algorithm can include but is not limited to traditional algorithms such as ORB algorithm, SIFT algorithm, SURF algorithm, or deep learning corner point extraction algorithms (such as SuperPoint), etc.

[0123] Exemplarily, the feature point matching method can include but is not limited to traditional algorithms such as RANSAC matching algorithm, brute-force matching algorithm, or deep learning algorithms such as SuperGlue and LightGlue, etc.

[0124] In one example, in order to obtain a more robust matching effect, the feature point extraction and matching technology based on deep learning can be used to implement the feature point extraction and matching of the image to be calibrated and the reference frame image.

[0125] Exemplarily, the feature point extraction network supports the matching of the center position of the circular area. The feature point extraction network outputs the position and descriptor of the feature points, and the feature matching network outputs the position coordinates of the matching points.

[0126] Exemplarily, the feature points can include not only the "X" - type corner points of the checkerboard, the "L" - type corner points of the square, the "L" - type corner points of the double - square, but also the "O" - type corner points of the circle (the center of the circle pattern is the corner point), etc.

[0127] Exemplarily, through the above - mentioned feature - point matching technology, for the same calibration site, the to - be - calibrated images and reference - frame images collected by vehicles with different vehicle lengths can be correctly matched to the center position of the circular target. The schematic diagram can be seen in Figure 8 . As Figure 8 shown, in the order from left to right and from top to bottom, they are the schematic diagrams of the matching results of the front view, the rear view, the left view, and the right view respectively.

[0128] In addition, for the same calibration site, under different target materials and different calibration environments, the images can be correctly matched to the corner points of the checkerboard target. The schematic diagram can be seen in Figure 9 . As Figure 9 shown, in the order from left to right and from top to bottom, they are the schematic diagrams of the matching results of the front view, the rear view, the left view, and the right view respectively.

[0129] Exemplarily, according to the serial numbers and coordinate information of the specified target corner points already marked in the reference - frame image, using the image - matching information, the corner points in the to - be - calibrated image that are closest to the specified target corner points are selected, and the serial numbers of the corner points marked in the reference - frame image are assigned to the specified target corner points in the to - be - calibrated image.

[0130] Taking the front view as an example, after screening, only the corner points corresponding to the marked corner points on the reference - frame image are retained, and the abnormal results in the image - matching results can be eliminated. The schematic diagram can be seen in Figure 10 . The screening of other views is the same.

[0131] Among them, as Figure 10 shown, the left side is the front view of the reference - frame, and the right side is the target corner points screened out in the to - be - calibrated image. 4) According to the known physical coordinates of the specified target corner points and the camera internal parameters, use PNP to calculate the homography matrix.

[0132] Exemplarily, according to the image coordinates, corner - point serial numbers, target - size information, etc. of the specified target corner points in the to - be - calibrated image, the known physical coordinates corresponding to the specified target corner points (which can be the X and Y coordinates with the vehicle - body center as the origin) can be obtained, and the homography matrix of each view is calculated using PNP.

[0133] The following introduces an example of using DLT (Direct Linear Transformation) to solve the homography matrix:

[0134] Among them, X and Y represent positions in the world coordinate system, u and v represent the pixel coordinates of the target corner points on the undistorted image, and h0 to h8 are the values of the elements in the 3×3 homography matrix.

[0135] Eliminate s using the last row to obtain two constraints:

[0136] Arrange them into a system of linear equations:

[0137] When the number of matching points is greater than or equal to 4, the above equation has a solution. That is, the solution of the system of linear equations AH = 0 is the T eigenvector corresponding to the smallest eigenvalue of A.

[0138] Exemplarily, the eigenvalues and eigenvectors can be solved by the Jacobi iteration method.

[0139] In addition, other methods for solving systems of linear equations can also be used: SVD decomposition, QR decomposition, LU decomposition, Cholesky decomposition, etc.

[0140] 5) According to the current homography matrix, determine the reprojection error of the specified target corner points in the image to be calibrated, and when it is determined that the calibration is successful based on the reprojection error, determine the external parameters of the camera corresponding to the current homography matrix as the external parameters of the vehicle-mounted surround-view camera to be calibrated.

[0141] Exemplarily, when the homography matrix of the vehicle-mounted surround-view camera to be calibrated is obtained, the reprojected pixel points of the world coordinates of the specified target corner points on the image can be obtained through the relationship between the world coordinate points and the homography matrix.

[0142] Exemplarily, when the initial homography matrix of the vehicle-mounted surround-view camera to be calibrated is obtained, LM optimization can be used to optimize the homography matrix by optimizing the sum of the reprojection errors (RMS) of the corner points in the co-visible area, so as to obtain the final homography matrix and make the calibration accuracy higher.

[0143] When it is determined that the calibration of the vehicle-mounted surround-view camera is successful based on the reprojection error, the external parameters of the camera (including the camera position and installation angle) can be obtained by combining the homography matrix and the camera internal parameters using SVD decomposition.

[0144] Exemplarily, for the determined homography matrix, by setting a threshold, it can be judged whether the RMS of this calibration meets the requirements, so as to output whether the calibration is successful or failed. Among them:

[0145]

[0146] Among them, (u, v) represents the pixel coordinates in the image obtained by projecting the world coordinates of the specified target corner points, and (x, y) represents the pixel coordinates of the specified target corner points extracted from the image to be calibrated. N represents the number of specified target corner points in the image.

[0147] Exemplarily, calibration can be determined to be successful when the average image reprojection error of all specified target corner points is less than a threshold (e.g., 0.5 pix).

[0148] It can be seen that the camera extrinsic parameter calibration scheme provided by the embodiments of the present application obtains the target corner point information in the image to be calibrated by introducing a reference frame, thereby completing the camera extrinsic parameter calibration, and is compatible with different vehicle models and diverse business application scenarios. Among them: 1) When the calibration site changes, only the reference frame information needs to be re-annotated, and the algorithm does not need to be upgraded and adapted to support the project.

[0149] 2) When the application scenario, vehicle model, and camera model (including resolution) change, the reference frame information does not need to be changed, and the algorithm does not need to be upgraded and adapted to support the project.

[0150] 3) When the number of vehicle-mounted surround-view cameras changes, if the calibration site remains unchanged, there is no need to update. If the calibration site changes, only the reference frame information needs to be updated.

[0151] Among them, the calibration site refers to the laid target pattern and quantity. For example, assuming that the target laying scheme remains unchanged and the number of vehicle-mounted surround-view cameras changes, the reference frame can be reused, and only the reference frame corresponding to each vehicle-mounted surround-view camera needs to be selected respectively. That is, when the calibration site changes, only the reference frame information needs to be updated, and the algorithm does not need to be upgraded to support the camera extrinsic parameter calibration; when the camera model is changed or the vehicle model is changed, there is no need to re-upgrade the calibration algorithm, and the project scheme can be quickly supported. The usability is high and the calibration accuracy is high, which can well solve the pain points of the industry.

[0152] In addition, the reference frame information is used to obtain the target corner point information in the image to be calibrated, and then PNP is used for calibration. The calibration accuracy is high, it does not depend on the extrinsic parameter information of the reference frame, and there is also a certain tolerance for the annotation accuracy of the target corner points in the reference frame, and the algorithm has strong robustness.

[0153] The method provided by the present application has been described above. Next, the device provided by the present application will be described: Please refer to Figure 11 , which is a schematic structural diagram of a camera extrinsic parameter calibration device provided by the embodiments of the present application. As Figure 11 shown, the camera extrinsic parameter calibration device may include: A generating unit is configured to, for any calibration site, collect a reference frame image of the calibration site through an on-vehicle surround-view camera of a calibration vehicle, and generate reference frame information corresponding to the reference frame image according to a detected annotation instruction; wherein, the reference frame information includes identification information of specified target corner points in the reference frame image, physical coordinates of the specified target corner points, and position information of the specified target corner points in the reference frame image; the number of specified target corner points is greater than or equal to 4, and all are located in the co-viewing area of adjacent on-vehicle surround-view cameras of the calibration vehicle in the reference frame image and are not collinear. A processing unit is configured to, for any vehicle to be calibrated, obtain the reference frame information of the same calibration site according to the current calibration site, and perform external parameter calibration on the on-vehicle surround-view camera to be calibrated according to the obtained reference frame information.

[0154] Exemplarily, the specific implementation process of the generating unit and the processing unit for realizing camera external parameter calibration can refer to the relevant descriptions in the above embodiments, and the embodiments of the present application will not elaborate herein.

[0155] Please refer to Figure 12 , which is a schematic structural diagram of a camera external parameter calibration device provided by an embodiment of the present application. As Figure 12 shown, the camera external parameter calibration device may include: An acquisition unit is configured to acquire an image to be calibrated; wherein, the image to be calibrated is obtained by an on-vehicle surround-view camera to be calibrated collecting an image of a calibration site. A determination unit is configured to determine specified target corner point information in the image to be calibrated according to the reference frame information of the same calibration site; wherein, the specified target corner point information includes identification information of specified target corner points in the image to be calibrated, physical coordinates of the specified target corner points, and position information of the specified target corner points in the image to be calibrated; the reference frame information includes identification information of the specified target corner points in the reference frame image, physical coordinates of the specified target corner points, and position information of the specified target corner points in the reference frame image; during the calibration process, the reference frame image is obtained by each on-vehicle surround-view camera of the calibration vehicle collecting an image of the calibration site; the number of specified target corner points is greater than or equal to 4, and all are located in the co-viewing area of adjacent on-vehicle surround-view cameras of the calibration vehicle in the reference frame image and are not collinear. A processing unit is configured to determine the external parameters of the on-vehicle surround-view camera to be calibrated according to the specified target corner point information in the image to be calibrated and the internal parameters of the on-vehicle surround-view camera to be calibrated.

[0156] Exemplarily, the specific implementation process of the acquisition unit, the determination unit, and the processing unit for realizing camera external parameter calibration can refer to the relevant descriptions in the above embodiments, and the embodiments of the present application will not elaborate herein.

[0157] An embodiment of the present application provides an electronic device, including a processor and a memory. The memory stores machine-executable instructions that can be executed by the processor, and the processor is configured to execute the machine-executable instructions to implement the camera extrinsic parameter calibration method described above.

[0158] Please refer to Figure 13 , which is a schematic hardware structure diagram of an electronic device provided by an embodiment of the present application. The electronic device may include a processor 1301 and a memory 1302 storing machine-executable instructions. The processor 1301 and the memory 1302 may communicate via a system bus 1303. And by reading and executing the machine-executable instructions corresponding to the camera extrinsic parameter calibration logic in the memory 1302, the processor 1301 can execute the camera extrinsic parameter calibration method described above.

[0159] The memory 1302 mentioned in this article can be any electronic, magnetic, optical or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or a combination thereof.

[0160] In some embodiments, a machine-readable storage medium is also provided, such as Figure 13 the memory 1302 in , and the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by the processor, the camera extrinsic parameter calibration method described above is implemented. For example, the storage medium can be ROM, RAM, CD-ROM, magnetic tapes, floppy disks, and optical data storage devices, etc.

Claims

1. A method for calibrating the external parameters of a camera, characterized in that, Including: Obtain an image to be calibrated; wherein, the image to be calibrated is obtained by an on-vehicle surround-view camera to be calibrated collecting images of a calibration site; Determine the specified target corner point information in the image to be calibrated according to the reference frame information of the same calibration site; wherein, the specified target corner point information includes: The identification information of the specified target corner point in the image to be calibrated; The physical coordinates of the specified target corner point; and, The position information of the specified target corner point in the image to be calibrated; The reference frame information includes: The identification information of the specified target corner point in the reference frame image; The physical coordinates of the specified target corner point; and, The position information of the specified target corner point in the reference frame image; wherein, during the calibration process, the reference frame image is obtained by each on-vehicle surround-view camera of a calibration vehicle collecting images of the calibration site; the number of the specified target corner points is greater than or equal to 4, and all are located in the common view area of adjacent on-vehicle surround-view cameras of the calibration vehicle in the reference frame image and are not collinear; Determine the external parameters of the on-vehicle surround-view camera to be calibrated according to the specified target corner point information in the image to be calibrated and the internal parameters of the on-vehicle surround-view camera to be calibrated.

2. The method according to claim 1, wherein The determining the specified target corner point information in the image to be calibrated according to the reference frame information of the same calibration site includes: For any image to be calibrated, perform corner point matching on the image to be calibrated and the reference frame image according to the reference frame image of the corresponding view under the same calibration site, and determine the specified target corner point information in the image to be calibrated.

3. The method according to claim 2, characterized in that, The performing corner point matching on the image to be calibrated and the reference frame image according to the reference frame image of the corresponding view under the same calibration site and determining the specified target corner point information in the image to be calibrated includes: Extract feature points from the image to be calibrated and the reference frame image respectively, to obtain a first set of feature points in the image to be calibrated and a second set of feature points in the reference frame image; Perform feature point matching on the first set of feature points and the second set of feature points to obtain mutually matching first feature points and second feature points; Determine the target first feature point that matches the target second feature point as the specified target corner point in the image to be calibrated; wherein, the target second feature point is the second feature point closest to the specified target corner point in the reference frame image; Determine the specified target corner point information according to the specified target corner point in the image to be calibrated.

4. The method according to claim 1, characterized in that, The determining the external parameters of the on-vehicle surround-view camera to be calibrated according to the specified target corner point information in the image to be calibrated and the internal parameters of the on-vehicle surround-view camera to be calibrated includes: For any on-vehicle surround-view camera to be calibrated, determine the homography matrix corresponding to the on-vehicle surround-view camera to be calibrated according to the internal parameters of the on-vehicle surround-view camera to be calibrated, the position information of the target specified target corner point in the image to be calibrated collected by the on-vehicle surround-view camera to be calibrated, and the physical coordinates of the target specified target corner point; wherein, the target specified target corner point is the specified target corner point in the image to be calibrated collected by the on-vehicle surround-view camera to be calibrated. According to the homography matrix corresponding to the on-vehicle surround-view camera to be calibrated, and the physical coordinates of the target calibration target corner points, project the target calibration target corner points onto the calibration image collected by the on-vehicle surround-view camera to be calibrated, and obtain the projection positions of the target calibration target corner points in the calibration image; According to the projection positions of the target calibration target corner points in the calibration image, and the position information of the target calibration target corner points in the calibration image, determine the reprojection error of the target calibration target corner points; When it is determined that the calibration of the on-vehicle surround-view camera to be calibrated is successful based on the reprojection error of the target calibration target corner points, determine the external parameters of the on-vehicle surround-view camera to be calibrated according to the homography matrix corresponding to the on-vehicle surround-view camera to be calibrated.

5. The method according to claim 4, characterized in that The determining the homography matrix corresponding to the on-vehicle surround-view camera to be calibrated according to the internal parameters of the on-vehicle surround-view camera to be calibrated, the position information of the target calibration target corner points in the calibration image collected by the on-vehicle surround-view camera to be calibrated, and the physical coordinates of the target calibration target corner points includes: Determine the initial homography matrix corresponding to the on-vehicle surround-view camera to be calibrated according to the internal parameters of the on-vehicle surround-view camera to be calibrated, the position information of the target calibration target corner points in the calibration image collected by the on-vehicle surround-view camera to be calibrated, and the physical coordinates of the target calibration target corner points; Use a pre-optimization algorithm to optimize the initial homography matrix to obtain the final homography matrix corresponding to the on-vehicle surround-view camera to be calibrated; The projecting the target calibration target corner points onto the calibration image collected by the on-vehicle surround-view camera to be calibrated according to the homography matrix corresponding to the on-vehicle surround-view camera to be calibrated, and the physical coordinates of the target calibration target corner points, and obtaining the projection positions of the target calibration target corner points in the calibration image includes: Project the target calibration target corner points onto the calibration image collected by the on-vehicle surround-view camera to be calibrated according to the final homography matrix corresponding to the on-vehicle surround-view camera to be calibrated, and the physical coordinates of the target calibration target corner points, and obtain the projection positions of the target calibration target corner points in the calibration image.

6. The method according to claim 4, characterized in that The determining that the calibration of the on-vehicle surround-view camera to be calibrated is successful according to the reprojection error of the target calibration target corner points includes: When the reprojection errors of all target calibration target corner points are less than the first error threshold, and / or when the sum of the reprojection errors of the target calibration target corner points is less than the second error threshold, determine that the calibration of the on-vehicle surround-view camera to be calibrated is successful.

7. The method according to claim 1, wherein For the reference frame image of the first calibration site and the calibration image of the second calibration site obtained by the on-vehicle surround-view camera to be calibrated, when the target patterns and the number of targets in each calibration image are the same as those in the reference frame image of the same view, determine that the first calibration site and the second calibration site are the same calibration site.

8. An external camera parameter calibration device, characterized in that, including: An acquisition unit for acquiring a calibration image; wherein, the calibration image is obtained by the on-vehicle surround-view camera to be calibrated by collecting images of the calibration site; A determination unit, configured to determine the specified target corner point information in the image to be calibrated according to the reference frame information of the same calibration site; wherein, the specified target corner point information includes the identification information of the specified target corner point in the image to be calibrated, the physical coordinates of the specified target corner point, and the position information of the specified target corner point in the image to be calibrated; the reference frame information includes the identification information of the specified target corner point in the reference frame image, the physical coordinates of the specified target corner point, and the position information of the specified target corner point in the reference frame image; during the calibration process, the reference frame image is obtained by respectively collecting images of the calibration site through each on-vehicle surround view camera of the calibration vehicle; the number of the specified target corner points is greater than or equal to 4, and all of them are located in the co-viewing area of the adjacent on-vehicle surround view cameras of the calibration vehicle in the reference frame image and are not collinear; A processing unit, configured to determine the external parameters of the on-vehicle surround view camera to be calibrated according to the specified target corner point information in the image to be calibrated and the internal parameters of the on-vehicle surround view camera to be calibrated.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory stores machine-executable instructions that can be executed by the processor, and the processor is configured to execute the machine-executable instructions to implement the method according to any one of claims 1-7.

10. A machine-readable storage medium, characterized in that, Machine-executable instructions are stored in the machine-readable storage medium, and when the machine-executable instructions are executed by a processor, the method according to any one of claims 1-7 is implemented.

Images