Camera external parameter calibration method, device, equipment, storage medium and system

Abstract

The invention provides a camera external parameter calibration method, device and system, equipment and a storage medium. In one example, the method comprises the following steps: for any calibration site, performing reference frame image acquisition on the calibration site through a vehicle-mounted around-view camera of a calibration vehicle, and generating reference frame information corresponding to the reference frame image according to a detected marking instruction; and for any to-be-calibrated vehicle, acquiring reference frame information of the same calibration site according to the current calibration site, and performing external parameter calibration on the to-be-calibrated vehicle-mounted around-view camera according to the acquired reference frame information. According to the method, the universality of the calibration scheme can be improved under the condition that the calibration accuracy is ensured.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular, to a method, device, equipment, storage medium and system for calibrating the external parameters of a camera. Background Art

[0002] The external parameter calibration of on-vehicle surround-view cameras is usually carried out after the vehicle rolls off the production line in the factory, which is usually also called the off-line calibration of the vehicle. After knowing the internal parameters of the camera and the physical size information of the target on the calibration site, the corner detection technology is used to locate the target corners in the image, so as to complete the external parameter calibration of the camera and provide support for the intelligent driving of the vehicle.

[0003] However, due to the large number of suppliers of surround-view external parameter calibration algorithms on the market, resulting in a variety of calibration site patterns built by different vehicle manufacturers, when supporting mass production projects of different vehicle models, new calibration site patterns are always encountered.

[0004] How to meet the calibration requirements of different sites while ensuring high-precision external parameter calibration has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, this application provides a method, device, equipment, storage medium and system for calibrating the external parameters of a camera.

[0006] Specifically, this application is implemented through the following technical solutions: According to the first aspect of the embodiments of this application, a method for calibrating the external parameters of a camera is provided, including: For any calibration site, when it is determined that there is no stored reference frame information associated with this calibration site, the on-vehicle surround-view camera of the calibration vehicle is used to collect reference frame images of this calibration site; wherein, the calibration site is characterized by a target laying scheme, and the target laying scheme includes a target pattern and the number of targets. The same target laying scheme uses the same target pattern, and the number of targets in the images of the same view of the on-vehicle surround-view camera is the same; According to the detected annotation instruction, generate the reference frame information corresponding to the reference frame image, and associate and store this calibration site with the reference frame information; wherein, the reference frame information includes: The identification information of the specified target corners in the reference frame image; The physical coordinates of the specified target corners; and, The position information of the specified target corners in the reference frame image; wherein, the number of the specified target corners is greater than or equal to 4, and they are all located in the co-viewing area of the adjacent on-vehicle surround-view cameras of the calibration vehicle in the reference frame image and are not collinear; For any vehicle to be calibrated, obtain the reference frame information of the same calibration site according to the current calibration site, and perform external parameter calibration on the on-vehicle surround-view camera to be calibrated according to the obtained reference frame information.

[0007] According to the second aspect of the embodiments of the present application, there is provided a camera external parameter calibration device, including: A generating unit, configured to, for any calibration site, in the case of determining that the reference frame information associated with the calibration site is not stored, collect reference frame images of the calibration site through the on-vehicle surround-view camera of the calibration vehicle, generate reference frame information corresponding to the reference frame images according to the detected annotation instructions, and associate and store the calibration site and the reference frame information; wherein, the calibration site is characterized by a target laying scheme, the target laying scheme includes a target pattern and the number of targets, the same target laying scheme uses the same target pattern, and the number of targets in the images of the same view of the on-vehicle surround-view camera is the same; the reference frame information includes the identification information of the specified target corner points in the reference frame image, the physical coordinates of the specified target corner points, and the position information of the specified target corner points in the reference frame image; the number of the specified target corner points is greater than or equal to 4, and all are located in the co-visible area of the adjacent on-vehicle surround-view cameras of the calibration vehicle in the reference frame image and are not collinear; A processing unit, configured to, for any vehicle to be calibrated, obtain the reference frame information of the same calibration site according to the current calibration site, and perform external parameter calibration on the on-vehicle surround-view camera to be calibrated according to the obtained reference frame information.

[0008] According to the third aspect of the embodiments of the present application, there is provided an electronic device, including a processor and a memory, the memory stores machine-executable instructions that can be executed by the processor, and the processor is configured to execute the machine-executable instructions to implement the method provided in the first aspect.

[0009] According to the fourth aspect of the embodiments of the present application, there is provided a machine-readable storage medium, the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0010] According to the fifth aspect of the embodiments of the present application, there is provided a camera external parameter calibration system, including: a processor, a memory, and a display; wherein: The memory is used to store the association relationship between the calibration site and the reference frame information; The processor is configured to, for any calibration site, when it is determined that the memory does not store the reference frame information associated with the calibration site, collect a reference frame image of the calibration site through the on-vehicle surround-view camera of the calibration vehicle; wherein, the calibration site is characterized by a target laying scheme, the target laying scheme includes a target pattern and the number of targets, the same target laying scheme uses the same target pattern, and the number of targets in the images of the same view of the on-vehicle surround-view camera is the same; The display is configured to display the reference frame image; The processor is further configured to generate reference frame information corresponding to the reference frame image according to the detected annotation instruction for the reference frame image displayed on the display; wherein, the reference frame information includes: The identification information of the specified target corner points in the reference frame image; The physical coordinates of the specified target corner points; and, The position information of the specified target corner points in the reference frame image; wherein, the number of the specified target corner points is greater than or equal to 4, and they are all located in the co-view area of the adjacent on-vehicle surround-view cameras of the calibration vehicle in the reference frame image and are not collinear; The memory is further configured to associate and store the calibration site and the reference frame information; The processor is further configured to, for any vehicle to be calibrated, according to the current calibration site, obtain the reference frame information of the same calibration site from the memory, and perform external parameter calibration on the on-vehicle surround-view camera to be calibrated according to the obtained reference frame information.

[0011] The technical solution provided by this application can at least bring the following beneficial effects: By collecting a reference frame image for the calibration site and generating reference frame information corresponding to the reference frame image according to the detected annotation instruction, during the process of performing external parameter calibration on the camera of the vehicle to be calibrated, according to the current calibration site, the reference frame information of the same calibration site can be obtained, and the on-vehicle surround-view camera to be calibrated can be externally parameter calibrated according to the obtained reference frame information. By setting the reference frame information for the calibration site, in the case of the existence of the reference frame information of the same calibration site, for the changes of other parameters such as the physical environment, camera model, vehicle model, etc., there is no need to re-upgrade the calibration algorithm, and the versatility of the calibration scheme can be improved while ensuring the calibration accuracy; in addition, in the case of a change in the calibration site, only the reference frame information needs to be updated to support the external parameter calibration of the camera for the new calibration site, which improves the scalability of the scheme and effectively reduces the development cost of expanding the calibration site. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic flowchart of a method for calibrating the external parameters of a camera shown in an exemplary embodiment of this application; Figure 2A It is a schematic diagram of a vehicle with 4 on-vehicle surround-view cameras shown in an exemplary embodiment of the present application; Figure 2B It is a schematic diagram of a vehicle with 6 on-vehicle surround-view cameras shown in an exemplary embodiment of the present application; Figure 3 It is a schematic flowchart of a method for calibrating external parameters of a camera shown in an exemplary embodiment of the present application; Figure 4 It is a schematic diagram of a calibration system device shown in an exemplary embodiment of the present application; Figure 5 It is a schematic diagram of a specific implementation process for calibrating external parameters of a camera shown in an exemplary embodiment of the present application; Figure 6 It is a schematic diagram of an image to be calibrated shown in an exemplary embodiment of the present application; Figure 7 It is a schematic diagram of a reference frame image shown in an exemplary embodiment of the present application; Figure 8 It is a schematic diagram of the matching between an image to be calibrated and a reference frame image shown in an exemplary embodiment of the present application; Figure 9 It is a schematic diagram of the matching between an image to be calibrated and a reference frame image shown in an exemplary embodiment of the present application; Figure 10 It is a schematic diagram of the matching between an image to be calibrated and a reference frame image after feature point screening shown in an exemplary embodiment of the present application; Figure 11 It is a schematic structural diagram of a device for calibrating external parameters of a camera shown in an exemplary embodiment of the present application; Figure 12 It is a schematic structural diagram of a device for calibrating external parameters of a camera shown in an exemplary embodiment of the present application; Figure 13 It is a schematic hardware structure diagram of an electronic device provided in an embodiment of the present application. Detailed implementation manners

[0013] In order to enable those skilled in the art to better understand the technical solutions provided in the embodiments of the present application, and to make the above-mentioned objects, features, and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0014] It should be noted that the sequence numbers of the steps in the embodiments of the present application do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0015] Please refer toFigure 1 , which is a schematic flowchart of a method for calibrating the external parameters of a camera provided by an embodiment of the present application. As Figure 1 shown, the method for calibrating the external parameters of the camera may include the following steps: Step S100: For any calibration site, when it is determined that there is no stored reference frame information associated with the calibration site, collect reference frame images of the calibration site through the on-vehicle surround-view camera of the calibration vehicle; wherein, the calibration site is characterized by a target laying scheme, and the target laying scheme includes a target pattern and the number of targets. The same target laying scheme uses the same target pattern, and the number of targets in the images of the same view of the on-vehicle surround-view camera is the same.

[0016] Step S110: Generate reference frame information corresponding to the reference frame image according to the detected annotation instruction, and associate and store the calibration site with the reference frame information; wherein, the reference frame information includes the identification information of the specified target corner points in the reference frame image, the physical coordinates of the specified target corner points, and the position information of the specified target corner points in the reference frame image; the number of specified target corner points is greater than or equal to 4, and they are all located in the co-viewing area of adjacent on-vehicle surround-view cameras of the calibration vehicle in the reference frame image and are not collinear.

[0017] Step S120: For any vehicle to be calibrated, obtain the reference frame information of the same calibration site according to the current calibration site, and perform external parameter calibration on the on-vehicle surround-view camera to be calibrated according to the obtained reference frame information.

[0018] In an embodiment of the present application, the on-vehicle surround-view camera includes on-vehicle cameras installed on the vehicle body for obtaining the surrounding environment of the vehicle body to achieve 360° surround view.

[0019] Exemplarily, for any vehicle equipped with an on-vehicle surround-view camera, the number of on-vehicle surround-view cameras is usually multiple.

[0020] For example, the number of on-vehicle surround-view cameras can be 4 or 6.

[0021] Taking the number of on-vehicle surround-view cameras being 4 as an example, the on-vehicle surround-view cameras usually include on-vehicle cameras deployed in front of the vehicle, behind the vehicle, on the left side of the vehicle, and on the right side of the vehicle (which can be respectively called the front on-vehicle camera, the rear on-vehicle camera, the left on-vehicle camera, and the right on-vehicle camera), and its schematic diagram can be as Figure 2A shown.

[0022] Taking the number of in-vehicle surround-view cameras as six as an example, in-vehicle surround-view cameras usually include in-vehicle cameras deployed in the front of the vehicle, the rear of the vehicle, the left front side of the vehicle, the left rear side of the vehicle, the right front side of the vehicle, and the right rear side of the vehicle (which can be respectively called the front in-vehicle camera, the rear in-vehicle camera, the left front in-vehicle camera, the left rear in-vehicle camera, the right front in-vehicle camera, and the right rear in-vehicle camera), and its schematic diagram can be as Figure 2B shown.

[0023] In the embodiments of the present application, in order to improve the generality of the camera extrinsic parameter calibration scheme, reference frame information can be set for different calibration sites. Thus, during the process of calibrating the extrinsic parameters of the in-vehicle surround-view cameras of the vehicle to be calibrated (which can be called the in-vehicle surround-view cameras to be calibrated), the corresponding reference frame information can be determined according to the actual calibration site, and the extrinsic parameters of the in-vehicle surround-view cameras to be calibrated can be calibrated according to the reference frame information.

[0024] Correspondingly, for any calibration site, it is possible to determine whether the reference frame information associated with the calibration site is stored by querying the stored association relationship between the calibration site and the reference frame information, and in the case where it is determined that the reference frame information associated with the calibration site is not stored, generate the reference frame information associated with the calibration site.

[0025] For example, assuming that a factory or a 4S store needs to add a service for calibrating the extrinsic parameters of in-vehicle surround-view cameras, relevant staff can submit a request for a new calibration site according to the actual target laying scheme.

[0026] When the system receives the request for the new calibration site, it can determine whether the reference frame information associated with the calibration site is stored, and in the case where it is determined that the reference frame information associated with the calibration site is not stored, generate the reference frame information associated with the calibration site.

[0027] Exemplarily, a calibration site can be characterized by a target laying scheme. The same target laying scheme corresponds to the same calibration site, and different target laying schemes correspond to different calibration sites.

[0028] That is, in the embodiments of the present application, for different physical environments, different camera models, and different vehicle models, when the target laying schemes are the same, the same reference frame information can be used for calibrating the extrinsic parameters of the cameras.

[0029] Exemplarily, the target laying scheme can include the target pattern and the number of targets. The same target laying scheme uses the same target pattern, and the number of targets in the images of the same view of the in-vehicle surround-view cameras is the same.

[0030] Exemplarily, the images of the same view of the in-vehicle surround-view cameras refer to the images collected by the in-vehicle surround-view cameras installed in the same orientation.

[0031] For example, in the case where the number of in-vehicle surround-view cameras is 4, the in-vehicle surround-view camera views include a front view, a rear view, a left view, and a right view.

[0032] For the case where the number of in-vehicle surround-view cameras is 6, the in-vehicle surround-view camera views include a front view, a rear view, a left front view, a left rear view, a right front view, and a right rear view.

[0033] Taking the number of in-vehicle surround-view cameras being 4 as an example, assuming that checkerboard targets are used in both target laying scheme 1 and target laying scheme 2, and the number of targets in the images of the front view / rear view / left view / right view of the in-vehicle surround-view camera is the same under target laying scheme 1 and target laying scheme 2, then target laying scheme 1 and target laying scheme 2 are the same.

[0034] In one example, for the to-be-calibrated image of the first calibration site and the reference frame image of the second calibration site obtained by the to-be-calibrated in-vehicle surround-view camera, when the target pattern and the number of targets in each to-be-calibrated image are the same as those in the reference frame image of the same view, it is determined that the first calibration site and the second calibration site are the same calibration site.

[0035] In the embodiments of the present application, the reference frame information may include the identification information (such as serial number) of the specified target corner points in the reference frame image, and the position information (such as image coordinates) of the specified target corner points in the reference frame image.

[0036] Wherein, the specified target corner points are the target corner points marked in the reference frame image.

[0037] Exemplarily, for any calibration site, the image of the calibration site (i.e., the reference frame image) can be collected by the in-vehicle surround-view camera of any vehicle (which can be called the marking vehicle), and according to the detected marking instruction, the target corner points of the reference frame images of each view are respectively marked (the marked target corner points are recorded as the above-mentioned specified target corner points), the reference frame information of the calibration site is generated, and the calibration site and the reference frame information are stored in an associated manner.

[0038] Exemplarily, the marked specified target corner points can be selected within the co-viewing area of the adjacent in-vehicle surround-view cameras of the calibration vehicle in the reference frame image.

[0039] For example, taking the number of in-vehicle surround-view cameras being 4 as an example, the front in-vehicle camera can respectively have a co-viewing area with the left in-vehicle camera and the right in-vehicle camera. For the reference frame image collected by the front in-vehicle camera, the specified target corner points can be selected from the co-viewing area of the front in-vehicle camera and the left in-vehicle camera, and / or the target corner points within the co-viewing area of the front in-vehicle camera and the right in-vehicle camera.

[0040] Exemplarily, the annotation of the target corner points may include the identification of the designated target corner points, the physical coordinates of the designated target corner points, and the position information of the designated target corner points.

[0041] Among them, the physical coordinates of the target corner points may include the physical coordinates of the target corner points in the target world coordinate system.

[0042] Exemplarily, the target world coordinate system (which may also be referred to as the calibration site world coordinate system) may include a world coordinate system established with the center of the calibration site as the coordinate origin.

[0043] Exemplarily, the X / Y axes of this world coordinate system are usually parallel to the main direction of the site (such as the edge of the wall or the target row and column direction), and the Z axis is perpendicular to the ground.

[0044] It should be noted that when using a large target (such as an extra-large checkerboard) in the calibration site, the center of this target can be used as the center of the calibration site.

[0045] In the case of using multiple scattered small targets in the calibration site, the center point of the calibration site can be the geometric center of these targets.

[0046] In addition, the center of the calibration site can also be manually marked by using high-precision measuring devices (such as total stations, laser trackers) to set fixed marking points at the center of the site.

[0047] Exemplarily, for any reference frame image, the number of the designated target corner points to be annotated is greater than or equal to 4, and the designated target corner points to be annotated are not collinear (that is, there is no straight line passing through all the designated target corner points in the image plane).

[0048] In the embodiments of the present application, for any vehicle to be calibrated, the reference frame information of the same calibration site can be obtained according to the current calibration site, and the external parameters of the vehicle-mounted surround-view camera to be calibrated can be calibrated according to the obtained reference frame information.

[0049] Exemplarily, in the process of calibrating the external parameters of the camera of the vehicle to be calibrated, on the one hand, the vehicle-mounted surround-view camera of the vehicle to be calibrated (which can be called the vehicle-mounted surround-view camera to be calibrated) can be used to collect images of the current calibration scene to obtain the images to be calibrated; on the other hand, the reference frame information of the same calibration site can be obtained according to the current calibration scene, and the external parameters of the vehicle-mounted surround-view camera to be calibrated can be calibrated according to the obtained images to be calibrated and the reference frame information of the same calibration site to determine the external parameters of the vehicle-mounted surround-view camera to be calibrated.

[0050] It should be noted that in the embodiments of the present application, during the process of calibrating the external parameters of the camera for any vehicle to be calibrated, when it is determined according to the current calibration site that there is no reference frame information for the same calibration site, the reference frame information for the current calibration site can also be generated by real-time annotation. Furthermore, based on the reference frame information for the current calibration site, the external parameters of the on-vehicle surround-view camera of the vehicle to be calibrated are calibrated.

[0051] It can be seen that in Figure 1 the method flow shown, by collecting reference frame images for the calibration site and generating reference frame information corresponding to the reference frame images according to the detected annotation instructions, during the process of calibrating the external parameters of the camera for the vehicle to be calibrated, the reference frame information for the same calibration site can be obtained according to the current calibration site, and the external parameters of the on-vehicle surround-view camera to be calibrated are calibrated based on the obtained reference frame information. By setting the reference frame information for the calibration site, in the case where there is reference frame information for the same calibration site, for changes in other parameters such as the physical environment, camera model, and vehicle model, there is no need to re-upgrade the calibration algorithm, and the versatility of the calibration scheme can be improved while ensuring the calibration accuracy. In addition, in the case where the calibration site changes, only the reference frame information needs to be updated to support the external parameter calibration of the camera for the new calibration site, improving the scalability of the scheme and effectively reducing the development cost of expanding the calibration site.

[0052] Next, the implementation of calibrating the external parameters of the on-vehicle surround-view camera based on the reference frame map information will be described.

[0053] Please refer to Figure 3 , which is a schematic flow diagram of a method for calibrating the external parameters of a camera provided by an embodiment of the present application. As Figure 3 shown, the method for calibrating the external parameters of the camera may include the following steps: Step S300: Obtain an image to be calibrated; wherein, the image to be calibrated is obtained by the on-vehicle surround-view camera to be calibrated collecting an image of the calibration site.

[0054] In the embodiments of the present application, for any vehicle to be calibrated, during the process of calibrating the external parameters of the camera for the vehicle to be calibrated, an image of the calibration site can be collected by the on-vehicle surround-view camera to be calibrated to obtain an image to be calibrated.

[0055] Step S310: Determine the specified target corner point information in the image to be calibrated according to the reference frame information for the same calibration site; wherein, the specified target corner point information includes the identification information of the specified target corner point in the image to be calibrated, the physical coordinates of the specified target corner point, and the position information of the specified target corner point in the image to be calibrated.

[0056] In the embodiments of the present application, when the to-be-calibrated image is obtained, the reference frame information of the same calibration site can be obtained according to the calibration site where the to-be-calibrated image is collected, and according to the obtained reference frame information, the specified target corner point information in the to-be-calibrated image can be determined.

[0057] Exemplarily, for the to-be-calibrated images of different views, in the reference frame information of the same calibration site obtained, there is reference frame information corresponding to each view.

[0058] The specified target corner point information of the to-be-calibrated image of this view can be determined according to the reference frame information of the same view.

[0059] It should be noted that in the embodiments of the present application, for any calibration site, when it is determined that there is no reference frame information of the same calibration site currently, the reference frame information of this calibration site can also be generated by means of real-time annotation. Furthermore, according to the reference frame information of this calibration site, the extrinsic parameters of the on-vehicle surround-view camera of the vehicle to be calibrated at this calibration site can be calibrated.

[0060] Step S320: Determine the extrinsic parameters of the to-be-calibrated on-vehicle surround-view camera according to the specified target corner point information in the to-be-calibrated image and the intrinsic parameters of the to-be-calibrated on-vehicle surround-view camera.

[0061] In the embodiments of the present application, when the specified target corner point information in the to-be-calibrated image is determined as above, during the reference frame calibration process, the physical coordinates of the specified target corner points are also marked, that is, the physical coordinates of the specified target corner points are known conditions. Therefore, the extrinsic parameters of the to-be-calibrated on-vehicle camera can be determined according to the physical coordinates of each specified target corner point, the position information of the specified target corner point in the to-be-calibrated image, and the intrinsic parameters of the to-be-calibrated on-vehicle surround-view camera.

[0062] It should be noted that in the embodiments of the present application, since the reference frame information is marked in the target world coordinate system, therefore, the extrinsic parameters of the to-be-calibrated on-vehicle camera directly determined according to the physical coordinates of each specified target corner point, the position information of the specified target corner point in the to-be-calibrated image, and the intrinsic parameters of the to-be-calibrated on-vehicle surround-view camera are the extrinsic parameters in the target world coordinate system, and it can be further converted into the extrinsic parameters in the vehicle body world coordinate system.

[0063] Exemplarily, when the coordinate origin of the vehicle body world coordinate system coincides with the coordinate origin of the target world coordinate system (a tolerable deviation is allowed), for example, for the factory calibration environment, since there is a centering device in the factory, generally the vehicle will be parked to ensure that the center of the vehicle body coincides with the center of the target world coordinate system. In this case, the extrinsic parameters in the target world coordinate system are usually the same as the extrinsic parameters in the vehicle body world coordinate system.

[0064] When the coordinate origin of the vehicle body world coordinate system does not coincide with the coordinate origin of the target world coordinate system, the offset between the vehicle body center and the center of the target world coordinate system can be obtained. For example, the offset between the vehicle body center and the center of the target world coordinate system can be obtained by measurement. Furthermore, based on the offset between the vehicle body center and the center of the target world coordinate system, the external parameters in the target world coordinate system can be converted into the external parameters in the vehicle body world coordinate system.

[0065] It can be seen that in Figure 3 In the method flow shown, by pre-setting the corresponding reference frame information for different calibration sites, during the process of calibrating the external parameters of the on-vehicle surround-view camera, for the to-be-calibrated image obtained by collecting images of the calibration site through the to-be-calibrated on-vehicle surround-view camera, the specified target corner point information in the to-be-calibrated image can be determined based on the reference frame information of the same calibration site. Furthermore, based on the specified target corner point information in the to-be-calibrated image and the internal parameters of the to-be-calibrated on-vehicle surround-view camera, the external parameters of the to-be-calibrated on-vehicle surround-view camera can be determined. By setting the reference frame information for the calibration site, in the case of the existence of the reference frame information of the same calibration site, for the changes in other parameters such as the physical environment, camera model, vehicle model, etc., there is no need to re-upgrade the calibration algorithm, and the versatility of the calibration scheme can be improved while ensuring the calibration accuracy; in addition, in the case of a change in the calibration site, only the reference frame information needs to be updated to support the calibration of the external parameters of the camera for the new calibration site, improving the scalability of the scheme and effectively reducing the development cost of expanding the calibration site.

[0066] In some embodiments, the above-mentioned determining the specified target corner point information in the to-be-calibrated image based on the reference frame information of the same calibration site may include: For any to-be-calibrated image, based on the reference frame image of the corresponding view under the same calibration site, perform corner point matching between the to-be-calibrated image and the reference frame image to determine the specified target corner point information in the to-be-calibrated image.

[0067] Exemplarily, by performing corner point matching between the to-be-calibrated image and the reference frame image under the same calibration site, based on the specified target corner points marked in the reference frame image, the specified target corner points in the to-be-calibrated image can be determined.

[0068] Correspondingly, for any to-be-calibrated image, based on the reference frame image of the corresponding view under the same calibration site, perform corner point matching between the to-be-calibrated image and the reference frame image to determine the specified target corner points in the to-be-calibrated image. Furthermore, determine the specified target corner point information in the to-be-calibrated image, that is, determine the identification information of the specified target corner points in the to-be-calibrated image and the position information of the specified target corner points in the to-be-calibrated image.

[0069] For example, taking the number of vehicle-mounted surround-view cameras as 4 as an example, for the front-view image to be calibrated (i.e., the image to be calibrated collected by the front vehicle-mounted camera), the corner points of the front-view image to be calibrated and the front-view reference frame image (in the reference frame calibration process, the image collected by the front vehicle-mounted camera and used for specifying the target corner point annotation) can be matched based on the same calibration site, so as to determine the specified target corner points in the front-view image to be calibrated, and further, determine the specified target corner point information in the front-view image to be calibrated.

[0070] Among them, assuming that the front-view reference frame image includes specified target corner points 1 to 4, the position information of the specified target corner points 1 to 4 in the front-view image to be calibrated can be determined respectively in the front-view image to be calibrated.

[0071] In some embodiments, the above-mentioned corner point matching of the image to be calibrated and the reference frame image based on the reference frame image of the corresponding view under the same calibration site to determine the specified target corner point information in the image to be calibrated may include: Extract feature points from the image to be calibrated and the reference frame image respectively to obtain the first feature point set in the image to be calibrated and the second feature point set in the reference frame image; Perform feature point matching on the first feature point set and the second feature point set to obtain the mutually matching first feature points and second feature points; Determine the target first feature point that matches the target second feature point as the specified target corner point in the image to be calibrated; where the target second feature point is the second feature point in the reference frame image that is closest to the specified target corner point; Determine the specified target corner point information based on the specified target corner point in the image to be calibrated.

[0072] Exemplarily, for any image to be calibrated, feature points can be extracted from the image to be calibrated and the reference frame image of the same view under the same calibration site respectively to obtain the feature point set in the image to be calibrated (which can be called the first feature point set) and the feature point set in the reference frame image (which can be called the second feature point set).

[0073] Exemplarily, the feature point extraction algorithm may include but is not limited to traditional ORB (Oriented FAST and Rotated BRIEF) algorithm, SIFT (Scale-Invariant Feature Transform) algorithm, SURF (Speeded-Up Robust Features) algorithm, or deep learning corner point extraction algorithm (such as SuperPoint), etc.

[0074] For the above-mentioned first set of feature points and the second set of feature points, feature point matching can be performed between the first set of feature points and the second set of feature points to determine the mutually matching first feature points and second feature points.

[0075] Exemplarily, the feature point matching method can include but is not limited to traditional RANSAC (Random Sample Consensus) matching algorithm, brute-force matching algorithm, or algorithms such as deep learning-based SuperGlue, LightGlue (image matching algorithm based on deep learning), etc.

[0076] In one example, a deep learning-based feature point extraction and matching technology can be used to implement the feature point extraction and matching of the image to be calibrated and the reference frame image.

[0077] That is, a deep learning-based feature point extraction method can be used to extract feature points from the image to be calibrated and the reference frame image respectively; and, A deep learning-based feature point matching method is used to perform feature point matching between the first set of feature points and the second set of feature points.

[0078] Exemplarily, the feature point extraction network supports the matching of the center position of the circular region. The feature point extraction network outputs the positions and descriptors of the feature points, and the feature matching network outputs the position coordinates of the matching points.

[0079] In this example, the feature points can include not only the "X"-type corner points of the checkerboard, the "L"-type corner points of the square, the "L"-type corner points of the double-square, but also the circular "O"-type corner points, etc.

[0080] That is, in the embodiments of the present application, the target pattern is no longer limited to the checkerboard target, the square target, the double-square target, but can also be a circular target (the corner points of the circular target are the centers).

[0081] Exemplarily, in the case where the mutually matching first feature points and second feature points are determined, the second feature point (which can be called the target second feature point) closest to the specified target corner point in the reference frame image can be used as the specified target corner point in the reference frame image, and the first feature point (which can be called the target first feature point) in the image to be calibrated that matches the target feature point is determined as the specified target corner point in the image to be calibrated.

[0082] In some embodiments, determining the external parameters of the vehicle-mounted surround-view camera to be calibrated based on the information of the specified target corner points in the image to be calibrated and the internal parameters of the vehicle-mounted surround-view camera to be calibrated includes: For any on-vehicle surround-view camera to be calibrated, based on the internal parameters of the on-vehicle surround-view camera to be calibrated, the position information of the target specified calibration target corner points in the calibration image collected by the on-vehicle surround-view camera to be calibrated, and the physical coordinates of the target specified calibration target corner points, determine the homography matrix corresponding to the on-vehicle surround-view camera to be calibrated; where the target specified calibration target corner points are the specified calibration target corner points in the calibration image collected by the on-vehicle surround-view camera to be calibrated; Based on the homography matrix corresponding to the on-vehicle surround-view camera to be calibrated, and the physical coordinates of the target specified calibration target corner points, project the target specified calibration target corner points onto the calibration image collected by the on-vehicle surround-view camera to be calibrated, and obtain the projection position of the target specified calibration target corner points in the calibration image; Based on the projection position of the target specified calibration target corner points in the calibration image, and the position information of the target specified calibration target corner points in the calibration image, determine the reprojection error of the target specified calibration target corner points; When it is determined that the calibration of the on-vehicle surround-view camera to be calibrated is successful based on the reprojection error of the target specified calibration target corner points, determine the external parameters of the on-vehicle surround-view camera to be calibrated based on the homography matrix corresponding to the on-vehicle surround-view camera to be calibrated.

[0083] Exemplarily, when the information of the specified calibration target corner points in the calibration image is determined in the above manner, for any on-vehicle surround-view camera, the physical coordinates of the target specified calibration target corner points (which can be called the target specified calibration target corner points) in the calibration image collected by the on-vehicle surround-view camera to be calibrated can be determined based on the identification information of the specified calibration target corner points.

[0084] Among them, during the calibration process of the reference frame, the physical coordinates of the specified calibration target corner points with different identification information are also marked, that is, they are known conditions. Therefore, based on the internal parameters of the on-vehicle surround-view camera to be calibrated, the position information of the target specified calibration target corner points in the calibration image collected by the on-vehicle surround-view camera to be calibrated, and the physical coordinates of the target specified calibration target corner points, the homography matrix corresponding to the on-vehicle surround-view camera to be calibrated can be determined, and its specific implementation will be described in the following with specific examples.

[0085] When the homography matrix corresponding to the on-vehicle surround-view camera to be calibrated is determined, based on the homography matrix corresponding to the on-vehicle surround-view camera to be calibrated, and the physical coordinates of the target specified calibration target corner points, project the target specified calibration target corner points onto the calibration image collected by the on-vehicle surround-view camera to be calibrated, obtain the projection position of the target specified calibration target corner points in the calibration image, and based on the projection position of the target specified calibration target corner points in the calibration image, and the position information of the target specified calibration target corner points in the calibration image, determine the reprojection error of the target specified calibration target corner points, and its specific implementation will be described in the following with specific examples.

[0086] Exemplarily, it is possible to determine whether the calibration of the in-vehicle surround-view camera to be calibrated is successful based on the reprojection error of the target-specified target corner points.

[0087] In one example, determining that the calibration of the in-vehicle surround-view camera to be calibrated is successful based on the reprojection error of the target-specified target corner points may include: Determining that the calibration of the in-vehicle surround-view camera to be calibrated is successful when the reprojection errors of all target-specified target corner points are less than a first error threshold, and / or when the average reprojection error of the target-specified target corner points is less than a second error threshold.

[0088] Exemplarily, it is possible to determine whether the calibration of the in-vehicle surround-view camera to be calibrated is successful based on the reprojection errors of the target-specified target corner points, and / or the average reprojection error of the target-specified target corner points.

[0089] As an example, it is possible to determine that the calibration of the in-vehicle surround-view camera to be calibrated is successful when the reprojection errors of all target-specified target corner points are less than a preset error threshold (which can be referred to as the first error threshold); otherwise, determine that the calibration of the in-vehicle surround-view camera to be calibrated fails.

[0090] As another example, it is possible to determine that the calibration of the in-vehicle surround-view camera to be calibrated is successful when the average reprojection error of the target-specified target corner points is less than a preset error threshold (which can be referred to as the second error threshold); otherwise, determine that the calibration of the in-vehicle surround-view camera to be calibrated fails.

[0091] As yet another example, it is possible to determine that the calibration of the in-vehicle surround-view camera to be calibrated is successful when the reprojection errors of all target-specified target corner points are less than the first error threshold and the average reprojection error of the target-specified target corner points is less than the second error threshold; otherwise, determine that the calibration of the in-vehicle surround-view camera to be calibrated fails.

[0092] Exemplarily, when it is determined that the calibration of the in-vehicle surround-view camera to be calibrated is successful based on the reprojection error of the target-specified target corner points, the external parameters of the in-vehicle surround-view camera to be calibrated are determined based on the homography matrix corresponding to the in-vehicle surround-view camera to be calibrated.

[0093] For example, based on the homography matrix corresponding to the in-vehicle surround-view camera to be calibrated and the internal parameters of the in-vehicle surround-view camera to be calibrated, the external parameters of the camera, such as the camera position (the coordinates of the camera in the vehicle body world coordinate system) and the installation angle, can be obtained using SVD (Singular Value Decomposition).

[0094] In one example, determining the homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated based on the internal parameters of the vehicle-mounted surround-view camera to be calibrated, the position information of the target specified calibration target corner points in the calibration image collected by the vehicle-mounted surround-view camera to be calibrated, and the physical coordinates of the target specified calibration target corner points may include: Determine the initial homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated based on the internal parameters of the vehicle-mounted surround-view camera to be calibrated, the position information of the target specified calibration target corner points in the calibration image collected by the vehicle-mounted surround-view camera to be calibrated, and the physical coordinates of the target specified calibration target corner points; Use a pre-optimization algorithm to optimize the initial homography matrix to obtain the final homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated; Projecting the target specified calibration target corner points onto the calibration image collected by the vehicle-mounted surround-view camera to be calibrated based on the homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated and the physical coordinates of the target specified calibration target corner points to obtain the projected position of the target specified calibration target corner points in the calibration image may include: Project the target specified calibration target corner points onto the calibration image collected by the vehicle-mounted surround-view camera to be calibrated based on the final homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated and the physical coordinates of the target specified calibration target corner points to obtain the projected position of the target specified calibration target corner points in the calibration image.

[0095] Exemplarily, in order to improve the calibration accuracy, when the initial homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated is determined in the above manner, a preset optimization algorithm may be used to optimize the initial homography matrix corresponding to the vehicle-mounted surround-view camera to be calibrated to obtain the final homography matrix.

[0096] For example, when the initial homography matrix of the vehicle-mounted surround-view camera to be calibrated is determined in the above manner, the LM (Levenberg-Marquardt) algorithm can be used to optimize the initial homography matrix of the vehicle-mounted surround-view camera to be calibrated to obtain the final homography matrix.

[0097] After obtaining the final homography matrix of the vehicle-mounted surround-view camera to be calibrated, the reprojection error of the target specified calibration target corner points can be determined based on the final homography matrix, and whether the vehicle-mounted surround-view camera to be calibrated is successfully calibrated can be determined based on the reprojection error of the target specified calibration target corner points.

[0098] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of the present application, the technical solutions provided in the embodiments of the present application will be described below with specific examples.

[0099] This embodiment provides a general camera extrinsic parameter calibration scheme. By setting reference frame information to assist in obtaining the corner point information (the above-mentioned specified target corner point information) in the image to be calibrated, the camera extrinsic parameter calibration of the vehicle can be completed, which can be compatible with different calibration sites, different vehicle models, and different application scenarios.

[0100] Exemplarily, the extrinsic parameters of the on-vehicle surround-view cameras of each camera can be calibrated by using the PNP algorithm by detecting the ordered corner points of the target on the image and combining the known physical size information of the target (such as the length and width of the target, the distance from the center of the vehicle body, etc.).

[0101] Exemplarily, the reference frame information and image matching technology can be used to detect and sort the corner points of the target on the image to be calibrated (taking the identification information of the target corner points as the corner point serial numbers as an example, the specified target corner points can be identified by the serial numbers).

[0102] The calibration system device will be described first below.

[0103] In this embodiment, the schematic diagram of the calibration system device can be referred to Figure 4 As shown in Figure 4 This calibration system device may include: a camera (i.e., an on-vehicle surround-view camera), a controller, a memory, and a display device. Among them: The camera is used to collect the image to be calibrated; the controller is used to execute corresponding control processing according to the detected control instructions; the memory is used to store the reference frame information; the display device is used to display the calibration operation interface.

[0104] Exemplarily, based on Figure 4 the calibration system device shown, the camera extrinsic parameter calibration process can be as follows: S1. According to the detected calibration site selection operation instruction, determine the corresponding calibration site pattern.

[0105] Exemplarily, the calibration operation interface can provide a "calibration site selection" function button. When the selection instruction of this function button is detected, the candidate calibration site patterns can be displayed in the calibration operation interface. Furthermore, the selected calibration site pattern can be determined according to the selection instruction for the calibration site pattern.

[0106] S2. According to the detected reference frame information selection instruction, determine the corresponding reference frame information.

[0107] Exemplarily, when the selected calibration site pattern is determined, the reference frame information corresponding to the calibration scene pattern can be displayed in the display interface. When the selection instruction for this reference frame information is detected, it can be determined to use the selected reference frame information for camera extrinsic parameter calibration.

[0108] S3. When a start calibration operation instruction is detected, perform camera extrinsic parameter calibration on the vehicle to be calibrated according to the determined reference frame information.

[0109] Exemplarily, when it is determined that the calibration of the vehicle to be calibrated is successful, the bird's-eye view effect of the calibration image can be displayed.

[0110] The following describes the specific implementation process of camera extrinsic parameter calibration.

[0111] In this embodiment, the specific implementation process of camera extrinsic parameter calibration can refer to Figure 5 , such as Figure 5 shown. The specific implementation process of this camera extrinsic parameter calibration may include: 1) Obtain the image to be calibrated.

[0112] Exemplarily, taking the number of on-vehicle surround-view cameras as 4 as an example, the schematic diagram of the image to be calibrated can be referred to Figure 6 , such as Figure 6 shown. Figure 6 From left to right and from top to bottom in , they are the front-view image to be calibrated (which can be simply referred to as the to-be-calibrated front view), the rear-view image to be calibrated (which can be simply referred to as the to-be-calibrated rear view), the left-view image to be calibrated (which can be simply referred to as the to-be-calibrated left view), and the right-view image to be calibrated (which can be simply referred to as the to-be-calibrated right view).

[0113] 2) Obtain the reference frame information under the same calibration site.

[0114] Exemplarily, the reference frame information may include the serial numbers and position coordinates of the specified target corner points in the reference frame image.

[0115] Exemplarily, the schematic diagram of the reference frame image can be referred to Figure 7 , such as Figure 7 shown. Figure 7 From left to right and from top to bottom in , they are the front-view reference frame image (which can be simply referred to as the reference frame front view), the rear-view reference frame image (which can be simply referred to as the reference frame rear view), the left-view reference frame image (which can be simply referred to as the reference frame left view), and the right-view reference frame image (which can be simply referred to as the reference frame right view).

[0116] Exemplarily, the reference frame image can be the original fisheye image collected by the on-vehicle surround-view camera during the calibration process, or the undistorted image obtained by performing undistortion processing on the original fisheye image.

[0117] It should be noted that when the reference frame image is the original fisheye image, the image to be calibrated also uses the original fisheye image; when the reference frame image is the undistorted image, the image to be calibrated also uses the undistorted image.

[0118] Exemplarily, the number of reference frames in each reference frame image is greater than or equal to 4 ( Figure 7 taking 8 as an example), non - collinear, and within the co - viewing area of adjacent vehicle surround cameras.

[0119] Exemplarily, the vehicle models, acquisition environments, and camera resolutions of the reference frame images and the images to be calibrated can be the same or different.

[0120] For example, the reference frame images and the images to be calibrated can be acquired by the vehicle surround cameras of the same vehicle model, or can be images acquired by the vehicle surround cameras of different vehicle models.

[0121] The reference frame images (or the images to be calibrated) can be acquired in the factory calibration board environment, or can be acquired by laying calibration cloth in the outdoor environment of a 4S store; the resolution of the acquisition camera can be 1280*720, or can be other resolutions such as 1280*960.

[0122] For example, for any calibration site, in the process of obtaining the reference frame images of the calibration site, a set of fisheye original images (including front, rear, left, and right views) of the calibration site can be obtained through a calibration vehicle, and the sequence numbers and position information of the specified target corner points can be determined according to the detected annotation instructions.

[0123] 3) Determine the specified target corner point information in the image to be calibrated based on the reference frame information.

[0124] Exemplarily, image matching technology can be used to perform image matching for each image to be calibrated and the reference frame image of the corresponding view, and obtain the corner point matching relationship between the two images.

[0125] Exemplarily, the above - mentioned image matching technology can adopt image feature point matching technology or image dense matching technology.

[0126] In one example, in order to reduce the computational time consumption of image matching and improve the processing efficiency, the image matching technology can adopt image feature point matching technology.

[0127] Exemplarily, the feature point matching technology can include feature point extraction and feature point matching.

[0128] Exemplarily, the feature point extraction algorithm can include but is not limited to traditional algorithms such as ORB algorithm, SIFT algorithm, SURF algorithm, or deep - learning corner point extraction algorithms (such as SuperPoint), etc.

[0129] Exemplarily, the feature point matching method can include but is not limited to traditional algorithms such as RANSAC matching algorithm, brute - force matching algorithm, or deep - learning algorithms such as SuperGlue and LightGlue, etc.

[0130] In one example, in order to obtain a more robust matching effect, a feature point extraction and matching technology based on deep learning can be used to extract and match the feature points of the image to be calibrated and the reference frame image.

[0131] Exemplarily, the feature point extraction network supports the matching of the center position of the circular region. The feature point extraction network outputs the position and descriptor of the feature points, and the feature matching network outputs the position coordinates of the matching points.

[0132] Exemplarily, the feature points can include not only the "X"-type corner points of the checkerboard, the "L"-type corner points of the square, the "L"-type corner points of the double-square, but also the "O"-type corner points of the circle (the corner point of the circular pattern is the center of the circle), etc.

[0133] Exemplarily, through the above feature point matching technology, for the same calibration site, the images to be calibrated and the reference frame images collected by vehicles with different vehicle lengths can be correctly matched to the center position of the circular target. The schematic diagram can be seen in Figure 8 . Such as Figure 8 shown, in the order from left to right and from top to bottom, they are the schematic diagrams of the front view matching result, the rear view matching result, the left view matching result, and the right view matching result.

[0134] In addition, for the same calibration site, under different target materials and different calibration environments, the images can be correctly matched to the corner points of the checkerboard target. The schematic diagram can be seen in Figure 9 . Such as Figure 9 shown, in the order from left to right and from top to bottom, they are the schematic diagrams of the front view matching result, the rear view matching result, the left view matching result, and the right view matching result.

[0135] Exemplarily, according to the serial numbers and coordinate information of the specified target corner points already marked in the reference frame image, using the image matching information, the corner points in the image to be calibrated that are closest to the specified target corner points are selected, and the serial numbers of the corner points marked in the reference frame image are assigned to the specified target corner points in the image to be calibrated.

[0136] Taking the front view as an example, after screening, only the corner points corresponding to the marked corner points on the reference frame image are retained, and the abnormal results in the image matching results can be eliminated. The schematic diagram can be seen in Figure 10 . The screening of other views is the same.

[0137] Among them, as Figure 10 shown, the left side is the front view of the reference frame, and the right side is the target corner points selected from the image to be calibrated. 4) According to the known physical coordinates of the specified target corner points and the camera internal parameters, use PNP to calculate the homography matrix.

[0138] Exemplarily, based on the image coordinates, corner numbers, target size information, etc. of the specified target corner points in the image to be calibrated, the known physical coordinates corresponding to the specified target corner points (which can be the X and Y coordinates with the vehicle body center as the origin) can be obtained, and the homography matrix of each view can be calculated using PNP.

[0139] The following introduces an example of using DLT (Direct Linear Transformation) to solve the homography matrix:

[0140] Among them, X and Y represent positions in the world coordinate system, u and v represent the pixel coordinates of the target corner points on the undistorted image, and h0 to h8 are the values of the elements in the 3*3 homography matrix.

[0141] Use the last row to eliminate s, obtaining two constraints:

[0142] Form a linear equation system:

[0143] When the number of matching points is greater than or equal to 4, the above equation has a solution. That is, the solution of the linear equation system AH = 0 is the eigenvector corresponding to the smallest eigenvalue of A T of A.

[0144] Exemplarily, the eigenvalues and eigenvectors can be solved by the Jacobi iterative method.

[0145] In addition, other methods for solving linear equation systems can also be used: SVD decomposition, QR decomposition, LU decomposition, Cholesky decomposition, etc.

[0146] 5) Based on the current homography matrix, determine the reprojection error of the specified target corner points in the image to be calibrated, and when it is determined that the calibration is successful based on the reprojection error, determine the external camera parameters corresponding to the current homography matrix as the external parameters of the vehicle-mounted surround-view camera to be calibrated.

[0147] Exemplarily, when the homography matrix of the vehicle-mounted surround-view camera to be calibrated is obtained, the reprojected pixel points of the world coordinates of the specified target corner points on the image can be obtained through the relationship between the world coordinate points and the homography matrix.

[0148] Exemplarily, when the initial homography matrix of the vehicle-mounted surround-view camera to be calibrated is obtained, LM optimization can be used to optimize the homography matrix by optimizing the sum of the reprojection errors (RMS) of the corner points in the co-visible area, obtaining the final homography matrix, thereby making the calibration accuracy higher.

[0149] When it is determined that the reprojection error indicates the success of the calibration of the vehicle surround-view camera, the extrinsic parameters of the camera (including the camera position and installation angle) can be obtained by combining the homography matrix and the camera intrinsic parameters and using SVD decomposition.

[0150] Exemplarily, for the determined homography matrix, by setting a threshold, it can be determined whether the RMS of this calibration meets the requirements, and thus the success or failure of the calibration is output. Wherein:

[0151]

[0152] Among them, (u, v) represents the pixel coordinates in the image where the world coordinates of the specified target corner points are projected, and (x, y) represents the pixel coordinates of the specified target corner points extracted from the image to be calibrated. N represents the number of specified target corner points in the image.

[0153] Exemplarily, it can be determined that the calibration is successful when the average image reprojection error of all specified target corner points is less than the threshold (for example, 0.5 pix).

[0154] It can be seen that the camera extrinsic parameter calibration scheme provided by the embodiments of the present application obtains the target corner point information in the image to be calibrated by introducing a reference frame, thereby completing the calibration of the camera extrinsic parameters and being compatible with different vehicle models and diverse business application scenarios. Wherein: 1) When the calibration site changes, only the reference frame information needs to be re-annotated, and the algorithm does not need to be upgraded and adapted to support the project.

[0155] 2) When the application scenario, vehicle model, and camera model (including resolution) change, the reference frame information does not need to be changed and the algorithm does not need to be upgraded and adapted to support the project.

[0156] 3) When the number of vehicle surround-view cameras changes, if the calibration site remains unchanged, there is no need to update; if the calibration site changes, only the reference frame information needs to be updated.

[0157] Among them, the calibration site refers to the laid target pattern and quantity. For example, assuming that the target laying scheme remains unchanged and the number of vehicle surround-view cameras changes, the reference frame can be reused, and only the reference frame corresponding to each vehicle surround-view camera needs to be selected respectively. That is, when the calibration site changes, only the reference frame information needs to be updated and the algorithm does not need to be upgraded to support the calibration of the camera extrinsic parameters; when the camera model is changed or the vehicle model is changed, there is no need to re-upgrade the calibration algorithm, and the project scheme can be quickly supported with high usability and high calibration accuracy, which can well solve the pain points of the industry.

[0158] In addition, the reference frame information is used to obtain the target corner point information in the image to be calibrated, and then PNP is used for calibration. The calibration accuracy is high, it does not depend on the external reference information of the reference frame, and it also has a certain tolerance for the annotation accuracy of the target corner points in the reference frame. The algorithm has strong robustness.

[0159] The method provided by the present application has been described above. Next, the device provided by the present application will be described: Please refer to Figure 11 , which is a schematic structural diagram of a camera external parameter calibration device provided by an embodiment of the present application. As Figure 11 shown, the camera external parameter calibration device may include: A generating unit, configured to, for any calibration site, when it is determined that the reference frame information associated with the calibration site is not stored, collect reference frame images of the calibration site through the on-vehicle surround-view camera of the calibration vehicle; wherein, the calibration site is characterized by a target laying scheme, the target laying scheme includes a target pattern and the number of targets, the same target laying scheme uses the same target pattern, and the number of targets in the images of the same view of the on-vehicle surround-view camera is the same; Generate the reference frame information corresponding to the reference frame image according to the detected annotation instruction; wherein, the reference frame information includes the identification information of the specified target corner points in the reference frame image, the physical coordinates of the specified target corner points, and the position information of the specified target corner points in the reference frame image; the number of specified target corner points is greater than or equal to 4, and they are all located in the co-visible area of the adjacent on-vehicle surround-view cameras of the calibration vehicle in the reference frame image and are not collinear; A processing unit, configured to, for any vehicle to be calibrated, obtain the reference frame information of the same calibration site according to the current calibration site, and perform external parameter calibration on the on-vehicle surround-view camera to be calibrated according to the obtained reference frame information.

[0160] Exemplarily, the specific implementation process of the generating unit and the processing unit for realizing camera external parameter calibration can refer to the relevant descriptions in the above embodiments, and the embodiments of the present application will not elaborate here.

[0161] Please refer to Figure 12 , which is a schematic structural diagram of a camera external parameter calibration device provided by an embodiment of the present application. As Figure 12 shown, the camera external parameter calibration device may include: An obtaining unit, configured to obtain an image to be calibrated; wherein, the image to be calibrated is obtained by the on-vehicle surround-view camera to be calibrated by collecting images of the calibration site. A determination unit is configured to determine the specified target corner point information in the image to be calibrated according to the reference frame information of the same calibration site. The specified target corner point information includes the identification information of the specified target corner points in the image to be calibrated, the physical coordinates of the specified target corner points, and the position information of the specified target corner points in the image to be calibrated. The reference frame information includes the identification information of the specified target corner points in the reference frame image, the physical coordinates of the specified target corner points, and the position information of the specified target corner points in the reference frame image. During the calibration process, the reference frame image is obtained by respectively collecting images of the calibration site through each on-vehicle surround camera of the calibration vehicle. The number of the specified target corner points is greater than or equal to 4, and all of them are located in the co-visible area of adjacent on-vehicle surround cameras of the calibration vehicle in the reference frame image and are not collinear. A processing unit is configured to determine the external parameters of the on-vehicle surround camera to be calibrated according to the specified target corner point information in the image to be calibrated and the internal parameters of the on-vehicle surround camera to be calibrated.

[0162] Exemplarily, the specific implementation process of the acquisition unit, the determination unit, and the processing unit for implementing the camera external parameter calibration can refer to the relevant descriptions in the above embodiments, and the embodiments of the present application will not elaborate herein.

[0163] An embodiment of the present application provides an electronic device, including a processor and a memory. The memory stores machine-executable instructions that can be executed by the processor, and the processor is configured to execute the machine-executable instructions to implement the camera external parameter calibration method described above.

[0164] Please refer to Figure 13 , which is a schematic hardware structure diagram of an electronic device provided by an embodiment of the present application. The electronic device may include a processor 1301 and a memory 1302 storing machine-executable instructions. The processor 1301 and the memory 1302 may communicate via a system bus 1303. And by reading and executing the machine-executable instructions corresponding to the camera external parameter calibration logic in the memory 1302, the processor 1301 may execute the camera external parameter calibration method described above.

[0165] The memory 1302 mentioned herein may be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or a combination thereof.

[0166] In some embodiments, a machine-readable storage medium is also provided, such as the memory 1302 in Figure 13 which machine-executable instructions are stored. When the machine-executable instructions are executed by a processor, the camera extrinsic parameter calibration method described above is implemented. For example, the storage medium may be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0167] An embodiment of the present application also provides a camera extrinsic parameter calibration system, including: a processor, a memory, and a display; wherein: The memory is used to store the association relationship between the calibration site and the reference frame information; The processor is configured to, for any calibration site, in the case where it is determined that the reference frame information associated with the calibration site is not stored in the memory, collect reference frame images of the calibration site through the on-vehicle surround-view camera of the calibration vehicle; wherein, the calibration site is characterized by a target laying scheme, the target laying scheme includes a target pattern and the number of targets, the same target laying scheme uses the same target pattern, and the number of targets in the images of the same view of the on-vehicle surround-view camera is the same; The display is used to display the reference frame image; The processor is further configured to generate reference frame information corresponding to the reference frame image according to a detected annotation instruction for the reference frame image displayed on the display; wherein, the reference frame information includes: The identification information of the specified target corner points in the reference frame image; The physical coordinates of the specified target corner points; and, The position information of the specified target corner points in the reference frame image; wherein, the number of the specified target corner points is greater than or equal to 4, and they are all located in the co-viewing area of the adjacent on-vehicle surround-view cameras of the calibration vehicle in the reference frame image and are not collinear; The memory is further configured to store the calibration site and the reference frame information in an associated manner; The processor is further configured to, for any vehicle to be calibrated, obtain the reference frame information of the same calibration site from the memory according to the current calibration site, and perform extrinsic parameter calibration on the on-vehicle surround-view camera to be calibrated according to the obtained reference frame information.

[0168] Exemplarily, the specific implementation process for the processor to implement camera extrinsic parameter calibration can refer to the relevant descriptions in the above embodiments, and the embodiments of the present application will not elaborate herein.

Claims

1. A method for calibrating the external parameters of a camera, characterized in that, Including: For any calibration site, when it is determined that there is no reference frame information associated with the calibration site stored, the on-vehicle surround-view camera of the calibration vehicle is used to collect reference frame images of the calibration site; wherein, the calibration site is characterized by a target laying scheme, the target laying scheme includes a target pattern and the number of targets, the same target laying scheme uses the same target pattern, and the number of targets in the images of the same view of the on-vehicle surround-view camera is the same; According to the detected annotation instruction, generate reference frame information corresponding to the reference frame image, and associate and store the calibration site and the reference frame information; wherein, the reference frame information includes: The identification information of the specified target corner points in the reference frame image; The physical coordinates of the specified target corner points; and, The position information of the specified target corner points in the reference frame image; wherein, the number of the specified target corner points is greater than or equal to 4, and they are all located in the co-view area of the adjacent on-vehicle surround-view cameras of the calibration vehicle in the reference frame image and are not collinear; For any vehicle to be calibrated, according to the current calibration site, obtain the reference frame information of the same calibration site, and perform external parameter calibration on the on-vehicle surround-view camera to be calibrated according to the obtained reference frame information.

2. The method according to claim 1, wherein For the reference frame image of the first calibration site and the to-be-calibrated image of the second calibration site obtained through the on-vehicle surround-view camera to be calibrated, when the target pattern and the number of targets in each to-be-calibrated image are the same as those in the reference frame image of the same view, determine that the first calibration site and the second calibration site are the same calibration site.

3. The method according to claim 2, wherein The target pattern includes a checkerboard target, a square target, a double-square target, or a circular target.

4. The method according to claim 1, wherein The obtaining the reference frame information of the same calibration site according to the current calibration site and performing external parameter calibration on the on-vehicle surround-view camera to be calibrated according to the obtained reference frame information includes: According to the detected calibration site selection instruction, determine the current calibration site; According to the detected reference frame image selection instruction for the current calibration site, determine the target reference frame image of the current calibration site; According to the target reference frame image, determine the reference frame information of the current calibration site; Perform external parameter calibration on the on-vehicle surround-view camera to be calibrated according to the reference frame information of the current calibration site.

5. The method according to claim 1, characterized in that, After performing external parameter calibration on the on-vehicle surround-view camera to be calibrated according to the obtained reference frame information, it further includes: When the on-vehicle surround-view camera to be calibrated is successfully calibrated, display the bird's-eye view effect of the vehicle to be calibrated.

6. The method according to claim 1, characterized in that The performing external parameter calibration on the on-vehicle surround-view camera to be calibrated according to the obtained reference frame information includes: For any to-be-calibrated image, according to the reference frame image of the corresponding view under the same calibration site, perform corner matching on the to-be-calibrated image and the reference frame image to determine the specified target corner point information in the to-be-calibrated image; wherein, the to-be-calibrated image is obtained by the on-vehicle surround-view camera to be calibrated to collect images of the calibration site; According to the specified target corner point information in the to-be-calibrated image and the internal parameters of the on-vehicle surround-view camera to be calibrated, determine the external parameters of the on-vehicle surround-view camera to be calibrated.

7. The method according to claim 6, wherein Based on the reference frame images corresponding to the views in the same calibration site, performing corner matching on the image to be calibrated and the reference frame image to determine the specified target corner point information in the image to be calibrated, including: Performing feature point extraction on the image to be calibrated and the reference frame image respectively to obtain a first feature point set in the image to be calibrated and a second feature point set in the reference frame image; Performing feature point matching on the first feature point set and the second feature point set to obtain mutually matching first feature points and second feature points; Determining the target first feature point that matches the target second feature point as the specified target corner point in the image to be calibrated; wherein, the target second feature point is the second feature point closest to the specified target corner point in the reference frame image; Determining the specified target corner point information based on the specified target corner point in the image to be calibrated.

8. An external camera parameter calibration device, characterized in that, Including: A generating unit, configured to, for any calibration site, in the case of determining that there is no stored reference frame information associated with the calibration site, collect a reference frame image of the calibration site through an on-vehicle panoramic camera of a calibration vehicle, and generate reference frame information corresponding to the reference frame image according to a detected annotation instruction; wherein, the calibration site is characterized by a target laying scheme, the target laying scheme includes a target pattern and the number of targets, the same target laying scheme uses the same target pattern, and the number of targets in the images of the same view of the on-vehicle panoramic camera is the same; the reference frame information includes identification information of specified target corner points in the reference frame image, physical coordinates of the specified target corner points, and position information of the specified target corner points in the reference frame image; the number of the specified target corner points is greater than or equal to 4, and all are located in the co-view area of adjacent on-vehicle panoramic cameras of the calibration vehicle in the reference frame image and are not collinear; A processing unit, configured to, for any vehicle to be calibrated, obtain reference frame information of the same calibration site according to the current calibration site, and perform external parameter calibration on the on-vehicle panoramic camera to be calibrated according to the obtained reference frame information.

9. An electronic device, characterized in that, Including a processor and a memory, the memory stores machine-executable instructions that can be executed by the processor, and the processor is configured to execute the machine-executable instructions to implement the method according to any one of claims 1-7.

10. A machine-readable storage medium, characterized in that, Machine-executable instructions are stored in the machine-readable storage medium, and when the machine-executable instructions are executed by a processor, the method according to any one of claims 1-7 is implemented.

11. An external camera calibration system, characterized in that, Including: A processor, a memory, and a display; wherein: The memory is configured to store the association relationship between the calibration site and the reference frame information; The processor is configured to, for any calibration site, in the case of determining that there is no stored reference frame information associated with the calibration site in the memory, collect a reference frame image of the calibration site through an on-vehicle panoramic camera of a calibration vehicle; wherein, the calibration site is characterized by a target laying scheme, the target laying scheme includes a target pattern and the number of targets, the same target laying scheme uses the same target pattern, and the number of targets in the images of the same view of the on-vehicle panoramic camera is the same; The display is used to display the reference frame image; The processor is further configured to generate reference frame information corresponding to the reference frame image according to a detected annotation instruction for the reference frame image displayed on the display; wherein, the reference frame information includes: The identification information of the specified target corner points in the reference frame image; The physical coordinates of the specified target corner points; and The position information of the specified target corner points in the reference frame image; wherein, the number of the specified target corner points is greater than or equal to 4, and they are all located in the common view area of adjacent vehicle-mounted surround-view cameras of the calibration vehicle in the reference frame image and are not collinear; The memory is further configured to associate and store the calibration site and the reference frame information; The processor is further configured to, for any vehicle to be calibrated, obtain the reference frame information of the same calibration site from the memory according to the current calibration site, and perform external parameter calibration on the vehicle-mounted surround-view camera to be calibrated according to the obtained reference frame information.

Images