Camera internal reference calibration method, device, equipment, medium and program

Abstract

The embodiment of the invention provides a camera internal reference calibration method and device, equipment, a medium and a program, and relates to the camera calibration technology, and the calibration method comprises the steps: obtaining a calibration image sequence obtained through the shooting of a calibration plate by a camera, determining an initial value of a camera internal reference and a camera pose of each calibration image in the calibration image sequence according to the calibration image sequence, and screening the calibration image sequence according to at least one of the following parameters of each calibration image: the camera pose and definition to obtain an effective calibration image sequence; calculating the distance between each effective calibration image in the effective calibration image sequence and the calibration plate, and determining the weight of each effective calibration image according to the distance between each effective calibration image and the calibration plate; and according to the initial value of the internal reference of the camera, the weight of each effective calibration image and the camera pose, optimizing the internal reference of the camera by using a nonlinear optimization algorithm with the weight to obtain an optimized value of the internal reference of the camera. The method improves the calibration precision of the internal reference of the camera.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of camera calibration, and in particular to a camera intrinsic parameter calibration method, device, equipment, medium and program. BACKGROUND

[0002] In image measurement and machine vision applications, in order to establish an accurate correspondence between a spatial object and its image, a geometric model of camera imaging needs to be established, and the parameters of these models are camera parameters. The process of establishing a geometric model of camera imaging is also called camera calibration, which includes intrinsic parameter calibration and extrinsic parameter calibration. Camera intrinsic parameters are parameters related to the characteristics of the camera itself, such as the focal length, optical center, distortion parameters, etc.

[0003] The traditional camera calibration method uses a calibration board with a fixed pitch pattern array for calibration, and a camera is used to take multiple calibration images containing the calibration board images from multiple different positions and angles. Feature points (i.e. corner points) are extracted from the calibration images, and the feature points of each calibration image are used as input for the calibration algorithm. The calibration algorithm first calculates the initial value of the camera intrinsic parameters using a linear algorithm, and then calculates the accurate value of the camera intrinsic parameters using a nonlinear optimization algorithm.

[0004] In this traditional camera calibration method, the quality of the calibration images is not good, resulting in low accuracy of the calibration results. SUMMARY

[0005] Embodiments of the present application provide a camera intrinsic parameter calibration method, device, equipment, medium and program, which improve the calibration accuracy of camera intrinsic parameters compared to the prior art.

[0006] In a first aspect, embodiments of the present application provide a camera intrinsic parameter calibration method, which includes: obtaining a calibration image sequence taken by a camera on a calibration board, determining an initial value of camera intrinsic parameters and camera poses of each calibration image in the calibration image sequence according to the calibration image sequence; selecting the calibration image sequence according to at least one of the following parameters of each calibration image: camera pose and sharpness, to obtain an effective calibration image sequence; calculating the distance between each effective calibration image in the effective calibration image sequence and the calibration board, and determining the weight of each effective calibration image according to the distance between the effective calibration image and the calibration board; and using a weighted nonlinear optimization algorithm to optimize the camera intrinsic parameters according to the initial value of the camera intrinsic parameters, the weight and camera pose of each effective calibration image, to obtain the optimized value of the camera intrinsic parameters.

[0007] In some example embodiments, the screening the sequence of calibration images according to at least one of the following parameters of each calibration image: camera pose and sharpness, to obtain a sequence of effective calibration images comprises: establishing a spatial grid with a fixed grid size in a calibration board coordinate system, projecting the camera pose of each calibration image into the spatial grid to obtain a grid in which the camera pose of each calibration image is located; for each target grid in which there is a calibration image in the spatial grid, screening the calibration images in the target grid according to at least one of the following parameters of the calibration images in the target grid: shooting angle difference between images and sharpness, to obtain effective calibration images in the target grid, wherein the shooting angle difference is determined according to the camera poses of two calibration images.

[0008] In some example embodiments, the screening the calibration images in the target grid according to at least one of the following parameters of the calibration images in the target grid: shooting angle difference between images and sharpness, to obtain effective calibration images in the target grid comprises: when the number of calibration images in the target grid is greater than a first number, screening out the first number of calibration images with the greatest sharpness from the target grid as candidate calibration images of the target grid; when the number of calibration images in the target grid is less than or equal to the first number, determining all the calibration images in the target grid as the candidate calibration images of the target grid; calculating the shooting angle difference between any two candidate calibration images in the target grid according to the camera poses of the candidate calibration images of the target grid; when the number of candidate calibration images in the target grid is less than or equal to a second number, determining all the candidate calibration images in the target grid as the effective calibration images of the target grid, the second number being less than the first number; when the number of candidate calibration images in the target grid is greater than the second number, determining the second number of calibration images with the greatest shooting angle difference from the target grid as the effective calibration images of the target grid.

[0009] In some example embodiments, the method further comprises: obtaining the size of a pattern in the calibration board, and determining the grid size according to the size of the pattern, wherein the size of the grid is greater than the size of the pattern.

[0010] In some example embodiments, the screening the sequence of calibration images according to at least one of the following parameters of each calibration image: camera pose and sharpness, to obtain a sequence of effective calibration images comprises: grouping images in the sequence of calibration images according to a preset fixed shooting time length according to a shooting time of each calibration image, to obtain a plurality of image groups; for each image group, screening calibration images in the image group according to at least one of the following parameters of the calibration images in the image group: shooting angle difference between images and sharpness, to obtain effective calibration images in the image group, wherein the shooting angle difference is determined according to camera poses of two calibration images.

[0011] In some example embodiments, the screening calibration images in the image group according to at least one of the following parameters of the calibration images in the image group: shooting angle difference between images and sharpness, to obtain effective calibration images in the image group comprises: when a number of calibration images in the image group is greater than a third number, screening the third number of calibration images with the greatest sharpness from the image group as candidate calibration images of the image group; when the number of calibration images in the image group is less than or equal to the third number, determining all calibration images in the image group as the candidate calibration images of the image group; calculating a shooting angle difference between any two candidate calibration images in the image group according to poses of the candidate calibration images in the image group; when the number of candidate calibration images in the image group is less than or equal to a fourth number, determining all candidate calibration images in the image group as effective calibration images of the image group, the fourth number being less than the third number; when the number of candidate calibration images in the image group is greater than the fourth number, determining the fourth number of candidate calibration images with the greatest shooting angle difference from the image group as the effective calibration images of the image group.

[0012] In some example embodiments, the sequence of calibration images is obtained by shooting the calibration board or the camera moving on at least one hemisphere.

[0013] In some example embodiments, the determining an initial value of camera intrinsic parameters and a camera pose of each calibration image in the sequence of calibration images according to the sequence of calibration images comprises: performing feature point detection on each calibration image in the sequence of calibration images, to obtain pixel coordinates and three-dimensional coordinates of feature points of each calibration image; determining the initial value of the camera intrinsic parameters and the camera pose of each calibration image using a pre-estimation algorithm according to the pixel coordinates and the three-dimensional coordinates of the feature points of each calibration image.

[0014] In some example embodiments, before the initial value of the camera intrinsic parameter and the camera pose of each calibration image are determined using a pre-estimation algorithm according to the pixel coordinates and the three-dimensional coordinates of the feature points of each calibration image, the method further comprises: deleting a calibration image with a number of feature points less than a preset number from the sequence of calibration images.

[0015] In some example embodiments, the clarity of the calibration image is an average gradient of the calibration image.

[0016] In some example embodiments, the method further comprises: for each feature point in the calibration image, determining a pixel block of a preset size centered on the feature point, calculating an average gradient of the pixels in the pixel block, and taking the average gradient of the pixels in the pixel block as the gradient of the feature point; and calculating an average value of the gradients of all the feature points in the calibration image to obtain the average gradient of the calibration image.

[0017] In some example embodiments, the greater the distance between the effective calibration image and the calibration board, the smaller the weight of the effective calibration image.

[0018] In some example embodiments, the camera intrinsic parameter is optimized using a weighted nonlinear optimization algorithm according to the initial value of the camera intrinsic parameter and the weight and camera pose of each effective calibration image to obtain an optimized value of the camera intrinsic parameter, which comprises: projecting the three-dimensional coordinates of the feature points of each effective calibration image onto each effective calibration image according to the camera pose of each effective calibration image and the initial value of the camera intrinsic parameter to obtain an estimated value of the pixel coordinates of the feature points of each effective calibration image; obtaining a projection error of each effective calibration image according to the difference between the observed value and the estimated value of the pixel coordinates of the feature points of each effective calibration image; performing a weighted operation on the projection errors of all the effective calibration images according to the weights of the effective calibration images to obtain a total projection error; and optimizing the camera intrinsic parameter using a nonlinear optimization algorithm to obtain an optimized value of the camera intrinsic parameter with the minimum total projection error as the optimization objective.

[0019] In a second aspect, an embodiment of the present application provides a camera intrinsic parameter calibration device, the device comprising: an initial estimation module, a screening module, a weight determination module, and an optimization module; the initial estimation module is configured to obtain a calibration image sequence captured by a camera on a calibration board, determine an initial value of a camera intrinsic parameter and a camera pose of each calibration image in the calibration image sequence according to the calibration image sequence; the screening module is configured to screen the calibration image sequence according to at least one of the following parameters of each calibration image: camera pose and definition, and obtain an effective calibration image sequence; the weight determination module is configured to calculate a distance between each effective calibration image in the effective calibration image sequence and the calibration board, and determine a weight of each effective calibration image according to the distance between each effective calibration image and the calibration board; and the optimization module is configured to optimize the camera intrinsic parameter using a weighted nonlinear optimization algorithm according to the initial value of the camera intrinsic parameter, the weight and the camera pose of each effective calibration image, and obtain an optimized value of the camera intrinsic parameter.

[0020] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory, the memory being configured to store a computer program, and the processor being configured to invoke and run the computer program stored in the memory to execute the method according to the first aspect or any of the implementation manners.

[0021] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium being configured to store a computer program, and the computer program being configured to enable a computer to execute the method according to the first aspect or any of the implementation manners.

[0022] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computer program, the computer program being configured to enable a processor to implement the method according to the first aspect or any of the implementation manners.

[0023] Through the technical scheme provided by the embodiment of the present application, the calibration device automatically screens the calibration image sequence according to the camera pose and / or definition of the calibration image, improves the efficiency of camera calibration, reduces the influence of overfitting of image quality and camera pose on the accuracy of camera calibration, and further improves the accuracy of camera intrinsic parameter calibration by assigning different weights to different calibration images according to the distance between the camera and the calibration board and using the weight of the calibration image to optimize the camera intrinsic parameter. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative effort based on these drawings are within the scope of the present application.

[0025] Figure 1 The schematic diagram of the architecture of the camera calibration system applicable to the embodiments of the present application;

[0026] Figure 2 The flow chart of the camera intrinsic parameter calibration method provided by the first embodiment of the present application;

[0027] Figure 3 The schematic diagram of the camera movement;

[0028] Figure 4 The flow chart of the camera intrinsic parameter calibration method provided by the second embodiment of the present application;

[0029] Figure 5 The schematic diagram of the space grid;

[0030] Figure 6 The flow chart of the camera intrinsic parameter calibration method provided by the third embodiment of the present application;

[0031] Figure 7 The structural schematic diagram of the camera intrinsic parameter calibration device provided by the fourth embodiment of the present application;

[0032] Figure 8 The structural schematic diagram of the calibration device provided by the fifth embodiment of the present application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of the present application.

[0034] It is to be understood that the terms "first", "second", and the like, used in the description and the claims of the present application as well as the above description of the drawings merely refer to categories and do not necessarily imply a sequence or order unless otherwise indicated by the context. It is to be understood that the use of the term "or" in the description and the claims of the present application is used either in the inclusive sense and for an open ended description of coverage or in the exclusive sense, when the description using this term is immediately preceded by the term "either". In addition, the use of "one embodiment", "an implementation", "an implementation include", "one implementation include" throughout this application are not to be construed to mean that this embodiment is the only one implemented. The terms "the process", "the process include", "the method", "the method include", "the article", "the article include", "the device", "the device include", "the system", "the system include", "the server", "the server include" and the like used in the description and the claims of the present application are intended to refer to at least one of the respective processes, methods, articles, devices and systems unless otherwise indicated by the context. Furthermore, the terms "comprise", "have" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, article, or apparatus that comprises a list of steps or units is not necessarily limited to those steps or units which are expressly listed or to method combinations consisting only of the steps or units but can include additional steps or units which are not expressly listed or consist only of sub-combinations of the steps or units.

[0035] In order to facilitate the understanding of the embodiments of the present application, before describing the various embodiments of the present application, first, some concepts involved in all embodiments of the present application are appropriately described.

[0036] Computer vision (CV) technology is a science that studies how to make machines "see". More further, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further to do image processing, so that the computer processing becomes more suitable for human eye observation or image transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, trying to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, intelligent transportation, etc. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0037] Camera calibration: in the CV technology, in order to determine the mapping relationship between the three-dimensional coordinates of a certain point in the world coordinate system and the two-dimensional coordinates in the image coordinate system, a geometric model of camera imaging needs to be established. The parameters of the geometric model are camera parameters, which include camera intrinsic parameters and camera extrinsic parameters. The process of solving camera parameters is called camera calibration.

[0038] Camera intrinsic parameters are parameters related to the characteristics of the camera itself, such as the focal length of the camera, the pixel size, etc., which are mainly used to describe the conversion relationship from the camera coordinate system to the pixel coordinate system. Different types of cameras have different camera intrinsic parameters, and the types of cameras include but are not limited to: pinhole cameras, fisheye cameras, etc.

[0039] Taking a pinhole camera as an example, the pinhole camera intrinsic parameters include: f x , f y , c x, c y , k1, k2, p1 and p2. Wherein, f x , f y represents the focal length of the camera, c x , c y represents the position of the optical center, k1, k2 represents the radial distortion parameters, p1, p2 represents the tangential distortion parameters. Optionally, in some cameras, only the radial distortion can be considered, and the tangential distortion can not be considered.

[0040] The camera extrinsic parameters can include a rotation matrix R and a translation vector t. The rotation matrix R is used to describe the direction of the coordinate axis of the world coordinate system relative to the camera coordinate axis, and the translation matrix is used to describe the position of the space origin in the camera coordinate system.

[0041] The traditional camera calibration adopts a calibration board to assist in camera calibration, Figure 1 The architecture of the camera calibration system suitable for the embodiments of the present application is shown in FIG. 1, which includes a calibration board 10, a camera 20 and a calibration device 30. Figure 1

[0042] The calibration board 10 is a device for assisting camera calibration, and a plurality of calibration points are arranged on the calibration board 10 at a fixed interval. The calibration points are also called mark points, corner points or feature points, and the camera 20 performs parameter calibration through the calibration points on the calibration board.

[0043] For example, the calibration board 10 adopts an international chessboard pattern (referred to as a chessboard), an Apritag board, a grid, a circular board, etc.

[0044] The camera 20 is a monocular camera, and before camera calibration, the camera 20 needs to shoot a plurality of calibration images containing the image of the calibration board 10 at a plurality of camera poses to form a calibration image sequence. The calibration image for calibration needs to contain the image of the calibration board 10, and the images not containing the image of the calibration board 10 need to be deleted.

[0045] During the shooting process, the camera 20 can shoot calibration images at different camera poses in the following two ways: way one, the calibration board 10 is installed at a fixed position, and the camera 20 is moved to shoot calibration images at different camera poses. Way two, the camera 20 is installed at a fixed position, and the calibration board 10 is moved to make the camera 20 shoot calibration images at different camera poses. In order to improve the camera calibration accuracy, calibration images with rich camera poses are obtained as much as possible.

[0046] The calibration image sequence shot by the camera 20 is sent to the calibration device 30, and the calibration device 30 is used to perform camera calibration using the calibration image sequence to obtain the extrinsic and intrinsic parameters of the camera.

[0047] ​The traditional camera calibration process mainly includes the following steps: (1) preparing a calibration board 10; (2) moving the camera 20 or the calibration board 10 to capture calibration images at different angles and distances; (3) providing the calibration images to the calibration device 30, which detects feature points in the calibration images and determines the projection points of the feature points in the calibration images; (4) the calibration device 30 estimates the initial values of the camera intrinsic parameters and the extrinsic parameters based on the feature point detection results; (5) the calibration device 30 optimizes the camera intrinsic parameters based on the initial values of the camera intrinsic parameters and the extrinsic parameters estimated in step (4) to obtain the final camera intrinsic parameters.

[0048] In the embodiments of the present application, the calibration device 30 performs the following steps: obtaining a sequence of calibration images captured by the camera on the calibration board, determining the initial values of the camera intrinsic parameters and the camera poses of each calibration image in the sequence of calibration images; selecting the sequence of calibration images based on the camera poses and / or the clarity of each calibration image to obtain a sequence of effective calibration images; calculating the distances between each effective calibration image and the calibration board in the sequence of effective calibration images, and determining the weights of each effective calibration image based on the distances between the effective calibration images and the calibration board; and optimizing the camera intrinsic parameters using a weighted nonlinear optimization algorithm based on the initial values of the camera intrinsic parameters, the weights of each effective calibration image, and the camera poses to obtain the optimized values of the camera intrinsic parameters.

[0049] The calibration device 30 can be a terminal or a server, and the terminal and the server can be directly or indirectly connected with the camera 20 through wired or wireless communication, which is not limited in the present application. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart television, a vehicle-mounted terminal, etc., but is not limited thereto. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0050] The camera 20 and the calibration device 30 can be two separately arranged devices, or can be an integrated device.

[0051] The camera 20 can be a vehicle camera, and the camera 20 is detached from the vehicle for calibration at the time of calibration. By using the calibration method provided in the embodiments of the present application, the accuracy of calibration of the vehicle camera can be improved. The method of the embodiments of the present application can be applied in the field of autonomous driving. The camera intrinsic parameter will affect the positioning accuracy of the autonomous driving vehicle based on the visual positioning algorithm. By improving the calibration accuracy of the camera of the autonomous driving vehicle, the overall positioning accuracy of the autonomous driving vehicle can be improved.

[0052] The camera 20 can also be a camera of a mobile terminal using Simultaneous Localization and Mapping (SLAM) positioning. The mobile terminal using SLAM positioning can be a smartphone, a tablet computer, an Extended Reality (XR) device, a robot, etc. The XR device includes a Virtual Reality (VR) device, an Augmented Reality (AR) device, and a Mixed Reality (MR) device. When the camera 20 is a mobile terminal using SLAM, the calibration device 30 described above can be the mobile terminal, i.e., the camera 20 and the calibration device 30 are integrated devices.

[0053] After introducing some concepts related to the embodiments of the present application, the calibration method of the camera intrinsic parameter provided in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0054] Figure 2 The flowchart of the calibration method of the camera intrinsic parameter provided in Embodiment One of the present application is executed by a calibration device. As shown in Figure 2 The method provided in the embodiment includes the following steps:

[0055] S101, acquire a calibration image sequence obtained by the camera shooting a calibration board, and determine initial values of the camera intrinsic parameter and camera poses of each calibration image in the calibration image sequence.

[0056] The camera transmits the acquired calibration image sequence to the calibration device. The calibration image sequence includes calibration images shot by the camera at different positions and angles (i.e., different camera poses). When calibrating the camera, the calibration images used need to include calibration board images. However, some images in the calibration image sequence may not have shot the calibration board. Therefore, before determining the initial values of the camera intrinsic parameter and the camera poses of each calibration image in the calibration image sequence, the images in the calibration image sequence that do not have shot the calibration board need to be filtered out.

[0057] In order to improve the accuracy of camera calibration, the camera is shot at various angles and poses as much as possible to obtain calibration images with rich camera poses. Optionally, the calibration image sequence is obtained by moving the calibration board or the camera on at least one hemisphere.

[0058] Figure 3 An example of camera movement is shown in FIG. 1, where an Apriltag calibration board is fixed at a certain position, and the camera is first controlled to move on a first hemisphere and shot calibration images; after the movement on the first hemisphere, the camera is moved backward by a suitable distance (e.g., 0.5 meters or 1 meter), and then moves on a second hemisphere and shot calibration images. The radius of the second hemisphere is greater than that of the first hemisphere, and the camera is controlled to move on the two hemispheres with different radii to make the camera positions more diverse. Figure 3

[0059] In one way, the camera is moved up, down, left and right in a hemisphere or an approximate hemisphere by a hand or a camera holder. In another way, the camera is automatically controlled by a mechanical arm to move up, down, left and right in a hemisphere or an approximate hemisphere. The camera poses are more diverse by moving up, down, left and right in a hemisphere.

[0060] Figure 3 The lines on the two hemispheres represent the approximate movement trajectory of the camera, which can be understood that Figure 3 is only a schematic diagram and does not constitute a limitation on the movement trajectory of the camera. The movement trajectory formed by moving up, down, left and right in a hemisphere or an approximate hemisphere is approximately arc-shaped.

[0061] When the camera is controlled to move on the hemisphere, the orientation of the camera is always towards the calibration board to ensure that the calibration board is always displayed in the image during the movement. For example, when the camera is above the calibration board, the camera needs to be tilted downward so that the lens is directed at the calibration board. When the camera is to the right of the calibration board, the camera needs to be tilted to the left.

[0062] Figure 3 Taking the camera movement as an example, in actual situations, the calibration board can also be moved up, down, left and right on one or more hemispheres to obtain more calibration images corresponding to camera poses.

[0063] In this embodiment, the calibration image sequence is obtained by automatically shooting calibration images by the camera at a fixed acquisition frequency. For example, after the shooting starts, the camera is turned on to the video mode, and then the camera automatically shoots calibration images.

[0064] ​The initial value of the camera intrinsic parameter and the camera pose of each calibration image in the calibration image sequence can be determined by using an existing camera calibration method, wherein the camera pose of each calibration image is the camera extrinsic parameter corresponding to the calibration image, and the camera pose includes a camera position and an attitude, the camera position refers to the coordinates of the camera in the three-dimensional space at the shooting time, and the camera attitude refers to the orientation of the camera at the shooting time.

[0065] The initial value of the camera intrinsic parameter and the camera pose of each calibration image in the calibration image sequence are determined according to the calibration image sequence, including the following steps: performing feature point detection on each calibration image in the calibration image sequence to obtain the pixel coordinates and three-dimensional coordinates of the feature points of each calibration image; and determining the initial value of the camera intrinsic parameter and the camera pose of each calibration image by using a pre-estimation algorithm according to the pixel coordinates and three-dimensional coordinates of the feature points of each calibration image.

[0066] The points on the calibration board for calibration are referred to as feature points (also referred to as corner points or calibration points), when the Apriltag is used for the calibration board, as shown in FIG. 1, the calibration board is composed of a 6-row and 6-column two-dimensional code grid, and the feature points are four small grids at the corner points or vertexes of the two-dimensional code grid. There are usually multiple feature points on the calibration board, and the number of feature points on the calibration board is selected according to the actual application scenario. Figure 3

[0067] The feature point detection is performed on each calibration image to determine the pixel coordinates and three-dimensional coordinates of the feature points on the calibration image, the pixel coordinates are also referred to as uv coordinates, and any pixel on the image can be located through the uv coordinates. The three-dimensional coordinates of a feature point in the three-dimensional space are usually obtained through three transformations, i.e., transforming from the world coordinate system to the camera coordinate system, performing perspective projection from the camera coordinate system to the image coordinate system, and performing twice conversion on the image coordinate system to obtain the pixel coordinate system.

[0068] The three-dimensional coordinates of the feature points refer to the coordinates of the feature points in the three-dimensional space (i.e., the world coordinate system), and the pixel coordinates of the feature points refer to the coordinates of the feature points in the pixel coordinate system. The feature point detection can be performed by using an existing detection method to obtain the three-dimensional coordinates and pixel coordinates of the feature points.

[0069] Since the camera poses of each calibration image are different, the number of feature points in each calibration image is also different, and the number of feature points in some calibration images is relatively large, and the number of feature points in some calibration images is relatively small. The pixel coordinates of the feature points of the calibration image obtained by the detection method are also referred to as the observation values of the pixel coordinates of the feature points.

[0070] ​Optionally, before determining the initial value of the camera intrinsic parameter and the camera pose of each calibration image using the pre-estimation algorithm according to the pixel coordinates and the three-dimensional coordinates of the feature points of each calibration image, the calibration image sequence can also be deleted from the calibration image sequence in which the number of feature points is less than a preset number. The preset number is related to the number of feature points on the calibration board. When the number of feature points on the calibration board is relatively large, the preset number can also be set to be relatively large. When the number of feature points on the calibration board is relatively small, the preset number is correspondingly set to be relatively small.

[0071] For example, for the calibration board in Figure 3 , the preset number can be set to 16. When the number of feature points on a calibration image is less than 16, the calibration image is deleted, and only the calibration image in which the number of feature points is greater than 16 is retained.

[0072] After the pixel coordinates and the three-dimensional coordinates of the feature points of each calibration image are determined, the pixel coordinates and the three-dimensional coordinates of the feature points of each calibration image are used as inputs to determine the initial value of the camera intrinsic parameter and the camera pose of each calibration image using an existing pre-estimation algorithm. The pre-estimation algorithm can be a linear algorithm, for example, Zhang Zhengyou calibration method.

[0073] Taking the Zhang Zhengyou calibration method as an example, the homography matrix is first calculated according to the pixel coordinates and the three-dimensional coordinates of the feature points of each calibration image. The homography matrix can be used to describe the position mapping relationship between the object in the world coordinate system and the pixel coordinate system. The homography matrix contains the camera intrinsic parameter and the camera extrinsic parameter. It can be understood that the camera intrinsic parameter of each image is fixed, while the camera extrinsic parameter (i.e. the camera pose) of each image is variable, so the camera intrinsic parameter is easier to solve. Therefore, the camera intrinsic parameter can be solved according to the homography matrix, and the camera extrinsic parameter of each calibration image can be obtained according to the camera intrinsic parameter. When solving the camera intrinsic parameter, the homography matrix is solved according to the constraint relationship of the rotation vector, and the specific solving process is not described here.

[0074] S102, according to at least one of the following parameters of each calibration image: camera pose and clarity, the calibration image sequence is screened to obtain an effective calibration image sequence.

[0075] In the calibration image sequence, there can be some calibration images with the same or very similar camera poses, i.e. there are calibration images with camera pose overfitting, which will result in low precision of the camera intrinsic parameter obtained by final optimization. In the traditional calibration method, the appropriate calibration image is usually manually selected by the engineering personnel, which takes a long time, and manual selection may introduce errors. In order to solve this problem, the method of the embodiments of the present application can automatically screen the calibration image sequence according to the camera pose and / or clarity of the calibration image to obtain an effective calibration image sequence, and the automatic screening improves the screening efficiency of the image.

[0076] In this embodiment, the pose difference between the calibration images can be determined according to the camera poses of the calibration images, and the calibration images with similar poses can be automatically filtered out according to the pose difference between the calibration images, and the calibration images with larger pose difference can be retained.

[0077] Alternatively, the pose difference between the calibration images can be calculated according to the difference in shooting angles of the two calibration images. The pose of the calibration image includes the position and the attitude, and the attitude of the calibration image is used to represent the orientation of the camera at the shooting moment. Therefore, the difference in shooting angles of the two calibration images can be calculated according to the attitudes of the two calibration images.

[0078] The definition of the image is a key parameter for measuring the image quality. Generally, the clearer the image is, the higher the image quality is. The definition and the blur of the image are two opposite concepts for describing the definition of the image. The larger the definition of the image is, the smaller the blur of the image is. The smaller the definition of the image is, the larger the blur of the image is. Therefore, the definition of the calibration image in the embodiments of the present application can also be replaced by the blur of the image.

[0079] There are many factors that can cause image blur in the process of image acquisition, transmission and processing. For example, incorrect focusing can cause defocus blur when acquiring the image, the relative motion between the scene and the camera can cause motion blur, and the loss of high frequency after image compression can cause blur. The blur reduces the definition of the image and seriously affects the image quality, resulting in inaccurate image analysis and processing results. In this embodiment, the calibration image with low definition can reduce the calibration accuracy of the camera. Therefore, the definition of each calibration image is acquired, the calibration image sequence is selected according to the definition of the image, the calibration image with low definition is filtered out, and the calibration image with high definition and good quality is retained.

[0080] Exemplarily, the definition of the calibration image can be determined by the following methods: (1) pixel-based technology, including analyzing the statistical characteristics of the pixel gray value and the correlation between the pixels; (2) transform domain-based technology, which utilizes the principle that the more high-frequency components in the transform domain, the clearer the image is, and the fewer high-frequency components, the more blurred the image is; (3) image gradient-based technology, which utilizes the gradient of the image edge to measure the degree of image blur, and the larger the gradient is, the clearer the image is.

[0081] Taking the image gradient-based technology as an example, the definition of the calibration image can be the average gradient of the calibration image. Exemplarily, the average gradient of the calibration image can be calculated by the following method: for each feature point in the calibration image, a pixel block with a preset size is determined with the feature point as the center, the average gradient of the pixels in the pixel block is calculated, and the average gradient of the pixels in the pixel block is taken as the gradient of the feature point; and the average value of the gradients of all the feature points in the calibration image is calculated to obtain the average gradient of the calibration image.

[0082] After feature point detection is performed on the calibration image, the feature points of each calibration image are obtained. Optionally, the gradient of each feature point can be calculated in the feature point detection process in this embodiment. The preset size can be, for example, 10 pixels. For each feature point in the calibration image, a 10*10 pixel block is determined with the feature point as the center, the average gradient of the pixels in the pixel block is calculated, the average gradient of the pixels in the pixel block is taken as the gradient of the feature point, the gradients of all the feature points in the calibration image are averaged to obtain the average gradient of the calibration image, and the average gradient of the calibration image is taken as the definition of the calibration image. The larger the average gradient of the calibration image is, the higher the definition of the calibration image is.

[0083] In this embodiment, the calibration device can automatically perform image screening according to the camera pose of the calibration image, can automatically perform image screening according to the definition of the calibration image, or can perform image screening according to the camera pose and the definition of the calibration image.

[0084] In this embodiment, the calibration image sequence is automatically screened according to the camera pose and the definition of the calibration image, and the calibration image with a higher definition and a larger pose difference is retained. The time for manually selecting images is saved, the calibration efficiency is improved, and the influence of image quality on the calibration accuracy and the influence of camera pose overfitting on the calibration accuracy are reduced.

[0085] S103, the distance between each effective calibration image and the calibration board in the effective calibration image sequence is calculated, and the weight of each effective calibration image is determined according to the distance between each effective calibration image and the calibration board.

[0086] The camera position, i.e., the three-dimensional coordinates of the camera, is obtained from the camera pose of each effective calibration image, the distance between the effective calibration image and the calibration board is calculated according to the camera position of the effective calibration image, the weight of each effective calibration image is determined according to the distance between each effective calibration image and the calibration board, the weight of the effective calibration image is smaller when the distance between the effective calibration image and the calibration board is larger, i.e., the weight of the effective calibration image is smaller when the effective calibration image is closer to the calibration board.

[0087] Optionally, the weight of the effective calibration image is in the range of [0, 1], the distance between each effective calibration image and the calibration board in the effective calibration image sequence can be normalized to obtain the weight of each effective calibration image.

[0088] S104, the camera intrinsic parameter is optimized using a weighted nonlinear optimization algorithm according to the initial value of the camera intrinsic parameter, the weight and the camera pose of each effective calibration image, and an optimized value of the camera intrinsic parameter is obtained.

[0089] The initial value of the camera intrinsic parameter estimated by step S101 is a rough value or an inaccurate value. The estimation result of S101 is used as an initial value, and the optimized value of the camera intrinsic parameter is obtained through multiple rounds of optimization iteration based on the maximum likelihood estimation.

[0090] In the nonlinear optimization process, the weight of each effective calibration image is used as the weight of the camera pose and the camera intrinsic parameter of each effective calibration image.

[0091] For example, the optimization process includes: projecting the three-dimensional coordinates of the feature points of each effective calibration image onto each effective calibration image according to the initial value of the camera pose and the camera intrinsic parameter of each effective calibration image, to obtain the estimated value of the pixel coordinates of the feature points of each effective calibration image; obtaining the projection error of each effective calibration image according to the difference between the observed value and the estimated value of the pixel coordinates of the feature points of each effective calibration image; performing weighted operation on the projection errors of all effective calibration images according to the weight of each effective calibration image, to obtain the total projection error; using a nonlinear optimization algorithm to optimize the camera intrinsic parameter with the minimum total projection error as the optimization target, to obtain the optimized value of the camera intrinsic parameter.

[0092] Suppose there are n effective calibration images, and each effective calibration image has m feature points. The total projection error L can be expressed as:

[0093]

[0094] wherein m ij represents the observed value of the pixel coordinates of the jth feature point in the ith effective calibration image, represents the estimated value of the pixel coordinates of the jth feature point in the ith effective calibration image, W i represents the weight of the ith effective calibration image, K represents the camera intrinsic parameter, k1 and k2 represent the radial distortion parameters, R i represents the rotation matrix of the ith effective calibration image, t i represents the translation vector of the ith effective calibration image, R i and t i i.e. the pose of the ith effective calibration image, P ij represents the three-dimensional coordinates (i.e. the three-dimensional coordinates in the temporal coordinate system) of the jth feature point in the ith effective calibration image.

[0095] represents the projection model or the projection function of the camera. The three-dimensional point is projected onto the two-dimensional image through the projection function. The projection result of the three-dimensional point obtained through the projection function is the estimated value of the pixel coordinates of the feature point. The projection result of the three-dimensional point is the result estimated according to the camera intrinsic parameter and the extrinsic parameter, so the projection result is the estimated value. And m ijis the pixel coordinate of the feature point detected from the calibration image, that is, the observation value of the pixel coordinate of the feature point is directly obtained from the calibration image. The projection error of the feature point is the mean square error of the observation value and the estimated value of the pixel coordinate of the feature point.

[0096] The total projection error L is taken as the objective function, and the minimum value of the objective function is taken as the optimization target (that is, the minimum total projection error). The optimization value of the camera intrinsic parameter is obtained through multiple iterations. In an optional manner, when the total projection error no longer decreases, the iteration process is stopped, and the camera intrinsic parameter obtained through iteration is taken as the optimization value of the camera intrinsic parameter. In another optional manner, when the number of iterations reaches the maximum number, the iteration process is stopped, and the camera intrinsic parameter obtained through iteration is taken as the optimization value of the camera intrinsic parameter.

[0097] In the traditional calibration method, because the optimization process is fully automatic, the weight of each pose of the camera in the optimization process is averagely distributed (or described as no weight is distributed for the image of each pose), therefore, overfitting occurs in the optimization of the pose in some positions, resulting in low precision of the finally optimized intrinsic parameter. In the optimization process of the embodiment, when the total projection error is calculated, the weights of the feature points in the same effective calibration image are the same, and the weights of different effective calibration images are different. Under normal circumstances, the greater the distance between the effective calibration image and the calibration board, the smaller the weight of the effective calibration image. By distributing different weights to different calibration images, overfitting in the optimization of the pose in some positions can be avoided, thereby improving the calibration precision of the camera intrinsic parameter.

[0098] For example, the weighted nonlinear optimization algorithm can be a weighted least squares method. The least squares method is a data optimization technique that finds the best function match of data by minimizing the sum of squares of errors. The Levenberg-Marquardt (LM) algorithm is a commonly used algorithm for solving least squares problems. The LM algorithm combines the advantages of the steepest descent method and the Gauss-Newton method, and can automatically adjust the step size in the iteration process, thereby more effectively approaching the optimal solution. In practical applications, the LM algorithm usually updates the parameters through iteration until the convergence condition is reached.

[0099] The method of the embodiment obtains a calibration image sequence captured by a camera on a calibration board, determines initial values of camera intrinsic parameters and camera poses of each calibration image in the calibration image sequence according to the calibration image sequence, filters the calibration image sequence according to at least one of the following parameters of each calibration image: the camera pose and the definition, and obtains an effective calibration image sequence; calculates distances between each effective calibration image in the effective calibration image sequence and the calibration board, and determines weights of each effective calibration image according to the distances between the effective calibration image and the calibration board; and optimizes the camera intrinsic parameters using a weighted nonlinear optimization algorithm according to the initial values of the camera intrinsic parameters, the weights of each effective calibration image and the camera poses, and obtains optimized values of the camera intrinsic parameters. The method automatically filters the calibration image sequence according to the camera poses and / or the definition of the calibration images, reduces the influence of overfitting of the image quality and the camera pose on the camera calibration accuracy, and assigns different weights to different calibration images according to the distances between the camera and the calibration board, further avoids overfitting of the camera pose, and thus improves the calibration accuracy of the camera intrinsic parameters.

[0100] Figure 4 The flowchart of the camera intrinsic parameter calibration method provided in Embodiment Two of the present application, and the embodiment is mainly used for describing various implementation manners of step S102 in Embodiment One. As shown in Figure 4 the method provided by the embodiment includes the following steps:

[0101] S1021, a space grid is established with a fixed grid size in the calibration board coordinate system, and the camera poses of each calibration image are projected into the space grid to obtain the grid where the camera poses of each calibration image are located.

[0102] Figure 5 A schematic diagram of the space grid is shown in Figure 5 the space grid is a 3D cubic grid, and the calibration board is a 2D flat board. For example, the upper left corner of the calibration board can be taken as the origin, and the left lower corner or other corners of the calibration board can also be taken as the origin. The length direction of the calibration board is taken as the X axis, the width direction of the calibration board is taken as the Y axis, a calibration board coordinate system is established, a 3D coordinate system is established based on the calibration board coordinate system and in the direction perpendicular to the calibration board, a space grid is established in the 3D coordinate system with a fixed grid size, the space grid includes a plurality of grids with the same size, each grid is a cube or cuboid, the size of the space grid is managed according to the camera pose, and the space where the space grid is located can cover at least all camera poses of the calibration images in the calibration image sequence.

[0103] Optionally, before the space grid is established, the size of the pattern in the calibration board is obtained, and the grid size is determined according to the size of the pattern, wherein the size of the grid is greater than the size of the pattern. For example, the size of the grid is a preset multiple of the size of the pattern in the calibration board.

[0104] For example, if the size of the pattern in the calibration board is 0.08 meters (i.e. 8 centimeters), the size of the grid can be 0.3 meters (i.e. 30 centimeters).

[0105] After the space grid is established, the camera poses of each calibration image are projected into the space grid to obtain the grid in which the camera poses of each calibration image are located. When the coordinate system of the camera pose is the coordinate system of the calibration board, the camera poses of each calibration image are directly projected into the space grid. When the coordinate system of the camera pose is not the coordinate system of the calibration board, the coordinate system of the camera pose is converted to the coordinate system of the calibration board, and then the camera poses of each calibration image are projected into the space grid. The projection of the camera poses of each calibration image into the space grid refers to determining which grid the camera poses of each calibration image belong to.

[0106] S1022, for each target grid in which the calibration images exist in the space grid, screening the calibration images in the target grid according to at least one of the following parameters of the calibration images in the target grid: the shooting angle difference and the sharpness between images, to obtain the effective calibration images in the target grid, wherein the shooting angle difference is determined according to the camera poses of two calibration images.

[0107] After the camera poses of each calibration image are projected into the space grid, some grids in the space grid have calibration images and some grids do not have calibration images. The grids with calibration images are called target grids, and the calibration images in the target grids are automatically screened. The grids without calibration images are not processed.

[0108] For each target grid, the sharpness of each calibration image in the target grid and the shooting angle difference between any two calibration images are determined. The specific determination method is described in Embodiment One, which will not be described here.

[0109] For example, for each target grid, the automatic screening is performed by the following method:

[0110] (1) determining whether the number of calibration images in the target grid is greater than or equal to a first number. When the number of calibration images in the target grid is less than or equal to the first number, all the calibration images in the target grid are determined as the candidate calibration images of the target grid. When the number of calibration images in the target grid is greater than the first number, the first number of calibration images with the greatest sharpness are selected from the target grid as the candidate calibration images of the target grid.

[0111] It can be understood that the number of calibration images in each target grid is independent. The number of calibration images in some target grids is relatively large, and the number of calibration images in some target grids is relatively small.

[0112] For example, the first number is 5. When the number of calibration images in the target grid is less than or equal to 5, all the calibration images in the target grid are retained as the candidate calibration images of the target grid. When the number of calibration images in the target grid is greater than 5, the 5 calibration images with the largest shooting angle difference are determined from the target grid as the effective calibration images of the target grid.

[0113] (2) Determine whether the number of candidate calibration images in the target grid is greater than a second number, and the second number is less than the first number. When the number of candidate calibration images in the target grid is less than or equal to the second number, all the candidate calibration images in the target grid are determined as the effective calibration images of the target grid. When the number of candidate calibration images in the target grid is greater than the second number, the second number of calibration images with the largest shooting angle difference are determined from the target grid as the effective calibration images of the target grid.

[0114] The candidate calibration images of each target grid are obtained through step (1). In this step, the camera poses of the candidate calibration images in the target grid are further screened. The shooting angle difference between any two candidate calibration images in the target grid is calculated according to the camera poses of the candidate calibration images in the target grid.

[0115] For any two candidate calibration images, the orientations of the cameras corresponding to the two candidate calibration images can be obtained according to the camera poses of the two candidate calibration images. The shooting angle difference of the cameras is calculated according to the orientations of the cameras corresponding to the two candidate calibration images. The shooting angle difference of the cameras is the shooting angle difference between the two candidate calibration images.

[0116] When the first number is 5, the second number can be 2. When the number of candidate calibration images in the target grid is less than 2, all the candidate calibration images in the target grid are determined as the effective calibration images of the target grid. When the number of candidate calibration images in the target grid is greater than 2, the two candidate calibration images with the largest shooting angle difference are determined from the target grid as the effective calibration images of the target grid.

[0117] In this embodiment, by establishing a space grid, the camera poses of each image in the calibration image sequence are mapped to different 3D grids, the calibration image sequence is grouped in 3D space, and the camera poses of the calibration images between different 3D grids are greatly different. For the calibration images in each grid, the camera poses and the clarity are automatically screened, and the calibration images with large clarity and large pose difference in each grid are retained as the effective calibration images, thereby reducing the influence of overfitting of image quality and camera pose on the camera calibration accuracy.

[0118] In the manner of the second embodiment, the calibration images are automatically screened in the time dimension. In other embodiments of the present application, the calibration images can also be automatically screened in the time dimension, because the camera poses of calibration images taken at similar times are usually less different.

[0119] For example, the calibration images can be automatically screened in the time dimension in the following manner: according to the shooting times of the calibration images, the images in the calibration image sequence are grouped according to a preset fixed shooting time length, obtaining a plurality of image groups; for each image group, the calibration images in the image group are screened according to at least one of the following parameters of the calibration images in the image group: shooting angle difference and sharpness between the images, obtaining effective calibration images in the image group, wherein the shooting angle difference is determined according to the camera poses of the two calibration images.

[0120] The fixed shooting time length can be 20 seconds, 30 seconds, or 1 minute, etc. In general, the camera poses of calibration images taken at similar times are relatively similar, so the calibration image sequence can be grouped in the time dimension according to the fixed shooting time length, and the camera poses of calibration images in different image groups are significantly different. For the calibration images in each image group, the calibration images are automatically screened according to the camera poses and the sharpness, and the calibration images with large sharpness and large pose difference in each image group are retained as effective calibration images.

[0121] The calibration images taken within the fixed shooting time length are screened according to the camera poses, and the calibration images with large camera pose difference are retained.

[0122] For example, for each image group, the calibration images in the image group are screened in the following manner:

[0123] (1) When the number of calibration images in the image group is greater than a third number, the third number of calibration images with the largest sharpness are screened from the image group as candidate calibration images of the image group; when the number of calibration images in the image group is less than or equal to the third number, all the calibration images in the image group are determined as the candidate calibration images of the image group.

[0124] (2) According to the poses of the candidate calibration images in the image group, the shooting angle difference between any two candidate calibration images in the image group is calculated, when the number of candidate calibration images in the image group is less than or equal to a fourth number, all the candidate calibration images in the image group are determined as the effective calibration images of the image group, and the fourth number is less than the third number; when the number of candidate calibration images in the image group is greater than the fourth number, the fourth number of candidate calibration images with the largest shooting angle difference are determined from the image group as the effective calibration images of the image group.

[0125] Figure 6 The flowchart of the camera intrinsic parameter calibration method provided in Embodiment Three of the present application is shown in FIG. 3. The same content is referred to the description of Embodiments One and Two. As shown in the figure, the method provided in the present embodiment includes the following steps: Figure 6

[0126] S201, obtaining a calibration image sequence captured by a camera pair on a calibration board. The calibration image sequence is captured by moving the calibration board or the camera on at least one hemisphere.

[0127] For example, the calibration board is fixed at a certain position, the video recording mode of the monocular camera is started, the camera is controlled to move up, down, left and right on a first hemisphere, and calibration images are captured. Then, the camera is moved back 0.5 meters, and the camera is controlled to move up, down, left and right on a second hemisphere, and calibration images are captured. The images captured on the two hemispheres are taken as the calibration image sequence.

[0128] S202, detecting feature points in each calibration image in the calibration image sequence to obtain pixel coordinates and three-dimensional coordinates of the feature points in each calibration image. Calibration images with a number of feature points less than a preset number are deleted from the calibration image sequence. The gradient of the feature points in each calibration image is calculated.

[0129] In this step, the pixel coordinates of the feature points in the calibration image obtained by feature point detection are the observed values of the pixel coordinates of the feature points, and the three-dimensional coordinates of the feature points are the coordinates of the feature points in the world coordinate system.

[0130] Optionally, in the feature point detection process, the gradient of the feature points in each calibration image is calculated. The gradient of the feature points is used to represent the sharpness of the feature points. For example, for each feature point in the calibration image, a pixel block with a preset size is determined with the feature point as the center. The average gradient of the pixels in the pixel block is calculated, and the average gradient of the pixels in the pixel block is taken as the gradient of the feature point.

[0131] S203, determining the initial value of the camera intrinsic parameter and the camera pose of each calibration image using a pre-estimation algorithm according to the pixel coordinates and the three-dimensional coordinates of the feature points in each calibration image.

[0132] The pixel coordinates and the three-dimensional coordinates of the feature points in each calibration image are taken as the input of the pre-estimation algorithm to estimate the initial value of the camera intrinsic parameter and the camera pose of each calibration image. The existing linear pre-estimation algorithm can be used to estimate the initial value of the camera intrinsic parameter, and the present embodiment does not limit this.

[0133] S204, establishing a space grid with a fixed grid size in the calibration board coordinate system, and projecting the camera pose of each calibration image into the space grid to obtain the grid where the camera pose of each calibration image is located.

[0134] ​The grid size is a preset fixed size or a size determined according to the size of the pattern in the calibration board. The camera poses of the calibration images are projected into the spatial grid by establishing the spatial grid.

[0135] In S205, for each target grid in which the calibration images exist, the sharpness of the calibration images is calculated according to the gradients of the feature points of the calibration images in the target grid, and the candidate calibration images of the target grid are determined according to the sharpness of the calibration images in the target grid.

[0136] In S202, the gradients of the feature points of the calibration images are calculated. For each calibration image in the target grid, the average value of the gradients of all the feature points of the calibration image is calculated to obtain the average gradient of the calibration image, and the average gradient of the calibration image is taken as the sharpness of the calibration image.

[0137] When the number of the calibration images in the target grid is greater than the first number, the first number of calibration images with the greatest sharpness are selected from the target grid as the candidate calibration images of the target grid; when the number of the calibration images in the target grid is less than or equal to the first number, all the calibration images in the target grid are determined as the candidate calibration images of the target grid.

[0138] In S206, the shooting angle difference between any two candidate calibration images in the target grid is calculated according to the camera poses of the candidate calibration images of the target grid, and the effective calibration images of the target grid are determined according to the shooting angle difference between the images in the target grid.

[0139] For the candidate calibration images in the target grid, the shooting angle difference is calculated between any two candidate calibration images. When the number of the candidate calibration images in the target grid is less than or equal to the second number, all the candidate calibration images in the target grid are determined as the effective calibration images of the target grid, and the second number is less than the first number; when the number of the candidate calibration images in the target grid is greater than the second number, the second number of calibration images with the greatest shooting angle difference are determined from the target grid as the effective calibration images of the target grid.

[0140] In S207, the distances between the effective calibration images in the effective calibration image sequence and the calibration board are calculated, and the weights of the effective calibration images are determined according to the distances between the effective calibration images and the calibration board.

[0141] After processing all the target grids, all the effective calibration images obtained form an effective calibration image sequence. The distances between the effective calibration images and the calibration board are calculated according to the poses of the effective calibration images, and the weights of the effective calibration images are determined according to the distances between the effective calibration images and the calibration board. The greater the distance between the effective calibration image and the calibration board, the smaller the weight of the effective calibration image.

[0142] S208, according to the initial value of the camera internal parameter, and the weight and the camera pose of each effective calibration image, the camera internal parameter is optimized by using a weighted nonlinear optimization algorithm, to obtain the optimized value of the camera internal parameter.

[0143] In the embodiment, the weights of the effective calibration images are different, and therefore, in the nonlinear optimization process of the camera internal parameter, the accuracy of the optimization result of the camera internal parameter can be avoided due to overfitting of the camera pose.

[0144] In order to better implement the camera internal parameter calibration method of the embodiment of the application, the embodiment of the application further provides a camera internal parameter calibration device. Figure 7 The structure diagram of the camera internal parameter calibration device provided by the fourth embodiment of the application is shown in FIG. 3, which can include: Figure 7

[0145] The initial estimation module 31 is configured to acquire a calibration image sequence obtained by a camera shooting a calibration board, determine an initial value of a camera internal parameter and a camera pose of each calibration image in the calibration image sequence according to the calibration image sequence.

[0146] The screening module 32 is configured to screen the calibration image sequence according to at least one of the following parameters of each calibration image: camera pose and definition, to obtain an effective calibration image sequence.

[0147] The weight determination module 33 is configured to calculate the distance between each effective calibration image in the effective calibration image sequence and the calibration board, and determine the weight of each effective calibration image according to the distance between each effective calibration image and the calibration board.

[0148] The optimization module 34 is configured to optimize the camera internal parameter by using a weighted nonlinear optimization algorithm according to the initial value of the camera internal parameter, and the weight and the camera pose of each effective calibration image, to obtain the optimized value of the camera internal parameter.

[0149] In some exemplary modes, the screening module 32 is specifically configured to: establish a space grid with a fixed grid size under the calibration board coordinate system, project the camera pose of each calibration image into the space grid to obtain a grid where the camera pose of each calibration image is located; for each target grid where there is a calibration image in the space grid, screen the calibration images in the target grid according to at least one of the following parameters of the calibration images in the target grid: shooting angle difference value between images and definition, to obtain effective calibration images in the target grid, wherein the shooting angle difference value is determined according to the camera pose of two calibration images.

[0150] ​In some example manners, the screening module 32 is specifically configured to: when the number of the calibration images in the target grid is greater than a first number, screen out a first number of calibration images with the greatest clarity from the target grid as candidate calibration images of the target grid; when the number of the calibration images in the target grid is less than or equal to the first number, determine all the calibration images in the target grid as the candidate calibration images of the target grid; calculate a shooting angle difference between any two candidate calibration images in the target grid according to camera poses of the candidate calibration images of the target grid; when the number of the candidate calibration images in the target grid is less than or equal to a second number, determine all the candidate calibration images in the target grid as effective calibration images of the target grid, the second number being less than the first number; and when the number of the candidate calibration images in the target grid is greater than the second number, determine the second number of calibration images with the greatest shooting angle difference from the target grid as the effective calibration images of the target grid.

[0151] In some example manners, the device further includes a size determination module configured to acquire a size of the pattern in the calibration board, and determine a grid size according to the size of the pattern, wherein the size of the grid is greater than the size of the pattern.

[0152] In some example manners, the screening module 32 is specifically configured to: group the images in the sequence of calibration images according to a preset fixed shooting time length according to shooting times of the calibration images, to obtain a plurality of image groups; and for each image group, screen the calibration images in the image group according to at least one of the following parameters of the calibration images in the image group: a shooting angle difference between the images and a clarity, to obtain effective calibration images in the image group, wherein the shooting angle difference is determined according to camera poses of two calibration images.

[0153] In some example modes, the screening module 32 is specifically configured to: when the number of calibration images in the image group is greater than a third number, screen out the third number of calibration images with the largest clarity from the image group as the candidate calibration images of the image group; when the number of calibration images in the image group is less than or equal to the third number, determine all the calibration images in the image group as the candidate calibration images of the image group; calculate the shooting angle difference between any two candidate calibration images in the image group according to the poses of the candidate calibration images in the image group; when the number of candidate calibration images in the image group is less than or equal to a fourth number, determine all the candidate calibration images in the image group as the effective calibration images of the image group, the fourth number being less than the third number; when the number of candidate calibration images in the image group is greater than the fourth number, determine the fourth number of candidate calibration images with the largest shooting angle difference from the image group as the effective calibration images of the image group.

[0154] In some example modes, the calibration image sequence is obtained by moving the calibration board or the camera to shoot on at least one hemisphere.

[0155] In some example modes, the initial estimation module 31: performs feature point detection on each calibration image in the calibration image sequence to obtain the pixel coordinates and three-dimensional coordinates of the feature points of each calibration image; and determines the initial value of the camera intrinsic parameter and the camera pose of each calibration image using a pre-estimation algorithm according to the pixel coordinates and three-dimensional coordinates of the feature points of each calibration image.

[0156] In some example modes, the initial estimation module 31 is further configured to: delete a calibration image with a number of feature points less than a preset number from the calibration image sequence.

[0157] In some example modes, the clarity of the calibration image is the average gradient of the calibration image.

[0158] In some example modes, the screening module 32 is further configured to: for each feature point in the calibration image, determine a pixel block of a preset size centered on the feature point, calculate the average gradient of the pixels in the pixel block, and take the average gradient of the pixels in the pixel block as the gradient of the feature point; and calculate the average value of the gradients of all the feature points in the calibration image to obtain the average gradient of the calibration image.

[0159] In some example modes, the greater the distance between the effective calibration image and the calibration board, the smaller the weight of the effective calibration image.

[0160] In some example manners, the optimization module 34 is specifically configured to: project the three-dimensional coordinates of the feature points of each effective calibration image onto each effective calibration image according to the initial values of the camera pose of each effective calibration image and the camera intrinsic parameters, to obtain estimated values of pixel coordinates of the feature points of each effective calibration image; obtain projection errors of each effective calibration image according to the difference between the observed values and the estimated values of the pixel coordinates of the feature points of each effective calibration image; perform weighted operation on the projection errors of all effective calibration images according to the weights of the effective calibration images, to obtain a total projection error; and use a nonlinear optimization algorithm to optimize the camera intrinsic parameters with the minimum total projection error as an optimization target, to obtain optimized values of the camera intrinsic parameters.

[0161] It should be understood that the device embodiments and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, no further description is given here.

[0162] The device 300 of the embodiments of the present application is described above from the perspective of functional modules in combination with the drawings. It should be understood that the functional modules can be implemented in the form of hardware, or in the form of instructions of software, or in the form of a combination of hardware and software modules. Specifically, the steps of the method embodiments in the embodiments of the present application can be completed by integrated logic circuits of hardware in a processor and / or instructions of software, the steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processing performed by a processor, or executed by a combination of hardware and software modules in a code processing processor. Alternatively, the software modules can be located in mature storage media in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage media is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps in the above method embodiments.

[0163] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as processing circuit or memory) or a combination thereof. Similarly, one processor (or multiple processors or memory) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the functions of the module or unit.

[0164] The embodiments of the present application also provide a calibration device. Figure 8 A structural schematic diagram of the calibration device provided in the fifth embodiment of the present application is shown in FIG. 4, which can include: Figure 8

[0165] ​a memory 41 for storing a computer program and transmitting the program code to the processor 42. In other words, the processor 42 can call and run the computer program from the memory 41 to implement the method as follows: obtaining a sequence of calibration images captured by a camera on a calibration board, determining initial values of camera intrinsic parameters and camera poses of each calibration image in the sequence of calibration images according to the sequence of calibration images; screening the sequence of calibration images according to at least one of the following parameters of each calibration image: camera pose and definition, to obtain a sequence of effective calibration images; calculating distances between each effective calibration image and the calibration board, and determining weights of each effective calibration image according to the distances between each effective calibration image and the calibration board; and optimizing the camera intrinsic parameters using a weighted nonlinear optimization algorithm according to the initial values of the camera intrinsic parameters, the weights and the camera poses of each effective calibration image, to obtain optimized values of the camera intrinsic parameters.

[0166] For example, the processor 42 can be configured to execute all or part of the steps of the camera intrinsic parameter calibration method according to the instructions in the computer program.

[0167] In some embodiments of the present application, the processor 42 can include but is not limited to: a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0168] In some embodiments of the present application, the memory 41 includes, but is not limited to, a volatile memory and / or a non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0169] In some embodiments of the present application, the computer program can be divided into one or more modules, which are stored in the memory 41 and executed by the processor 42 to complete the method provided by the present application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the server.

[0170] As shown in Figure 8 The calibration device 400 can further include a transceiver 43, which can be connected to the processor 42 or the memory 41.

[0171] The processor 42 can control the transceiver 43 to communicate with other devices, specifically, can send information or data to other devices, or receive information or data sent by other devices. The transceiver 43 can include a transmitter and a receiver. The transceiver 43 can further include an antenna, and the number of antennas can be one or more.

[0172] In some embodiments of the present application, the calibration device 400 can further include a basic input / output system (Input / Output, I / O system) to help transmit information between various devices in the computer device, and a mass storage device for storing operating systems, application programs and other program modules.

[0173] In some embodiments, the basic input / output system includes a display for displaying information and an input device such as a mouse, keyboard, etc. for user input information. Wherein the display and input device are connected to the processor 42 through the input / output controller connected to the system bus. The basic input / output system can also include an input / output controller for receiving and processing input from a keyboard, mouse, or electronic stylus, and other devices. Similarly, the input / output controller also provides output to the display screen, printer or other types of output devices.

[0174] It can be understood that although Figure 8 The calibration device 400 can further include more devices, such as cameras, Bluetooth modules, etc., which are not shown in the calibration device 400 and will not be described here.

[0175] It should be understood that the various components in the calibration device 400 are connected through a bus system, wherein the bus system includes a data bus, a power bus, a control bus and a status signal bus in addition to the data bus.

[0176] The present application also provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the method of the above method embodiments. Alternatively, the present application embodiment also provides a computer program product containing instructions, which, when executed by a computer, causes the computer to perform the method of the above method embodiments.

[0177] The present application also provides a computer program product, which includes a computer program stored in a computer readable storage medium. The processor of the electronic device reads the computer program from the computer readable storage medium, and the processor executes the computer program to make the electronic device execute the corresponding processes in the above method embodiments, which will not be described here for brevity.

[0178] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the division of the above-described device embodiments is only a logical function division, and there can be another division manner for the actual implementation, for example, multiple devices or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different parts can be indirect couplings or communication connections through some interfaces, devices or modules, and can be electrical, mechanical or in other forms.

[0179] The modules explained as separated components can or can not be physically separated, and the components shown as modules can or can not be physical modules, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. For example, the functional modules in the embodiments of the present application can be integrated into a processing module, or each module can be physically present separately, or two or more modules can be integrated into one module.

[0180] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for calibrating camera intrinsic parameters, characterized in that, The method includes: Acquire a sequence of calibration images captured by the camera on the calibration board, and determine the initial values ​​of the camera intrinsic parameters and the camera pose of each calibration image in the calibration image sequence based on the calibration image sequence; The calibration image sequence is filtered based on at least one of the following parameters of each calibration image: camera pose and sharpness, to obtain a valid calibration image sequence; Calculate the distance between each valid calibration image in the valid calibration image sequence and the calibration board, and determine the weight of each valid calibration image based on the distance between each valid calibration image and the calibration board; Based on the initial values ​​of the camera intrinsic parameters, the weights of each valid calibration image, and the camera pose, a weighted nonlinear optimization algorithm is used to optimize the camera intrinsic parameters to obtain optimized values.

2. The method according to claim 1, characterized in that, The step of filtering the calibration image sequence based on at least one of the following parameters of each calibration image to obtain a valid calibration image sequence includes: In the calibration plate coordinate system, a spatial grid is established with a fixed grid size, and the camera pose of each calibration image is projected into the spatial grid to obtain the grid where the camera pose of each calibration image is located. For each target grid containing a calibration image in the spatial grid, the calibration images in the target grid are filtered according to at least one of the following parameters of the calibration images in the target grid: the shooting angle difference between the images and the sharpness, to obtain the valid calibration images in the target grid, wherein the shooting angle difference is determined based on the camera pose of the two calibration images.

3. The method according to claim 2, characterized in that, The step of filtering the calibration images within the target grid based on at least one of the following parameters: the difference in shooting angle between images and sharpness, to obtain valid calibration images within the target grid, includes: When the number of calibration images in the target grid is greater than a first number, the first number of calibration images with the highest clarity are selected from the target grid as candidate calibration images for the target grid. When the number of calibration images within the target grid is less than or equal to the first number, all calibration images within the target grid are determined as candidate calibration images for the target grid. Based on the camera pose of the candidate calibration images of the target grid, calculate the shooting angle difference between any two candidate calibration images within the target grid; When the number of candidate calibration images within the target grid is less than or equal to the second number, all candidate calibration images within the target grid are determined as valid calibration images of the target grid, wherein the second number is less than the first number; When the number of candidate calibration images in the target grid is greater than the second number, the second number of calibration images with the largest shooting angle difference in the target grid are determined as the valid calibration images of the target grid.

4. The method according to claim 2, characterized in that, The method further includes: Obtain the size of the pattern in the calibration plate, and determine the grid size based on the size of the pattern, wherein the size of the grid is larger than the size of the pattern.

5. The method according to claim 1, characterized in that, The step of filtering the calibration image sequence based on at least one of the following parameters of each calibration image to obtain a valid calibration image sequence includes: Based on the shooting time of each calibration image, the images in the calibration image sequence are grouped according to a preset fixed shooting duration to obtain multiple image groups; For each image group, the calibration images within the image group are filtered based on at least one of the following parameters: the difference in shooting angle between the images and the sharpness, to obtain the valid calibration images within the image group, wherein the difference in shooting angle is determined based on the camera pose of the two calibration images.

6. The method according to claim 5, characterized in that, The step of filtering the calibration images within the image group based on at least one of the following parameters: the difference in shooting angle between images and the sharpness, to obtain valid calibration images within the image group, includes: When the number of calibration images in the image group is greater than the third number, the calibration image with the highest clarity in the third number is selected from the image group as the candidate calibration image of the image group; When the number of calibration images in the image group is less than or equal to the third number, all calibration images in the image group are determined as candidate calibration images of the image group; Based on the pose of the candidate calibration images within the image group, calculate the shooting angle difference between any two candidate calibration images within the image group; When the number of candidate calibration images in the image group is less than or equal to the fourth number, all candidate calibration images in the image group are determined as valid calibration images of the image group, and the fourth number is less than the third number; When the number of candidate calibration images in the image group is greater than the fourth number, the candidate calibration image with the largest shooting angle difference in the fourth number is determined as the valid calibration image of the image group.

7. The method according to claim 1, characterized in that, The calibration image sequence is obtained by moving the calibration plate or the camera on at least one hemisphere.

8. The method according to claim 1, characterized in that, The step of determining the initial values ​​of camera intrinsic parameters and the camera pose of each calibration image in the calibration image sequence based on the calibration image sequence includes: Feature point detection is performed on each calibration image in the calibration image sequence to obtain the pixel coordinates and three-dimensional coordinates of the feature points of each calibration image; Based on the pixel coordinates and 3D coordinates of the feature points of each calibration image, the initial values ​​of the camera intrinsic parameters and the camera pose of each calibration image are determined using a pre-estimation method.

9. The method according to claim 8, characterized in that, Before determining the initial values ​​of the camera intrinsic parameters and the camera pose of each calibration image using a pre-estimation method based on the pixel coordinates and three-dimensional coordinates of the feature points of each calibration image, the method further includes: Delete calibration images from the calibration image sequence that have fewer than a preset number of feature points.

10. The method according to any one of claims 1-9, characterized in that, The sharpness of the calibration image is the average gradient of the calibration image.

11. The method according to claim 10, further comprising: For each feature point in the calibration image, a pixel block of a preset size is determined with the feature point as the center. The average gradient of the pixels in the pixel block is calculated, and the average gradient of the pixels in the pixel block is used as the gradient of the feature point. The average gradient of the calibration image is obtained by calculating the average gradient of all feature points in the calibration image.

12. The method according to any one of claims 1-9, characterized in that, The greater the distance between the effective calibration image and the calibration plate, the smaller the weight of the effective calibration image.

13. The method according to any one of claims 1-9, characterized in that, The step of optimizing the camera intrinsics using a weighted nonlinear optimization algorithm based on the initial values ​​of the camera intrinsics, the weights of each valid calibration image, and the camera pose, to obtain optimized values ​​for the camera intrinsics, includes: Based on the camera pose of each valid calibration image and the initial values ​​of the camera intrinsic parameters, the three-dimensional coordinates of the feature points of each valid calibration image are projected onto each valid calibration image to obtain the estimated values ​​of the pixel coordinates of the feature points of each valid calibration image. The projection error of each valid calibration image is obtained by the difference between the observed and estimated values ​​of the pixel coordinates of the feature points of each valid calibration image. Based on the weights of each valid calibration image, the projection errors of all valid calibration images are weighted and calculated to obtain the total projection error; With the goal of minimizing the total projection error, a nonlinear optimization algorithm is used to optimize the camera intrinsic parameters to obtain optimized values ​​for the camera intrinsic parameters.

14. A calibration device for camera intrinsic parameters, characterized in that, include: The initial estimation module is used to acquire a sequence of calibration images captured by the camera on the calibration board, and to determine the initial values ​​of the camera intrinsic parameters and the camera pose of each calibration image in the calibration image sequence based on the calibration image sequence. A filtering module is used to filter the calibration image sequence based on at least one of the following parameters of each calibration image: camera pose and sharpness, to obtain a valid calibration image sequence; The weight determination module is used to calculate the distance between each valid calibration image in the valid calibration image sequence and the calibration board, and to determine the weight of each valid calibration image based on the distance between each valid calibration image and the calibration board. The optimization module is used to optimize the camera intrinsic parameters using a weighted nonlinear optimization algorithm based on the initial values ​​of the camera intrinsic parameters, the weights of each valid calibration image, and the camera pose, to obtain the optimized values ​​of the camera intrinsic parameters.

15. An electronic device, characterized in that, include: A processor and a memory, the memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the method of any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 13.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 13.

Images