Calibration method, image correction method, device, electronic equipment and storage medium

Abstract

The application provides a calibration method, an image correction method, a device, electronic equipment and a storage medium. The calibration method comprises: acquiring a calibration board image; determining the extrinsic parameter of the target camera according to the calibration board image, and determining an angle θ according to the extrinsic parameter of the target camera; updating the rotation matrix in the extrinsic parameter of the target camera according to the angle θ, and obtaining an updated extrinsic parameter; mapping the specified feature points of the calibration board to a pixel plane according to the updated extrinsic parameter, and obtaining corrected specified feature point pixel coordinates; and determining the homography matrix corresponding to the target camera according to the original specified feature point pixel coordinates and the corrected specified feature point pixel coordinates, wherein the homography matrix is used for image correction of the image collected by the target camera. The method can improve the accuracy of image correction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a calibration method, an image correction method, a device, an electronic device and a storage medium. BACKGROUND

[0002] Generally, the imaging result of a visual camera on a three-dimensional space, in the pixel coordinate system of an image, a straight line on a certain target plane in the three-dimensional space does not maintain the original angle relationship in the two-dimensional image, that is, a special vertical or parallel straight line does not remain unchanged in the image.

[0003] In some operations, it is necessary to calculate the angle relationship of some straight lines in the image in the three-dimensional space, and in the actual operation of the camera, it cannot be absolutely guaranteed that the z-axis of the camera is parallel to the normal direction of the target object, and it is necessary to artificially restore the angle relationship of the straight lines on the corresponding three-dimensional target plane in the image. SUMMARY

[0004] Therefore, the present application provides a calibration method, an image correction method, a device, an electronic device and a storage medium.

[0005] According to a first aspect of an embodiment of the present application, a calibration method is provided, comprising:

[0006] acquiring a calibration board image; wherein the calibration board image is collected by a target camera under the condition that the calibration board is parallel to a target plane, the target plane is a correction plane of a target object, and the target object is a shooting object of the target camera;

[0007] determining an extrinsic parameter of the target camera according to the calibration board image, and determining an angle θ according to the extrinsic parameter of the target camera; the angle θ is a rotation angle of a camera coordinate system of the target camera relative to a z-axis of a calibration board coordinate system;

[0008] updating a rotation matrix in the extrinsic parameter of the target camera according to the angle θ, to obtain an updated extrinsic parameter;

[0009] mapping a specified feature point of the calibration board to a pixel plane according to the updated extrinsic parameter, to obtain a corrected specified feature point pixel coordinate;

[0010] determining a homography matrix corresponding to the target camera according to an original specified feature point pixel coordinate and the corrected specified feature point pixel coordinate; wherein the original specified feature point pixel coordinate is obtained by performing specified feature point identification on the calibration board image, and the homography matrix is used for image correction on an image collected by the target camera.

[0011] According to a second aspect of an embodiment of the present application, an image correction method is provided, comprising:

[0012] obtain an image to be corrected captured by a target camera;

[0013] perform image correction on the image to be corrected by using a homography matrix corresponding to the target camera, wherein the homography matrix corresponding to the target camera is obtained by using the method provided in the first aspect.

[0014] According to a third aspect of the embodiment of the present application, a calibration device is provided, comprising an image acquisition module and a processor, wherein:

[0015] The image acquisition module is configured to acquire a calibration board image, wherein the calibration board image is captured by a target camera when the calibration board is parallel to a target plane, the target plane is a correction plane of a target object, and the target object is a shooting object of the target camera.

[0016] The processor is configured to determine an extrinsic parameter of the target camera according to the calibration board image, and determine an angle θ according to the extrinsic parameter of the target camera, wherein the angle θ is a rotation angle of a camera coordinate system of the target camera relative to a z-axis of a calibration board coordinate system.

[0017] The processor is further configured to update a rotation matrix in the extrinsic parameter of the target camera according to the angle θ, to obtain an updated extrinsic parameter.

[0018] The processor is further configured to map a specified feature point of the calibration board to a pixel plane according to the updated extrinsic parameter, to obtain a corrected specified feature point pixel coordinate.

[0019] The processor is further configured to determine a homography matrix corresponding to the target camera according to an original specified feature point pixel coordinate and the corrected specified feature point pixel coordinate, wherein the original specified feature point pixel coordinate is obtained by performing specified feature point identification on the calibration board image, and the homography matrix is used for performing image correction on an image captured by the target camera.

[0020] According to a fourth aspect of the embodiment of the present application, an image correction device is provided, comprising an image acquisition module and a processor, wherein:

[0021] The image acquisition module is configured to acquire an image to be corrected captured by a target camera.

[0022] The processor is configured to perform image correction on the image to be corrected by using a homography matrix corresponding to the target camera, wherein the homography matrix corresponding to the target camera is obtained by using the method provided in the first aspect.

[0023] According to a fifth aspect of the embodiments of the present application, an electronic device is provided, comprising a processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions which can be executed by the processor, and the processor is configured to execute the machine executable instructions to implement the method provided in the first aspect or the second aspect.

[0024] According to a sixth aspect of the embodiments of the present application, a storage medium is provided, the storage medium stores machine executable instructions, and the machine executable instructions are executed by a processor to implement the method provided in the first aspect.

[0025] The calibration method provided in the embodiments of the present application determines the transformation matrix of the target camera in the calibration board coordinate system (i.e. the extrinsic parameter of the target camera in the case of taking the calibration board coordinate system as the world coordinate system), and determines the angle θ according to the determined extrinsic parameter, updates the rotation matrix in the extrinsic parameter of the target camera according to the angle θ, obtains the updated extrinsic parameter, removes the component in the original extrinsic parameter that affects the perpendicularity between the calibration board and the imaging plane, and then projects the calibration board into the image according to the updated extrinsic parameter to obtain the corrected pixel coordinates of the specified feature points, and determines the homography matrix of the target camera according to the original pixel coordinates of the specified feature points and the corrected pixel coordinates of the specified feature points. With the precision of the calibration board, the homography matrices before and after correction are obtained through the distance information in the three-dimensional space. In the subsequent image correction process, the homography matrix can be used to correct the image collected by the target camera, and the calculation accuracy is high and the calculation process is simple and easy to operate. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a flowchart of a calibration method provided in the embodiments of the present application;

[0027] Figure 2 is a flowchart of an image correction method provided in the embodiments of the present application;

[0028] Figure 3A is a schematic diagram of an image before correction provided in the embodiments of the present application;

[0029] Figure 3B is a schematic diagram of an image after correction provided in the embodiments of the present application; Figure 3A is a schematic diagram of a correction effect corresponding to the image shown in the embodiments of the present application;

[0030] Figure 3C is a schematic diagram of another correction effect corresponding to the image shown in the embodiments of the present application; Figure 3A

[0031] Figure 4A and Figure 4B are schematic diagrams of two image correction effects provided in the embodiments of the present application;

[0032] ​Figure 5 is a flowchart of a calibration device provided by an embodiment of the present application;

[0033] Figure 6 is a flowchart of an image correction device provided by an embodiment of the present application;

[0034] Figure 7 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0035] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, unless otherwise indicated, like numbers refer to like elements throughout the several drawings. The following description of exemplary embodiments is not meant to represent all embodiments in accordance with the present application. Rather, they are merely examples in accordance with some aspects of the present application as detailed in the appended claims.

[0036] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0037] In order for those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, some technical terms in the embodiments of the present application will be briefly described first.

[0038] 1. Calibration board coordinate system: taking the first designated feature point on the calibration board as the coordinate origin, taking the direction perpendicular to the calibration board outward as the z-axis, and giving the long and short sides of the calibration board as the x and y axes based on the right-hand rule, the coordinate system thus formed is the calibration board coordinate system.

[0039] Taking a checkerboard as an example, taking the first corner point on the checkerboard as the coordinate origin, taking the direction perpendicular to the calibration board outward as the z-axis, and giving the long and short sides of the checkerboard as the x and y axes based on the right-hand rule, the coordinate system thus formed is the calibration board coordinate system.

[0040] 2. Camera coordinate system: the camera coordinate system refers to a coordinate system taking the camera optical center as the coordinate origin, the axis perpendicular to the image plane passing through the optical center as the z-axis, the axis parallel to and in the same direction as the long side of the image passing through the optical center as the x-axis, and the axis parallel to and in the same direction as the short side of the image passing through the optical center as the y-axis.

[0041] 3. Camera pose: the transformation matrix from the world coordinate system to the camera coordinate system, that is, the camera extrinsic parameter.

[0042] 4, Camera intrinsic: the transformation matrix from 3D points in camera coordinate system to image plane points.

[0043] 5, PNP (Perspective-n-Point) algorithm: the projection transformation matrix from 3D points to 2D normalized pixel plane.

[0044] 6, Projection rectification: rectify the non-parallel lines in image (which are parallel in 3D space) to parallel lines.

[0045] 7, Affine rectification: rectify the vertical line relationship in image, on the basis of projection rectification.

[0046] In order to make the above-mentioned purposes, features and advantages of the embodiments of the present application more apparent and easy to understand, the technical solutions in the embodiments of the present application will be further described in detail below with reference to the drawings.

[0047] Please refer to Figure 1 , a flowchart of a calibration method provided by the embodiments of the present application is shown, which can include the following steps: Figure 1

[0048] It should be noted that the sequence numbers of the steps in the embodiments of the present application do not mean the order of execution, the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0049] Step S100, acquiring a calibration board image; wherein the calibration board image is acquired by a target camera in the case that the calibration board is parallel to a target plane, the target plane is a rectification plane of a target object, and the target object is a shooting object of the target camera.

[0050] In the embodiments of the present application, for any camera (referred to as target camera in this paper) which needs to perform image rectification, after the target camera is installed, the calibration board can be placed in the acquisition range of the target camera, and the calibration board is placed parallel to the target plane, and the calibration board image is acquired by the target camera.

[0051] For example, the target plane is the rectification plane of the target object, and the target object is the shooting object of the target camera.

[0052] For example, taking an indoor monitoring scene as an example, the target plane can be a ceiling or a floor, i.e. the calibration board is placed parallel to the ceiling or the floor.

[0053] ​It should be noted that, in actual scenarios, due to the placement position of the calibration board and the hardware conditions of the reference plane (such as the target plane), it is possible that the calibration board and the target plane are not absolutely parallel. Therefore, in the embodiments of the present application, the parallelism between the calibration board and the target plane can allow an angle error that can be tolerated.

[0054] In step S110, the extrinsic parameter of the target camera is determined according to the calibration board image, and the angle θ is determined according to the extrinsic parameter of the target camera. The angle θ is the rotation angle of the camera coordinate system of the target camera relative to the z-axis of the calibration board coordinate system.

[0055] In the embodiments of the present application, after the calibration board image is obtained in the manner described in step S100, the extrinsic parameter of the target camera can be determined according to the obtained calibration board image.

[0056] For example, the extrinsic parameter of the target camera can be determined by using the PNP algorithm according to the obtained calibration board image.

[0057] The extrinsic parameter of the target camera can be a transformation matrix from the world coordinate system to the camera coordinate system of the target camera.

[0058] For example, in the embodiments of the present application, the calibration board coordinate system is used as the world coordinate system, and the extrinsic parameter of the target camera can be a transformation matrix from the calibration board coordinate system to the camera coordinate system of the target camera.

[0059] It should be noted that, in the following description, the camera coordinate system refers to the camera coordinate system of the target camera unless otherwise specified.

[0060] In one example, in order to improve the accuracy of calibration, for the obtained calibration board image, the calibration board image can be de-distorted according to the intrinsic parameter of the target camera to obtain a processed calibration board image.

[0061] In the embodiments of the present application, in order to make the corrected image meet the condition that the z-axis of the camera (i.e., the z-axis of the camera coordinate system) is parallel to the normal direction of the target object (i.e., the z-axis of the calibration board coordinate system) in the calibration process (i.e., the calibration board), it is necessary to first determine the included angle between the xy plane of the camera coordinate system and the xy plane of the calibration board coordinate system, so as to correct the image of the target camera according to the included angle.

[0062] In addition, in order to ensure the position relationship of the target plane in the three-dimensional space, it is also necessary to ensure that the rotation angle of the target plane around the z-axis in the image before and after correction is consistent.

[0063] Correspondingly, after the extrinsic parameter of the target camera is determined, the rotation angle (which can be denoted as angle θ) of the camera coordinate system of the target camera relative to the z-axis of the calibration board coordinate system can be determined according to the extrinsic parameter of the target camera.

[0064] For example, the rotation angle of the camera coordinate system of the target camera relative to the z-axis of the calibration board coordinate system can be the yaw angle in the Euler angle of the transformation matrix from the calibration board coordinate system to the camera coordinate system.

[0065] In step S120, the rotation matrix in the extrinsic parameter of the target camera is updated according to the angle θ, and an updated extrinsic parameter is obtained.

[0066] In the embodiment of the present application, in order to make the corrected image meet the parallelism between the z-axis of the camera (i.e. the z-axis of the camera coordinate system) and the normal direction of the target object (i.e. the z-axis of the calibration board coordinate system) in the calibration process (i.e. the calibration board), and ensure the consistency of the rotation angle of the target plane around the z-axis in the image before and after the correction, in the case of obtaining the angle θ in the manner described in step S110, the rotation matrix in the extrinsic parameter of the target camera can be updated according to the angle θ determined in step S110, and an updated extrinsic parameter is obtained, so as to perform coordinate transformation between the calibration board coordinate system and the camera coordinate system according to the updated extrinsic parameter.

[0067] In the embodiment of the present application, in order to make the corrected image meet the parallelism between the z-axis of the camera (i.e. the z-axis of the camera coordinate system) and the normal direction of the target object (i.e. the z-axis of the calibration board coordinate system) in the calibration process (i.e. the calibration board), and ensure the consistency of the rotation angle of the target plane around the z-axis in the image before and after the correction, in the case of obtaining the angle θ in the manner described in step S110, the rotation matrix in the extrinsic parameter of the target camera can be updated according to the angle θ determined in step S110, and an updated extrinsic parameter is obtained, so as to perform coordinate transformation between the calibration board coordinate system and the camera coordinate system according to the updated extrinsic parameter.

[0068] A new rotation matrix can be determined according to the angle θ, and the rotation matrix is replaced by the rotation matrix in the extrinsic parameter of the target camera, and an updated extrinsic parameter of the target camera is obtained.

[0069] In step S130, the specified feature points of the calibration board are mapped to the pixel plane according to the updated extrinsic parameter, and the corrected specified feature point pixel coordinates are obtained.

[0070] In step S140, a homography matrix corresponding to the target camera is determined according to the original specified feature point pixel coordinates and the corrected specified feature point pixel coordinates; wherein the original specified feature point pixel coordinates are obtained by identifying the specified feature points of the calibration board image, and the homography matrix is used for image correction of the image collected by the target camera.

[0071] In the embodiment of the present application, in the case of determining the updated extrinsic parameter of the target camera, two groups of specified feature point pixel coordinates can be determined.

[0072] One group of pixel specified feature point coordinates can be obtained by mapping the specified feature points of the calibration board to the pixel plane according to the updated extrinsic parameter (which can be referred to as the corrected specified feature point pixel coordinates).

[0073] The other group of pixel specified feature point coordinates can be obtained by identifying the specified feature points of the calibration board image (which can be referred to as the original specified feature point pixel coordinates).

[0074] Based on the original specified feature point pixel coordinates and the corrected specified feature point pixel coordinates, the transformation relationship between these two sets of specified feature point pixel coordinates can be determined to obtain the homography matrix corresponding to the target camera, so as to perform image correction on the image acquired by the target camera based on the homography matrix.

[0075] For example, when the calibration board is a checkerboard, the specified feature point can be a corner point of the checkerboard; when the calibration board is an apirltag calibration board, the specified feature point can be a vertex of a QR code quadrilateral; when the calibration board is a dot calibration board, the specified feature point can be the center of the dot calibration board.

[0076] For ease of description and understanding, the following text will use the specified feature points as the corner points of the chessboard (hereinafter referred to as corner points) as an example.

[0077] It can be seen that, in Figure 1 In the method described, the transformation matrix of the target camera in the calibration plate coordinate system (i.e., the extrinsic parameters of the target camera when the calibration plate coordinate system is the world coordinate system) is determined, and the angle θ is determined based on the determined extrinsic parameters. The rotation matrix in the extrinsic parameters of the target camera is updated based on the angle θ to obtain the updated extrinsic parameters. The components in the original extrinsic parameters that affect the perpendicularity between the calibration plate and the imaging plane are removed. Then, the calibration plate is projected onto the image based on the updated extrinsic parameters to obtain the corrected pixel coordinates of the specified feature points. Based on the original and corrected pixel coordinates of the specified feature points, the homography matrix of the target camera is determined. With the help of the precision of the calibration plate, the homography matrix before and after correction is obtained through the distance information in three-dimensional space. In the subsequent image correction process, the correction of the image acquired by the target camera can be achieved based on the homography matrix. The calculation accuracy is high, and the calculation process is simple and easy to operate.

[0078] In some embodiments, determining the angle θ based on the extrinsic parameters of the target camera may include:

[0079] Based on the extrinsic parameters of the target camera, the angle θ is determined using the following formula:

[0080]

[0081] Where PR(1) represents the projection vector of R(1) onto the xy plane of the camera coordinate system, and R(1) is the first column of the rotation matrix in the extrinsic parameters of the target camera. This represents the x-axis direction vector of the camera coordinate system.

[0082] For example, in the camera's extrinsic parameters, the geometric meaning of the first column (which can be denoted as R(1)) is the coordinate of the point of disappearance in the x-axis direction of the world coordinate system in the camera coordinate system.

[0083] The vanishing point of a world straight line is obtained by the intersection of a ray parallel to the straight line and passing through the camera center and the image plane.

[0084] The angle θ can be characterized as the angle between the projection vector of R(1) on the xy plane of the camera coordinate system and the x axis of the camera coordinate system.

[0085] PR(1) represents the projection vector of R(1) on the xy plane of the camera coordinate system, i.e., the z coordinate of the vector is 0.

[0086] R(1) is the first column of the rotation matrix in the extrinsic parameters of the target camera, which is a unit vector.

[0087] Therefore, according to the above formula, θ = R(1, 1), which is the element in the first row and the first column of the extrinsic parameters of the target camera.

[0088] Exemplarily, since PR(1) can belong to one of the four quadrants on the xy plane, under the condition that the positive direction of the angle is clockwise, the positive and negative signs can be determined by the following formula:

[0089]

[0090] wherein R(1) y represents the corresponding y-axis coordinate of the vector R(1).

[0091] In some embodiments, updating the rotation matrix in the extrinsic parameters of the target camera according to the angle θ to obtain the updated extrinsic parameters can include:

[0092] According to the angle θ, a rotation vector [0, 0, signθ] is obtained; wherein sign is the positive and negative of θ;

[0093] The rotation vector is converted into a target rotation matrix, and the rotation matrix in the extrinsic parameters of the target camera is replaced with the target rotation matrix to obtain the updated extrinsic parameters.

[0094] Exemplarily, a rotation vector [0, 0, signθ] can be generated according to the determined angle θ, and the rotation vector is converted into a rotation matrix (referred to as a target rotation matrix herein).

[0095] In the case where the target rotation matrix is obtained, the rotation matrix in the extrinsic parameters of the target camera can be replaced with the target rotation matrix to obtain the updated extrinsic parameters of the target camera.

[0096] In some embodiments, the above mapping of the specified feature points of the calibration board to the pixel plane to obtain the corrected specified feature point pixel coordinates according to the updated extrinsic parameters can include:

[0097] For any corner point of the calibration board, the three-dimensional coordinates of the corner point are mapped to the camera coordinate system of the target camera according to the updated extrinsic parameter to determine the camera coordinates of the specified feature point;

[0098] According to the intrinsic parameter of the target camera, the camera coordinates of the specified feature point are mapped to the pixel coordinate system to obtain the corrected pixel coordinates of the specified feature point.

[0099] For example, in the case where the updated extrinsic parameter of the target camera is determined in the manner described in the above embodiment, for any corner point of the calibration board, the three-dimensional coordinates (coordinates in the calibration board coordinate system) of the corner point can be mapped to the camera coordinate system of the target camera according to the updated extrinsic parameter to determine the camera coordinates of the corner point, and then, according to the intrinsic parameter of the target camera, the camera coordinates of the corner point are mapped to the pixel coordinate system to obtain the pixel coordinates of the corner point (i.e. the above-mentioned corrected corner pixel coordinates).

[0100] Please refer to Figure 2 A flowchart of an image correction method provided by an embodiment of the present application is shown in Figure 2 The image correction method can include the following steps:

[0101] Step S200: acquiring a to-be-corrected image captured by a target camera.

[0102] Step S210: performing image correction on the to-be-corrected image by using a homography matrix corresponding to the target camera.

[0103] In the embodiment of the present application, in the case where the homography matrix of the target camera is determined in the manner shown in Figure 1 The to-be-corrected image captured by the target camera can be corrected by using the homography matrix of the target camera.

[0104] For example, the to-be-corrected image captured by the target camera can include but is not limited to a real-time image captured by the target camera, or an image captured by the target camera and stored in a storage device.

[0105] It should be noted that, considering that the homography matrix corresponding to different cameras is usually different, the homography matrix corresponding to different cameras can be determined and stored in advance, for example, the correspondence between the camera identifier (such as the camera ID number) and the corresponding homography matrix is stored, and for any to-be-corrected image of a camera, the corresponding homography matrix can be determined according to the identifier of the camera, and the to-be-corrected image is corrected according to the homography matrix.

[0106] In order for those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application will be described in conjunction with specific application scenarios.

[0107] In this embodiment, in the case of image rectification of the target camera, the viewing angle is converted to observing the target plane from the front of the target plane, and it is ensured that the rotation angle of the target plane around the z axis is consistent with that of the original image in the case of observing the target plane directly. In this case, the xy plane of the camera coordinate system and the image plane are parallel to the target plane, so the x axis or y axis of the camera coordinate system, that is, the x axis or y axis of the pixel plane, can be used as the reference straight line of the image plane.

[0108] As Figures 3A-3C , Figure 3B and Figure 3C are images after image rectification of Figure 3A , but Figure 3B has a certain rotation of the viewing angle relative to Figure 3C , while Figure 3C and Figure 3A maintain the same z direction rotation.

[0109] The image rectification process of the embodiment of the present application is described below.

[0110] I. Determination of the included angle θ

[0111] Place the calibration board parallel to the target plane (in the example herein, make the calibration board parallel to the indoor ceiling), and collect the calibration board image. Perform distortion removal processing on the calibration board image using the intrinsic parameters (previously calibrated) of the target camera to obtain the processed calibration board image.

[0112] According to the processed calibration board image, the extrinsic parameters T cw of the target camera are calculated using the PNP algorithm (the extrinsic parameters are the transformation matrix between the calibration board coordinate system and the camera coordinate system).

[0113] R(1) represents the first column of T cw , and in the extrinsic parameters of the camera, the geometric meaning of the first column is the coordinates of the vanishing point in the x axis direction of the world coordinate system in the camera coordinate system.

[0114] Based on R(1), the angle θ can be obtained by the following processing:

[0115] First, calculate the included angle between the projection vector of R(1) on the xy plane of the camera and the x axis:

[0116]

[0117] where PR(1) represents the projection vector of R(1) on the xy plane of the camera coordinate system, represents the x axis direction vector of the camera coordinate system, and R(1) is a unit vector, so the result of the calculation is R(1, 1), that is, the element of the first row and the first column of T cw .

[0118] Secondly, PR(1) can belong to one of the four quadrants on the xy plane, and its positive or negative sign can be determined by the following formula under the condition that the positive direction of the specified angle is clockwise:

[0119]

[0120] wherein R(1) y represents the corresponding y-axis coordinate of the vector R(1).

[0121] II. Determination of the homography matrix corresponding to the target camera

[0122] The homography matrix is essentially a two-dimensional point to two-dimensional point transformation, and the calculation of the matrix requires the corresponding relationship of two sets of two-dimensional points.

[0123] The first set is the corresponding point of the corner point on the image of the calibration board, which can be obtained by the corner point recognition method of the chessboard (taking the calibration board as an example).

[0124] The second set of two-dimensional points is obtained after correcting the first set of two-dimensional points, which can be determined by the following method:

[0125] First, the corresponding rotation vector [0, 0, signθ] is obtained according to the determined angle θ (plus or minus sign is signθ), and the rotation vector is converted into a rotation matrix (i.e. the above target rotation matrix) R = Exp([0, 0, signθ]).

[0126] wherein the function Exp represents the transformation function of converting the rotation vector into the rotation matrix.

[0127] Then, the target rotation matrix is used to replace the rotation matrix in T cw to obtain the updated external parameter.

[0128] Finally, according to the updated external parameter, the three-dimensional coordinates of the corner points of the calibration board are projected to the pixel plane to obtain the pixel coordinates of the second set of two-dimensional points (i.e. the corrected corner point pixel coordinates mentioned above).

[0129] According to the pixel coordinates of the above two sets of two-dimensional points, the homography matrix corresponding to the target camera can be calculated.

[0130] III. Image correction

[0131] For the image collected by the target camera, image correction can be performed according to the homography matrix corresponding to the target camera.

[0132] wherein the two sets of correction result schematic diagrams can be respectively as Figure 4A and Figure 4BThe image is shown (the left side is the original image, and the right side is the corrected image).

[0133] The method provided in the present application is described above. The device provided in the present application is described below:

[0134] See Figure 5 The structure diagram of a calibration device provided in an embodiment of the present application is shown in the figure, and the calibration device can include an image acquisition module 510 and a processor 520; wherein: Figure 5

[0135] The image acquisition module 510 is configured to acquire a calibration board image; wherein the calibration board image is acquired by a target camera in the case that the calibration board is parallel to a target plane, the target plane is a correction plane of a target object, and the target object is a shooting object of the target camera;

[0136] The processor 520 is configured to determine an extrinsic parameter of the target camera according to the calibration board image, and determine an angle θ according to the extrinsic parameter of the target camera; the angle θ is a rotation angle of a camera coordinate system of the target camera relative to a z-axis of a calibration board coordinate system;

[0137] The processor 520 is further configured to update a rotation matrix in the extrinsic parameter of the target camera according to the angle θ, to obtain an updated extrinsic parameter;

[0138] The processor 520 is further configured to map a specified feature point of the calibration board to a pixel plane according to the updated extrinsic parameter, to obtain a corrected specified feature point pixel coordinate;

[0139] The processor 520 is further configured to determine a homography matrix corresponding to the target camera according to an original specified feature point pixel coordinate and the corrected specified feature point pixel coordinate; wherein the original specified feature point pixel coordinate is obtained by performing specified feature point identification on the calibration board image, and the homography matrix is used for image correction on an image acquired by the target camera.

[0140] For example, in the case that the calibration device is deployed in a front-end image acquisition device (such as a camera), the image acquisition module 510 can be an image acquisition module, such as a lens module (including a lens and an image sensor, etc.); in the case that the calibration device is deployed in a back-end image processing device, the image acquisition module 510 can be a communication module in communication with the front-end image acquisition device, and the communication module can acquire image data from the front-end image acquisition device.

[0141] In some embodiments, the processor 520 determines the angle θ according to the extrinsic parameter of the target camera, including:

[0142] ​According to the extrinsic parameter of the target camera, the angle θ is determined by the following formula:

[0143]

[0144] wherein PR(1) represents a projection vector of R(1) on the xy plane of the camera coordinate system, R(1) is the first column of a rotation matrix in the extrinsic parameter of the target camera, represents a direction vector of the x axis of the camera coordinate system;

[0145] wherein the positive and negative sign of θ is determined by the following formula:

[0146]

[0147] wherein R(1) y represents the corresponding y axis coordinate of the vector R(1), and the clockwise direction is the positive direction of the angle.

[0148] In some embodiments, the processor updates the rotation matrix in the extrinsic parameter of the target camera according to the angle θ, to obtain an updated extrinsic parameter, including:

[0149] According to the angle θ, a rotation vector [0, 0, signθ] is obtained; wherein θ is the rotation angle, and sign is the positive and negative of θ;

[0150] The rotation vector is converted into a target rotation matrix, and the target rotation matrix is used to replace the rotation matrix in the extrinsic parameter of the target camera to obtain an updated extrinsic parameter.

[0151] In some embodiments, the processor 520 maps the specified feature points of the calibration board to the pixel plane according to the updated extrinsic parameter, to obtain corrected specified feature point pixel coordinates, including:

[0152] For any specified feature point of the calibration board, the three-dimensional coordinates of the specified feature point are mapped to the camera coordinate system of the target camera according to the updated extrinsic parameter, to determine the camera coordinates of the specified feature point;

[0153] According to the intrinsic parameter of the target camera, the camera coordinates of the specified feature point are mapped to the pixel coordinate system to obtain the corrected pixel coordinates of the specified feature point.

[0154] Please refer to Figure 6 , a structural schematic diagram of an image correction device provided by the embodiments of the present application, as Figure 6 shown, the image correction device can include an image acquisition module 610 and a processor 620; wherein:

[0155] The image acquisition module 610 is configured to acquire a to-be-corrected image captured by a target camera.

[0156] The processor 620 is configured to perform image correction on the to-be-corrected image by using a homography matrix corresponding to the target camera.

[0157] For example, the homography matrix corresponding to the target camera is obtained by using the calibration method described in the method flow. Figure 1 The calibration method described in the method flow is obtained.

[0158] For example, in the case where the image correction device is deployed in a front-end image acquisition device (such as a camera), the image acquisition module 610 can be an image acquisition module, such as a lens module (including a lens and an image sensor, etc.); in the case where the image correction device is deployed in a back-end image processing device, the image acquisition module 610 can be a communication module that communicates with the front-end image acquisition device, and the communication module can acquire image data from the front-end image acquisition device.

[0159] For example, the image acquisition module 610 can be a lens module (including a lens and an image sensor, etc.) in the case where the image correction device is deployed in a front-end image acquisition device (such as a camera). Figure 7 A hardware structure schematic diagram of an electronic device is provided for the embodiments of the present application. The electronic device can include a processor 701 and a memory 702 storing machine executable instructions. The processor 701 and the memory 702 can communicate via a system bus 703. By reading and executing the machine executable instructions in the memory 702, the processor 701 can perform the calibration method or the image correction method described above.

[0160] The memory 702 mentioned herein can be any electronic, magnetic, optical or other physical storage device, and can contain or store information such as executable instructions, data, etc. For example, the machine readable storage medium can be RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard drive), solid state disk, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0161] In some embodiments, a storage medium, such as the memory 702 in the electronic device, is also provided, and machine executable instructions are stored in the storage medium, which, when executed by a processor, implement the calibration method or the image correction method described above. For example, the machine readable storage medium can be ROM, RAM, CD-ROM, magnetic tape, floppy disk and optical data storage device, etc. Figure 7

[0162] ​It is to be noted that, as used in this specification and the appended claims, the singular forms "a," "an" and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component" can include a combination of two or more components, and the term "an element" can include comparable reference to a plurality of elements. Also, as used in this specification and the appended claims, the term "or" as used in the context of "A / B" or "A / B / C" means any of the possibilities; for example, the term "or" means "any of A, B, or C; or any of A, B, or C." Further, the term "comprising" as used in this specification and the appended claims, whether in the description of the application or the claims themselves, is used in the sense of "including" and means "including but not limited to." Further, the term "consisting essentially of" as used in this specification and the appended claims, whether in the description of the application or the claims themselves, means "including at least the members identified, but excluding others not specifically identified." Further, the term "consisting of" as used in this specification and the appended claims, whether in the description of the application or the claims themselves, means "including and limited to."

[0163] The foregoing is considered as illustrative only of the principles of the application. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the application to the exact construction and practice described. Accordingly, all such variations are intended to be included within the scope of the present application as defined in the following claims.

Claims

1. A calibration method characterized by, The method comprises the following steps: acquiring a calibration board image; wherein the calibration board image is acquired by a target camera when the calibration board is parallel to a target plane, the target plane is a rectification plane of a target object, and the target object is a shooting object of the target camera; determining an extrinsic parameter of the target camera according to the calibration board image, and determining an angle θ according to the extrinsic parameter of the target camera; the angle θ is a rotation angle of a camera coordinate system of the target camera relative to a z-axis of a calibration board coordinate system; updating a rotation matrix in the extrinsic parameter of the target camera according to the angle θ to obtain an updated extrinsic parameter; mapping a specified feature point of the calibration board to a pixel plane according to the updated extrinsic parameter to obtain a rectified specified feature point pixel coordinate; determining a homography matrix corresponding to the target camera according to an original specified feature point pixel coordinate and the rectified specified feature point pixel coordinate; wherein the original specified feature point pixel coordinate is obtained by performing specified feature point identification on the calibration board image, and the homography matrix is used for image rectification on an image acquired by the target camera; wherein the determination of the angle θ according to the extrinsic parameter of the target camera comprises: determining the angle θ according to the extrinsic parameter of the target camera by the following formula: wherein PR(1) represents a projection vector of R(1) on the xy plane of the camera coordinate system, R(1) is the first column of the rotation matrix in the extrinsic parameter of the target camera, represents a direction vector of the x axis of the camera coordinate system; wherein the positive or negative sign of θ is determined by the following formula: wherein R(1) y represents the corresponding y-axis coordinate of the vector R(1), the clockwise direction being the positive direction of the angle; the updating of the rotation matrix in the extrinsic parameter of the target camera according to the angle θ to obtain the updated extrinsic parameter comprises: obtaining a rotation vector [0, 0, signθ] according to the angle θ; wherein θ is the rotation angle, and sign is the positive or negative sign of θ; converting the rotation vector into a target rotation matrix, and replacing the rotation matrix in the extrinsic parameter of the target camera with the target rotation matrix to obtain the updated extrinsic parameter.

2. The method of claim 1, wherein, the mapping of the specified feature point of the calibration board to the pixel plane according to the updated extrinsic parameter to obtain the rectified specified feature point pixel coordinate comprises: for any specified feature point of the calibration board, mapping a three-dimensional coordinate of the specified feature point to a camera coordinate system of the target camera according to the updated extrinsic parameter to determine a camera coordinate of the specified feature point; mapping the camera coordinate of the specified feature point to a pixel coordinate system according to an intrinsic parameter of the target camera to obtain a rectified pixel coordinate of the specified feature point.

3. An image correction method characterized by, The method comprises the following steps: acquiring a to-be-rectified image acquired by a target camera; performing image rectification on the to-be-rectified image by using a homography matrix corresponding to the target camera; wherein the homography matrix corresponding to the target camera is obtained by the method of claim 1 or 2.

4. A calibration device, characterized by The method comprises the following steps: an image acquisition module and a processor; wherein: the image acquisition module is configured to acquire a calibration board image; wherein the calibration board image is acquired by a target camera when the calibration board is parallel to a target plane, the target plane is a rectification plane of a target object, and the target object is a shooting object of the target camera; The processor is configured to determine the extrinsic parameter of the target camera according to the calibration board image, and determine an angle θ according to the extrinsic parameter of the target camera; the angle θ is a rotation angle of a camera coordinate system of the target camera relative to a z-axis of a calibration board coordinate system. The processor is further configured to update a rotation matrix in the extrinsic parameter of the target camera according to the angle θ, to obtain an updated extrinsic parameter. The processor is further configured to map a specified feature point of the calibration board to a pixel plane according to the updated extrinsic parameter, to obtain a corrected specified feature point pixel coordinate. The processor is further configured to determine a homography matrix corresponding to the target camera according to an original specified feature point pixel coordinate and the corrected specified feature point pixel coordinate; the original specified feature point pixel coordinate is obtained by performing specified feature point identification on the calibration board image, and the homography matrix is used for image correction on an image collected by the target camera. The processor determines the angle θ according to the extrinsic parameter of the target camera, and includes: The angle θ is determined according to the extrinsic parameter of the target camera by the following formula: wherein PR(1) represents a projection vector of R(1) on the xy plane of the camera coordinate system, R(1) is the first column of the rotation matrix in the extrinsic parameter of the target camera, represents a direction vector of the x axis of the camera coordinate system; The positive and negative sign of θ is determined by the following formula: wherein R(1) y represents the corresponding y-axis coordinate of the vector R(1), the clockwise direction being the positive direction of the angle; The processor updates the rotation matrix in the extrinsic parameter of the target camera according to the angle θ, to obtain an updated extrinsic parameter, and includes: A rotation vector [0, 0, signθ] is obtained according to the angle θ; θ is the rotation angle, and sign is the positive and negative sign of θ; The rotation vector is converted into a target rotation matrix, and the target rotation matrix is used to replace the rotation matrix in the extrinsic parameter of the target camera, to obtain the updated extrinsic parameter.

5. The apparatus of claim 4, wherein The processor maps the specified feature point of the calibration board to the pixel plane according to the updated extrinsic parameter, to obtain the corrected specified feature point pixel coordinate, and includes: For any specified feature point of the calibration board, the three-dimensional coordinate of the specified feature point is mapped to the camera coordinate system of the target camera according to the updated extrinsic parameter, to determine the camera coordinate of the specified feature point; The camera coordinate of the specified feature point is mapped to a pixel coordinate system according to the intrinsic parameter of the target camera, to obtain the corrected pixel coordinate of the specified feature point.

6. An image correction apparatus characterized by comprising: It includes an image acquisition module and a processor; wherein: The image acquisition module is configured to acquire a to-be-corrected image collected by a target camera; The processor is configured to perform image correction on the to-be-corrected image by using a homography matrix corresponding to the target camera; the homography matrix corresponding to the target camera is obtained by using the method in claim 1 or 2. The storage medium stores machine executable instructions, and the machine executable instructions are executed by the processor to implement the method in any one of claims 1-3.

7. An electronic device, comprising: The storage medium stores machine executable instructions, and the machine executable instructions are executed by the processor to implement the method in any one of claims 1-3.

8. A storage medium, characterized by ​

Images