Data processing method, device, system and storage medium

Abstract

The embodiments of the present application provide a data processing method, device, system and storage medium, which relate to the technical field of gimbal equipment and can improve the problem of tilt of the shooting picture caused by the low calibration accuracy of the gimbal camera. A data processing method based on a gimbal camera includes: obtaining a main calibration photo taken by the gimbal camera in an initial posture, the initial posture is the posture of the gimbal camera relative to the main calibration plate when the code disk of the gimbal camera is in the zero position to be calibrated, and the main calibration photo contains the main calibration plate; calculating the initial rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera in the initial posture according to the main calibration photo; obtaining the calibration Euler angle corresponding to the relative rotation matrix between the initial rotation matrix and the standard rotation matrix; and sending the calibration Euler angle to the gimbal camera.

Description

Technical Field

[0001] The present application relates to the technical field of pan / tilt devices, and in particular to a data processing method, device, system and storage medium. Background Art

[0002] With the development of technology, gimbal cameras are gradually becoming popular among users because they can make the transition of shooting pictures smooth and reduce picture shaking. A gimbal camera consists of a gimbal and a camera installed on the gimbal. The gimbal needs to be calibrated before leaving the factory to determine the zero position of the code disk. However, the current calibration method has low accuracy, which can easily cause the pictures taken by the gimbal camera to be tilted. Summary of the invention

[0003] A data processing method, device, system and storage medium can improve the problem of tilted shooting images caused by low calibration accuracy of a pan-tilt camera.

[0004] In a first aspect, a data processing method based on a gimbal camera is provided, comprising: obtaining a main calibration photo taken by the gimbal camera in an initial posture, the initial posture being the posture of the gimbal camera relative to a main calibration plate when a code disk of the gimbal camera is in a zero position to be calibrated, the main calibration photo comprising a main calibration plate; calculating an initial rotation matrix corresponding to a camera extrinsic parameter of the gimbal camera in the initial posture according to the main calibration photo; obtaining a calibration Euler angle corresponding to a relative rotation matrix between the initial rotation matrix and a standard rotation matrix; and sending the calibration Euler angle to the gimbal camera.

[0005] In a possible implementation, the process of calculating the initial rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera in the initial posture according to the main calibration photo includes: obtaining the camera intrinsic parameters of the gimbal camera; calculating the initial rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera in the initial posture according to the camera intrinsic parameters and the main calibration photo.

[0006] In a possible implementation, before obtaining the camera intrinsic parameters of the gimbal camera, it also includes: obtaining a plurality of auxiliary calibration photos taken of the gimbal camera relative to the main calibration plate at a plurality of different postures different from the initial posture, each auxiliary calibration photo including the main calibration plate; obtaining the camera intrinsic parameters of the gimbal camera includes: calculating the camera intrinsic parameters of the gimbal camera based on the plurality of auxiliary calibration photos.

[0007] In a possible implementation, after the calibration Euler angles are sent to the gimbal camera, it also includes: obtaining a verification photo taken by the gimbal camera in an initial position after calibration, the initial position after calibration being the position of the gimbal camera relative to the main calibration plate when the code disk of the gimbal camera is in a zero position after calibration, and the verification photo includes the main calibration plate; calculating the calibrated rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera in the initial position after calibration according to the verification photo; obtaining the verification Euler angle corresponding to the relative rotation matrix between the calibrated rotation matrix and the standard rotation matrix; determining whether the verification Euler angle is less than the preset Euler angle, if not, taking the current zero position of the code disk of the gimbal camera as the zero position to be calibrated, and executing the process of obtaining the main calibration photo taken by the gimbal camera in the initial position.

[0008] In a possible implementation, before obtaining the main calibration photo taken by the gimbal camera in the initial posture, it also includes: obtaining the initialization calibration photo taken by the gimbal camera in the standard posture relative to the main calibration plate; calculating the initialization rotation matrix corresponding to the camera extrinsic parameters of the current gimbal camera according to the initialization calibration photo, and using the initialization rotation matrix as the standard rotation matrix.

[0009] In a possible implementation, before obtaining the main calibration photo taken by the gimbal camera in the initial posture, it also includes: obtaining the initialization calibration photo taken by the gimbal camera relative to the main calibration plate in the initialization posture; calculating the initialization rotation matrix corresponding to the camera extrinsic parameter of the current gimbal camera according to the initialization calibration photo; reassigning each value r of the initialization rotation matrix to obtain a standard rotation matrix, wherein, if r∈[r1,r2], r is assigned to 0, if r∈[-1,r3], r is assigned to -1, if r∈[r4,1], r is assigned to 1, -1<r3<-0.5<r1<0<r2<0.5<r4<1.

[0010] In a possible implementation, if r∈(r3,r1), or r∈(r2,r4), a placement error prompt instruction is generated.

[0011] In a possible implementation, r3=-0.7, r1=-0.3, r2=0.3, and r4=0.7.

[0012] In a second aspect, a data processing device based on a gimbal camera is provided, including: a photo acquisition unit, used to acquire a main calibration photo taken by the gimbal camera in an initial posture, the initial posture being the posture of the gimbal camera relative to the main calibration plate when the code disk of the gimbal camera is in a zero position to be calibrated, and the main calibration photo includes the main calibration plate; a matrix calculation unit, used to calculate an initial rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera in the initial posture according to the main calibration photo; an Euler angle acquisition unit, used to acquire the calibration Euler angle corresponding to the relative rotation matrix between the initial rotation matrix and the standard rotation matrix; and a sending unit, used to send the calibration Euler angle to the gimbal camera.

[0013] In a third aspect, a data processing device based on a gimbal camera is provided, comprising: a processor and a memory, the memory being used to store at least one instruction, and when the instruction is loaded and executed by the processor, the data processing method based on the gimbal camera of the first aspect is implemented.

[0014] In a fourth aspect, a gimbal camera calibration system is provided, comprising: a data processing device based on the gimbal camera of the third aspect; a gimbal camera, the gimbal camera is used to receive the calibration Euler angle sent by the data processing device, and adjust the code disk of the gimbal camera from the zero position to be calibrated to the zero position after calibration according to the calibration Euler angle; a main calibration plate, when the gimbal camera is in an initial position, the camera of the gimbal camera is facing the main calibration plate.

[0015] In a possible implementation, the gimbal camera calibration system further includes at least two auxiliary calibration plates, and any one of the main calibration plate and the at least two auxiliary calibration plates is tangent to a virtual sphere, which is a virtual sphere with the optical center of the gimbal camera as its center.

[0016] In a fifth aspect, a computer-readable storage medium is provided, characterized in that a computer program is stored in the computer-readable storage medium, and when the computer program is run on a computer, the computer executes the data processing method based on the gimbal camera of the first aspect.

[0017] The data processing method, device, gimbal camera calibration system and storage medium based on the gimbal camera in the embodiments of the present application calculate the corresponding initial rotation matrix based on the main calibration photos taken by the gimbal camera in the initial position, and obtain the calibration Euler angle according to the relative rotation matrix between the initial rotation matrix and the standard rotation matrix, so that the gimbal camera can calibrate the zero position of the polar code disk according to the calibration Euler angle, that is, the camera calibration method is used to calibrate the gimbal camera, which is simple and convenient, and improves the calibration accuracy. In addition, the technical solution of the present application takes into account the installation error between the camera and the gimbal, making the calibration more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1This is a schematic diagram of the structure of a gimbal camera calibration system in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of a data processing method based on a pan-tilt camera in an embodiment of the present application;

[0020] Figure 3 Schematic diagram of another data processing method based on a gimbal camera in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0022] like Figure 1 As shown, an embodiment of the present application provides a gimbal camera calibration system, including: a data processing device 1 based on a gimbal camera, the data processing device 1 is used to obtain the calibration Euler angle, and send the calibration Euler angle to the gimbal camera 2. The specific process and principle will be described in detail in the subsequent content. The data processing device 1 can be, for example, a server; a gimbal camera 2, the gimbal camera 2 includes a gimbal and a camera, and a code disk is provided in the gimbal. The code disk is also called an encoder, which is used to control the rotation of the three axes of the gimbal X, Y, and Z. The gimbal camera 2 is used to receive the calibration Euler angle sent by the data processing device 1, and adjust the code disk of the gimbal camera 2 from the zero position to be calibrated to the zero position after calibration according to the calibration Euler angle. The gimbal camera 2 and the data processing device 1 are communicated and connected, for example, by a cable or wirelessly; a main calibration plate 31, when the gimbal camera 2 is in an initial position, the camera of the gimbal camera 2 faces the main calibration plate 31, and the main calibration plate 31 has a specific pattern, so that the camera can be calibrated by shooting the main calibration plate 31.

[0023] In a possible implementation, the gimbal camera calibration system further includes at least two auxiliary calibration plates 32, such as Figure 1 Four auxiliary calibration plates 32 are shown in the figure. When the gimbal camera 2 is in the initial position, the camera of the gimbal camera 2 faces the main calibration plate 31. The main calibration plate 31 and any one of the at least two auxiliary calibration plates 32 are tangent to the virtual sphere. The virtual sphere is a virtual sphere with the optical center of the gimbal camera 2 as the center. The main calibration plate 31 and the at least two auxiliary calibration plates 32 are arranged in a hemispherical shape. It should be noted that in addition to the method in which the gimbal camera calibration system can include the main calibration plate 31 and the auxiliary calibration plates 32, in other possible implementations, the gimbal camera calibration system can also have only one main calibration plate 31. The specific calibration methods under the two forms will be described in the subsequent content.

[0024] The present application embodiment provides a data processing method based on a pan-tilt camera, and the execution subject can be Figure 1The data processing device 1 in the data processing method is used based on Figure 1 The pan-tilt camera calibration system shown realizes the calibration of the pan-tilt camera 2. Before calibrating the pan-tilt camera 2, a fixture is first set to place the pan-tilt camera 2 and the main calibration plate 31. The pan-tilt camera 2 is set in an initial position relative to the main calibration plate 31, that is, in the initial position, the camera of the pan-tilt camera 2 faces the main calibration plate 31, and the code disk of the pan-tilt camera 2 in the initial position is in the zero position to be calibrated. Then, the pan-tilt camera 2 is calibrated, that is, the zero position of the code disk of the pan-tilt camera 2 is calibrated. During the calibration process, the data processing device 1 is required to execute a data processing method to obtain a calibration Euler angle for calibration, such as Figure 2 As shown, the data processing method includes:

[0025] Step 101, obtaining a main calibration photo taken by the gimbal camera 2 in an initial posture, the initial posture being the posture of the gimbal camera 2 relative to the main calibration plate 31 when the code disk of the gimbal camera 2 is in the zero position to be calibrated, the main calibration photo including the main calibration plate 31, that is, firstly controlling the gimbal camera 2 to start, and taking a main calibration photo in the initial posture, and sending the main calibration photo to the data processing device 1;

[0026] Step 102, calculating the initial rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera 2 in the initial position according to the main calibration photo;

[0027] Among them, step 101 and step 102 actually obtain the camera extrinsic parameters through the camera calibration method, and then obtain the corresponding rotation matrix. The camera extrinsic parameters can be used to describe the position and posture of the gimbal camera 2. The image taken by the camera uses the image coordinate system. The camera coordinate system is a coordinate system established based on the camera. The world coordinate system is a coordinate system established based on the real world. There is a corresponding conversion relationship between the image coordinate system and the camera coordinate system, and there is a corresponding conversion relationship between the camera coordinate system and the world coordinate system.

[0028] The following formula describes the conversion relationship between the camera coordinate system and the world coordinate system:

[0029]

[0030] in, is the coordinate of a point in the world coordinate system. is the coordinate of the point converted to the camera coordinate system, is the coordinate of the point in the image coordinate system, R is the rotation matrix, T is the translation vector, R and T are the camera extrinsics, and the camera extrinsics can be expressed as a matrix: K is the camera intrinsic parameter, and the camera extrinsic parameter is the rotation and translation transformation relationship of the camera coordinate system relative to the world coordinate system. The camera intrinsic parameter is the transformation relationship of the camera coordinate system relative to the image coordinate system. In the embodiment of the present application, the rotation matrix corresponding to the camera extrinsic parameter of the gimbal camera 2 in the initial posture is defined as the initial rotation matrix, and the initial rotation matrix is ​​defined as R1. In some scenarios, the camera intrinsic parameter K is a certain known item. The case where the camera intrinsic parameter K is an unknown item will be explained in the subsequent content. First, the case where the camera intrinsic parameter K is a known item will be explained. A specific calibration image is set on the main calibration plate 31, and the world coordinate system position corresponding to each feature point in the calibration image is a known coordinate. According to the main calibration photo, the position coordinates of the camera coordinate system corresponding to each feature point in the calibration image on the main calibration plate 31 taken by the gimbal camera 2 can be extracted, that is, the equation group can be established according to the above formula, through the known camera intrinsic parameter K, the known position coordinates of the feature point in the main calibration photo in the image coordinate system And the position coordinates in the world coordinate system By setting R in the formula as the initial rotation matrix R1 corresponding to the camera extrinsic parameters, the camera extrinsic parameters and the initial rotation matrix R1 corresponding to the camera extrinsic parameters can be solved.

[0031] Step 103, obtaining the calibration Euler angles corresponding to the relative rotation matrix between the initial rotation matrix and the standard rotation matrix;

[0032] Among them, let the standard rotation matrix be R0 and the relative rotation matrix be R 1-0 , the standard rotation matrix R0 is predetermined. The specific method of determining the standard rotation matrix R0 will be described in the subsequent content. 1-0 =R0×R1', R1' is the inverse matrix of R1. The calibration Euler angle refers to the relative rotation matrix R 1-0 The three Euler angles obtained by decomposing them in a certain order (for example, the order of yzx) are rotated according to the angles corresponding to the three axes of the gimbal camera 2 in this order, so that the gimbal camera 2 can be changed from one posture to another.

[0033] Step 104: Send the calibrated Euler angles to the gimbal camera 2.

[0034] After receiving the calibration Euler angle sent by the data processing device 1, the gimbal camera 2 adjusts the code disk of the gimbal camera 2 from the zero position to be calibrated to the zero position after calibration according to the calibration Euler angle, thereby realizing the calibration of the gimbal camera 2. Through the above method, the calibration of batches of gimbal cameras 2 can be realized.

[0035] The data processing method based on the gimbal camera and the gimbal camera calibration system in the embodiment of the present application calculate the corresponding initial rotation matrix based on the main calibration photos taken by the gimbal camera in the initial position, and obtain the calibration Euler angle according to the relative rotation matrix between the initial rotation matrix and the standard rotation matrix, so that the gimbal camera can calibrate the zero position of the polar code disk according to the calibration Euler angle, that is, the camera calibration method is used to calibrate the gimbal camera, which is simple and convenient, and improves the calibration accuracy. In addition, the technical solution of the present application takes into account the installation error between the camera and the gimbal, making the calibration more accurate.

[0036] In a possible implementation, step 102, the process of calculating the initial rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera 2 at the initial posture according to the main calibration photo, includes:

[0037] Step 1021, obtaining the camera internal parameters of the PTZ camera 2;

[0038] If the camera intrinsic parameters are unknown, the camera intrinsic parameters need to be calculated first in the process of calculating the camera extrinsic parameters. The process of calculating the camera intrinsic parameters can also be obtained by calibration based on calibration photos taken by the gimbal camera 2.

[0039] Step 1022: Calculate the initial rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera 2 in the initial position according to the camera intrinsic parameters and the main calibration photo.

[0040] Specifically, based on the above formula, after obtaining the camera intrinsic parameter K, the coordinates of the feature points in the main calibration photo in the image coordinate system and the coordinates in the world coordinate system can be substituted into the equation group after establishing the equation group to solve the camera extrinsic parameters and the initial rotation matrix R1 corresponding to the camera extrinsic parameters.

[0041] If the camera intrinsic parameters are known, the calibration calculation process of the camera extrinsic parameters only requires one main calibration photo taken by the main calibration plate 31 in the initial position to be implemented. If the camera intrinsic parameters are unknown, the camera intrinsic parameters need to be calibrated and calculated. There are two calibration calculation methods for the camera intrinsic parameters, which are described below.

[0042] In a first possible implementation, the gimbal camera calibration system may include only one main calibration plate 31, but not an auxiliary calibration plate 32, or may include auxiliary calibration plates 32 to improve the calibration accuracy by using more auxiliary calibration plates 32. Before step 1021, obtaining the camera internal parameters of the gimbal camera 2, it also includes: obtaining a plurality of auxiliary calibration photos taken by the gimbal camera 2 relative to the main calibration plate 31 at a plurality of different postures different from the initial posture, each auxiliary calibration photo including the main calibration plate. For example, before taking every two auxiliary calibration photos, the gimbal camera 2 is controlled to The pan-tilt camera 2 is pitched, yawed and rolled at a certain angle so that the pan-tilt camera 2 can take auxiliary calibration photos in different postures; step 1021, obtaining the camera intrinsic parameters of the pan-tilt camera 2 includes: calculating the camera intrinsic parameters of the pan-tilt camera 2 according to multiple auxiliary calibration photos. For the case where there is only one main calibration plate 31, the camera intrinsic parameters of the pan-tilt camera 2 can be calculated according to multiple auxiliary calibration photos taken in different postures. For the case where the auxiliary calibration plate 32 is included, the camera intrinsic parameters of the pan-tilt camera 2 are calibrated and calculated by a similar method, which can improve the calibration accuracy.

[0043] In a second possible implementation, the gimbal camera calibration system may include a main calibration plate 31 and multiple auxiliary calibration plates 32. Due to the function of the auxiliary calibration plates 32, the camera intrinsic parameters can be directly calibrated and calculated based on the main calibration photos taken by the gimbal camera 2 in the initial posture.

[0044] In a possible implementation, Figure 3 As shown, after step 104, the calibrated Euler angles are sent to the gimbal camera 2, the following further includes:

[0045] Step 105, obtaining a verification photo taken by the gimbal camera 2 in the initial position after calibration, the initial position after calibration being the position of the gimbal camera 2 relative to the main calibration plate 31 when the code disk of the gimbal camera 2 is in the zero position after calibration, and the verification photo includes the main calibration plate 31;

[0046] Among them, step 105 is specifically executed after the code disk of the gimbal camera 2 is adjusted from the zero position to be calibrated to the zero position after calibration. For example, the operator can control the execution of step 105 after determining that the code disk of the gimbal camera 2 has completed the calibration, or, after the code disk of the gimbal camera 2 is adjusted from the zero position to be calibrated to the zero position after calibration, an indication message of calibration completion is sent to the data processing device 1. When the data processing device 1 receives the indication message of calibration completion, step 105 is executed.

[0047] Step 106, calculating the calibrated rotation matrix R3 corresponding to the camera extrinsic parameters of the gimbal camera 2 in the initial position after calibration according to the verification photo;

[0048] The calculation principle of step 106 is the same as that of step 102, except that step 102 calculates the initial rotation matrix R1 corresponding to the state to be calibrated, while step 106 calculates the calibrated rotation matrix R3 corresponding to the state after calibration.

[0049] Step 107: Obtain the relative rotation matrix R between the calibrated rotation matrix R3 and the standard rotation matrix R0 3-0 The corresponding verification Euler angle;

[0050] The calculation principle of step 107 is the same as that of step 103. The relative rotation matrix R 3-0 =R0×R3', R3' is the inverse matrix of R3, according to the relative rotation matrix R 3-0 The corresponding verification Euler angle can be obtained.

[0051] Step 108, determine whether the verified Euler angle is less than the preset Euler angle. If not, proceed to step 109, use the current zero position of the gimbal camera's code disk as the zero position to be calibrated, and execute step 101, obtain the main calibration photo taken by the gimbal camera in the initial posture. If so, the calibration of gimbal camera 2 is completed.

[0052] Specifically, the process of steps 105 to 109 is a verification process after the calibration is completed. The verification Euler angle corresponding to the relative rotation matrix between the calibrated rotation matrix R3 and the standard rotation matrix R0 can be obtained by taking calibration photos and then calculating the Euler angle. If the verification Euler angle is less than the preset Euler angle, it means that the calibration has met the requirements and the calibration of the gimbal camera is completed. If the verification Euler angle is greater than or equal to the preset Euler angle, it means that the previous calibration did not meet the requirements, so it will be recalibrated. Multiple calibrations will make the code disk zero position of the gimbal camera 2 converge to the standard code disk zero position.

[0053] The following describes how to determine the standard rotation matrix. For example, there are two ways to determine the standard rotation matrix:

[0054] The first method is that before step 101, obtaining the main calibration photo taken by the gimbal camera 2 in the initial posture, it also includes: obtaining the initialization calibration photo taken by the gimbal camera 2 in the standard posture relative to the main calibration plate 31; calculating the initialization rotation matrix corresponding to the camera extrinsic parameters of the current gimbal camera 2 according to the initialization calibration photo, and using the initialization rotation matrix as the standard rotation matrix R0. Among them, the standard posture between the gimbal camera 2 and the main calibration plate 31 is manually adjusted, and the photos taken between the gimbal camera 2 and the main calibration plate 31 in a certain posture are manually judged to meet the requirements, then the current posture is used as the standard posture, the main calibration plate 31 is photographed in the standard posture and the standard rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera 2 is calculated, and in the subsequent calibration process of other gimbal cameras 2, the standard rotation matrix is ​​used as the standard for calibration. It should be noted that after the gimbal camera calibration system is arranged, the standard rotation matrix determination process only needs to be performed once, and then the determined standard rotation matrix R0 can be recorded in a file. When calibrating other gimbal cameras 2 subsequently, it is only necessary to directly read the predetermined standard rotation matrix R0 in the file.

[0055] The second method is that before step 101, obtaining the main calibration photo taken by the gimbal camera 2 in the initial posture, it also includes:

[0056] Step 001, obtaining the initialization calibration photo taken by the gimbal camera 2 relative to the main calibration plate 31 in the initialization posture, wherein, in the initialization posture, the main calibration plate 31 is manually adjusted to be strictly horizontal in the horizontal direction and strictly vertical in the vertical direction, that is, to ensure that one of the x and y axes of the main calibration plate 31 is perpendicular to the horizontal plane, and the other is parallel to the horizontal plane, then the gimbal camera 2 is directed toward the main calibration plate 31, and the posture is used as the initialization posture, and then the initialization calibration photo in the posture is taken;

[0057] Step 002, calculating the initialization rotation matrix R4 corresponding to the camera extrinsic parameters of the current gimbal camera 2 according to the initialization calibration photo;

[0058] Step 003, reassign each value r of the initialized rotation matrix R4 to obtain the standard rotation matrix R0, where if r∈[r1,r2], r is assigned to 0, if r∈[-1,r3], r is assigned to -1, if r∈[r4,1], r is assigned to 1, -1<r3<-0.5<r1<0<r2<0.5<r4<1, if r∈(r3,r1), or r∈(r2,r4), a placement error prompt instruction is generated, for example, r3=-0.7, r1=-0.3, r2=0.3, r4=0.7.

[0059] Specifically, the rotation matrix R4 is initialized to a 3×3 matrix with a total of nine values ​​r. For each value r, it is in the range of [-1,1]. For example, if r∈[-0.3,0.3], it means that it is closer to 0 than 1 and -1, so r is corrected to 0. If r∈[0.7,1], it means that it is closer to 1 than -1 and 0, so r is corrected to 1. If r∈[-1,-0.7], it means that it is closer to -1 than 1 and 0, so r is corrected to - 1. If r∈(r3,r1), or r∈(r2,r4), it means that the relative placement posture between the gimbal camera 2 and the main calibration plate 31 is wrong, that is, the placement between the gimbal camera 2 and the main calibration plate 31 is more crooked, so a placement error prompt instruction is generated, which can be prompted on the data processing device 1 side or on the gimbal camera 2 side to prompt manual adjustment of the placement between the gimbal camera 2 and the main calibration plate 31, and the adjusted posture is used as the initialization posture and re-enters step 001. It should be noted that r3=-0.7, r1=-0.3, r2=0.3, r4=0.7 are only examples, and the specific values ​​can be adjusted within a certain range nearby. The main function is to determine whether this position should be -1, 0 or 1, and whichever value is closer. By re-assigning the initialization rotation matrix R4 to obtain the standard rotation matrix R0, it can be ensured that the coordinates of the gimbal camera 2 and the coordinates of the main calibration plate are parallel to each other in the position of the gimbal camera 2 corresponding to the standard rotation matrix R0 (the three-axis parallelism here is not necessarily x-to-x parallelism, y-to-y parallelism, and z-to-z parallelism, but may be x-to--y parallelism, or x-to-z parallelism). After the standard rotation matrix R0 is determined in advance, each subsequent gimbal camera 2 can calibrate the zero position of the code disk based on the standard rotation matrix R0. In the second method, there is no need to manually determine whether the initialization calibration photo meets the requirements. It only needs to ensure that the position of the main calibration plate meets the requirements. The calibrated camera coordinate system will be parallel to the three axes of the main calibration plate.

[0060] It should be noted that after the gimbal camera calibration system is arranged, the standard rotation matrix determination process only needs to be performed once, and then the determined standard rotation matrix R0 can be recorded in a file. When calibrating other gimbal cameras 2 subsequently, it is only necessary to directly read the predetermined standard rotation matrix R0 in the file.

[0061] An embodiment of the present application also provides a data processing device based on a gimbal camera, including: a photo acquisition unit, used to acquire a main calibration photo taken by the gimbal camera in an initial posture, the initial posture being the posture of the gimbal camera relative to the main calibration plate when the code disk of the gimbal camera is in the zero position to be calibrated, and the main calibration photo includes the main calibration plate; a matrix calculation unit, used to calculate the initial rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera in the initial posture according to the main calibration photo; an Euler angle acquisition unit, used to acquire the calibration Euler angle corresponding to the relative rotation matrix between the initial rotation matrix and the standard rotation matrix; a sending unit, used to send the calibration Euler angle to the gimbal camera.

[0062] The data processing device may apply the data processing method in any of the above embodiments, and the specific process and principle thereof will not be described in detail herein.

[0063] In a possible implementation, the matrix calculation unit is specifically used to: obtain the camera intrinsic parameters of the gimbal camera; and calculate the initial rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera in the initial posture according to the camera intrinsic parameters and the main calibration photo.

[0064] In a possible implementation, the photo acquisition unit is also used to: acquire a plurality of auxiliary calibration photos taken by the gimbal camera relative to the main calibration plate at a plurality of different postures different from the initial posture, each auxiliary calibration photo including the main calibration plate; acquiring the camera intrinsic parameters of the gimbal camera includes: calculating the camera intrinsic parameters of the gimbal camera based on the plurality of auxiliary calibration photos.

[0065] In a possible implementation, the data processing device based on the gimbal camera also includes: a verification unit, wherein the verification unit is used to: obtain a verification photo taken by the gimbal camera in an initial position after calibration, the initial position after calibration being the position of the gimbal camera relative to the main calibration plate when the code disk of the gimbal camera is in a zero position after calibration, and the verification photo includes the main calibration plate; calculate the calibrated rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera in the initial position after calibration according to the verification photo; obtain the verification Euler angle corresponding to the relative rotation matrix between the calibrated rotation matrix and the standard rotation matrix; determine whether the verification Euler angle is less than the preset Euler angle, if not, use the current zero position of the code disk of the gimbal camera as the zero position to be calibrated, and execute the process of obtaining the main calibration photo taken by the gimbal camera in the initial position.

[0066] In a possible implementation, the data processing device based on the gimbal camera also includes: an initialization unit, used to: obtain an initialization calibration photo taken by the gimbal camera in a standard posture relative to the main calibration plate; calculate the initialization rotation matrix corresponding to the camera extrinsic parameters of the current gimbal camera based on the initialization calibration photo, and use the initialization rotation matrix as the standard rotation matrix.

[0067] In a possible implementation, the data processing device based on the gimbal camera also includes: an initialization unit, used to: obtain an initialization calibration photo taken by the gimbal camera relative to the main calibration plate in an initialization posture; calculate the initialization rotation matrix corresponding to the camera extrinsic parameters of the current gimbal camera according to the initialization calibration photo; reassign each value r of the initialization rotation matrix to obtain a standard rotation matrix, wherein, if r∈[r1,r2], r is assigned to 0, if r∈[-1,r3], r is assigned to -1, if r∈[r4,1], r is assigned to 1, -1<r3<-0.5<r1<0<r2<0.5<r4<1.

[0068] In a possible implementation, if r∈(r3,r1), or r∈(r2,r4), a placement error prompt instruction is generated.

[0069] In a possible implementation, r3=-0.7, r1=-0.3, r2=0.3, and r4=0.7.

[0070] It should be understood that the above division of the data processing device based on the pan-tilt camera is only a division of logical functions. In actual implementation, all or part of it can be integrated into one physical entity, or it can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, any one of the photo acquisition unit, matrix calculation unit, Euler angle acquisition unit and sending unit can be a separately established processing element, or it can be integrated in the data processing device, such as integrated in a chip of the data processing device. In addition, it can also be stored in the memory of the data processing device in the form of a program, and called and executed by a processing element of the data processing device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0071] For example, the modules such as the photo acquisition unit, the matrix calculation unit, the Euler angle acquisition unit and the sending unit may be one or more integrated circuits configured to implement the above method, such as: one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when one of the above modules is implemented in the form of a processing element scheduler, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call a program. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0072] An embodiment of the present application also provides a data processing device based on a gimbal camera, including: a processor and a memory, the memory is used to store at least one instruction, and when the instruction is loaded and executed by the processor, it is used to implement the data processing method based on the gimbal camera in any of the above embodiments. The specific process and principle of the data processing method are the same as those in the above embodiments and will not be repeated here.

[0073] The number of processors may be one or more, and the processor and memory may be connected via a bus or other means. The memory, as a non-transitory computer-readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the data processing device in the embodiment of the present application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions and modules stored in the memory, that is, implements the method in any of the above method embodiments. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; and necessary data, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device.

[0074] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the data processing method based on the gimbal camera in any of the above embodiments.

[0075] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk), etc.

[0076] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0077] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method based on a pan-tilt camera, characterized in that: include: Acquire a main calibration photo taken by the gimbal camera in an initial posture, wherein the initial posture is the posture of the gimbal camera relative to the main calibration plate when the code disk of the gimbal camera is in a zero position to be calibrated, and the main calibration photo includes the main calibration plate; Calculate the initial rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera at the initial posture according to the main calibration photo; Obtaining the calibration Euler angles corresponding to the relative rotation matrix between the initial rotation matrix and the standard rotation matrix; Sending the calibrated Euler angles to the gimbal camera; Before obtaining the main calibration photo taken by the gimbal camera in the initial posture, the method further includes: Get the initialization calibration photos taken by the gimbal camera at the initialization position relative to the main calibration plate; Calculate the initialization rotation matrix corresponding to the camera extrinsic parameters of the current gimbal camera according to the initialization calibration photo; Reassign each value r of the initialized rotation matrix to obtain the standard rotation matrix, where if r∈[r1,r2], r is assigned to 0, if r∈[-1,r3], r is assigned to -1, if r∈[r4,1], r is assigned to 1, -1<r3<-0.5<r1<0<r2<0.5<r4<1.

2. The method according to claim 1, characterized in that The process of calculating the initial rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera in the initial posture according to the main calibration photo includes: Obtaining the camera internal parameters of the gimbal camera; An initial rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera in the initial posture is calculated according to the camera intrinsic parameters and the main calibration photo.

3. The method according to claim 2, characterized in that Before obtaining the camera internal parameters of the pan-tilt camera, the method further includes: Acquire a plurality of auxiliary calibration photos taken by the gimbal camera relative to the main calibration plate at a plurality of different postures different from the initial posture, each of the auxiliary calibration photos including the main calibration plate; The step of obtaining the camera internal parameters of the pan / tilt camera comprises: The camera intrinsic parameters of the gimbal camera are calculated according to the multiple auxiliary calibration photos.

4. The method according to claim 1, characterized in that: After sending the calibrated Euler angles to the gimbal camera, the method further includes: Acquire a verification photo taken by the gimbal camera in an initial position after calibration, wherein the initial position after calibration is the position of the gimbal camera relative to the main calibration plate when the code disk of the gimbal camera is in a zero position after calibration, and the verification photo includes the main calibration plate; Calculate the calibrated rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera in the initial position after calibration according to the verification photo; Obtaining the verification Euler angles corresponding to the relative rotation matrix between the calibrated rotation matrix and the standard rotation matrix; Determine whether the verified Euler angle is less than the preset Euler angle. If not, use the current zero position of the code disk of the gimbal camera as the zero position to be calibrated, and execute the process of obtaining the main calibration photo taken by the gimbal camera in the initial posture.

5. The method according to claim 1, characterized in that Before obtaining the main calibration photo taken by the gimbal camera in the initial posture, the method further includes: Get the initial calibration photos taken by the gimbal camera in the standard posture relative to the main calibration plate; An initialization rotation matrix corresponding to the camera extrinsic parameters of the current gimbal camera is calculated according to the initialization calibration photo, and the initialization rotation matrix is ​​used as the standard rotation matrix.

6. The method according to claim 1, characterized in that If r∈(r3,r1), or r∈(r2,r4), a placement error prompt instruction is generated.

7. The method according to claim 6, characterized in that r3=-0.7，r1=-0.3，r2=0.3，r4=0.7。 8. A data processing device based on a pan-tilt camera, characterized in that: include: An initialization unit, used to obtain an initialization calibration photo taken by the gimbal camera relative to the main calibration plate at an initialization posture; Calculate the initialization rotation matrix corresponding to the camera extrinsic parameters of the current gimbal camera according to the initialization calibration photo; Reassign each value r of the initialization rotation matrix to obtain a standard rotation matrix, wherein if r∈[r1,r2], r is assigned to 0, if r∈[-1,r3], r is assigned to -1, if r∈[r4,1], r is assigned to 1, -1<r3<-0.5<r1<0<r2<0.5<r4<1; A photo acquisition unit, used to acquire a main calibration photo taken by the gimbal camera in an initial posture, wherein the initial posture is the posture of the gimbal camera relative to the main calibration plate when the code disk of the gimbal camera is in a zero position to be calibrated, and the main calibration photo includes the main calibration plate; A matrix calculation unit, used for calculating an initial rotation matrix corresponding to the camera extrinsic parameters of the gimbal camera at the initial posture according to the main calibration photo; An Euler angle acquisition unit, used to acquire a calibration Euler angle corresponding to a relative rotation matrix between the initial rotation matrix and the standard rotation matrix; A sending unit is used to send the calibrated Euler angle to the gimbal camera.

9. A data processing device based on a pan-tilt camera, characterized in that: include: A processor and a memory, wherein the memory is used to store at least one instruction, and when the instruction is loaded and executed by the processor, the data processing method based on the gimbal camera as described in any one of claims 1 to 7 is implemented.

10. A pan / tilt camera calibration system, characterized in that: include: The data processing device based on the pan-tilt camera as claimed in claim 9; A pan-tilt camera, the pan-tilt camera being used to receive the calibration Euler angles sent by the data processing device, and to adjust the code disk of the pan-tilt camera from a zero position to be calibrated to a zero position after calibration according to the calibration Euler angles; A main calibration plate, when the gimbal camera is in an initial position, the camera of the gimbal camera faces the main calibration plate.

11. The pan / tilt camera calibration system according to claim 10, characterized in that: Also includes: At least two auxiliary calibration plates, any one of the main calibration plate and the at least two auxiliary calibration plates is tangent to a virtual sphere, and the virtual sphere is a virtual sphere with the camera optical center of the gimbal camera as the sphere center.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the data processing method based on a gimbal camera as claimed in any one of claims 1 to 7.

Images