A method, device, equipment and medium for calculating external parameters of a pan-tilt camera

Abstract

The present invention discloses a method, device, equipment and medium for calculating the extrinsic parameters of a pan-tilt camera, which relates to the field of camera calibration technology to solve the problem that the extrinsic parameters of a camera cannot be calculated without an optical positioning system. A method for calculating the extrinsic parameters of a pan-tilt camera comprises: determining a first extrinsic parameter component based on the offset range and rotation angle range within which a physical camera can move in three dimensions in the pan-tilt; adjusting the position of the physical camera on the pan-tilt, and determining a second extrinsic parameter component based on the overlapping edges of a benchmark and a reference object in the physical camera image; when the first extrinsic parameter component and the second extrinsic parameter component are both zero, correcting a third extrinsic parameter component having a non-zero value to obtain the value of the corrected third extrinsic parameter component; and determining the extrinsic parameters of the physical camera based on the first extrinsic parameter component, the second extrinsic parameter component and the third extrinsic parameter component. The method for calculating the extrinsic parameters of a pan-tilt camera provided by the present invention is used to calculate camera extrinsics without an optical positioning system.

Description

Technical Field

[0001] The present invention relates to the technical field of camera calibration, and in particular to a method, device, equipment and medium for calculating extrinsic parameters of a pan-tilt camera. Background Art

[0002] Virtual live broadcast and virtual studio are unique TV program production technologies developed in recent years. In essence, they are the digital real-time synthesis of computer-generated virtual three-dimensional scenes and live images of people shot by TV cameras, so that the people and virtual backgrounds can change synchronously, thereby realizing the fusion of the two to obtain a perfect composite picture. This requires obtaining the position and posture information of the camera, and then transmitting it to the virtual rendering engine in real time to provide the virtual camera in the engine with the position and posture information of the physical camera in three-dimensional space and lens parameter information, etc.; optical positioning systems are usually used for positioning. Due to the existence of positioning points, the coordinate origins of the gimbal and the camera are the same, and the external parameters can be calculated using marker points to obtain the camera's position and posture information and send it to the virtual rendering engine.

[0003] However, optical positioning systems are often expensive and are not suitable for low-cost virtual production and virtual studios. Without an optical positioning system, the origin of the virtual camera in the virtual rendering engine is at the center of the lens, and the origin of the physical camera is on the rotation axis of the gimbal. Due to the different reference coordinate systems, the camera's external parameters cannot be calculated, and the virtual camera cannot obtain the accurate position and posture information of the physical camera. Summary of the Invention

[0004] The purpose of the present invention is to provide a method, device, equipment and medium for calculating the extrinsic parameters of a pan-tilt camera, which are used to solve the problem that the extrinsic parameters of the camera cannot be calculated without an optical positioning system.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] In a first aspect, the present invention provides a method for calculating extrinsic parameters of a pan-tilt camera, comprising:

[0007] Determine a first extrinsic parameter component based on an offset range and a rotation angle range within which the physical camera can move in three dimensions in the gimbal;

[0008] Adjusting the position of the physical camera on the gimbal, and determining a second extrinsic parameter component based on the overlapping edges of the benchmark object and the reference object in the physical camera image; the benchmark object and the reference object have the same regular shape;

[0009] When the first extrinsic parameter component and the second extrinsic parameter component are both zero, the third extrinsic parameter component having a non-zero value is corrected to obtain a value of the corrected third extrinsic parameter component; the third extrinsic parameter component is an offset parameter;

[0010] Determine the extrinsic parameters of a physical camera based on the first extrinsic parameter component, the second extrinsic parameter component, and the third extrinsic parameter component.

[0011] Compared with the prior art, the present invention provides a method for calculating the extrinsic parameters of a gimbal camera, comprising: determining a first extrinsic parameter component based on an offset range and a rotation angle range within which a physical camera can move in three dimensions in the gimbal; adjusting the position of the physical camera on the gimbal, and determining a second extrinsic parameter component based on the overlapping edges of a benchmark object and a reference object in an image of the physical camera; when the first extrinsic parameter component and the second extrinsic parameter component are both zero, correcting a third extrinsic parameter component having a non-zero value to obtain a value of the corrected third extrinsic parameter component; and determining the extrinsic parameters of the physical camera based on the first extrinsic parameter component, the second extrinsic parameter component, and the third extrinsic parameter component. By determining the first extrinsic parameter component and the second extrinsic parameter component of the physical camera in the present invention, the position where the five extrinsic parameter components of the physical camera are all zero can be determined, and the five extrinsic parameter components that do not need to be measured in the extrinsic parameter components can be quickly determined. Only one third extrinsic parameter component needs to be measured to obtain six extrinsic parameter component values ​​to calculate the extrinsic parameters of the physical camera, which greatly reduces the complexity of measuring the extrinsic parameters. The first extrinsic parameter component can be determined by the offset range and the rotation angle range. The second extrinsic parameter component can be determined by moving the position of the physical camera in the gimbal and the coincidence of the edges of the reference object and the benchmark object. The method for determining the axis pivot point is simple; at the same time, it solves the problem in the prior art that the extrinsic parameters of the physical camera cannot be measured without an optical positioning system, thereby reducing costs; the benchmark object and the reference object have the same and regular shapes, which makes it easy to judge whether the edges coincide.

[0012] In a second aspect, the present invention further provides a device for calculating extrinsic parameters of a pan-tilt camera, comprising:

[0013] A first extrinsic parameter component determination module is used to determine the first extrinsic parameter component according to an offset range and a rotation angle range within which the physical camera can move in three dimensions in the gimbal;

[0014] A second extrinsic parameter component determination module is used to adjust the position of the physical camera on the gimbal and determine the second extrinsic parameter component based on the overlapping edges of the benchmark object and the reference object in the physical camera image; the benchmark object and the reference object have the same regular shape;

[0015] a third extrinsic parameter component determination module, configured to correct a third extrinsic parameter component having a non-zero value when the first extrinsic parameter component and the second extrinsic parameter component are both zero, to obtain a value of the corrected third extrinsic parameter component; the third extrinsic parameter component being an offset parameter;

[0016] An extrinsic parameter calculation module is used to determine the extrinsic parameters of the physical camera based on the first extrinsic parameter component, the second extrinsic parameter component, and the third extrinsic parameter component.

[0017] Compared with the prior art, the beneficial effects of the extrinsic parameter calculation device for a gimbal camera provided by the present invention are the same as the beneficial effects of the extrinsic parameter calculation method for a gimbal camera described in the above technical solution, and will not be elaborated here.

[0018] In a third aspect, the present invention further provides a device for calculating extrinsic parameters of a pan-tilt camera, comprising:

[0019] A communication unit / communication interface is used to obtain the offset range and rotation angle range of the camera that can move in three dimensions in the gimbal;

[0020] A processing unit / processor, configured to determine a first extrinsic parameter component based on an offset range and a rotation angle range within which the physical camera can move in three dimensions in the gimbal;

[0021] Adjusting the position of the physical camera on the gimbal, and determining a second extrinsic parameter component based on the overlapping edges of the benchmark object and the reference object in the physical camera image; the benchmark object and the reference object have the same regular shape;

[0022] When the first extrinsic parameter component and the second extrinsic parameter component are both zero, the third extrinsic parameter component having a non-zero value is corrected to obtain a value of the corrected third extrinsic parameter component; the third extrinsic parameter component is an offset parameter;

[0023] Determine the extrinsic parameters of a physical camera based on the first extrinsic parameter component, the second extrinsic parameter component, and the third extrinsic parameter component.

[0024] Compared with the prior art, the beneficial effects of the gimbal camera extrinsic parameter measurement device provided by the present invention are the same as the beneficial effects of the gimbal camera extrinsic parameter measurement method described in the above technical solution, and will not be repeated here.

[0025] In a fourth aspect, the present invention also provides a computer storage medium, in which instructions are stored. When the instructions are executed, the above-mentioned gimbal camera external parameter measurement method is implemented.

[0026] Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present invention are the same as the beneficial effects of the method for measuring the external parameters of a gimbal camera described in the above technical solution, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0028] Figure 1 A flow chart of a method for calculating extrinsic parameters of a pan-tilt camera provided by the present invention;

[0029] Figure 2 A schematic diagram of the positional relationship between the benchmark object and the reference object provided by the present invention when the corresponding edges in the y-axis direction coincide with each other;

[0030] Figure 3 A schematic structural diagram of a device for calculating extrinsic parameters of a pan-tilt camera provided by the present invention;

[0031] Figure 4 This is a structural schematic diagram of a pan-tilt camera extrinsic parameter measurement device provided by the present invention. DETAILED DESCRIPTION

[0032] To facilitate a clear description of the technical solutions of the embodiments of the present invention, the words "first" and "second" are used in the embodiments of the present invention to distinguish between identical or similar items with substantially the same functions and effects. For example, the first threshold and the second threshold are merely used to distinguish between different thresholds and do not limit their order. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or execution order, and the words "first" and "second" do not necessarily mean different.

[0033] It should be noted that, in the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0034] In the present invention, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b, c can be single or multiple.

[0035] Virtual studio technology includes camera tracking, computer virtual scene design, color key technology, lighting technology, etc. Generally, an optical positioning system is used for camera positioning and tracking, but the cost of the optical positioning system is high, and without the optical positioning system, the extrinsic parameters of the physical camera cannot be calculated.

[0036] To solve the above technical problems, the present invention provides a method, device, equipment, and medium for calculating the extrinsic parameters of a pan-tilt camera, which calculate the extrinsic parameters of a physical camera without using an optical positioning system. This will be described below with reference to the accompanying drawings.

[0037] Figure 1 The present invention provides a flow chart of a method for calculating the external parameters of a pan-tilt camera, such as Figure 1 As shown, the method includes the following steps:

[0038] Step 101: Determine a first extrinsic parameter component based on an offset range and a rotation angle range within which a physical camera can move in three dimensions in a gimbal.

[0039] The role of the camera extrinsic parameter is to convert the coordinates from the world coordinate system to the virtual camera coordinate system. In the gimbal system, the world coordinate system is the gimbal coordinate system, and the coordinate origin of the physical camera is on the rotation axis of the gimbal. In a gimbal system, the horizontal direction of the physical camera lens is the y-axis of the gimbal three-dimensional coordinate system, the vertical direction of the physical camera lens is the z-axis of the gimbal three-dimensional coordinate system, and the direction perpendicular to the lens plane is the x-axis; the origin of the virtual camera coordinate system of the virtual camera is the lens center point, the direction perpendicular to the lens plane is the z-axis, the horizontal direction of the lens is the x-axis, and the vertical direction of the lens is the y-axis; assuming that the coordinates of a point P in the gimbal coordinate system are Pw and the coordinates in the virtual camera coordinate system are Pc, the conversion relationship between the two is shown in formula (1):

[0040]

[0041] in, is the camera external parameter matrix, the rotation matrix R is three matrices Rx(ψ), The product of Rz(θ), ψ is the rotation parameter corresponding to the rotation angle around the x-axis, is the rotation parameter corresponding to the rotation angle around the y-axis, θ is the rotation parameter corresponding to the rotation angle around the z-axis; the translation vector T represents the offset of point P from the gimbal coordinate system to the virtual camera coordinate system, T is composed of the offset parameters Tx, Ty, and Tz corresponding to the three axes of the gimbal coordinate system, so the external parameters of the physical camera are composed of the x-axis offset parameter Tx, the y-axis offset parameter Ty, the z-axis offset parameter Tz, the x-axis rotation parameter ψ, and the y-axis rotation parameter The z-axis rotation parameter θ consists of six external parameter components.

[0042] As an optional manner, determining the first extrinsic parameter component according to the offset range and rotation angle range of the physical camera that can move in three dimensions in the gimbal includes:

[0043] Get the offset range and rotation angle range of the physical camera that can move in the three dimensions of the x-axis, y-axis, and z-axis of the gimbal;

[0044] Determine whether the offset range and rotation angle range corresponding to each dimension are zero;

[0045] If any one of the offset range and the rotation angle range is zero, the corresponding offset parameter and / or rotation parameter is determined as the first extrinsic parameter component.

[0046] Specifically, in actual applications, not all gimbal systems can enable the physical camera to translate on three axes and rotate around three axes. For example, if the gimbal system cannot rotate the physical camera around the x-axis, that is, the obtained rotation angle range of the physical camera that can move on the x-axis in the gimbal is zero, then the x-axis rotation parameter ψ is determined as the first extrinsic parameter component and the x-axis rotation parameter ψ is zero; if the gimbal system cannot translate the physical camera in the x-axis direction, that is, the obtained offset range of the physical camera that can move in the x-axis direction in the gimbal is zero, then the x-axis offset parameter Tx is determined as the first extrinsic parameter component, and the x-axis offset parameter Tx is zero. This specification continues to explain by taking a gimbal system that cannot rotate the physical camera on the x-axis, can translate on the x-axis, y-axis, and z-axis, and can rotate around the y-axis and z-axis as an example, then the determined first extrinsic parameter component is the x-axis rotation parameter ψ.

[0047] Step 102: Adjust the position of the physical camera on the gimbal, and determine the second extrinsic parameter component based on the overlapping edges of the benchmark object and the reference object in the physical camera image.

[0048] As an optional manner, adjusting the position of the physical camera on the gimbal and determining the second extrinsic parameter component according to the overlapping edges of the benchmark object and the reference object in the physical camera image may include:

[0049] Obtaining the fiducial object image and the reference object image taken by the physical camera;

[0050] Move the physical camera in each dimension of the gimbal to determine whether the edges of the benchmark object image and the reference object image can overlap.

[0051] If they can overlap, the offset parameter and rotation parameter of the dimension corresponding to the overlapping edge are determined as the second external parameter component.

[0052] Specifically, after the physical camera is installed in the pan-tilt system, the positional relationship between the benchmark and the reference object can be combined with Figure 2To explain, first place a fiducial in front of the physical camera and a reference behind it. Both the fiducial and the reference are parallel to the lens plane. The edges of the fiducial and the reference do not overlap. The fiducial and the reference have the same shape but different sizes. The corresponding edges of the fiducial and the reference have the same direction. In order to facilitate comparison and observation of whether the edges of the fiducial and the reference overlap, objects of regular shape are selected as the fiducial and the reference, such as rectangles, positive directions, etc. First, move the physical camera in the y-axis direction so that the left sides of the fiducial and the reference overlap in the image captured by the physical camera, as shown in the following example. Figure 2 As shown, the corresponding edges of the datum object and the reference object in the y-axis direction coincide with each other. At this time, the y-axis offset parameter Ty and the y-axis rotation parameter Determined as the second external parameter component, and the y-axis offset parameter Ty and the y-axis rotation parameter are all zero; then move the physical camera in the z-axis direction so that the upper edges of the benchmark and the reference object in the image taken by the physical camera coincide with each other, and determine the z-axis offset parameter Tz and the z-axis rotation parameter θ as the second extrinsic parameter component, and the z-axis offset parameter Tz and the z-axis rotation parameter θ are zero, and then move the physical camera in the x-axis direction so that the other two edges of the benchmark and the reference object in the image taken by the physical camera coincide with each other. If the other two edges can also coincide, then record the position as the axis pivot point of the physical camera. At this time, the six extrinsic parameter components of the physical camera are all zero. However, since the gimbal system illustrated in this specification limits the range of offset of the physical camera in the x-axis direction, it is impossible to make the other two edges of the benchmark and the reference object in the image taken by the physical camera coincide with each other, and the value of the x-axis offset parameter Tx is not zero. Therefore, the second extrinsic parameter component determined by moving the position of the physical camera in the gimbal includes the y-axis offset parameter Ty, the y-axis rotation parameter The z-axis offset parameter Tz and the z-axis rotation parameter θ require adjustment and calculation of the x-axis offset parameter Tx to determine the axis pivot point. The above method for determining the first and second extrinsic parameter components allows for the rapid determination of the five unnecessary extrinsic parameter components. In this case, only the x-axis offset parameter Tx needs to be calculated to calculate the camera extrinsics, significantly reducing the difficulty and complexity of extrinsic parameter calculation.

[0053] It should be understood that the first and second extrinsic parameter components determined above are only examples and are not specifically limited. For example, the pan-tilt system can be a rocker system. During installation, the physical camera and the rocker need to be adjusted horizontally and vertically. That is, the physical camera cannot rotate in the three dimensions of the rocker system. Therefore, the corresponding three rotation parameters ψ1, and θ1 are both 0, the offset parameters Tx1, Ty1 and Tz1 have a greater impact. However, since the physical camera cannot move on the Y axis and Z axis in the rocker system, the offset parameters Ty1 and Tz1 of the physical camera are also 0. Therefore, ψ1, The five extrinsic parameter components θ1, Ty1, Tz1 are determined as the first extrinsic parameter components, and then the offset parameter Tx1 is determined by the method of step 102. If the offset parameter Tx1 is not zero, the offset parameter is calculated by step 103.

[0054] Next, step 103 is used to describe the method for calculating the x-axis offset parameter Tx.

[0055] Step 103: When the first extrinsic parameter component and the second extrinsic parameter component are both zero, the third extrinsic parameter component having a non-zero value is corrected to obtain the value of the corrected third extrinsic parameter component. The third extrinsic parameter component is an x-axis offset parameter.

[0056] Since the x-axis offset parameter is the offset parameter corresponding to the coordinate axis perpendicular to the lens, the position of the axis pivot point will change with the focus value and depth of field value of the lens, so it is necessary to add data points and draw the Tx function curve to calculate the extrinsic parameter component Tx.

[0057] As an optional manner, when the first extrinsic parameter component and the second extrinsic parameter component are both zero, correcting the third extrinsic parameter component having a non-zero value to obtain the value of the corrected third extrinsic parameter component includes:

[0058] The posture information of the adjusted physical camera is transmitted to the virtual engine in real time to synchronize the movement and composition of the virtual camera with the physical camera; the posture information is the position and posture information.

[0059] Specifically, before calculating the x-axis offset parameter Tx, the hardware equipment must be connected. The hardware equipment includes a computer installed with a virtual engine and a computer vision system such as OpenCV. The virtual engine is connected to a tracking system. The tracking system is used to obtain the position and posture information of the physical camera, and then transmits the position and posture information to the virtual engine in real time. The virtual engine uses rendering and synthesis technology based on the position and posture information to synchronize the movement and composition of the virtual camera and the physical camera. The image captured by the physical camera and the corresponding image of the virtual camera are displayed on the computer display. The computer vision system is used to process and calculate the physical camera extrinsic parameters in real time based on the extrinsic parameter components, and perform curve fitting calculations on the extrinsic parameter components. The hardware equipment also includes a memory for storing all data, such as image information and parameter information. At the same time, the computer obtains the image information captured by the physical camera through an integrated acquisition unit.

[0060] A physical target is placed in front of the physical camera; its center aligns with the center of the physical camera's viewport. The distance from the physical target to the physical camera is recorded. A virtual target is placed at the same distance in front of the virtual camera, with its center aligning with the center of the virtual camera's viewport. The image of the physical target captured by the physical camera and the corresponding image of the virtual target are then transmitted to the display screen for display. Specifically, the coordinates of the physical target image are mapped into the virtual camera's coordinate system, and the image corresponding to the virtual target is constructed using a computer. Both images are displayed in the same coordinate system.

[0061] Then, the focus parameters and x-axis offset parameters of the physical camera are adjusted so that the physical target image and the virtual target image on the display screen are completely aligned under any focus parameters, and the x-axis offset parameter value corresponding to each focus parameter is obtained.

[0062] Specifically, the focus parameter of the physical camera is adjusted to zero, the depth of field value of the physical camera is adjusted to zero, and then the physical camera is rotated left and right respectively. At this time, the physical target image and the virtual target image do not overlap, and the x-axis offset parameter is adjusted until the physical target image and the virtual target image on the display screen are completely aligned, thereby obtaining a first x-axis offset parameter; then the focus parameter of the physical camera is adjusted to one, the depth of field value of the physical camera is zero, and then the physical camera is rotated left and right respectively. At this time, the physical target image and the virtual target image do not overlap, and the x-axis offset parameter is adjusted until the physical target image and the virtual target image on the display screen are completely aligned, thereby obtaining a second x-axis offset parameter; according to the first x-axis offset parameter, the second x-axis offset parameter and the corresponding focus parameter and depth of field value, the interpolation method is used to calculate the x-axis offset parameter value corresponding to the focus parameter from zero to each point in a pair of focal segments, thereby obtaining the x-axis offset parameter value corresponding to each focus parameter, generating a curve of the x-axis offset parameter value changing with the focus parameter, and completing the preliminary curve fitting.

[0063] Then adjust the focus parameter of the physical camera to an arbitrary value, rotate the physical camera left and right, and determine whether the physical target image and the virtual target image on the display screen completely overlap. If they overlap, it means that the current curve is reasonable, and the adjustment and correction of the x-axis offset parameter value of the third extrinsic parameter component is completed;

[0064] If they do not overlap, adjust the x-axis offset parameter value corresponding to the current focus parameter until the physical target image and the virtual target image on the display screen completely overlap to obtain a data point; for example, if the current focus parameter is 0.6 and the depth of field value is 0, the x-axis offset parameter value corresponding to the adjusted focus parameter 0.6 is 5, then the data point is (0.6, 5).

[0065] The data points are added to the preliminary curve and the curve is fitted again. The focus parameter values ​​are adjusted again on the resulting curve. The physical camera is rotated left and right to test whether the physical target image and the virtual target image on the display completely overlap. If they do, the correction of the third extrinsic parameter component is complete. With each measured Tx value, the OpenCV computational vision library calculates the camera's extrinsic parameters in real time based on the known extrinsic parameters and makes adjustments.

[0066] Step 104: Determine the extrinsic parameters of the physical camera based on the first extrinsic parameter component, the second extrinsic parameter component, and the third extrinsic parameter component.

[0067] Determine a rotation matrix based on the rotation parameters in the first extrinsic parameter component and the second extrinsic parameter component;

[0068] Specifically, the x-axis rotation parameter ψ in the first extrinsic parameter component is zero, and the y-axis rotation parameter in the second extrinsic parameter component is and the z-axis rotation parameter θ is zero, and the matrix Rx(ψ) is calculated according to formula (2), formula (3) and formula (4), Rz(θ), as shown in formula (2), formula (3), and formula (4):

[0069]

[0070]

[0071]

[0072] The rotation matrix R is calculated according to formula (5), as shown in formula (5):

[0073]

[0074] Determine a translation matrix according to the offset parameters in the first extrinsic parameter component, the second extrinsic parameter component, and the third extrinsic parameter component;

[0075] Specifically, the y-axis offset parameter is zero, the z-axis offset parameter is zero, and the value of the x-axis offset parameter changes with the focus parameter. The translation matrix T is calculated according to formula (6), as shown in formula (6):

[0076] T=(Tx、Ty、Tz) (6)

[0077] The extrinsic parameters of the physical camera are calculated according to the rotation matrix and the translation matrix.

[0078] Specifically, substitute the rotation matrix R and translation matrix T into In , the external parameters of the physical camera are calculated.

[0079] In order to ensure the accuracy of the measurement results, final testing and extrinsic parameter adjustment are required after adjusting the third extrinsic parameter component: specifically, place a calibration plate on the ground and align the physical camera with the calibration plate; when a virtual calibration plate is placed at the same position and angle as the virtual camera, if the physical calibration plate and the virtual calibration plate overlap, the relationship between the ground under the calibration plate and the camera is basically the same as the relationship between the camera and the scene ground in VPW, indicating that the current extrinsic parameter measurement is reasonable and no adjustment is required; if the physical calibration plate and the virtual calibration plate do not overlap, find a point next to the calibration plate as a reference point, move the physical camera to determine the relationship between the calibration plate and the reference point, and then adjust the focus parameter fcous and the depth of field value zoom to continue testing until the position relationship changes are basically consistent and the extrinsic parameter adjustment is completed.

[0080] The present invention can determine the position where the five extrinsic parameter components of the physical camera are all zero through the steps of determining the first extrinsic parameter component and the second extrinsic parameter component of the physical camera in the present invention, and can quickly determine the five extrinsic parameter components that do not need to be measured in the extrinsic parameter components. Only one third extrinsic parameter component needs to be measured to obtain six extrinsic parameter component values ​​to calculate the extrinsic parameters of the physical camera, which greatly reduces the complexity of measuring the extrinsic parameters. The first extrinsic parameter component can be determined by the offset range and the rotation angle range. The second extrinsic parameter component can be determined by moving the position of the physical camera in the pan-tilt head and the coincidence of the edges of the reference object and the benchmark object. The method for determining the axis pivot point is simple; at the same time, it solves the problem in the prior art that the extrinsic parameters of the physical camera cannot be measured without an optical positioning system, thereby reducing costs; the benchmark object and the reference object have the same and regular shapes, which makes it easy to judge whether the edges coincide.

[0081] Based on the same idea, the present invention also provides a device for calculating the external parameters of a pan-tilt camera. Figure 3 As shown, the device may include:

[0082] A first extrinsic parameter component determination module 301 is configured to determine a first extrinsic parameter component based on an offset range and a rotation angle range within which a physical camera can move in three dimensions in a gimbal;

[0083] A second extrinsic parameter component determination module 302 is configured to adjust the position of the physical camera on the gimbal and determine the second extrinsic parameter component based on the overlapping edges of the benchmark object and the reference object in the physical camera image; the benchmark object and the reference object have the same regular shape;

[0084] A third extrinsic parameter component determination module 303 is configured to correct a third extrinsic parameter component having a non-zero value when both the first extrinsic parameter component and the second extrinsic parameter component are zero, to obtain a corrected value of the third extrinsic parameter component; the third extrinsic parameter component is an offset parameter;

[0085] The extrinsic parameter calculation module 304 is configured to determine the extrinsic parameters of the physical camera based on the first extrinsic parameter component, the second extrinsic parameter component, and the third extrinsic parameter component.

[0086] Optionally, the extrinsic parameter components of the physical camera include an x-axis offset parameter, a y-axis offset parameter, a z-axis offset parameter, an x-axis rotation parameter, a y-axis rotation parameter, and a z-axis rotation parameter; the first extrinsic parameter component determination module 301 can be specifically used to:

[0087] Get the offset range and rotation angle range of the physical camera that can move in the three dimensions of the x-axis, y-axis, and z-axis of the gimbal;

[0088] Determine whether the offset range and rotation angle range corresponding to each dimension are zero;

[0089] If any one of the offset range and the rotation angle range is zero, the corresponding offset parameter and / or rotation parameter is determined as the first extrinsic parameter component.

[0090] Optionally, the second extrinsic parameter component determining module 302 may be specifically configured to:

[0091] Obtaining the fiducial object image and the reference object image taken by the physical camera;

[0092] Move the physical camera in each dimension of the gimbal to determine whether the edges of the benchmark object image and the reference object image can overlap.

[0093] If they can overlap, the values ​​of the offset parameter and the rotation parameter of the dimension corresponding to the overlapping edge are zero, and the offset parameter and the rotation parameter of the dimension corresponding to the overlapping edge are determined as the second extrinsic parameter component.

[0094] Optionally, the third extrinsic parameter component is an x-axis offset parameter, and the x-axis is perpendicular to the physical camera lens. The third extrinsic parameter component determining module 303 may include:

[0095] The system tracking and rendering unit is used to transmit the position information of the adjusted physical camera to the virtual engine in real time, so that the movement and composition of the virtual camera are synchronized with the physical camera;

[0096] A physical target placement unit, configured to place a physical target in front of the physical camera; the center point of the physical target corresponds to the center of the viewport of the physical camera;

[0097] A virtual target placement unit is used to place a virtual target at the same position in front of the virtual camera, wherein the center point of the virtual target corresponds to the center of the viewport of the virtual camera;

[0098] A display unit is used to transmit the physical target image and the image corresponding to the virtual target taken by the physical camera to a display screen for display;

[0099] The third extrinsic parameter component adjustment unit is used to adjust the focus parameters and x-axis offset parameters of the physical camera so that the physical target image and the virtual target image on the display screen are completely aligned under any focus parameters, and obtain the x-axis offset parameter value corresponding to each focus parameter.

[0100] Optionally, the third extrinsic parameter component adjustment unit may be specifically configured to:

[0101] Adjust the focus parameter of the physical camera to zero, and then adjust the x-axis offset parameter until the physical target image and the virtual target image on the display screen are completely aligned, thereby obtaining a first x-axis offset parameter;

[0102] Adjust the focus parameter of the physical camera to 1, and then adjust the x-axis offset parameter until the physical target image and the virtual target image on the display screen are completely aligned, to obtain a second x-axis offset parameter;

[0103] According to the first x-axis offset parameter, the second x-axis offset parameter and the corresponding focus parameter, an interpolation method is used to calculate the x-axis offset parameter value corresponding to each focus parameter.

[0104] Optionally, the third extrinsic parameter component adjustment unit may also be used to:

[0105] Adjust the focus parameter of the physical camera to any value, rotate the physical camera left and right, and determine whether the physical target image and the virtual target image on the display screen completely overlap. If so, the correction of the third extrinsic parameter component is completed;

[0106] If they do not coincide, adjust the x-axis offset parameter value corresponding to the current focus parameter until the physical target image and the virtual target image on the display screen completely coincide, and obtain the data point;

[0107] The data points are added to a curve consisting of x-axis offset parameter values ​​corresponding to each focusing parameter to complete the correction of the third extrinsic parameter component.

[0108] Optionally, the external parameter calculation module 304 may be specifically used to:

[0109] Determine a rotation matrix based on the rotation parameters in the first extrinsic parameter component and the second extrinsic parameter component;

[0110] Determine a translation matrix according to the offset parameters in the first extrinsic parameter component, the second extrinsic parameter component, and the third extrinsic parameter component;

[0111] The extrinsic parameters of the physical camera are calculated according to the rotation matrix and the translation matrix.

[0112] Based on the same idea, the present invention also provides a device for measuring and calculating the external parameters of a pan-tilt camera. Figure 4 As shown, this may include:

[0113] A communication unit / communication interface is used to obtain the offset range and rotation angle range of the camera that can move in three dimensions in the gimbal;

[0114] A processing unit / processor, configured to determine a first extrinsic parameter component based on an offset range and a rotation angle range within which the physical camera can move in three dimensions in the gimbal;

[0115] Adjusting the position of the physical camera on the gimbal, and determining a second extrinsic parameter component based on the overlapping edges of the benchmark object and the reference object in the physical camera image; the benchmark object and the reference object have the same regular shape;

[0116] When the first extrinsic parameter component and the second extrinsic parameter component are both zero, the third extrinsic parameter component having a non-zero value is corrected to obtain a value of the corrected third extrinsic parameter component; the third extrinsic parameter component is an offset parameter;

[0117] Determine the extrinsic parameters of a physical camera based on the first extrinsic parameter component, the second extrinsic parameter component, and the third extrinsic parameter component.

[0118] like Figure 4 As shown, the processor can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present invention. There can be one or more communication interfaces. The communication interface can use any device such as a transceiver for communicating with other devices or a communication network.

[0119] like Figure 4 As shown, the terminal device may further include a communication line. The communication line may include a path for transmitting information between the components.

[0120] like Figure 4As shown, the memory can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory can exist independently and be connected to the processor through a communication line. The memory can also be integrated with the processor.

[0121] In a specific implementation, as an embodiment, Figure 4 As shown, the processor may include one or more CPUs, such as Figure 4 CPU0 and CPU1 in.

[0122] In a specific implementation, as an embodiment, Figure 4 As shown, the terminal device may include multiple processors, such as Figure 4 Each of these processors can be a single-core processor or a multi-core processor.

[0123] Based on the same idea, the present invention also provides a computer-readable storage medium, which stores instructions. When the instructions are executed, the following are achieved:

[0124] Determine a first extrinsic parameter component based on an offset range and a rotation angle range within which the physical camera can move in three dimensions in the gimbal;

[0125] Adjusting the position of the physical camera on the gimbal, and determining a second extrinsic parameter component based on the overlapping edges of the benchmark object and the reference object in the physical camera image; the benchmark object and the reference object have the same regular shape;

[0126] When the first extrinsic parameter component and the second extrinsic parameter component are both zero, the third extrinsic parameter component having a non-zero value is corrected to obtain a value of the corrected third extrinsic parameter component; the third extrinsic parameter component is an offset parameter;

[0127] Determine the extrinsic parameters of a physical camera based on the first extrinsic parameter component, the second extrinsic parameter component, and the third extrinsic parameter component.

[0128] The above mainly introduces the solution provided by the embodiment of the present invention from the perspective of the interaction between the various modules. It can be understood that in order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0129] The embodiments of the present invention can be divided into functional modules according to the above-mentioned method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated into a single processing module. The above-mentioned integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the division of modules in the embodiments of the present invention is illustrative and is only a logical functional division. In actual implementation, other division methods may be used.

[0130] Optionally, the computer-executable instructions in the present invention may also be referred to as application code, which is not specifically limited in the present invention.

[0131] The methods disclosed herein can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the method can be performed by hardware integrated logic circuits within the processor or by software instructions. The processor may be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor or by a combination of hardware and software modules within the decoding processor. The software modules can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in a memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the method.

[0132] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are performed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device, or other programmable device. The computer program or instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions may be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video disc (DVD); or a semiconductor medium, such as a solid-state drive (SSD).

[0133] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0134] Although the present invention has been described with reference to specific features and embodiments thereof, it will be apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the invention as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the invention. It will be apparent that various modifications and variations may be made to the present invention by those skilled in the art without departing from the spirit and scope of the invention. Thus, the present invention is intended to include such modifications and variations as fall within the scope of the claims of the present invention and their equivalents.

Claims

1. A method for calculating extrinsic parameters of a pan-tilt camera, characterized in that: include: Determine a first extrinsic parameter component based on an offset range and a rotation angle range within which the physical camera can move in three dimensions in the gimbal; Adjust the position of the physical camera on the gimbal, and determine the second extrinsic parameter component based on the overlapping edges of the fiducial object and the reference object in the physical camera image; The shapes of the benchmark object and the reference object are the same, both being regular shapes; When the first extrinsic parameter component and the second extrinsic parameter component are both zero, the third extrinsic parameter component having a non-zero value is corrected to obtain a value of the corrected third extrinsic parameter component; The third extrinsic parameter component is an offset parameter; Determine the extrinsic parameters of the physical camera based on the first extrinsic parameter component, the second extrinsic parameter component, and the third extrinsic parameter component; The extrinsic parameter components of the physical camera include an x-axis offset parameter, a y-axis offset parameter, a z-axis offset parameter, an x-axis rotation parameter, a y-axis rotation parameter, and a z-axis rotation parameter; determining the first extrinsic parameter component according to the offset range and rotation angle range that the physical camera can move in three dimensions in the gimbal includes: Get the offset range and rotation angle range of the physical camera that can move in the three dimensions of the x-axis, y-axis, and z-axis of the gimbal; Determine whether the offset range and rotation angle range corresponding to each dimension are zero; If any one of the offset range and the rotation angle range is zero, the corresponding offset parameter and / or rotation parameter is determined as the first extrinsic parameter component; The adjusting the position of the physical camera on the gimbal and determining the second extrinsic parameter component according to the coincident edge of the benchmark object and the reference object in the physical camera image include: Obtaining the fiducial object image and the reference object image taken by the physical camera; Move the physical camera in each dimension of the gimbal to determine whether the edges of the benchmark object image and the reference object image can overlap. If they can overlap, the values ​​of the offset parameter and the rotation parameter of the dimension corresponding to the overlapping edge are zero, and the offset parameter and the rotation parameter of the dimension corresponding to the overlapping edge are determined as the second extrinsic parameter component.

2. The method for calculating extrinsic parameters of a pan-tilt camera according to claim 1, wherein: The third extrinsic parameter component is an x-axis offset parameter, and the x-axis is perpendicular to the physical camera lens. When the first extrinsic parameter component and the second extrinsic parameter component are both zero, the third extrinsic parameter component having a non-zero value is corrected to obtain a value of the corrected third extrinsic parameter component, including: The position information of the adjusted physical camera is transmitted to the virtual engine in real time, so that the movement and composition of the virtual camera are synchronized with the physical camera; A physical target is placed in front of the physical camera; the center point of the physical target corresponds to the center of the viewport of the physical camera; placing a virtual target at the same position in front of the virtual camera, wherein the center point of the virtual target corresponds to the center of the viewport of the virtual camera; The physical target image and the image corresponding to the virtual target captured by the physical camera are transmitted to the display screen for display; The focus parameters and x-axis offset parameters of the physical camera are adjusted so that the physical target image and the virtual target image on the display screen are completely aligned under any focus parameters, and the x-axis offset parameter value corresponding to each focus parameter is obtained.

3. The method for calculating extrinsic parameters of a pan-tilt camera according to claim 2, wherein: The adjusting of the focus parameters and x-axis offset parameters of the physical camera so that the physical target image and the virtual target image on the display screen are completely aligned under any focus parameters, and obtaining the x-axis offset parameter value corresponding to each focus parameter, includes: Adjust the focus parameter of the physical camera to zero, and then adjust the x-axis offset parameter until the physical target image and the virtual target image on the display screen are completely aligned, thereby obtaining a first x-axis offset parameter; Adjust the focus parameter of the physical camera to 1, and then adjust the x-axis offset parameter until the physical target image and the virtual target image on the display screen are completely aligned, to obtain a second x-axis offset parameter; According to the first x-axis offset parameter, the second x-axis offset parameter and the corresponding focus parameter, an interpolation method is used to calculate the x-axis offset parameter value corresponding to each focus parameter.

4. The method for calculating extrinsic parameters of a pan-tilt camera according to claim 3, wherein: The method further includes calculating the x-axis offset parameter value corresponding to each focusing parameter using an interpolation method based on the first x-axis offset parameter, the second x-axis offset parameter and the corresponding focusing parameter, and then further including: Adjust the focus parameter of the physical camera to any value, rotate the physical camera left and right, and determine whether the physical target image and the virtual target image on the display screen completely overlap. If so, the correction of the third extrinsic parameter component is completed; If they do not coincide, adjust the x-axis offset parameter value corresponding to the current focus parameter until the physical target image and the virtual target image on the display screen completely coincide with each other to obtain the data point; The data points are added to a curve consisting of x-axis offset parameter values ​​corresponding to each focusing parameter to complete the correction of the third extrinsic parameter component.

5. The method for calculating extrinsic parameters of a pan-tilt camera according to claim 1, wherein: The determining of the extrinsic parameters of the physical camera based on the first extrinsic parameter component, the second extrinsic parameter component, and the third extrinsic parameter component includes: Determine a rotation matrix based on the rotation parameters in the first extrinsic parameter component and the second extrinsic parameter component; Determine a translation matrix according to the offset parameters in the first extrinsic parameter component, the second extrinsic parameter component, and the third extrinsic parameter component; The extrinsic parameters of the physical camera are calculated according to the rotation matrix and the translation matrix.

6. A device for calculating extrinsic parameters of a pan-tilt camera, applied to the method for calculating extrinsic parameters of a pan-tilt camera according to any one of claims 1 to 5, characterized in that: include: A first extrinsic parameter component determination module is used to determine the first extrinsic parameter component according to an offset range and a rotation angle range within which the physical camera can move in three dimensions in the gimbal; A second extrinsic parameter component determination module is used to adjust the position of the physical camera on the gimbal and determine the second extrinsic parameter component based on the overlapping edges of the benchmark object and the reference object in the physical camera image; the benchmark object and the reference object have the same regular shape; a third extrinsic parameter component determination module, configured to correct a third extrinsic parameter component having a non-zero value when the first extrinsic parameter component and the second extrinsic parameter component are both zero, to obtain a value of the corrected third extrinsic parameter component; The third extrinsic parameter component is an offset parameter; An extrinsic parameter calculation module is used to determine the extrinsic parameters of the physical camera based on the first extrinsic parameter component, the second extrinsic parameter component, and the third extrinsic parameter component.

7. A device for calculating extrinsic parameters of a pan-tilt camera, applied to the method for calculating extrinsic parameters of a pan-tilt camera according to any one of claims 1 to 5, characterized in that: include: A communication unit / communication interface, used to obtain the offset range and rotation angle range of the physical camera that can move in three dimensions in the gimbal; A processing unit / processor, configured to determine a first extrinsic parameter component based on an offset range and a rotation angle range within which the physical camera can move in three dimensions in the gimbal; Adjust the position of the physical camera on the gimbal, and determine the second extrinsic parameter component based on the overlapping edges of the fiducial object and the reference object in the physical camera image; The shapes of the benchmark object and the reference object are the same, both being regular shapes; When the first extrinsic parameter component and the second extrinsic parameter component are both zero, the third extrinsic parameter component having a non-zero value is corrected to obtain a value of the corrected third extrinsic parameter component; The third extrinsic parameter component is an offset parameter; Determine the extrinsic parameters of a physical camera based on the first extrinsic parameter component, the second extrinsic parameter component, and the third extrinsic parameter component.

8. A computer storage medium, characterized in that The computer storage medium stores instructions, and when the instructions are executed by the computer, the method for calculating the external parameters of the pan-tilt camera described in any one of claims 1 to 5 is implemented.

Images