Method and apparatus for controlling a camera

Abstract

A method and device for controlling a camera, the method comprising: obtaining multiple sets of calibration data of the camera, each set of calibration data including: PTZ data output by the camera and satellite map data; solving a camera pose estimation model according to the multiple sets of calibration data to obtain six-degree-of-freedom pose parameters of the camera; obtaining PTZ control parameters of the camera according to the coordinates and pose parameters of a target to be observed in the world coordinate system; and outputting the PTZ control parameters of the camera. This solution does not rely on external sensors, but relies on satellite maps for camera calibration and linkage, with a lower cost, and models the six-degree-of-freedom pose of the camera. Only a small amount of calibration data needs to be obtained to complete the calibration process, improving the camera calibration efficiency and control accuracy.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular, to a method and device for controlling a camera. Background Art

[0002] Observation cameras can directly obtain real-time and historical on-site situation information and have been widely used as an important means of modern supervision in various public transportation scenarios, such as highway, urban road, maritime port and other areas. Traditional supervision means mainly manually control the rotation angle and zoom ratio of a PTZ camera for manual observation, with low work efficiency and easy loss of the observed target. Therefore, currently, the camera intelligent linkage method is mostly adopted to automatically control the PTZ camera, which can obtain the detailed picture of the target scene more quickly and efficiently.

[0003] In the existing intelligent observation scenarios, the mapping relationship between the coordinate systems of externally fixed sensors (such as fixed cameras, radars) and the angles of the PTZ camera to be controlled is usually calibrated, and the input coordinates under the external sensor are converted into PTZ control commands. For example, in the fixed camera-PTZ camera linkage scheme, a target to be observed can be selected in a fixed camera, and the PTZ camera is turned to the target to be observed and tracked. This fixed camera-PTZ camera linkage scheme requires the combined deployment of a fixed camera and a PTZ camera, which requires additional hardware costs for some scenarios of reusing cameras. At the same time, the rotation angle of the PTZ camera is limited to the visible range of the fixed camera and cannot fully utilize its 360-degree rotation ability. Moreover, the fixed camera-PTZ camera linkage scheme mostly uses the grid method for calibration, which requires calibrating a large number of preset positions and estimating the rotation angle of the PTZ camera through interpolation, resulting in problems such as cumbersome calibration, inaccurate control angle, and low reliability.

[0004] Therefore, there is an urgent need in the industry for a method for controlling a camera to reduce the complexity of the camera calibration and camera linkage processes. Summary of the Invention

[0005] This application provides a method and device for controlling a camera to improve the calibration efficiency and control accuracy of the camera.

[0006] In a first aspect, a method for controlling a camera is provided, including: obtaining multiple groups of calibration data of the camera, each group of calibration data including: PTZ data output by the camera and satellite map data, the PTZ data including the pan value and tilt value of the camera corresponding to the feature point, and the satellite map data including the coordinates of the feature point in the world coordinate system; solving a camera pose estimation model according to the multiple groups of calibration data to obtain the six-degree-of-freedom pose parameters of the camera; obtaining the PTZ control parameters of the camera according to the coordinates of the target to be observed in the world coordinate system and the pose parameters; and outputting the PTZ control parameters of the camera.

[0007] In a second aspect, the present application provides a device for controlling a camera, including: a communication module, configured to obtain multiple sets of calibration data of the camera, each set of calibration data including: PTZ data output by the camera and satellite map data, the PTZ data including the pan value and tilt value of the camera corresponding to feature points, and the satellite map data including the coordinates of the feature points in the world coordinate system; a processing module, configured to solve a camera pose estimation model according to the multiple sets of calibration data to obtain six-degree-of-freedom pose parameters of the camera; the processing module is further configured to obtain PTZ control parameters of the camera according to the coordinates of a target to be observed in the world coordinate system and the pose parameters; the communication module is further configured to output the PTZ control parameters of the camera.

[0008] In a third aspect, a computer device is provided, including a processor, which is configured to call a computer program from a memory, and when the computer program is executed, the processor is configured to execute the method in the first aspect above.

[0009] In a fourth aspect, a computer-readable storage medium is provided, configured to store a computer program, and the computer program includes code for executing the method in the first aspect above.

[0010] In a fifth aspect, a computer program product is provided, including a computer program, and the computer program includes code for executing the method in the first aspect above.

[0011] In an embodiment of the present application, a solution for controlling a camera is provided. This solution does not rely on an external sensor as the input for camera calibration, but relies on a satellite map for camera calibration and linkage. It has a lower cost and can ensure full coverage angle control in the horizontal and vertical directions of the camera, and realizes zoom control within the maximum magnification range according to the distance between the target to be observed and the camera. This solution fully considers the installation state of the camera, models its six-degree-of-freedom pose, and only needs to obtain a small amount of calibration data to complete the calibration process, improving the efficiency of camera calibration and the accuracy of control. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0013] Figure 1 is a schematic diagram of an application scenario of an embodiment of the present application;

[0014] Figure 2 is a schematic diagram of an application scenario of another embodiment of the present application;

[0015] Figure 3 is a schematic diagram of the internal structure of a computing device 100 according to an embodiment of the present application;

[0016] Figure 4 It is a schematic diagram of the scene of the camera pose in an embodiment of the present application;

[0017] Figure 5 It is a schematic diagram of the scene of the camera pose in an embodiment of the present application;

[0018] Figure 6 It is a schematic flowchart of a method for controlling a camera in an embodiment of the present application;

[0019] Figure 7 It is a schematic structural diagram of device 700 in an embodiment of the present application;

[0020] Figure 8 It is a schematic structural diagram of device 800 in an embodiment of the present application. Detailed implementation manners

[0021] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0022] First, the nouns involved in the present application will be explained.

[0023] Six-degree-of-freedom pose: An object has six degrees of freedom in a three-dimensional coordinate system, including: translational degrees of freedom in the directions of the three rectangular coordinate axes x, y, and z and rotational degrees of freedom about the three axes.

[0024] Camera calibration: The process of constructing a mapping model from the world coordinate system to the pixel coordinate system and obtaining the parameters required in the model.

[0025] PTZ: In security monitoring applications, it is an abbreviation for pan / tilt / zoom, which respectively represent the left-right and up-down movements of the camera pan-tilt head and the lens zoom factor.

[0026] Pan-tilt head: It refers to the support device for installing and fixing a mobile phone, camera, or video camera. The pan-tilt head can rotate up and down or left and right to adapt to the application scenario.

[0027] World coordinate system: It refers to the absolute coordinate system of the system. The world coordinate system can be a geographic coordinate system, for example, the World Geodetic System (WGS84), or any fixedly defined world coordinate system.

[0028] Camera coordinate system: The camera coordinate system takes the optical center of the camera as the origin. The X-axis points to the left and right direction of the camera, the Y-axis points to the up and down direction of the camera, and the Z-axis points to the direction the camera observes. It changes with the movement of the camera, so it is a relative coordinate system.

[0029] Pixel coordinate system: The image pixel coordinate system is a plane rectangular coordinate system fixed on the image with pixels as the unit. Its origin is located at the upper left corner of the image. The X-axis and Y-axis are parallel to the X and Y axes of the camera coordinate system. It depends on the camera coordinate system and is a relative coordinate system.

[0030] Body coordinate system of the camera: Taking the optical center of the camera as the origin, the X-axis points to the direction the camera observes, the Y-axis is the left and right direction of the camera, and the Z-axis is the up and down direction of the camera.

[0031] Figure 1 It is a schematic diagram of the application scenario of an embodiment of the present application. As Figure 1 shown, in a public transportation video system, it usually includes observation cameras, a switch system, and a supervision center. Among them, the observation cameras are widely arranged at various traffic intersections, ports, etc. to obtain images or videos of vehicles or ships in the public transportation scenario, and transmit the obtained images or videos to the supervision center through the switch system. The supervision center usually includes display devices, storage devices, and computing devices. Among them, the storage device can be used to store the obtained image or video information, the display device is used to display the observation images, and the computing device can analyze and process the obtained images or videos. In some examples, the above computing device may include a server.

[0032] Optionally, as Figure 1 shown, the above observation cameras usually include two types: dome cameras and bullet cameras. A bullet camera is a bullet-type camera with a fixed installation position and can only face a certain observation position, so the observation azimuth is limited. A dome camera is a spherical camera that integrates a camera system, a zoom lens, and an electronic pan-tilt head. The dome camera can set preset cruises according to the on-site situation and user needs, and set the dwell time and lens zoom in at important points for key observation.

[0033] It should be understood that Figure 1 the description of the application scenario in

[0034] To solve the above technical problems of the prior art, an embodiment of the present application provides a solution for controlling a camera. This solution does not rely on an external sensor as the input for camera calibration, but relies on a satellite map for camera calibration and linkage. First, through the collected calibration data, a six-degree-of-freedom camera pose estimation model is determined, and the mapping relationship between the satellite map and the control angle of the camera pan-tilt is established and calibrated. Then, based on the position of the target to be observed on the satellite map, the rotation angle of the camera pan-tilt and the zoom ratio of the camera lens are calculated, so as to control the camera linkage in real time. This solution does not need to rely on an external sensor, and only needs to obtain less calibration data to complete the six-degree-of-freedom camera pose modeling, improving the calibration efficiency and control accuracy.

[0035] The following uses specific embodiments to elaborate in detail on the technical solution of the present application and how the technical solution of the present application solves the above technical problems. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0036] Figure 2 is a schematic diagram of the application scenario of another embodiment of the present application. As Figure 2 shown, the computing device 100 can obtain calibration data through the camera 200, and at the same time obtain satellite map data through the satellite positioning system. The computing device 100 can determine a six-degree-of-freedom camera pose estimation model according to the calibration data and the satellite map data. This model is used to establish and calibrate the mapping relationship between the satellite map and the control angle of the camera, and then based on the position of the target to be observed on the satellite map, the rotation angle of the camera pan-tilt and the zoom ratio of the camera lens are calculated, so as to control the camera linkage in real time.

[0037] The embodiment of the present application does not limit the type of satellite positioning system that provides the satellite map. For example, the satellite map can come from the Beidou positioning system, the Automatic Identification System (AIS), the Global Positioning System (GPS), etc.

[0038] It should be noted that the camera 200 in the embodiment of the present application can include a dome camera, a hemispherical camera, or any type of camera equipped with a pan-tilt.

[0039] Figure 3 is a schematic diagram of the internal structure of the computing device 100 in an embodiment of the present application. As Figure 3As shown in the figure, the computing device 100 includes a calibration data acquisition module 110, a calibration model solving module 120, an angle calculation module 130, and a magnification calculation module 140. The input information of the computing device 100 includes satellite map data and calibration data collected by the camera 100. The output information of the computing device 100 includes the PTZ control parameters of the camera. The above PTZ control parameters may include the angle parameters (e.g., pan and tilt) for controlling the camera pan-tilt head and the magnification parameter (zoom) of the camera lens.

[0040] The solution for controlling the camera in the embodiments of the present application mainly includes two parts: camera calibration and camera linkage. The input of the camera calibration part includes the PTZ values of the camera and satellite map data. The core modules of the camera calibration include a calibration data acquisition module 110 and a calibration model solving module 120, and the output of the camera calibration part is the pose parameters.

[0041] The input of the camera linkage part is the coordinate information and pose parameters of the target to be observed on the satellite map. The core modules include an angle calculation module 130 and a magnification calculation module 140. Among them, selecting the satellite map coordinates as the input can achieve stable and convenient acquisition. As an example, the acquisition methods of satellite map coordinates include vehicle GPS information, ship AIS data, or selecting the pixel coordinates of a certain point in the satellite map screenshot and converting them into longitude and latitude. Finally, the output of the computing device 100 is the PTZ control parameters of the camera, and the angle and magnification control operations of the camera can be completed through a preset camera control protocol.

[0042] Next, in combination with Figure 3 , the functions of each module in the computing device 100 will be described in detail.

[0043] A1. Calibration data acquisition module 110

[0044] It is used to obtain the calibration data required for camera calibration and provide input information for the subsequent calibration model solving module 120. Each camera can collect multiple groups of calibration data. Each group of calibration data includes the PTZ values of the camera corresponding to the feature points and the coordinate information of the feature points on the satellite map. Among them, the above camera PTZ values include the pan value and tilt value of the camera. As an example, each camera can collect at least 3 groups or more of data.

[0045] As an example, the zoom value of the camera can be set to 1, and the angle of the camera 100 can be adjusted until a feature point that can find a specific location in the satellite map appears in the displayed image. For example, the above feature point can be a marker, a building edge, etc., and the feature point is above the ground plane. After finding the feature point, in order to make the calibration data more accurate, the zoom value of the camera can be enlarged (for example, enlarged by 10 times or more), and the angle of the camera pan-tilt head can be finely adjusted until the feature point is at the center of the displayed image, and then take a screenshot and record the pan and tilt values of the camera pan-tilt head at this time, as well as the coordinates (x, y) of the feature point in the satellite map.

[0046] Among them, the coordinates (x, y) of the above satellite map can refer to the coordinates in the world coordinate system. In some examples, the coordinates (x, y) of the satellite map can be longitude and latitude coordinates.

[0047] A2. Calibration model solving module 120

[0048] The calibration model solving module 120 is used to establish a six-degree-of-freedom camera pose estimation model, and perform optimization and solution based on multiple groups of calibration data obtained by the calibration data acquisition module 110 to obtain pose parameters.

[0049] Figure 4 and Figure 5 are schematic diagrams of the camera pose in an embodiment of the present application. Among them, Figure 4 is a schematic diagram of the camera pan-tilt head not rotated, Figure 5 is a schematic diagram of the camera pan-tilt head rotated. Among them, the camera rotation angle is expressed as (pan, tilt).

[0050] Refer to Figure 4 and Figure 5 , the purpose of the calibration model solving module 120 is to obtain the pose parameters for converting the body coordinate system of the camera to the world coordinate system, including: the origin O b of the body coordinate system of the camera relative to the origin O w of the world coordinate system 0 translation vector (x 0 , y 0 ), and the angle vector (roll 0 , pitch 0 , yaw 0 ) of the three-axis rotation of the body coordinate system of the camera relative to the world coordinate system. Among them, roll 0 refers to the angle of rotation around the X w axis, pitch 0 refers to the angle of rotation around the Y w axis, and yaw 0 refers to the angle of rotation around the Z w axis.

[0051] The function of the calibration model solving module 120 includes two parts. The first part is to establish a camera pose estimation model, and the second part is to optimize and solve the model. Next, these two parts will be described.

[0052] 1). Establish a camera pose estimation model

[0053] Before establishing the camera pose estimation model, different modeling schemes can be selected according to whether there is an inclination when the camera is installed.

[0054] a). There is no inclination when the camera is installed

[0055] Continue to refer to Figure 4 , Figure 4 , which shows a schematic diagram of the camera without inclination when installed. Assuming that there is no inclination when the camera is installed, that is, the camera does not tilt in the front-back direction and the up-down direction, then roll 0 and pitch 0 are 0. When solving the model, only the values of x 0 , y 0 , z 0 , and yaw 0 need to be obtained. The camera pose estimation model can be established according to Formulas (1) and (2).

[0056] x = x 0 + z 0 * tan(tilt) * cos(yaw 0 + pan); (1)

[0057] y = y 0 + z 0 * tan(tilt) * sin(yaw 0 + pan); (2)

[0058] Among them, (x, y) represents the coordinates of the feature point in the world coordinate system, and (tilt, pan) represents the pan value and tilt value of the camera corresponding to when the feature point is located at the center of the camera image.

[0059] b). There is an inclination when the camera is installed

[0060] If there is an inclination when the camera is installed, then roll 0 and pitch 0 are not zero. The method of coordinate system transformation can be used to rotate the current body coordinate system to a body coordinate system without inclination. Specifically, the rotation angle relative to the body coordinate system without inclination can be obtained according to Formula (3):

[0061] q(roll', tilt', pan') = q(roll 0 , pitch 0 , yaw 0 ) * q(0, tilt, pan); (3)

[0062] Among them, (roll', tilt', pan') represents the rotation angle of the camera's body coordinate system relative to the world coordinate system when the feature point is at the center of the camera image, (roll 0 , pitch 0 , yaw 0 ) represents the angle vector of the three-axis rotation of the camera's body coordinate system relative to the world coordinate system of the tilted camera, and (0, tilt, pan) represents the pan value and tilt value of the corresponding camera when the feature point is at the center of the camera image, and also represents the rotation angle of the camera's body coordinate system relative to the tilted camera coordinate system when the feature point is at the center of the camera image.

[0063] q(m, n, l) represents the quaternion (w, x, y, z) constructed by the three-axis rotation angle, which can be represented by formula (4).

[0064]

[0065] After rotating the body coordinate system, the obtained tilt' and pan' from formula (3) can be substituted into formulas (5) and (6) to establish a camera pose estimation model.

[0066] x = x 0 + z 0 * tan(tilt') * cos(pan'); (5)

[0067] y = y 0 + z 0 * tan(tilt') * sin(pan'); (6)

[0068] 2). Optimize and solve the model

[0069] After establishing the camera pose estimation model, the camera pose estimation model can be used for solving to obtain the pose parameters (x 0 , y 0 , z 0 ) and (roll 0 , pitch 0 , yaw 0 ). The embodiments of the present application do not limit the way of solving the model. For example, if the camera pose estimation model solves a system of nonlinear equations, methods such as the interval iteration method or the fixed-point method can be used for solving.

[0070] In some examples, a function to be optimized for the least squares method can be constructed using a camera pose estimation model, and solved by the Gauss-Newton iteration method.

[0071] For example, for the case where the camera is not tilted during installation, that is, for the model established according to formulas (1) and (2), the following formulas (7) and (8) can be used to solve the model.

[0072]

[0073]

[0074] For another example, for the case where the camera is tilted during installation, that is, for the model established according to formulas (5) and (6), the following formulas (9) and (10) can be used to solve the model.

[0075] e 3 = ∑(x - x 0 - z 0 * tan(tilt') * cos(pan')) 2 + (y - y 0 - z 0 * tan(tilt') * sin(pan')) 2 ; (9)

[0076]

[0077] For another example, for a scenario where only the pan value of the camera changes, that is, a scenario where the camera only moves in the left-right direction, only the three degrees of freedom parameters x 0 、y 0 and yaw 0 need to be calculated. Regardless of whether the camera is tilted during installation, the following formula (11) can be used to solve the model.

[0078]

[0079] Among them, (x, y) represents the coordinates of the feature point in the world coordinate system, (tilt, pan) represents the pan value and tilt value of the camera corresponding to the feature point; (tilt', pan') represents the rotation angle of the body coordinate system of the camera relative to the world coordinate system when the feature point is located at the center of the camera image.

[0080] Among them, the functions e 1 、e 3 are applicable to scenarios where the camera is installed at a relatively high position and the tilt degree is relatively large during data acquisition. As an example, it can be applicable to scenarios where the tilt degree is greater than 1 degree. The functions e 2 、e 4Applicable to scenarios where the camera installation position is low or the tilt is severe, or the tilt value is small during data acquisition. For example, it can be applicable to scenarios where the tilt value is less than 1 degree in many cases. Function e 5 Applicable to scenarios where the dome camera installation position is very low and all tilt values are around 0 degrees during data acquisition, or scenarios where the tilt value is fixed and only the pan value needs to be controlled in a planar scene, or as e 1 , e 3 , e 3 , e 4 Initial value acquisition method of the function.

[0081] A3. Angle calculation module 130

[0082] Used to obtain the coordinates (x w , y w ) of the target to be observed in the world coordinate system, and use the camera pose estimation model and pose parameters obtained by the calibration model solving module 120 to output the pan and tilt angle parameters for controlling the camera pan-tilt head.

[0083] As an example, the pan value and tilt value for controlling the camera pan-tilt head can be obtained according to the following formulas (12) to (14).

[0084]

[0085]

[0086] q(roll w , tilt w , pan w ) = q -1 (roll 0 , pitch 0 , yaw 0 ) * q(0, tilt w , pan w '); (14)

[0087] Among them, (x w , y w ) represents the coordinates of the target to be observed in the world coordinate system; (tilt w , pan w ) represents the pan value and tilt value of the camera corresponding to the target to be observed, that is, the pan value and tilt value of the camera when the target to be observed is located at the center of the camera image; (roll w , tilt w , pan w ) represents the rotation angle of the body coordinate system of the camera relative to the world coordinate system when the target to be observed is located at the center of the camera image.

[0088] A4. Zoom calculation module 140

[0089] For obtaining the coordinates (x, y) of the target to be observed, and calculating the zoom factor zoom for controlling the camera 200 based on the distance between the coordinates (x, y) and the calibrated camera position (x 0 , y 0 ).

[0090] As an example, the zoom factor zoom for controlling the camera 200 can be obtained according to the following formula (15).

[0091]

[0092] Where (x w , y w ) represents the coordinates of the target to be observed in the world coordinate system, s represents the scale of the satellite map, which is used to convert the coordinate system used by the satellite map to the metric scale, and k represents the empirical coefficient of the magnification factor, which can be adjusted according to the size of the target to be observed. For example, it can be defaulted to double the magnification for every 100-meter distance.

[0093] Finally, the computing device 100 can send the PTZ control parameters (pan w , tilt w , zoom) of the camera to the camera 200 according to the preset camera control protocol, so as to control the camera 200 to complete functions such as turning and zooming, and realize the camera linkage for the target to be observed.

[0094] In the embodiment of the present application, a solution for controlling a camera is provided. This solution does not rely on external sensors as the input for camera calibration, but relies on satellite maps for camera calibration and linkage, with lower costs, and can ensure the full coverage angle control of the camera horizontally and vertically, and realize the zoom control within the maximum magnification range according to the distance between the target to be observed and the camera. This solution fully considers the installation state of the camera, models its six-degree-of-freedom pose, and only needs to obtain a small amount of calibration data to complete the calibration process. For example, at least only 3 groups of calibration data are required, improving the camera calibration efficiency and control accuracy.

[0095] In addition, in the embodiment of the present application, a variety of optimization functions applied to the calibration process are designed, which can be flexibly adapted to different camera installation scenarios with different heights and postures, improving the calibration efficiency and application flexibility.

[0096] Figure 6 It is a schematic flowchart of the method for controlling a camera according to an embodiment of the present application. This method can be executed by the Figures 1 to 3 computing device in. Figure 6 As shown, this method includes the following contents.

[0097] S601. Obtain multiple sets of calibration data. Each set of calibration data includes: PTZ data output by the camera and satellite map data. The PTZ data includes the pan value and tilt value of the camera corresponding to the feature point, and the satellite map data includes the coordinates of the feature point in the world coordinate system.

[0098] As an example, each camera can collect at least 3 sets or more of data.

[0099] As an example, the zoom value of the camera can be set to 1, and the angle of the camera 100 can be adjusted until a feature point that can find a specific position in the satellite map appears in the displayed image. For example, the above-mentioned feature point can be a marker, the edge of a building, etc., and the feature point is above the ground plane. After finding the feature point, in order to make the calibration data more accurate, the zoom value of the camera can be enlarged (for example, enlarged by 10 times or more), and the angle of the camera pan-tilt head can be finely adjusted until the feature point is at the center of the displayed image, and then take a screenshot and record the pan and tilt values of the camera pan-tilt head at this time, as well as the coordinates (x, y) of the feature point in the satellite map.

[0100] Among them, the coordinates (x, y) of the above-mentioned satellite map can refer to the coordinates in the world coordinate system. In some examples, the coordinates (x, y) of the satellite map can be longitude and latitude coordinates.

[0101] S602. Solve the camera pose estimation model according to multiple sets of calibration data to obtain the six-degree-of-freedom pose parameters of the camera.

[0102] Among them, the camera pose estimation model is used to obtain the pose parameters for converting the body coordinate system of the camera to the world coordinate system. The pose parameters include the origin O b of the body coordinate system of the camera relative to the origin O w of the world coordinate system 0 translation vector (x 0 , y 0 ), and the angle vector (roll 0 , pitch 0 , yaw 0 ) of the three-axis rotation of the body coordinate system of the camera relative to the world coordinate system.

[0103] Among them, for the specific principle of the establishment and solution of the camera pose estimation model, reference can be made to the relevant description in Figures 3 to 5 . For the sake of brevity, it will not be elaborated here.

[0104] Optionally, in the S602 part, if the installation of the camera does not produce tilt, the camera pose estimation model meets the following conditions:

[0105] x = x 0 + z0 *tan(tilt)*cos(yaw 0 +pan); (1)

[0106] y = y 0 +z 0 *tan(tilt)*sin(yaw 0 +pan); (2)

[0107] roll 0 = 0; pitch 0 = 0;

[0108] where (x, y) represents the coordinates of the feature point in the world coordinate system, and (tilt, pan) represents the pan value and tilt value of the camera corresponding to when the feature point is at the center of the camera image;

[0109] If the camera is installed with an inclination, the camera pose estimation model meets the following conditions:

[0110] x = x 0 +z 0 *tan(tilt')*cos(pan'); (5)

[0111] y = y 0 +z 0 *tan(tilt')*sin(pan'); (6)

[0112] where (tilt', pan') represents the rotation angle of the body coordinate system of the camera relative to the world coordinate system when the feature point is at the center of the camera image.

[0113] Optionally, (tilt', pan') meets the following conditions:

[0114] q(roll', tilt', pan') = q(roll 0 , pitch 0 , yaw 0 ); (3)

[0115] where (roll', tilt', pa)n' represents the rotation angle of the body coordinate system of the camera relative to the world coordinate system when the feature point is at the center of the camera image, and (roll 0 , pitc 0 h, ya 0 w) represents the angle vector of the three-axis rotation of the body coordinate system of the tilted camera relative to the world coordinate system (roll 0 , pitch 0 , yaw 0 );

[0116] q(m,n,l) represents the quaternion (w,x,y,z) constructed by the three-axis rotation angles, which satisfies the following conditions:

[0117]

[0118] In part S602, the embodiments of the present application do not limit the way of solving the model. For example, the camera pose estimation model solves a system of nonlinear equations, and methods such as the interval iteration method or the fixed-point method can be used for solving. Or, a function to be optimized by the least squares method is constructed using the camera pose estimation model, and the pose parameters are obtained by an iterative method.

[0119] As an example, if the camera is installed without tilt, the function to be optimized by the least squares method is constructed using the following formula for solving:

[0120]

[0121]

[0122] As an example, if the camera is installed with tilt, the function to be optimized by the least squares method is constructed using the following formula for solving:

[0123] e 3 = ∑(x - x 0 - z 0 * tan(tilt') * cos(pan')) 2 +(y - y 0 - z 0 * tan(tilt') * sin(pan')) 2 ; (9)

[0124]

[0125] As an example, if the camera only controls the change of the pan value, the function to be optimized by the least squares method is constructed using the following formula for solving:

[0126]

[0127] where (x, y) represents the coordinates of the feature point in the world coordinate system, (tilt, pan) represents the pan value and tilt value of the camera corresponding to the feature point; (tilt', pan') represents the rotation angle of the body coordinate system of the camera relative to the world coordinate system when the feature point is located at the center of the camera image.

[0128] where the function e 1 、e 3Applicable to scenarios where the camera installation position is relatively high and the tilt degree is relatively large during data acquisition. As an example, it can be applicable to scenarios where the tilt degree is greater than 1 degree. Function e 2 e 4 Applicable to scenarios where the camera installation position is relatively low, the tilt is relatively severe, or the tilt value is relatively small during data acquisition. For example, it can be applicable to scenarios where many tilt values are less than 1 degree. Function e 5 Applicable to scenarios where the dome camera installation position is very low, all tilt values are around 0 degrees during data acquisition, or scenarios where the tilt value is fixed and only the pan value needs to be controlled, or as e 1 e 3 e 3 e 4 Initial value acquisition method of the function.

[0129] In the embodiments of the present application, by designing a variety of optimization functions applied to the calibration process, different camera installation scenarios with different heights and postures can be flexibly adapted, improving the calibration efficiency and application flexibility.

[0130] S603. Obtain the PTZ control parameters of the camera according to the coordinates and pose parameters of the target to be observed in the world coordinate system.

[0131] Optionally, the PTZ control parameters of the camera include at least one of the following: the pan value, tilt value, and zoom value of the camera corresponding to the target to be observed.

[0132] Optionally, in part S603, obtaining the PTZ control parameters of the camera according to the coordinates and pose parameters of the target to be observed in the world coordinate system includes:

[0133] Obtain the pan value and tilt value of the camera corresponding to the target to be observed according to the following formula:

[0134]

[0135]

[0136] q(roll w ',tilt w ,pan w ) = q -1 (roll 0 ,pitch 0 ,yaw 0 ) * q(0,tilt w ',pan w ); (14)

[0137] Wherein, (x w ,y w) represents the coordinates of the target to be observed in the world coordinate system, (tilt w , pan w ) represents the pan value and tilt value of the camera corresponding to the target to be observed; (roll w ', tilt w ', pan w ) represents the rotation angle of the body coordinate system of the camera relative to the world coordinate system when the target to be observed is located at the center of the camera image.

[0138] And, according to the following formula, obtain the zoom value of the camera corresponding to the target to be observed:

[0139]

[0140] Wherein, zoom represents the zoom value of the camera corresponding to the target to be observed, (x w , y w ) represents the coordinates of the target to be observed in the world coordinate system, s represents the scale of the satellite map, and k represents the empirical coefficient of the magnification ratio.

[0141] S604. Output the PTZ control parameters of the camera.

[0142] Finally, the computing device can, according to the preset camera control protocol, send the PTZ control parameters (pan w , tilt w , zoom) of the camera to the camera 200 to control the camera to complete functions such as turning and zooming, and realize the camera linkage for the target to be observed.

[0143] In the embodiment of the present application, a scheme for controlling a camera is provided. This scheme does not rely on an external sensor as the input for camera calibration, but relies on a satellite map for camera calibration and linkage, with lower costs, and can ensure the full coverage angle control in the horizontal and vertical directions of the camera, and achieve variable magnification control within the maximum magnification range according to the distance between the target to be observed and the camera. This scheme fully considers the installation state of the camera, models its six-degree-of-freedom pose, and only needs to obtain a small amount of calibration data to complete the calibration process, improving the camera calibration efficiency and control accuracy.

[0144] Figure 7 It is a schematic structural diagram of a device 700 according to an embodiment of the present application. The device 700 is used to execute the method executed by the computing device 100 in the above text.

[0145] The device 700 includes a communication module 710 and a processing module 720. The device 700 is used to implement the operations executed by the computing device 100 in the above method embodiments.

[0146] For example, the communication module 710 is used to obtain multiple sets of calibration data. Each set of calibration data includes: PTZ data output by the camera and satellite map data. The PTZ data includes the pan value and tilt value of the camera corresponding to the feature point, and the satellite map data includes the coordinates of the feature point in the world coordinate system. The processing module 720 is used to solve the camera pose estimation model according to the multiple sets of calibration data to obtain the six-degree-of-freedom pose parameters of the camera. The processing module 720 is further used to obtain the PTZ control parameters of the camera according to the coordinates and pose parameters of the target to be observed in the world coordinate system. The communication module 710 is further used to output the PTZ control parameters of the camera.

[0147] Figure 8 FIG. 4 is a schematic structural diagram of a device 800 according to an embodiment of the present application. The device 800 is used to execute the method performed by the computing device 100 in the foregoing text.

[0148] The device 800 includes a processor 810. The processor 810 is used to execute the computer program or instruction stored in the memory 820, or read the data stored in the memory 820 to execute the methods in the foregoing method embodiments. Optionally, the processor 810 is one or more.

[0149] Optionally, as Figure 8 shown, the device 800 further includes a memory 820. The memory 820 is used to store computer programs or instructions and / or data. The memory 820 can be integrated with the processor 810 or can be separately provided. Optionally, the memory 820 is one or more.

[0150] Optionally, as Figure 8 shown, the device 800 further includes a communication interface 830. The communication interface 830 is used for receiving and / or sending signals. For example, the processor 810 is used to control the communication interface 830 to receive and / or send signals.

[0151] Optionally, the device 800 is used to implement the operations performed by the computing device 100 in the foregoing method embodiments.

[0152] For example, the processor 810 is used to execute the computer program or instruction stored in the memory 820 to implement the operations related to the computing device 100 in the foregoing method embodiments.

[0153] It should be noted that Figure 8 the device 800 in FIG. 4 can be the computing device 100 in the foregoing embodiment, or a component (such as a chip) of the computing device 100, which is not limited herein.

[0154] In the embodiments of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor can be a circuit with the ability to read and execute instructions, such as a CPU, a microprocessor, a GPU (which can be understood as a type of microprocessor), or a DSP, etc. In another implementation, the processor can implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of this hardware circuit is fixed or can be reconfigured. For example, the processor is a hardware circuit implemented by an ASIC or a PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the configuration of the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as a type of ASIC, such as an NPU, a TPU, a DPU, etc.

[0155] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method. For example: a CPU, a GPU, an NPU, a TPU, a DPU, a microprocessor, a DSP, an ASIC, an FPGA, or a combination of at least two of these processor forms.

[0156] In addition, each unit in the above device can be integrated in whole or in part, or can be independently implemented. In one implementation, these units are integrated together and implemented in the form of a system-on-a-chip (SOC). The SOC can include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The types of the at least one processor can be different. For example, it includes a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0157] Correspondingly, the embodiments of the present application also provide a computer-readable storage medium storing a computer program, which, when the computer program / instructions are executed by a processor, causes the processor to implement Figures 2 to 6 the steps in the method executed by the computing device 100 in

[0158] Correspondingly, the embodiments of the present application also provide a computer program product, including computer program / instructions, which, when the computer program / instructions are executed by a processor, causes the processor to implement Figures 2 to 6 the steps in the method executed by the computing device 100 in

[0159] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0160] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or a plurality of flows and / or blocks

[0161] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in Figure 1 one or more of the flows Figure 1 or a plurality of flows and / or blocks

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or a plurality of flows and / or blocks

[0163] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0164] The memory may include non-permanent memory in the computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

[0165] A computer-readable medium includes both permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0166] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or apparatus comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising the above elements.

[0167] The above are only embodiments of the present application and are not used to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A method for controlling a camera, characterized in that, comprising: obtaining multiple sets of calibration data of the camera, each set of calibration data including: PTZ data output by the camera and satellite map data, the PTZ data including the pan value and tilt value of the camera corresponding to the feature point, and the satellite map data including the coordinates of the feature point in the world coordinate system; solving a camera pose estimation model according to the multiple sets of calibration data to obtain the six-degree-of-freedom pose parameters of the camera; the camera pose estimation model is used to obtain the pose parameters for converting the body coordinate system of the camera to the world coordinate system; obtaining the PTZ control parameters of the camera according to the coordinates of the target to be observed in the world coordinate system and the pose parameters; outputting the PTZ control parameters of the camera; The pose parameters include the origin O of the body coordinate system of the camera b relative to the origin O of the world coordinate system w translation vector (x 0 , y 0 , z 0 ), and the angle vector (roll 0 , pitch 0 , yaw 0 ) of the three-axis rotation of the body coordinate system of the camera relative to the world coordinate system; if the installation of the camera does not produce tilt, the camera pose estimation model meets the following conditions: ； ； roll 0 = 0; pitch 0 = 0; wherein, (x, y) represents the coordinates of the feature point in the world coordinate system, and (tilt, pan) represents the pan value and tilt value of the camera corresponding to when the feature point is located at the center of the camera image; if the installation of the camera produces tilt, the camera pose estimation model meets the following conditions: ； ； wherein, (tilt', pan') represents the rotation angle of the body coordinate system of the camera relative to the world coordinate system when the feature point is located at the center of the camera image.

2. The method according to claim 1, characterized in that, the solving the camera pose estimation model according to the multiple sets of calibration data to obtain the six-degree-of-freedom pose parameters of the camera includes: constructing an optimization function to be optimized by the least squares method using the camera pose estimation model, and solving it by an iterative method to obtain the pose parameters.

3. A device for controlling a camera, characterized in that, comprising: a communication module for obtaining multiple sets of calibration data of the camera, each set of calibration data including: PTZ data output by the camera and satellite map data, the PTZ data including the pan value and tilt value of the camera corresponding to the feature point, and the satellite map data including the coordinates of the feature point in the world coordinate system; a processing module for solving a camera pose estimation model according to the multiple sets of calibration data to obtain the six-degree-of-freedom pose parameters of the camera; the camera pose estimation model is used to obtain the pose parameters for converting the body coordinate system of the camera to the world coordinate system; the processing module is further configured to obtain the PTZ control parameters of the camera according to the coordinates of the target to be observed in the world coordinate system and the pose parameters; the communication module is further configured to output the PTZ control parameters of the camera; The pose parameters include the origin O of the body coordinate system of the camera b relative to the origin O of the world coordinate system w translation vector (x 0 , y 0 , z 0 ), and the angle vector (roll 0 , pitch 0 , yaw 0 ) of the three-axis rotation of the body coordinate system of the camera relative to the world coordinate system; if the installation of the camera does not produce tilt, the camera pose estimation model meets the following conditions: ； ； roll 0 =0; pitch 0 =0; wherein, (x, y) represents the coordinates of the feature point in the world coordinate system, and (tilt, pan) represents the pan value and tilt value of the camera corresponding to when the feature point is located at the center of the camera image; if the installation of the camera produces tilt, the camera pose estimation model meets the following conditions: ； ； Among them, (tilt', pan') represents the rotation angle of the body coordinate system of the camera relative to the world coordinate system when the feature point is located at the center of the camera image.

4. The device according to claim 3, wherein, in terms of solving the camera pose estimation model according to the multiple sets of calibration data to obtain the six-degree-of-freedom pose parameters of the camera, the processing module is specifically configured to: construct an optimization function to be optimized by the least squares method using the camera pose estimation model, and solve it through an iterative method to obtain the pose parameters.

5. An electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 2.

6. A computer-readable storage medium, wherein, computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 2.

7. A computer program product, wherein, the computer program product includes a computer program, and when the computer program is executed by a processor, it is used to implement the method according to any one of claims 1 to 2.

Images