Camera Calibration Methods

Abstract

A method for calibrating a camera is disclosed. The method includes obtaining geographic coordinates of a selected physical point position within a camera image view and measuring an angle between an x-axis of a real-world coordinate system passing through the selected point position and true north. The method includes deriving corrected projection data using the obtained geographic coordinates, the measured angle, and projection data derived from characteristics of the camera for converting a two-dimensional pixel coordinate system of the camera image view into a three-dimensional geographic coordinate system of the point position in the camera image view.

Description

Technical Field

[0001] The invention relates to a camera calibration method to obtain the absolute position coordinates of an object within an image view of a camera or in an image taken by a camera. Background Art

[0002] Cameras are widely installed to perform different types of smart city applications, such as people / vehicle tracking, generating traffic statistics, vehicle speed detection, etc. Visual localization is a technique that provides the geographic location of objects within a region of interest by using roadside camera image views or images captured by these cameras. Many applications, such as autonomous vehicle control and automatic parking, can be further developed using the detected geographic location of objects within the region of interest. The improvement in localization accuracy and the reduction in cost are enhancing the demand for such visual localization systems, especially in smart city environments. However, known camera calibration algorithms often exhibit one or more of the following problems, such as the camera needs to have a movable field of view or two or more cameras are required to cover the same area. Other problems that may arise are that the output position data or the coordinates of the observed object are relative to the camera reference system rather than absolute position coordinates, such as global position geographic coordinates. In addition, the known calibration methods can be difficult to implement, time-consuming, and have low accuracy when determining the relative position coordinates of the observed object.

[0003] WO2014 / 068073 discloses an optical measuring device for determining the three-dimensional (3D) coordinates of an object. The measuring device includes a projection device for illuminating the object with at least one predetermined pattern; at least one camera for capturing a two-dimensional (2D) image of the pattern reflected from the object; and a computing device for measuring a brightness value sequence of at least one two-dimensional image point from the two-dimensional image, and calculating the three-dimensional coordinates of the object point associated with the measured brightness value sequence of the two-dimensional image point. The output three-dimensional coordinates are relative coordinates, and multiple cameras are required to determine the three-dimensional position coordinates of the observed object.

[0004] US9679382 discloses a method for geo-referencing a first image of a scene obtained from a first imaging device based on at least one second image of the scene obtained from a second imaging device. The method includes obtaining data indicating qualified parameters of one or more regions of the scene; selecting one or more pivot regions in the one or more regions of the scene, wherein the qualified parameters of the pivot regions meet a predetermined criterion; for at least some of the selected pivot regions, determining the connection points of the first and second images; and solving the external orientation of the first image using the determined connection points and the first imaging device model. In addition, the system requires more than one camera and requires a digital elevation model (DEM) map, which is time-consuming to obtain or generate.

[0005] US8934008 discloses a method for determining a geographic location in an image. Calibration is performed by using multiple images taken from different locations.

[0006] CN111461994 discloses a method for obtaining the pixel position and absolute coordinates of a reference point in an image using a transformation matrix based on camera's Global Positioning System (GPS) data.

[0007] The publication entitled "Coordinate Rotation Derivations RevE: Coordinate Transformations via Euler Angle Rotations" by J. Riggs, published on April 10, 2019, discloses an example of a rotation matrix for rotating an angle along an axis. It also discloses the transformation from a local horizontal (North-East-Down Coordinate System, NED) coordinate system to an Earth-Centered, Earth-Fixed (ECEF) coordinate system.

[0008] A December 2020 publication by Sandia National Laboratories titled “Stripmap SAR Data Collection Geometry on a Round Earth” discloses examples of conversion from the ECEF coordinate system to an absolute coordinate system, such as the World Geodetic System 1984 (WGS84).

[0009] An improved camera calibration method is needed to obtain high-precision absolute coordinates.

[0010] Purpose of the Invention

[0011] It is an object of the present invention to alleviate or obviate to some extent one or more problems associated with known camera calibration methods for determining the position of an object.

[0012] The above objects are achieved by the feature combination of the main claim; the dependent claims disclose further advantageous embodiments of the invention.

[0013] Another object of the present invention is to provide a novel camera system.

[0014] Another object of the present invention is to provide a method for obtaining the absolute position coordinates of an object in an image view of a camera.

[0015] Those skilled in the art will derive other objects of the present invention from the following description. Therefore, the above object statements are not exhaustive and are only used to illustrate some of the multiple objects of the present invention. Summary of the invention

[0016] The present invention relates to a method for calibrating a camera and a method for determining the absolute position coordinates of an object within a camera image view using the image data of the camera. The method comprises: obtaining the geographic coordinates of a selected physical point position within the camera image view, and measuring the angle between the x-axis of a real-world coordinate system passing through the selected point position and true north. The method comprises: using the obtained geographic coordinates, the measured angle and projection data derived from the characteristics of the camera to derive corrected projection data for converting a two-dimensional pixel coordinate system of the camera image view or image data into a three-dimensional geographic coordinate system of a point position in the camera image view or image data. The conversion is one-to-one based on the assumption that all objects are located on the Z=0 plane of the real-world coordinate system.

[0017] In a first main aspect, the present invention provides a method for calibrating a camera, comprising the steps of obtaining geographic coordinates of a selected point location within a camera image view; measuring the angle between an x-axis of a real-world coordinate system and true north, the x-axis passing through the selected point location; and deriving corrected projection data using the obtained geographic coordinates, the measured angle, and projection data derived from characteristics of the camera to transform a two-dimensional pixel coordinate system of the camera image view into a three-dimensional geographic coordinate system of the point location in the camera image view.

[0018] In a second main aspect, the present invention provides a camera comprising: a memory storing machine-readable instructions; and a processor for executing the machine-readable instructions, so that when the processor executes the machine-readable instructions, it configures the camera to: receive data including geographic coordinates of a selected point location within a camera image view; receive data including a measured angle between an x-axis of a real-world coordinate system and true north, the x-axis passing through the selected point location; and use the received data including the geographic coordinates, the received data including the measured angle, and projection data derived from characteristics of the camera to derive corrected projection data for converting a two-dimensional pixel coordinate system of the camera image view into a three-dimensional geographic coordinate system of the point location in the camera image view.

[0019] In a third main aspect, the present invention provides a method for determining the geographic coordinates of a point location within a camera image view, comprising the following steps: acquiring an image from a camera; selecting a point location in the image; obtaining two-dimensional pixel coordinates of the selected point location in the image; and converting the two-dimensional pixel coordinates of the selected point location in the image into three-dimensional geographic coordinates of the selected point location using projection data that maps the two-dimensional pixel coordinate system of the camera image view to a three-dimensional geographic coordinate system.

[0020] This summary does not necessarily disclose all features necessary to define the invention; the invention may lie in a sub-combination of the disclosed features.

[0021] The features of the present invention have been outlined quite broadly above so that the following detailed description of the invention may be better understood. Other features and advantages of the present invention will be described hereinafter, which form the subject of the claims of the present invention. It will be appreciated by those skilled in the art that the concepts and specific embodiments disclosed may be readily used as a basis for modifying or designing other structures to achieve the same objectives of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and further features of the present invention will be apparent from the following description of preferred embodiments, which are provided by way of example only with reference to the accompanying drawings, in which:

[0023] Figure 1 is a block diagram of a smart city scenario, in which a camera according to the present invention can implement the method of the present invention;

[0024] Figure 2 is a schematic block diagram of a camera improved according to the present invention;

[0025] Figure 3 is a flow chart of a set of methods according to the present invention;

[0026] Figure 4It is a schematic diagram of the relationship between the camera's three-dimensional coordinate system and the camera's two-dimensional pixel coordinate system;

[0027] Figure 5 It is a schematic diagram of the relationship between the three-dimensional real-world coordinate system of the camera's location and the camera's three-dimensional coordinate system;

[0028] Figure 6 Shows methods for obtaining camera extrinsics;

[0029] Figure 7 Shows methods for obtaining intrinsic characteristics of a camera;

[0030] Figure 8 Showing the camera calibration method according to the present invention;

[0031] Fig. 9 and 10 A calibration verification method according to the present invention is shown. DETAILED DESCRIPTION

[0032] The following description describes preferred embodiments by way of example only and does not limit the combination of features necessary to implement the invention.

[0033] References to "one embodiment" or "an embodiment" in this specification mean that a particular feature, structure, or characteristic associated with that embodiment is included in at least one embodiment of the present invention. The phrase "in one embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive of other embodiments. In addition, various features are described that may be exhibited by some embodiments and not by others. Similarly, various requirements are described that may be requirements of some embodiments and not others.

[0034] It should be understood that the elements shown in the figure can be implemented in various forms of hardware, software or a combination thereof. These elements can be implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, memory and input / output interface.

[0035] This specification illustrates the principles of the invention. It will therefore be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope.

[0036] In addition, the principles, aspects and embodiments of the present invention and specific examples thereof are described herein, and are intended to cover their structural and functional equivalents. In addition, such equivalents also include currently known equivalents and equivalents developed in the future, i.e., any elements developed, performing the same function, no matter how their structure.

[0037] Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herein represent conceptual views of systems and devices embodying the principles of the invention.

[0038] The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software together with appropriate software. When provided by a processor, these functions may be provided by a single dedicated processor, a single shared processor, or multiple separate processors, some of which may be shared. In addition, the explicit use of the term "processor" or "controller" should not be interpreted as referring only to hardware capable of executing software, and may implicitly include, but is not limited to, digital signal processor ("DSP") hardware, read-only memory ("ROM") for storing software, random access memory ("RAM"), and non-volatile memory.

[0039] In the claims herein, any element represented as a means for performing a particular function is intended to encompass any manner of performing that function, including, for example, a) a combination of circuit elements that perform that function or b) any form of software, thus including firmware, microcode, etc., combined with appropriate circuitry to execute that software to perform the function. The invention as defined by these claims resides in the functionality provided by the various recited means, combined and brought together in the manner required by the claims. Any means for providing those functions is therefore considered equivalent to the means shown herein.

[0040] One of the most commonly used sensors in the development of smart cities is the surveillance camera. Millions of such cameras are being installed in cities and towns around the world to perform functions such as human / vehicle tracking and location, speed detection, providing data to vehicles and pedestrians through appropriately enabled electronic devices, providing data to central control systems, etc. Such surveillance cameras can be effectively used to assist in traffic control and policing, pedestrian safety, and automatic parking, to identify just a few of the functions. However, some of these functions require vehicle and / or pedestrian position data to be highly accurate within a reference frame. In many existing systems, the relative reference frame of the camera provides position data of vehicles and / or pedestrians relative to the camera position, rather than absolute position data such as GPS geographic coordinates. The usefulness of such surveillance cameras would be greatly enhanced if they were adjusted to provide absolute position data of vehicles and / or pedestrians or any object that falls within the camera's field of view.

[0041] One of the difficulties in achieving the above goal is that it is difficult to calibrate such cameras to provide absolute position data of objects in the camera's image view.

[0042] In one of its main aspects, the present invention provides a novel and simple method for calibrating the relationship between the real world coordinate system of the camera location and an absolute coordinate system (such as a three-dimensional geographic coordinate system). This in turn enables the two-dimensional pixel coordinate system of the camera image view to be calibrated to a three-dimensional geographic coordinate system, i.e., an absolute positioning system.

[0043] The method may include calibrating one or both of intrinsic characteristics of the camera and extrinsic characteristics of the camera if these characteristics are not already known.

[0044] Additionally, the method may include a novel and simple method for validating the calibration results that requires only the same tools used for the main calibration step of the method.

[0045] The main purpose of the method of the first main aspect of the invention is to transform a two-dimensional pixel coordinate system of a camera image view into a three-dimensional geographic coordinate system of any selected point location in a plane (eg the ground) within the camera image view.

[0046] Figure 1 A smart city scene 100 is included, wherein one or more cameras 101 are appropriately enabled according to the present invention to capture and provide image data, from which absolute position data of objects 102 within the camera view are obtained, such as global positioning system (GPS) data, etc. Each camera 101 has a designated camera view, for example, a field of view (FOV) of a portion of the city scene 100, which in this example includes a road 103 with a crosswalk 104.

[0047] exist Figure 1 In the example of , the camera 101 is mounted on a streetlight 105, but it should be understood that the camera 101 can be mounted on any suitable structure, including a building, etc. The camera views can overlap, although this is not necessary, but the camera overlap area is useful for object recognition and tracking across cameras. Each camera 101 is suitable for obtaining image data of its respective camera image view, from which the absolute position data of the object 102 located within the camera image view can be obtained. Each camera 101 can enable object recognition software to enable it to recognize and classify the objects 102 within its camera view, but this is not the focus of the present invention. Each camera 101 is preferably able to select a point position on each object 102 in a plane (such as the ground) within its camera view, and the absolute position data of the point can be obtained from the camera image data. Preferably, the point position is selected at the bottom of one end of the object 102 closest to the camera 101 and at the center of the bottom of the object, and the absolute position data of the selected point position represents the absolute position of the object 102.

[0048] Each camera 101 may be enabled to autonomously determine absolute position data of each object 102 within its camera view and report the data to other entities in the smart city system, such as reporting the data to vehicle-based electronic devices, pedestrian handheld electronic devices, other system sensors including other cameras 101, a system gateway (not shown), and one or more system central controllers 106. Additionally or alternatively, the cameras 101 may each be connected to a system server 107 and a database 108 via a suitable communication network 109, such as an Ethernet network, wherein the cameras 101 are capable of providing camera image data to the server 107 and / or the database 108 via a real-time encoded video stream using a video compression method (such as the known video compression standard for high-definition digital video known as "H.264"). The server 107 may be configured to determine the absolute position data of the objects 102 within the camera view from the camera image data provided by each camera 101, store the absolute position data in the database 108, and transmit the absolute position data to other smart city system entities as needed.

[0049] Figure 2 An exemplary embodiment of a camera 101 improved according to the concept of the present invention is shown. In the illustrated embodiment, the camera 101 may include a communication device 110, such as a network node, a network card or a network circuit operating in a 5G communication system environment, but the improved camera 101 of the present invention is not limited to operating in a 5G communication system, but may include a communication device 110 for operating in any suitable communication network 109.

[0050] The camera 101 may include a plurality of functional blocks for performing its various functions. For example, the camera 101 may include a transceiver module 111 for sending and receiving data signals from other smart city system entities. The transceiver module 111 provides signal processing and is configured to provide signals and / or information extracted therefrom to a functional block module 112, which may include, for example, various data sinks, control elements, user interfaces, etc. Regardless of the specific configuration of the transceiver module 111, an embodiment includes a data processing module 113, which may be provided in conjunction with the transceiver module 111 to facilitate accurate processing of data according to the present invention.

[0051] Although the data processing module 113 is shown as being deployed as part of the transceiver module 111 (e.g., including part of the transceiver module control and logic circuits), there is no limitation on such deployment configuration according to the concepts of the present invention. For example, the data processing module 113 can be deployed as an independent functional block of the camera 101, which is different from but connected to the transceiver module 111. For example, the data processing module 113 can be implemented using logic circuits and / or executable code / machine readable instructions stored in the memory 114 of the camera 101 so as to be executed by the processor 115 to perform the functions described herein. For example, the executable code / machine readable instructions can be stored in one or more memories 114 (e.g., random access memory (RAM), read-only memory (ROM), flash memory, magnetic storage, optical storage, etc.), which are suitable for storing one or more instruction sets (e.g., application software, firmware, operating system, applets, etc.), data (e.g., configuration parameters, operating parameters and / or thresholds, collected data, processed data, etc.). The one or more memories 114 may include processor-readable memory for one or more processors 115 to operably execute code segments of the data processing module 113 and / or utilize data provided thereby to perform the functions of the data processing module 113 described herein. Additionally or alternatively, the data processing module 113 may include one or more dedicated processors (e.g., application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), graphics processing units (GPUs), and / or similar configurations) to perform the functions of the data processing module 113 described herein.

[0052] Figure 3 is a flow chart of a set of methods 120 implemented by the data processing module 113 of the camera 101 according to the first main aspect of the invention. The set of methods 120 comprises a first optional method 121 of calibrating intrinsic characteristics of the camera 101. The method 121 calibrates intrinsic characteristics of the camera 101, which include focal length of one or more camera lenses, camera optical center and optionally radial and tangential distortion coefficients of the camera lenses, from which a projective transformation of the camera 3D coordinate system into the camera 2D pixel coordinate system can be obtained for a camera image view or captured image.

[0053] The set of methods 120 comprises a second optional method 122 for calibrating the extrinsic characteristics of the camera 101. The method 122 calibrates the extrinsic characteristics of the camera 101 by calibrating the 3D real world coordinate system where the camera 101 is located with the 3D camera coordinate system.

[0054] The main non-optional method 123 of the present invention includes: calibrating the relationship between the three-dimensional real-world coordinate system of the location of the camera 101 and the absolute three-dimensional coordinate system, which will be described below.

[0055] The last optional method 124 in the group of methods 120 includes: obtaining the absolute position coordinates of multiple test points at the location of the camera 101, that is, within the camera image view, and comparing the absolute position coordinates with the absolute position data calculated from the corresponding point positions in the image data taken by the camera 101.

[0056] Figure 4 The relationship between the camera 3D coordinate system 130 and the camera 2D pixel coordinate system 131 is shown. The main intrinsic characteristics of the camera 101 include: the focal length of the camera lens in pixels and the optical center of the camera also in pixels. The intrinsic characteristics of the camera can be represented by a 3x3 matrix K:

[0057]

[0058] in,

[0059] [c x , c y ] is the optical center (principal point) in pixels;

[0060] [f x , f y ] is the focal length of the lens in pixels.

[0061] The matrix K represents the projective transformation from the camera 3D coordinate system 130 to the camera 2D pixel coordinate system 131, and vice versa. However, the matrix K does not take into account lens distortion, so optionally calibrating the intrinsic characteristics of the camera may include adding distortion correction coefficients for one or both of radial distortion and tangential distortion. Radial distortion occurs when light rays bend more near the edge of the lens than at the optical center of the lens. Tangential distortion occurs when the lens and image planes are not parallel. Both radial distortion and tangential distortion are non-linear. The relationships for correcting radial distortion include:

[0062] x 畸变 =x(1+k1*r 2 +k2*r 4 +k3r 6 )

[0063] v 畸变 =y(1+k1*r 2 +k2*r 4 +k3*r 6 )

[0064] Where x, y are the undistorted pixel positions;

[0065] k1, k2, k3 are the radial distortion coefficients of the camera lens;

[0066] r 2 :x2 +y 2 .

[0067] The relations for correcting tangential distortion include:

[0068] x 畸变 =x+[2*p1*x*y+p2*(r 2 +2*x 2 )]

[0069] y 畸变 =y+[p1*(r2+2*y 2 )+2*p2*x*y]

[0070] Where x, y are the undistorted pixel positions;

[0071] p1, p2 are the tangential distortion coefficients of the camera lens;

[0072] r 2 :x 2 +y 2 .

[0073] Figure 5 The relationship between the real world coordinate system 132 where the camera 101 is located and the camera's three-dimensional coordinate system 130 is shown. The camera's external relations include a 3x3 rotation matrix R and a 3x1 translation vector T. The origin of the camera's three-dimensional coordinate system 130 is its optical center, and its x-axis and y-axis define the camera's image plane. The 3x3 rotation matrix R and the 3x1 translation vector T are represented by the following matrices:

[0074]

[0075] The camera intrinsic properties and the camera extrinsic properties may be combined to provide projection data or a projection matrix P that transforms the real world coordinate system 132 at the location of the camera 101 into the camera two-dimensional pixel coordinate system 131:

[0076]

[0077] The above camera projection matrix P does not include distortion correction, but the distortion correction can be compensated as follows:

[0078]

[0079] x′=x / z

[0080] y′=y / z

[0081]

[0082]

[0083] where r 2 = x′ 2 +y′ 2

[0084] u=f x *x″+c x

[0085] v=f y *y″+c y

[0086] where x″ and y″ are the compensated camera image plane x and y values.

[0087] In the following description, it will be assumed that there is no lens distortion in the camera 101 .

[0088] One preferred method 122 of determining the camera's extrinsic properties uses a first pattern plane device 140 of known size, which is preferably placed completely within the image view of the camera 101 and preferably appears as large as possible in the image. The method 122 is preferably relative to a specified origin of the first pattern plane device 140 (at Figure 8 denoted as "world center (0, 0, 0)" in the figure), determines the position and orientation of the camera 101. The first pattern plane device 140 preferably includes a first planar chessboard pattern device 140, which defines a three-dimensional real world coordinate system 132 at the location of the camera 101, with axis Z = 0. However, the chessboard pattern of the first planar chessboard pattern device 140 is only locally meaningful to the camera 101 and cannot be directly used to obtain the absolute position data of a selected point in the camera image view.

[0089] The preferred method 122 of determining the extrinsic characteristics of a camera assumes that the same camera 101 is also used to separately determine the intrinsic characteristics of the camera, as described below. Figure 6As shown, method 122 includes: placing the first planar chessboard pattern device 140 on the ground within the FOV of the camera, that is, within the image view of the camera, so that the entire first planar chessboard pattern device 140 is within the FOV of the camera. The first planar chessboard pattern device 140 should be placed flat on the ground. Preferably, the size of the first planar chessboard pattern device 140 is at least 150 pixels by 150 pixels in the image view of the camera. In a typical smart city surveillance camera scene 100, this is usually a size of about 1.5 meters by 1.5 meters. Method 122 includes: taking at least one image of the first planar chessboard pattern device 140 with a camera 101. The elevation angle of the camera 101 depends on the region of interest (ROI) of the camera 101, which is used to detect the object 102 located within the camera FOV. The elevation angle can be about 30°. The camera 101 can provide the captured image to the server 107 and / or the database 108. The total square dimensions of the first planar checkerboard pattern device 140 and the number of its rows and columns in meters are input to the camera 101 and / or the server 107. The dimensions of the checkerboard may have been uploaded to the camera 101 and / or the server / database 107 / 108 before the imaging process of the first planar checkerboard pattern device 140 is performed, thereby saving time in the extrinsic property calibration method 122. In any case, the extrinsic property matrix is ​​determined from the captured image data and the dimensional data of the first planar checkerboard pattern device 140 using any suitable camera extrinsic property calibration method. A suitable tool for implementing camera extrinsic calibration methods is called "OpenCV TM ”, which includes publicly available open source computer vision libraries. You can use OpenCV TM SolvePnP function. The obtained extrinsic characteristic matrix is saved to the camera 101 and / or the server / database 107 / 108.

[0090] like Figure 7As shown, a preferred method 121 for determining the intrinsic characteristics of a camera includes: capturing multiple images of a second planar pattern device 150 having a size smaller than the first planar pattern device 140. The method may include: printing the second planar pattern device 150, which includes the second planar chessboard pattern device 150, on a paper or card of, for example, A4 size. The method preferably includes: placing or keeping the second planar chessboard pattern device 150 in the FOV of the camera at multiple different viewing angles and / or multiple different distances from the camera 101, and capturing multiple images of the second planar chessboard pattern device 150 with the camera 101. Preferably, up to 20 to 30 images are obtained. The size of the printed second planar chessboard pattern device 150 is stored in the camera 101 and / or the server / database 107 / 108, which can be done before the imaging step. The intrinsic characteristic matrix K is determined from a combination of the captured image data and the size data of the second planar chessboard pattern device 150 using any suitable camera intrinsic calibration method. A suitable tool for implementing the camera intrinsic calibration method 121 is OpenCV TM The camera function is calibrated. The resulting intrinsic characteristic matrix K is saved to the camera 101 and / or the server / database 107 / 108.

[0091] As described above, the preferred method 122 for determining camera extrinsic characteristics and the preferred method 121 for determining camera intrinsic characteristics are optional methods if the intrinsic and extrinsic characteristics of camera 101 are not already known. In addition, any suitable method may be implemented to obtain the intrinsic and extrinsic characteristics of camera 101.

[0092] According to the present invention, the key, non-optional method 123 is a method of modifying the projection matrix P so that it is suitable for converting the two-dimensional pixel coordinate system 131 of the camera image view into a three-dimensional global geographic coordinate system, i.e., an absolute positioning system or absolute coordinate system, including a one-to-one transformation of the Z=0 plane for mapping to the three-dimensional real-world coordinate system, for point positions within the camera's image view.

[0093] Method 123 should use the same first planar chessboard pattern device 140 as in method 122 to determine the external characteristics of the camera. The measurement of method 123 is simple and can be completed in a very short time, generally not more than one minute. Method 123 includes: measuring the absolute coordinates of a selected point position within the image view of the camera 101, that is, the origin of the chessboard coordinate system. The selected point position should be a selected corner point 141 on the first planar chessboard pattern device 140, and preferably an internal corner point 141 on the planar chessboard pattern device 140, that is, not a physical external corner point of the device 140, but a corner point 141 in the chessboard pattern of the device 140, such as Figure 8As shown. The selected corner point 141 can be considered as the "world center" or origin [0,0,0] of the three-dimensional real-world coordinate system where the camera 101 is located. The absolute coordinates of the origin [0,0,0], i.e., the global geographic coordinates, can be measured using any suitable global positioning equipment such as real-time kinematic (RTK) equipment.

[0094] In addition to obtaining the absolute coordinates of the origin [0,0,0], the method 123 further comprises: measuring an angle α between an x-axis 142 of a real-world coordinate system and a true north 144, the x-axis 142 passing through the origin [0,0,0]. Preferably, the method 123 comprises: measuring a counterclockwise angle α relative to the true north. The angle α may be measured using any suitable compass device, such as an electronic compass device, wherein the angle measurement error is preferably less than 0.5 degrees.

[0095] Method 123 also includes: using the obtained absolute coordinates of the origin [0,0,0], the measured angle α and the projection data derived from the intrinsic and extrinsic characteristics of the camera 101 to derive corrected projection data for converting the two-dimensional pixel coordinate system 131 of the camera image view into a three-dimensional geographic coordinate system so that global geographic coordinates can be derived for any selected point position (i.e., pixel position) within the image view of the camera 101 or within the image captured by the camera 101.

[0096] Thus, method 123 changes the previously described camera projection matrix P into a modified or integrated projection matrix P', enabling pixel positions in the two-dimensional pixel coordinate system 131 of camera 101 to be transformed or converted into global geocentric or geographic coordinates in a three-dimensional absolute position system.

[0097] The method 123 uses a first rotation matrix based on the measurement angle α and a second rotation matrix based on the latitude and longitude geographic coordinates of the measurement origin [0,0,0] to modify the camera projection matrix P to obtain a comprehensive projection matrix P′.

[0098] The camera projection matrix P shown below enables the three-dimensional real world coordinates [XYZ], i.e. the coordinate system of the first planar checkerboard pattern device 140, to be converted into two-dimensional pixel positions [u, v] on the camera image view or image data where the camera 101 is located.

[0099] The camera projection matrix P (assuming no camera lens distortion) consists of:

[0100]

[0101] When the Z value in the three-dimensional real-world coordinate system of the location of camera 101 is taken as zero, there is a one-to-one relationship or solution from the pixel position in the camera image view or image data to the three-dimensional real-world coordinate system of the location of camera 101.

[0102] Setting Z to zero and expanding row 3 of the camera projection matrix P can be expressed as:

[0103] S=r 13 *t1+r 23 *t2+r 33 *t3 / r 13 *u+r 23 *v+r 33 *1.0

[0104] This expression can be rewritten as:

[0105]

[0106] In this way, a unique X,Y position in the 3D real-world coordinate system can be resolved to a 2D pixel position [u, v] on the camera image view or image data.

[0107] Then, the camera projection matrix P is modified into a comprehensive projection matrix P' by the first and second rotation matrices in method 123 as shown below:

[0108]

[0109] Wherein [XE, YE, ZE] represents a three-dimensional absolute coordinate system (global geocentric or geographic coordinate system), [XC, YC, ZC] represents a three-dimensional real-world coordinate system where the camera 101 is located, and PRef is the absolute coordinate position of the origin [0,0,0].

[0110] because:

[0111]

[0112] Then the comprehensive projection matrix P' can be expressed as:

[0113]

[0114] The combined projection matrix P' makes it possible to find the 3D absolute coordinates for any 2D pixel position [u, v] on the camera image view or the image data system at the camera's location.

[0115] In the foregoing, the three-dimensional absolute coordinates may be geocentric coordinates based on an Earth-Centered Earth-Fixed (ECEF) coordinate system, which will be described in more detail below, but method 123 preferably includes: the ECEF coordinate system is ultimately transformed into a true global geographic coordinate system, such as the World Geodetic System 1984 (WGS84), although any absolute geographic coordinate system may be used.

[0116] As described above, the camera projection matrix P enables the two-dimensional pixel position [u, v] on the camera image view or image data to be converted to the three-dimensional real world coordinate system of the camera 101, i.e., the coordinate system of the first planar chessboard pattern device 140 when the constraint Z = 0 is applied. The method 123 modifies the camera projection matrix P to provide a comprehensive camera projection matrix P', which enables the two-dimensional pixel position [u, v] on the camera image view or image data to be converted to a three-dimensional global geocentric or geographic location system, i.e., an absolute location system.

[0117] The method 123 preferably includes: first converting the three-dimensional real world coordinate system of the location of the camera 101, i.e., the coordinate system of the first planar checkerboard pattern device 140, into an intermediate coordinate system, such as a local horizontal (NED) coordinate system. The NED coordinate system is the coordinate system closest to the coordinate system of the first planar checkerboard pattern device 140, because they both occupy a tangential plane of the earth. The difference between the two coordinate systems is only the rotation of the Z axis (lower axis). Therefore, converting or transforming the coordinate system of the first planar checkerboard pattern device 140 into the NED coordinate system makes it easier to subsequently convert or transform into other absolute coordinate systems, such as the ECEF geocentric coordinate system and / or the WGS84 global geographic coordinate system.

[0118] Method 123 preferably also includes: converting an intermediate coordinate system such as the NED coordinate system to the ECEF coordinate system. This is because the NED coordinate system is a relative coordinate system, while the ECEF is an absolute Euclidean coordinate system, which has been found to be a good stepping stone for further conversion to a commonly used global geographic coordinate system such as WGS84. WGS84 is preferred because it is the most commonly used global geographic coordinate system.

[0119] The NED coordinate system is a right-handed local tangent coordinate system on the surface of the earth. In the NED coordinate system, the normal, i.e., "downward", points to the center of the earth, and north points to true north / geocentric north, i.e., along the surface of the earth in the direction toward the geographic North Pole. For a small flat area, such as the area of ​​the first planar chessboard pattern device 140, the surface of the earth can be considered to be flat or horizontal from a local perspective. Therefore, at this scale, the tangent plane is essentially the same as the real surface of the earth. In view of this, the first planar chessboard pattern device 140 includes a local tangent plane on the surface of the earth. The conversion or transformation of the three-dimensional real-world coordinate system at the location of the camera 101 to the NED coordinate system includes: a rotation transformation on the Z axis (lower axis) based on the angular difference between the x-axis 142 and the true north 144. In other words, the conversion or transformation of the three-dimensional real-world coordinate system at the location of the camera 101 to the NED coordinate system includes a rotation transformation derived from the measured angle α, which provides the aforementioned first rotation matrix. The first rotation matrix is ​​given by the following formula:

[0120]

[0121] The method 123 preferably further comprises: transforming the NED coordinate system to the ECEF coordinate system. The NED coordinate system and the ECEF coordinate system are Euclidean coordinate systems, and the difference between them is a rigid transformation including rotation and translation. Therefore, the transformation from the NED coordinate system to the ECEF coordinate system is based on the following relationship:

[0122]

[0123] Where P NED is a three-dimensional position in the NED system, P ECEF is the corresponding ECEF position, P Ref is the reference ECEF position, i.e., the ECEF coordinate position of the origin [0,0,0] of the first planar chessboard pattern device 140, and R is the second rotation matrix, whose columns include the "North", "East" and "Down" axes. Ref A matrix of geographical coordinates measuring latitude Φ and longitude λ.

[0124] The transformation from the three-dimensional real-world coordinate system [XC, YC, ZC] of the camera 101 to the ECEF coordinate system (XE, YE, ZE) can be integrated into the following formula:

[0125]

[0126] Where α is the measured angle between the x-axis 142 of the three-dimensional real-world coordinate system and the true north 144 in the counterclockwise direction; Φ and λ are the longitude and latitude geographic coordinates measured by the RTK device at the origin [0,0,0] of the first planar chessboard pattern device 140; T = P Ref, which includes the ECEF coordinates of the origin [0,0,0].

[0127] In practice, people find it easier to measure T using a WGS84 device and then convert the resulting global geographic coordinates to ECEF coordinates.

[0128] Finally, method 123 preferably includes: converting or transforming the ECEF coordinate system into a preferably widely used global coordinate system, such as WGS84. The ECEF coordinate system is a Euclidean coordinate system with a one-to-one non-linear mapping to WGS84, which is a polar coordinate system. The conversion from the ECEF coordinate system to WGS84 can be calculated by a closed form algorithm published by JiJieZhu in 1993. Third-party software such as or pymap3d provides a suitable API for the conversion. Therefore, the integrated camera projection matrix P' can be further modified to include the conversion from the ECEF coordinate system to WGS84 as follows:

[0129]

[0130] It can be rewritten as:

[0131]

[0132] The set of methods 120 may comprise a further optional method 124 of verifying the camera calibration. Preferably, the same first planar checkerboard pattern device 140 is used for the verification method 124, or several copies of the same first planar checkerboard pattern device 140 are used.

[0133] like Fig. 9 As shown, two other first planar chessboard pattern devices 140 are placed at different distances from the camera 101 within the camera FOV. Preferably, the surrounding environment has minimal blocking of the GPS signal so as to obtain the most accurate global geographic coordinates for the test point as the "ground truth". Fig.10 As shown, the method 124 includes: in each of the other two first planar chessboard pattern devices 140, determining a plurality of points including a corner point 143 as test points (in Fig.101 to 8 in the figure). For each test point 143, a global geographic coordinate is obtained using a device such as an RTK device or the like, which can preferably obtain GPS with an error of no more than 1.5 cm. In addition, the pixel position of each of the test points 143 is obtained from the camera image view or image data, and the pixel position is converted into absolute position data, i.e., WGS84 geographic coordinates. The geographic coordinates obtained for the test point 143 using the RTK device are compared with the geographic coordinates of the test point 143 obtained from the pixel position in the camera image view or image data. By comparison, any projection error between the physically measured geographic coordinates of the test point 143 and the geographic coordinates of the test point 143 obtained from the pixel position in the camera image view or image data can be determined, thereby verifying the accuracy of the camera calibration under method 123. The size of any projection error may depend on the positioning accuracy required for the relevant application.

[0134] Preferably, the test points 143 include two additional internal corner points in the first planar checkerboard pattern device 140. Corner points are preferred because they are easier to detect, but other patterns or shapes, such as dots, are also possible, and suitable image processing algorithms can be used to detect them with sub-pixel accuracy. Corner points can be detected using the Harris corner detection algorithm or any other suitable corner detection algorithm.

[0135] The present invention also provides a method for determining the geographic coordinates of a point position within a camera image view, comprising the following steps: acquiring an image from a camera; selecting a point position in the image, wherein the point is located in the same plane as a first planar chessboard pattern device; obtaining the two-dimensional pixel coordinates of the selected point position in the image; and using projection data that maps the two-dimensional pixel coordinate system of the camera image view to a three-dimensional geographic coordinate system to convert the two-dimensional pixel coordinates of the selected point position in the image into three-dimensional geographic coordinates of the selected point position.

[0136] The present invention also provides a camera, comprising: a memory storing machine-readable instructions; and a processor for executing the machine-readable instructions, so that when the processor executes the machine-readable instructions, it configures the camera to implement the above method according to the present invention.

[0137] The above-mentioned means may be implemented at least in part by software. It will be appreciated by those skilled in the art that the above-mentioned means may be implemented at least in part by using general-purpose computer equipment or by using customized equipment.

[0138] Here, various aspects of the methods and devices described herein can be executed on any device including a communication system. The program aspects of the technology can be considered as a "product" or "article of manufacture", usually in the form of executable code and / or related data, carried or embodied in a machine-readable medium. "Storage" type media include any or all memories of mobile stations, computers, processors or similar devices, or their related modules, such as various semiconductor memories, tape drives, disk drives, etc., which can provide storage for software programming at any time. All or part of the software can sometimes communicate through the Internet or various other telecommunications networks. For example, such communication can enable software to be loaded from one computer or processor to another computer or processor. Therefore, another type of media that can carry software elements includes optical waves, radio waves, and electromagnetic waves, such as on physical interfaces between local devices, through wired and optical landline networks, and through various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be considered as media that carry software. As used herein, unless limited to tangible non-temporary "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions to the processor for execution.

[0139] Although the present invention has been described and illustrated in detail in the accompanying drawings and the foregoing description, it should be considered as illustrative rather than restrictive, and it should be understood that only exemplary embodiments are shown and described and the scope of the present invention is not limited in any way. It is understood that any feature described herein can be used in any embodiment. The illustrative embodiments do not exclude each other or other embodiments not described herein. Therefore, the present invention also provides an embodiment including a combination of one or more illustrative embodiments described above. Without departing from the spirit and scope of the present invention, the present invention can be modified and varied, and therefore, only the restrictions shown in the appended claims should be applied.

[0140] In the appended claims and the foregoing description of the invention, unless the context requires otherwise due to explicit language or necessary implication, the word "comprise" or variations such as "comprising" are used in an inclusive sense, i.e. specifying the presence of stated features but not excluding the presence or addition of further features in various embodiments of the invention.

[0141] It will be appreciated that, if any prior art publication is referred to herein, this reference does not constitute an admission that the publication forms part of the common general knowledge in the art.

Claims

1. A method for calibrating a camera, comprising the following steps: Obtaining geographic coordinates of a selected point position within a camera image view, comprising: placing a first pattern plane device of known size within the camera image view, selecting a point on the first pattern plane device as the selected point position within the camera image view, wherein the first pattern plane device defines a three-dimensional real-world coordinate system of the selected point position; measuring the angle between an x-axis of the three-dimensional real-world coordinate system and true north, the x-axis passing through the selected point location; and deriving modified projection data for converting a two-dimensional pixel coordinate system of the camera image view to a three-dimensional geographic coordinate system of a point location within the camera image view using the obtained geographic coordinates, the measured angles, and projection data derived from characteristics of the camera; wherein the projection data derived from the characteristics of the camera comprises a projection matrix of the camera that transforms the three-dimensional real-world coordinate system of the selected point location into the two-dimensional pixel coordinate system of the camera image view; wherein said step of deriving corrected projection data comprises: modifying said projection matrix derived from characteristics of said camera using a first rotation matrix based on said measured angles and a second rotation matrix based on said geographic coordinates of said selected point locations to provide a composite projection matrix; wherein the three-dimensional geographic coordinate system of the point location within the camera image view comprises an Earth-centered Earth-fixed (ECEF) coordinate system, and a combination of the first rotation matrix and the second rotation matrix of the integrated projection matrix transforms the three-dimensional real-world coordinate system of the selected point location to the ECEF coordinate system; The projection matrix derived from the characteristics of the camera is derived from a combination of the intrinsic characteristics of the camera and the extrinsic characteristics of the camera, including: obtaining the intrinsic characteristics of the camera from a second pattern plane device of known size, including: imaging the second pattern plane device at multiple different angles and / or different distances within the camera image view to obtain the intrinsic characteristic matrix of the camera, the intrinsic characteristic matrix including a projection transformation from the three-dimensional coordinate system of the camera image view to the two-dimensional pixel coordinate system of the camera image view.

2. The method of claim 1, wherein the first rotation matrix transforms the three-dimensional real-world coordinate system into a local horizontal (NED) coordinate system, and the second rotation matrix transforms the NED coordinate system into the ECEF coordinate system.

3. The method according to claim 1, wherein the method comprises: The integrated projection matrix is ​​modified to transform the ECEF coordinate system into an absolute coordinate system.

4. The method according to claim 1, wherein the step of obtaining the geographic coordinates of the selected point location comprises: Use a real-time kinematic (RTK) device.

5. The method according to claim 1, wherein the method comprises: A second pattern plane device of known dimensions smaller than the first pattern plane device is used.

6. The method according to claim 1, wherein the method comprises: The extrinsic properties of the camera are obtained from the first pattern plane device to obtain an extrinsic property matrix of the camera, which includes rotation and translation vectors, and the extrinsic property matrix includes a transformation of the three-dimensional real-world coordinate system of the selected point position to the three-dimensional coordinate system of the camera image view.

7. The method of claim 6, wherein the projection matrix derived from the characteristics of the camera comprises a matrix product of the intrinsic characteristics matrix and the extrinsic characteristics matrix.

8. The method according to claim 1, wherein the method comprises: Using RTK equipment to obtain geographic coordinates of multiple point positions on the first pattern plane device; detecting the plurality of point locations on the first pattern plane arrangement in the camera image view; using the integrated projection matrix, determining corresponding geographic coordinates for each detected point location in the camera image view; and comparing the geographic coordinates obtained using the RTK device with the geographic coordinates calculated using the integrated projection matrix to determine any projection errors between the geographic coordinates obtained using the RTK device and the geographic coordinates calculated using the integrated projection matrix.

9. The method of claim 8, wherein whether the calibration of the camera is verified depends on the size of any projection errors.

10. The method according to claim 8, wherein the step of detecting the plurality of point positions on the first pattern plane device comprises: Detecting the corner point positions on the first pattern plane device.

11. The method according to claim 8, wherein the step of detecting the plurality of point positions on the first pattern plane device comprises: Placing the first pattern plane device at different distances from the camera within the camera image view, and repeating the steps of claim 7 for each different distance.

12. A camera comprising: a memory storing machine-readable instructions; and A processor for executing the machine-readable instructions, such that when the processor executes the machine-readable instructions, the camera is configured to: Receiving data including geographic coordinates of a selected point location within a camera image view, comprising: placing a first pattern plane device of known size within the camera image view, selecting a point on the first pattern plane device as the selected point location within the camera image view, wherein the first pattern plane device defines a three-dimensional real-world coordinate system for the selected point location; receiving data comprising a measured angle between an x-axis of a three-dimensional real-world coordinate system and true north, the x-axis passing through the selected point location; and deriving modified projection data for converting a two-dimensional pixel coordinate system of the camera image view to a three-dimensional geographic coordinate system of point locations in the camera image view using the received data including the geographic coordinates, the received data including the measured angles, and projection data derived from characteristics of the camera; wherein the projection data derived from the characteristics of the camera comprises a projection matrix of the camera that transforms the three-dimensional real-world coordinate system of the selected point location into the two-dimensional pixel coordinate system of the camera image view; wherein said step of deriving corrected projection data comprises: modifying said projection matrix derived from characteristics of said camera using a first rotation matrix based on said measured angles and a second rotation matrix based on said geographic coordinates of said selected point locations to provide a composite projection matrix; wherein the three-dimensional geographic coordinate system of the point location within the camera image view comprises an Earth-centered Earth-fixed (ECEF) coordinate system, and a combination of the first rotation matrix and the second rotation matrix of the integrated projection matrix transforms the three-dimensional real-world coordinate system of the selected point location to the ECEF coordinate system; The projection matrix derived from the characteristics of the camera is derived from a combination of the intrinsic characteristics of the camera and the extrinsic characteristics of the camera, including: obtaining the intrinsic characteristics of the camera from a second pattern plane device of known size, including: imaging the second pattern plane device at multiple different angles and / or different distances within the camera image view to obtain the intrinsic characteristic matrix of the camera, the intrinsic characteristic matrix including a projection transformation from the three-dimensional coordinate system of the camera image view to the two-dimensional pixel coordinate system of the camera image view.

Images