A real-time global positioning and tracking method based on high-precision maps using a roadside monocular camera

Abstract

The present invention discloses a real-time global positioning and tracking method for high-precision maps based on a roadside monocular camera, including: roadside camera self-calibration, which learns semantic clues from traffic scenes to achieve camera self-calibration; target detection and tracking, which detects and tracks vehicles on the image plane to obtain pixel position changes over a period of time; target vehicle-camera angle estimation, which calculates the angle between the vehicle and the camera based on pixel positions and combined with calibrated camera parameters; target vehicle-camera distance estimation, which calculates the distance between the vehicle and the camera based on calibrated camera parameters; final positioning calculation, which uses the obtained distance and angle combined with the camera's longitude and latitude to achieve global positioning of the target; and real-time display of the global positioning results obtained by tracking on a high-precision map. This method automatically achieves monocular global positioning of any uncalibrated roadside camera without manual operation, and achieves positioning and tracking in the image and high-precision plane.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a camera calibration method based on deep learning. Background Art

[0002] With the rapid development of autonomous driving technology, autonomous driving safety is receiving increasing attention. As the first step in autonomous driving perception, accurate positioning is a prerequisite for ensuring autonomous driving safety.

[0003] Global positioning and tracking of vehicles using the Global Navigation Satellite System (GNSS) or various commercial products with built-in GNSS technology has been widely adopted. However, GNSS-based positioning solutions require line of sight (LOS) of at least four navigation satellites. In many areas, particularly in densely built-up areas where satellite signals are severely attenuated or even rejected, the availability of such a large number of navigation satellites cannot be guaranteed due to the urban canyon effect. Furthermore, the root mean square error (RMSE) of pure GNSS positioning in cities is typically greater than 5 meters. Even with enhancements such as inertial sensors, differential correction, or multi-constellation receivers, the required navigation performance in terms of availability and accuracy cannot be guaranteed. Furthermore, with the widespread deployment of roadside surveillance cameras, a method for achieving monocular global vehicle positioning on the roadside using large-scale traffic surveillance cameras in the absence of GNSS is needed. While a large number of intelligent roadside applications using roadside surveillance cameras have been implemented, including speed estimation, traffic flow counting and prediction, and illegal parking identification, to improve road safety and efficiency, few systematic approaches have been developed to specifically address the problem of roadside vehicle positioning.

[0004] Most current positioning methods rely on manually selecting at least four corresponding points in maps and images to convert from the camera coordinate system to the world coordinate system by solving the homography matrix using four points. While this method is simple to implement, it is prone to manual errors. More importantly, in the current large-scale deployment of intelligent traffic cameras, this method requires manually establishing initial point correspondences and solving the coordinate system transformation matrix for each camera, which severely limits its application scale and deployment efficiency.

[0005] Currently, monocular positioning is mostly tested in experimental scenarios, with camera calibration already performed by default or relying on manual methods, such as Zhang Zhengyou et al.'s calibration method using a chessboard. These methods also require on-site personnel, and traffic constraints make them unsuitable for real-world traffic scenarios. Furthermore, in some roadside scenarios, camera angles often fluctuate due to physical factors. Furthermore, with the prevalence of PTZ cameras, the focal length of cameras can change at any time, making manual camera calibration methods unsuitable.

[0006] There is also a self-calibration method for roadside camera calibration, which relies on vehicle vanishing point detection. However, this type of method requires the road to be approximately straight or the vehicle trajectory to be approximately straight, which cannot be used in some turning scenarios.

[0007] In summary, there is currently no complete solution to the above-mentioned roadside monocular positioning problems that can achieve automatic camera calibration without requiring scene prior information or any manual operation, and is suitable for monocular vehicle positioning in any unknown scene. Summary of the Invention

[0008] To solve the above practical problems, the present invention provides a real-time global positioning and tracking method for high-precision maps based on a roadside monocular camera. The technical solution is as follows:

[0009] The technical solution provided by this invention can automatically calibrate existing roadside surveillance cameras, eliminating the need for human intervention and ensuring no specific scene requirements. The algorithm can be applied to any intersection, directly positioning vehicles in the scene and simultaneously tracking the resulting global positioning results on a high-precision map.

[0010] A global positioning and tracking method based on a roadside monocular camera, the method comprising:

[0011] Step A: The camera is automatically calibrated to obtain the internal and external parameters of the camera;

[0012] Step B: The target detector detects the vehicle in the image and obtains the pixel position of the vehicle in the image at a certain frame;

[0013] Step C, the target tracker establishes a correlation between the previous and next frames based on the detection to obtain the pixel position change of the vehicle over time;

[0014] Step D: Vehicle positioning accepts pixel position changes to calculate the angle and distance between the vehicle and the camera to obtain the vehicle's global position;

[0015] Step E: High-precision map positioning tracking calculates the global positioning information and displays it on the high-precision map.

[0016] In step A, in the roadside scenario, a deep learning-based model is used to directly learn semantic clues from the scene and infer the vertical field of view angle, roll angle, and pitch angle without relying on manual input and scene prior information.

[0017] In step A, the focal length is not estimated directly, but is converted into the vertical field of view angle for indirect reasoning. The continuous values ​​of the pitch angle, roll angle, and vertical field of view angle are discretized into 256 buckets for classification prediction. Then, three fully connected layers are used to predict these three quantities respectively. Finally, the expected value of the probability distribution of each fully connected head is used as their predicted value. For the vertical field of view angle, the Softargmax biased-L2 loss is used, and for the roll and pitch, the standard Softargmax-L2 loss is used.

[0018] In steps B and C, only the bottom center point of the detection frame obtained by detection and tracking is required as the positioning input. The accurate positioning depends on the camera's global parameters, such as longitude and latitude, camera altitude, camera heading angle, and the camera's own parameters, including internal and external parameters.

[0019] In step D, the angle estimation between the vehicle and the camera depends on the camera's heading angle and horizontal field of view angle, and the longitudinal distance between the vehicle and the camera depends on the camera's height. Based on the camera parameters, the distance and angle between the vehicle and the camera are calculated simultaneously. Then, the camera's latitude and longitude are calculated using the distance and angle information combined with the camera's own heading angle and longitude and latitude.

[0020] A global high-precision map positioning and tracking system based on a roadside monocular camera, including: camera self-calibration, vehicle detection, vehicle tracking, vehicle positioning, and high-precision map positioning and tracking;

[0021] The camera self-calibration predicts the camera focal length, roll angle, and pitch angle information through a deep learning network;

[0022] The vehicle detection and tracking part detects and tracks the vehicle in real time, and obtains the change of the coordinates of the center point at the bottom of the vehicle detection frame over time;

[0023] The angle and distance between the vehicle and the camera are estimated, and then the latitude and longitude coordinates of the target vehicle are obtained by combining the angle and distance between the vehicle and the camera with the geographic location conversion formula;

[0024] The high-precision vehicle positioning and tracking input pixel changes and camera parameters obtained by camera calibration are combined with a positioning algorithm to display the geographic positioning results in real time on a high-precision map, and real-time positioning and tracking are performed on the image plane and the high-precision map plane.

[0025] The camera calibration algorithm only infers the vertical field of view angle, pitch angle and roll angle, does not require any manual input, and has no requirements for the scene.

[0026] In the target detection and tracking algorithm, the pixel position of the bottom center point of the 2D target detection box is used to calculate the distance and angle between the vehicle and the camera.

[0027] Angle estimation is achieved by combining the horizontal field of view angle obtained by camera calibration, the camera's own heading angle, and the vehicle pixel position obtained by target tracking. The angle between the vehicle and the camera is calculated as follows:

[0028]

[0029] ω d =ω h +ω c

[0030] Where, ω h Indicates the camera heading angle;

[0031] ω c Indicates the angle between the vehicle and the camera heading angle;

[0032] D represents the distance between the vehicle and the camera;

[0033] ωd represents the clockwise angle between the distance D between the vehicle and the camera and the north direction.

[0034] Distance estimation is achieved by combining the field of view angle obtained from camera calibration and the vehicle pixel position obtained from target tracking. The angle between the vehicle and the camera is calculated as follows:

[0035]

[0036] Where (x, y) is the pixel coordinate of the bottom center point of the vehicle detection box in the image;

[0037] H is the camera installation height;

[0038] Y represents the longitudinal distance between the vehicle and the camera;

[0039] w, h are the pixel width and pixel height of the image respectively;

[0040] θ represents the pitch angle of the camera;

[0041] f represents the focal length of the camera;

[0042] The final distance between the vehicle and the camera is determined by the longitudinal distance and the angle between the vehicle and the camera. The global positioning result is calculated by combining the distance and angle between the camera and the vehicle, and the camera's own longitude and latitude conversion. The final global positioning longitude and latitude result is calculated by the following formula:

[0043] R=6371.393×1000

[0044]

[0045]

[0046] Where R is the radius of the earth in meters;

[0047] l, g represent the latitude and longitude of the vehicle respectively;

[0048] l c 、g c Represents the latitude and longitude of the camera respectively;

[0049] High-precision map tracking and positioning is performed by displaying the longitude and latitude obtained by image calculation on the map without human intervention.

[0050] This solution significantly reduces the deployment process and manual operations required for roadside positioning, minimizing human error and improving the algorithm's adaptability to various scenarios and camera models. Furthermore, monocular positioning results can serve as a supplementary service for locations where GNSS signals are missing, providing real-time positioning results for all autonomous vehicles in traffic scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0052] Figure 1 This is a schematic diagram of an angle estimation method for monocular positioning according to an embodiment of the present invention;

[0053] Figure 2 Schematic diagram of a vehicle located on the right side of the camera's field of view in angle estimation according to an embodiment of the present invention

[0054] Figure 3 Schematic diagram of a vehicle located on the left side of the camera's field of view in angle estimation according to an embodiment of the present invention

[0055] Figure 4 Schematic diagram of a distance estimation method for monocular positioning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The purpose of this invention is to realize single-purpose positioning on the roadside without manual calibration of the camera, and to be applicable to different roadside scenes, to realize global vehicle positioning based on calibration, and to perform real-time positioning and tracking in high-precision maps.

[0057] To implement the above solutions, it is necessary to solve the camera calibration problem and achieve vehicle global positioning based on calibration without manual methods.

[0058] To address the above problems, the present invention provides a high-precision map real-time global positioning and tracking algorithm based on a roadside monocular camera, which does not rely on manual input, is not restricted in deployment scenarios, and is not restricted in camera type, installation position, and orientation.

[0059] In the present invention, for the geometric camera model, the 3D point p in the scene w and the image pixel position relationship p im It can be expressed as:

[0060] p im =[λu λv λ] T =K[R|t][p w |1] T ,

[0061] Where K is the camera projection matrix (camera intrinsic parameters), R and t are the camera rotation and translation in the world coordinate system (camera extrinsic parameters). This paper adopts the mainstream assumption that the pinhole camera model is used, the image center is the principal point, the camera is not tilted, and the horizontal and vertical scales are equal (aspect ratio of 1, square pixels), and the focal length is equal in both directions, then f x =f y =f, the projection matrix K can be expressed as K=diag([ff1]), where f is the focal length (in pixels).

[0062] Since the focal length has an infinite range and changes when we resize the image, we directly estimate the vertical field of view α in radians and convert it to focal length as follows, where h is the image pixel height:

[0063]

[0064] A rotation matrix can be described using three angles: roll, pitch, and yaw. Since there's no natural reference frame to estimate yaw (left and right) from any image, we don't estimate yaw. Instead, our rotation matrix consists of two angles: pitch θ and roll ψ. These two quantities are represented by a horizontal line. The camera's pitch angle θ is the intersection of the horizontal midpoint and the image's horizontal center, while the roll angle ψ is the rotation angle of the estimated line relative to the horizontal line.

[0065] The pitch angle θ, roll angle ψ, and vertical field of view angle α are discretized into 256 buckets in the spatial domain, converting the regression problem into a B-classification problem. This paper uses ResNet-50 as the backbone network and uses a separate fully connected layer to predict the pitch angle θ, roll angle ψ, and vertical field of view angle α. The center value of each bucket can be expressed as θ = [θ1, ...θ i ,...θ B],ψ=[ψ1,...ψ i ,...ψ B ], α=[α1,...α i , ...α B ],make Denotes the probability mass of the fully connected layer of each head. The expected value of the probability mass of each fully connected head can be expressed as:

[0066]

[0067]

[0068]

[0069] Next, for the vertical field of view angle α, the Softargmax biased-L2 loss is used, and for the pitch angle θ and the roll angle ψ, the standard Softargmax-L2 is used. The final loss function is expressed as:

[0070]

[0071] The model can be trained on some currently available camera calibration datasets, such as the pano360 dataset. After training, the camera field of view, pitch angle, and roll angle can be obtained.

[0072] During calibration, we also perform target detection and tracking on vehicles in the scene, obtaining the pixel position corresponding to each vehicle. The target detection and tracking algorithms can be replaced with any other algorithms. After calibration, we obtain the camera's pitch angle θ, roll angle ψ, and vertical field of view angle α. In traffic scenes, the deviation of the roll angle ψ has a smaller impact on the final positioning than the pitch angle θ. Therefore, the influence of the roll angle is not considered in this invention.

[0073] like Figure 1 As shown, given the position of the camera, we only need to know the distance D and angle ω between the camera and the target vehicle d , the geographical location of the target vehicle can be estimated. In the present invention, ω d Represents the clockwise angle between the distance D and the camera relative to the true north N of the map, ω c represents the orientation angle ω between the vehicle and the camera h Given the image width w and the camera horizontal field of view hfov, the coordinates of the bottom center of the vehicle detection frame are (x, y), ω c It can be expressed as:

[0074]

[0075] ω h Indicates the camera heading angle;

[0076] ω c Represents the angle between the vehicle and the camera heading angle;

[0077] D represents the distance between the vehicle and the camera;

[0078] ω d It represents the clockwise angle between the distance D between the vehicle and the camera and the north direction.

[0079] Figure 1 Two different scenarios such as Figure 2 and Figure 3 As shown in the figure, the vehicle's direction of travel is on the right and left of the camera's field of view. For both cases, ω d Can be expressed as:

[0080] ω d =ω h +ω c

[0081] Distance estimation models such as Figure 4 As shown, assuming that a car is detected in the road scene, its position is in the ground coordinate system, (X w , Y w , Z w ), let θ v is the projection ray (relative to the camera) of the intersection of the detected rear or front of the vehicle and the road plane, and H is the installation height of the camera. Assuming a horizontal road, the longitudinal distance Y can be expressed as follows based on the angular relationship:

[0082]

[0083] Among them, β can be expressed as:

[0084]

[0085] Then the longitudinal distance Y between the camera and the vehicle can be calculated as follows:

[0086]

[0087] (x, y) is the pixel coordinate of the bottom center point of the vehicle detection box in the image;

[0088] H is the camera installation height;

[0089] Y represents the longitudinal distance between the vehicle and the camera;

[0090] w, h are the pixel width and pixel height of the image respectively;

[0091] θ represents the pitch angle of the camera;

[0092] f represents the focal length of the camera;

[0093] The final distance D is:

[0094]

[0095] After obtaining the distance D and angle ω d After that, assume that the camera's latitude and longitude are: c , g c The vehicle latitude l and longitude g can be obtained by the following formula:

[0096] R=6371.393×1000,

[0097]

[0098]

[0099] Where R is the radius of the Earth in meters. l and g represent the latitude and longitude of the vehicle, respectively;

[0100] l c 、g c Represents the latitude and longitude of the camera respectively;

[0101] After determining the vehicle's global position, the positioning results can be displayed in real time on a high-precision map, ultimately enabling real-time positioning and tracking on a monocular high-precision roadside map. This method achieves positioning without manual intervention, eliminating the need for on-site calibration and enabling plug-and-play operation. Positioning accuracy can also be further improved by optimizing camera calibration accuracy.

[0102] The above description is only a specific embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A global positioning and tracking method based on a roadside monocular camera, characterized in that: The method includes: Step A: The camera is automatically calibrated to obtain the internal and external parameters of the camera; Step B: The target detector detects the vehicle in the image and obtains the pixel position of the vehicle in the image at a certain frame; Step C, the target tracker establishes a correlation between the previous and next frames based on the detection to obtain the pixel position change of the vehicle over time; Step D: Vehicle positioning accepts pixel position changes to calculate the angle and distance between the vehicle and the camera to obtain the vehicle's global position; Step E: High-precision map positioning tracking displays the global positioning information obtained through calculation on the high-precision map; In step A, in the roadside scenario, a deep learning-based model is used to directly learn semantic clues from the scene and infer the vertical field of view angle, roll angle, and pitch angle without relying on manual input and scene prior information; Angle estimation is achieved by combining the horizontal field of view angle obtained by camera calibration, the camera's own heading angle, and the vehicle pixel position obtained by target tracking. The angle between the vehicle and the camera is calculated as follows: oh d =ω h +oh c Where, ω h Indicates the camera heading angle; ω c Indicates the angle between the vehicle and the camera heading angle; D represents the distance between the vehicle and the camera; ωd represents the clockwise angle between the distance D between the vehicle and the camera and the north direction; Distance estimation is achieved by combining the field of view angle obtained from camera calibration and the vehicle pixel position obtained from target tracking. The angle between the vehicle and the camera is calculated as follows: Where (x, y) is the pixel coordinate of the bottom center point of the vehicle detection box in the image; H is the camera installation height; Y represents the longitudinal distance between the vehicle and the camera; w, h are the pixel width and pixel height of the image respectively; θ represents the pitch angle of the camera; f represents the focal length of the camera; The final distance between the vehicle and the camera is determined by the longitudinal distance and the angle between the vehicle and the camera. The global positioning result is calculated by combining the distance and angle between the camera and the vehicle, and the camera's own longitude and latitude conversion. The final global positioning longitude and latitude result is calculated by the following formula: R＝6371.393×1000 Where R is the radius of the earth in meters; l, g represent the latitude and longitude of the vehicle respectively; l c 、g c Represents the latitude and longitude of the camera respectively; High-precision map tracking and positioning is performed by displaying the longitude and latitude obtained by image calculation on the map without human intervention.

2. The global positioning and tracking method based on a roadside monocular camera according to claim 1, characterized in that: In step A, the focal length is not estimated directly, but is converted into the vertical field of view angle for indirect reasoning. The continuous values ​​of the pitch angle, roll angle, and vertical field of view angle are discretized into 256 buckets for classification prediction. Then, three fully connected layers are used to predict these three quantities respectively. Finally, the expected value of the probability distribution of each fully connected head is used as their predicted value. For the vertical field of view angle, the Softargmax biased-L2 loss is used, and for the roll and pitch, the standard Softargmax-L2 loss is used.

3. The global positioning and tracking method based on a roadside monocular camera according to claim 1, characterized in that: In steps B and C, only the bottom center point of the detection frame obtained by detection and tracking is required as the positioning input. The accurate positioning depends on the camera's global parameters, such as longitude and latitude, camera altitude, camera heading angle, and the camera's own parameters, including internal and external parameters.

4. The global positioning and tracking method based on a roadside monocular camera according to claim 1, characterized in that: In step D, the angle estimation between the vehicle and the camera depends on the camera's heading angle and horizontal field of view angle, and the longitudinal distance between the vehicle and the camera depends on the camera's height. Based on the camera parameters, the distance and angle between the vehicle and the camera are calculated simultaneously. Then, the camera's latitude and longitude are calculated using the distance and angle information combined with the camera's own heading angle and longitude and latitude.

5. A global high-precision map positioning and tracking system based on a roadside monocular camera, characterized in that: include: Camera self-calibration, vehicle detection, vehicle tracking, vehicle positioning, and high-precision map positioning and tracking; The camera self-calibration predicts the camera focal length, roll angle, and pitch angle information through a deep learning network; The vehicle detection and tracking part detects and tracks the vehicle in real time, and obtains the change of the coordinates of the center point at the bottom of the vehicle detection frame over time; The angle and distance between the vehicle and the camera are estimated, and then the latitude and longitude coordinates of the target vehicle are obtained by combining the angle and distance between the vehicle and the camera with the geographic location conversion formula; The high-precision vehicle positioning and tracking input pixel changes and camera parameters obtained by camera calibration are combined with the positioning algorithm to display the geolocation results in real time on the high-precision map, and real-time positioning and tracking are performed on the image plane and the high-precision map plane; Angle estimation is achieved by combining the horizontal field of view angle obtained by camera calibration, the camera's own heading angle, and the vehicle pixel position obtained by target tracking. The angle between the vehicle and the camera is calculated as follows: oh d =ω h +oh c Where, ω h Indicates the camera heading angle; ω c Indicates the angle between the vehicle and the camera heading angle; D represents the distance between the vehicle and the camera; ωd represents the clockwise angle between the distance D between the vehicle and the camera and the north direction; Distance estimation is achieved by combining the field of view angle obtained from camera calibration and the vehicle pixel position obtained from target tracking. The angle between the vehicle and the camera is calculated as follows: Where (x, y) is the pixel coordinate of the bottom center point of the vehicle detection box in the image; H is the camera installation height; Y represents the longitudinal distance between the vehicle and the camera; w, h are the pixel width and pixel height of the image respectively; θ represents the pitch angle of the camera; f represents the focal length of the camera; The final distance between the vehicle and the camera is determined by the longitudinal distance and the angle between the vehicle and the camera. The global positioning result is calculated by combining the distance and angle between the camera and the vehicle, and the camera's own longitude and latitude conversion. The final global positioning longitude and latitude result is calculated by the following formula: R＝6371.393×1000 Where R is the radius of the earth in meters; l, g represent the latitude and longitude of the vehicle respectively; l c 、g c Represents the latitude and longitude of the camera respectively; High-precision map tracking and positioning is performed by displaying the longitude and latitude obtained by image calculation on the map without human intervention.

6. The global high-precision map positioning and tracking system based on a roadside monocular camera according to claim 5 is characterized in that: The camera calibration algorithm only infers the vertical field of view angle, pitch angle and roll angle, does not require any manual input, and has no requirements for the scene.

7. The global high-precision map positioning and tracking system based on a roadside monocular camera according to claim 5 is characterized in that: In the target detection and tracking algorithm, the pixel position of the bottom center point of the 2D target detection box is used to calculate the distance and angle between the vehicle and the camera.

Images