A monitoring camera parameter calibration method and device
Through the target recognition and deep learning technology of monitoring video, the vanishing points and parameters of the monitoring camera are automatically calculated, which solves the problems of high calibration complexity and low efficiency in the existing technology, and realizes efficient camera parameter calibration and large-scale deployment. In various application scenarios.
Patent Information
- Application Number
- CN202110301461.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-22
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-03-22
AI Technical Summary
The existing monitoring camera parameter calibration method based on orthogonal vanishing points is highly complex and inefficient, making it difficult to achieve large-scale engineering deployment, and requires a lot of manual intervention.
By obtaining surveillance video, target recognition is performed to determine the set of line segments pointing to the vanishing point, automatically calculate the coordinates of the vanishing point, determine the camera focal length and internal reference matrix, combine deep learning to extract the line segments of the vehicle trajectory and headlights, calculate the rotation matrix and translation vector, Implement automatic calibration of camera parameters.
It improves the accuracy of vanishing points and line segment extraction, reduces the deviation of calculation results, and realizes large-scale camera parameter calibration without manual intervention, adapts to a variety of application scenarios, and can estimate the position and movement speed of ground plane points in the image.
Smart Images

Figure CN112950725B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a monitoring camera parameter calibration method and device. Background Art
[0002] With the rapid development of science and technology in modern society, the scale of road traffic continues to expand, and people's travel is becoming more and more convenient, but at the same time it has also brought a lot of negative effects, such as the number of traffic accidents is increasing year by year. In order to reduce the impact of traffic problems on people's travel and improve road management efficiency, various intelligent transportation products have emerged. Among them, video detection technology has the most attention due to its advantages of simplicity, convenience, strong intuitiveness and low cost, and has become an important research direction for the development of intelligent transportation systems.
[0003] In the prior art, when monitoring cameras are installed at high points or on the side of the road to capture road conditions, in order to improve detection accuracy and understand the scene, it is necessary to obtain the size, speed, and positional relationship of traffic participants when detecting various traffic events. The monitoring camera needs to be calibrated to establish an accurate mapping relationship from the pixels of the image to the world coordinate system. In the large-scale deployment of roadside cameras, it is necessary to minimize human intervention to improve efficiency and ensure safety. The current method with engineering feasibility is the calibration method based on orthogonal vanishing points. However, the existing method of calibrating camera parameters based on orthogonal vanishing points is highly complex and inefficient, making it difficult to achieve large-scale engineering deployment. Summary of the invention
[0004] The present invention aims to solve the technical problems in the related art at least to a certain extent. To achieve the above-mentioned purpose, in a first aspect, the present invention provides a monitoring camera parameter calibration method, which comprises:
[0005] Get surveillance video from surveillance cameras;
[0006] Performing target recognition according to the surveillance video to determine a first line segment set pointing to a first vanishing point and a second line segment set pointing to a second vanishing point;
[0007] determining the coordinates of the first vanishing point according to the first line segment set, and determining the coordinates of the second vanishing point according to the second line segment set;
[0008] Determine the focal length of the monitoring camera according to the coordinates of the first vanishing point and the second vanishing point, determine the intrinsic parameter matrix of the monitoring camera according to the focal length, and determine the coordinates of a third vanishing point according to the coordinates of the first vanishing point and the second vanishing point, wherein the three vanishing points are orthogonal to each other;
[0009] Determine the rotation matrix of the surveillance camera according to the coordinates of the three vanishing points and the intrinsic parameter matrix;
[0010] The translation vector of the surveillance camera is determined according to the center point of the image in the surveillance video, the intrinsic parameter matrix and the rotation matrix.
[0011] Furthermore, the performing target recognition according to the surveillance video to determine a first line segment set pointing to the first vanishing point and a second line segment set pointing to the second vanishing point includes:
[0012] Perform vehicle identification based on the surveillance video to determine the vehicle's trajectory segment and headlight connection segment;
[0013] The first line segment set is determined according to the trajectory line segments, and the second line segment set is determined according to the vehicle headlight connecting line segments.
[0014] Further, the performing of vehicle identification according to the surveillance video and determining the trajectory segment and the headlight connection segment of the vehicle includes:
[0015] Analyze the surveillance video frame by frame to determine the vehicle detection frame of each frame;
[0016] Determining the trajectory segment of the vehicle according to the vehicle detection frame;
[0017] Car lights are extracted according to the vehicle detection frame, and the car lights connecting line segment is determined according to the result of the car lights extraction.
[0018] Further, determining the trajectory segment of the vehicle according to the vehicle detection frame includes:
[0019] Determining the center of mass of the vehicle according to the vehicle detection frame;
[0020] The trajectory segment of the vehicle is determined based on the center of mass.
[0021] Further, determining the first line segment set according to the trajectory line segments includes:
[0022] Determining whether the trajectory segment meets a preset condition;
[0023] The first line segment set is determined according to all trajectory line segments that meet the preset condition.
[0024] Further, determining the focal length of the monitoring camera according to the coordinates of the first vanishing point and the coordinates of the second vanishing point, determining the intrinsic parameter matrix of the monitoring camera according to the focal length, and determining the coordinates of the third vanishing point according to the coordinates of the first vanishing point and the coordinates of the second vanishing point includes:
[0025] Constructing a pixel coordinate system according to the image in the surveillance video, and determining the pixel coordinates of the camera optical center of the surveillance camera in the pixel coordinate system;
[0026] determining pixel coordinates of the first vanishing point and the second vanishing point in the pixel coordinate system;
[0027] Determine a camera focal length according to the pixel coordinates of the camera optical center, the first vanishing point, and the second vanishing point, and determine an intrinsic parameter matrix of the surveillance camera according to the focal length;
[0028] The coordinates of the third vanishing point are determined according to the coordinates of the optical center of the camera, the first vanishing point, the second vanishing point in a world coordinate system, and the focal length of the camera.
[0029] Furthermore, determining the rotation matrix of the surveillance camera according to the coordinates of the three vanishing points and the intrinsic parameter matrix includes:
[0030] Determining a linear relationship between the three vanishing points and the rotation matrix according to the coordinates of the three vanishing points and the corresponding scale parameters;
[0031] The rotation matrix is solved according to the linear relationship.
[0032] Furthermore, the determining the translation vector of the surveillance camera according to the center point of the image in the surveillance video, the intrinsic parameter matrix and the rotation matrix includes:
[0033] The ground plane position of the center point of the image in the monitoring video is determined as the origin of the world coordinate system, and a conversion equation between the camera coordinate system and the world coordinate system is constructed;
[0034] The translation vector is solved according to the transformation equation.
[0035] Further, solving the translation vector according to the conversion equation includes:
[0036] Substituting the installation height of the monitoring camera into the conversion equation to solve the translation vector; or
[0037] Substituting the coordinates of any two points on the ground plane in the monitoring video in the pixel coordinate system and the actual horizontal distance difference between the two points into the conversion equation to solve the translation vector; or
[0038] The translation vector is solved by substituting the coordinates of any two points on the vertical ground plane in the monitoring video in the pixel coordinate system and the actual vertical height difference between the two points into the conversion equation.
[0039] To achieve the above object, in a second aspect, the present invention provides a monitoring camera parameter calibration device, which includes:
[0040] An acquisition module is used to acquire surveillance videos from surveillance cameras;
[0041] a vanishing point determination module, configured to perform target recognition according to the surveillance video, determine a first line segment set pointing to a first vanishing point and a second line segment set pointing to a second vanishing point; and further configured to determine the coordinates of the first vanishing point according to the first line segment set, and determine the coordinates of the second vanishing point according to the second line segment set;
[0042] A parameter calibration module, used to determine the focal length of the monitoring camera according to the coordinates of the first vanishing point and the second vanishing point, and to determine the intrinsic parameter matrix of the monitoring camera according to the focal length, and to determine the coordinates of the third vanishing point according to the coordinates of the first vanishing point and the second vanishing point, wherein the three vanishing points are orthogonal to each other; and to determine the rotation matrix of the monitoring camera according to the coordinates of the three vanishing points and the intrinsic parameter matrix; and to determine the translation vector of the monitoring camera according to the center point of the image in the monitoring video, the intrinsic parameter matrix and the rotation matrix.
[0043] The monitoring camera parameter calibration method or device of the present invention is used to detect targets in monitoring videos, and two vanishing points in the picture are determined based on the line segment set pointing to the vanishing points, which effectively improves the accuracy of extracting the line segments pointing to the vanishing points, thereby effectively reducing the deviation of the vanishing point calculation results, and then the third vanishing point is determined based on the two vanishing points and the optical center of the camera. The three vanishing points in the monitoring camera picture are automatically determined, and the camera internal parameters are calibrated based on the three vanishing points. The camera external parameters are flexibly calibrated through a variety of optional methods, which is less restricted by environmental references, suitable for more application scenarios, and does not require manual intervention, and is easy to achieve large-scale deployment. In addition, the present invention can estimate the position, height, and movement speed of any point on the ground plane in the image based on the calibration results and coordinate conversion.
[0044] To achieve the above-mentioned purpose, in a third aspect, the present invention provides a non-temporary computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the monitoring camera parameter calibration method according to the first aspect of the present invention is implemented.
[0045] To achieve the above-mentioned purpose, in a fourth aspect, the present invention provides a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the monitoring camera parameter calibration method according to the first aspect of the present invention is implemented.
[0046] The non-transitory computer-readable storage medium and computing device according to the present invention have similar beneficial effects as the monitoring camera parameter calibration method according to the first aspect of the present invention, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 A schematic diagram of a flow chart of a monitoring camera parameter calibration method according to an embodiment of the present invention;
[0048] Figure 2 A schematic diagram of a process of determining a line segment set according to an embodiment of the present invention;
[0049] Figure 3 A schematic diagram of a process for determining a vehicle trajectory segment and a vehicle headlight connection segment according to an embodiment of the present invention;
[0050] Figure 4 A schematic diagram of a process of determining a first line segment set according to an embodiment of the present invention;
[0051] Figure 5 A schematic diagram of a straight line trajectory screened according to vehicle tracking according to an embodiment of the present invention;
[0052] Figure 6 A schematic diagram of detecting and obtaining connection lines between the headlights and taillights of a vehicle according to an embodiment of the present invention;
[0053] Figure 7 A schematic diagram of a process for determining an intrinsic parameter matrix and coordinates of a third vanishing point according to an embodiment of the present invention;
[0054] Figure 8 A schematic diagram of automatically determining a vanishing point of a surveillance video according to an embodiment of the present invention;
[0055] Fig. 9 A schematic diagram of a process for determining a rotation matrix according to an embodiment of the present invention;
[0056] Fig.10 A schematic diagram of a process for determining a translation vector according to an embodiment of the present invention;
[0057] Fig.11 is a schematic diagram of a ground plane distance grid drawn in units of 2 meters according to an embodiment of the present invention;
[0058] Fig.12 Schematic diagram of the structure of a surveillance camera parameter calibration device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0059] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings. When describing the drawings, unless otherwise indicated, the same reference numerals in different drawings represent the same or similar elements. It should be noted that the embodiments described in the following exemplary embodiments do not represent all embodiments of the present invention. They are only examples of devices and methods consistent with some aspects disclosed in the present invention as detailed in the claims, and the scope of the present invention is not limited to this. The features of the various embodiments of the present invention can be combined with each other without contradiction.
[0060] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0061] In the current traffic monitoring scenario, monitoring cameras (cameras) can be used to capture road conditions at high points or on the side of the road. With the rapid development of deep learning, deep neural networks can be used to detect traffic participants such as vehicles and pedestrians in the scene, as well as various traffic events, but size, speed and spatial information cannot be obtained. Purely visual methods can be used to correlate image coordinates with world coordinates. For example, Zhang's calibration method and other methods that rely on calibration plates, place the calibration plates at various locations to take multiple photos, and convert camera calibration into a PnP (Perspective-n-Point) problem, that is, determine the position of the camera in space based on the relative position relationship of n points in a given space. Another method is to use a camera to capture the same point at multiple locations, and reconstruct the space based on the camera's movement and the relationship between three points. Neither of these two methods is feasible in engineering scenarios where thousands of cameras are deployed. The method of calibrating camera parameters based on orthogonal vanishing points on the road has higher flexibility and the potential for automated calibration. However, existing methods cannot achieve unsupervised calibration and require a lot of manual intervention and measurement.
[0062] The present invention extracts a set of line segments pointing to the vanishing point through a deep learning-based target detection method, automatically determines the vanishing point of the camera screen, and then calibrates the camera's internal and external parameters based on the vanishing point. This realizes automatic calibration of camera parameters in surveillance camera screens based on side or front shots, and does not require manual feature labeling and parameter adjustment steps, so large-scale engineering deployment of surveillance cameras can be carried out efficiently and quickly.
[0063] First, the following definitions of terms are given, which will not be repeated in the following text:
[0064] Vanishing point (VP point): refers to the intersection of parallel lines. In physical space, parallel lines can only intersect at infinity, so the VP point is at infinity. But in perspective, two parallel lines will easily intersect at one point, which is the VP point.
[0065] Pixel coordinate system: Pixel coordinates are the positions of pixels in an image. Generally, the top left vertex of a pixel coordinate system is the far point, the horizontal right is the u axis, and the vertical downward is the v axis.
[0066] World coordinate system: The world coordinate system is a mapping relationship between an image and a real object. In monocular vision, it is a mapping relationship between the size of a real object and the size of an image.
[0067] Camera intrinsic parameters: Intrinsic parameters are the inherent properties of the camera, which are actually the focal length and pixel size.
[0068] Camera extrinsic parameters: The extrinsic parameters are the rotation and translation transformation relationship of the camera relative to the world coordinate system.
[0069] Figure 1 FIG. 1 is a flow chart of a method for calibrating parameters of a surveillance camera according to an embodiment of the present invention, including steps S1 to S6 .
[0070] In step S1, a surveillance video of a surveillance camera is obtained. In an embodiment of the present invention, a surveillance video of a surveillance camera to be calibrated is obtained, and a video screen in the surveillance video may be recorded, for example, with a length of 5 minutes, for subsequent automatic determination of the vanishing point.
[0071] In step S2, target recognition is performed based on the surveillance video to determine a first line segment set pointing to the first vanishing point and a second line segment set pointing to the second vanishing point. Figure 2 It is a schematic diagram of a process of determining a line segment set according to an embodiment of the present invention, including steps S21 to S22.
[0072] In step S21, the vehicle is identified based on the monitoring video to determine the vehicle's trajectory segment and headlight segment. Figure 3 It is a schematic diagram of a process of determining a trajectory segment and a headlight connection segment of a vehicle according to an embodiment of the present invention, including steps S211 to S213.
[0073] In step S211, a frame-by-frame analysis is performed on the surveillance video to determine the vehicle detection frame of each frame. In an embodiment of the present invention, the recorded surveillance video can be input into a pre-trained target detection model for frame-by-frame analysis to determine the vehicle detection frame in each frame. It is understandable that the pre-trained target detection model can adopt an existing target detection model such as Yolo V3, and the present invention is not limited thereto. It is also understandable that each frame may contain multiple vehicle detection frames at the same time, which will be processed according to subsequent algorithms.
[0074] In step S212, the trajectory segment of the vehicle is determined according to the vehicle detection frame. In an embodiment of the present invention, the center of mass of the vehicle is determined according to the vehicle detection frame, and the trajectory segment of the vehicle is determined according to the center of mass. Specifically, in an embodiment of the present invention, the image at time T-1 is determined for target detection, and the vehicle detection frame contained therein is determined. Then, based on the vehicle motion state of the previous frame, the position of the vehicle at time T is predicted, and the vehicle features are extracted and matched and fused with the spatial information. According to the fused position prediction and the target features, the vehicle detection frame is updated to track the vehicle. All vehicle detection frames are tracked, the center of mass of the vehicle in the vehicle detection frame is determined, and the connection line of the center of mass is saved as the trajectory segment of the vehicle. For example, the DeepSort model can be used to track the vehicle, and the tracking is stopped when the vehicle detection frame contacts the edge of the picture to prevent the center of mass detection jump caused by only part of the vehicle body, which affects the accuracy of the detection. After all vehicle detection frames are tracked, all trajectory segments are saved as an alternative set for subsequent processing.
[0075] In step S213, the headlights are extracted according to the vehicle detection frame, and the headlight line segment is determined according to the result of the headlight extraction. In an embodiment of the present invention, the vehicle detection frame identified by the target detection model is subjected to headlight extraction using a pre-trained vehicle key point extraction model based on a deep neural network. The center of mass of the vehicle in the image is extracted from the vehicle detection frame, and the headlights are identified from the vehicle detection frame. It can be understood that the method of headlight recognition is based on vehicle detection, and headlight extraction is performed in the vehicle detection bounding box. This method is still based on target detection based on deep learning. The training samples include images and labels under various lighting conditions and whether the headlights are turned on. After extracting the headlights, the two endpoints of the headlight line are obtained by centroid selection to determine the headlight line segment.
[0076] It is understandable that in the embodiment of the present invention, if the vehicle lights cannot be extracted, it is most likely because the vehicle is blocked, and the vehicle detection frame is marked as blocked. If the headlights or taillights are extracted, the vehicle detection frame is marked as not blocked, and the vehicle light line segment determined therefrom is saved.
[0077] It is understandable that in the embodiment of the present invention, target detection can also be performed by dynamic foreground extraction, image segmentation, etc., and straight lines pointing to two vanishing points parallel to the ground plane in the picture can be extracted to calculate the positions of the vanishing points.
[0078] In step S22, a first line segment set is determined according to the trajectory line segments, and a second line segment set is determined according to the headlight line segments. Figure 4 It is a schematic diagram of a process of determining a first line segment set according to an embodiment of the present invention, including steps S221 to S222.
[0079] In step S221, it is determined whether the trajectory segment meets the preset conditions. In the embodiment of the present invention, since the trajectory segment is formed by tracking and determining the centroids of multiple vehicle detection frames obtained by the target detection model, various situations may occur during the vehicle detection process, such as occlusion leading to loss of tracking, deformation of the vehicle detection frame, and the tracking model not adapting to the characteristics. The trajectory segments obtained in these cases will have large jumps and bends, which are actually erroneous tracking results and will affect the accuracy of the vanishing point determination. Therefore, certain preset conditions can be set to screen the trajectory segments, thereby improving the accuracy of the overall calculation. It can be understood that the preset conditions can be set, for example, according to the inflection point characteristics or bending angles of the trajectory connection, or according to actual conditions, and the present invention does not limit this.
[0080] In step S222, the first segment set is determined based on all trajectory segments that meet the preset conditions. In the embodiment of the present invention, the trajectory segments that do not meet the preset conditions are deleted, and deleting the wrong results helps to improve the accuracy of the subsequent determination of the vanishing point. All trajectory segments that meet the preset conditions are retained as the first segment set. Figure 5 The figure shows a schematic diagram of a straight line trajectory screened according to vehicle tracking according to an embodiment of the present invention. It can be understood that since the trajectory segments all intersect in the lane direction in the perspective picture, all the trajectory segments in the first segment set point to the first vanishing point VP1.
[0081] In the embodiment of the present invention, the second line segment set is determined based on the line segments connecting the headlights. Since the line segments connecting the headlights are all line segments parallel to the ground plane and perpendicular to the lane direction, these line segments all intersect in the direction parallel to the ground plane and perpendicular to the lane in the perspective picture. Therefore, all the line segments connecting the headlights in the second line segment set point to the second vanishing point VP2. Figure 6 The figure shows a schematic diagram of obtaining connection lines of the headlights and taillights of a vehicle according to an embodiment of the present invention. It can be understood that preset conditions can also be set for filtering the headlight connection segments, and the headlight connection segments that do not meet the preset conditions are deleted, and only the headlight connection segments that meet the preset conditions are retained as the second segment set.
[0082] In step S3, the coordinates of the first vanishing point are determined according to the first line segment set, and the coordinates of the second vanishing point are determined according to the second line segment set. In an embodiment of the present invention, the coordinates of the first vanishing point are determined by a voting method according to the first line segment set, and the coordinates of the second vanishing point are determined by the voting method according to the second line segment set. The voting method includes establishing a pixel coordinate system and a diamond space coordinate system, converting the intersections of multiple parallel straight lines in the pixel coordinate system into the intersections of a broken line group in the diamond space, and converting the intersections of the broken line group in the diamond space into the pixel coordinate system to obtain the vanishing point coordinates in the pixel coordinate system.
[0083] In step S4, the focal length of the monitoring camera is determined according to the coordinates of the first vanishing point and the second vanishing point, the intrinsic parameter matrix of the monitoring camera is determined according to the focal length, and the coordinates of the third vanishing point are determined according to the coordinates of the first vanishing point and the second vanishing point, wherein the three vanishing points are orthogonal to each other. Figure 7 It is a schematic diagram of a process of determining the intrinsic parameter matrix and the coordinates of the third vanishing point according to an embodiment of the present invention, including steps S41 to S44.
[0084] In step S41, a pixel coordinate system is constructed according to the image in the surveillance video, and the pixel coordinates of the camera optical center of the surveillance camera in the pixel coordinate system are determined. In the embodiment of the present invention, a three-dimensional coordinate system is reconstructed according to the image in the surveillance video, with the upper left corner of the image as the origin of the pixel coordinate system, and the unit is pixel. Then, the coordinates of the camera optical center of the surveillance camera in the pixel coordinate system are determined as ,in, and represents the pixel coordinates of the camera optical center in the image, Indicates the focal length of the surveillance camera.
[0085] In step S42, the pixel coordinates of the first vanishing point and the second vanishing point in the pixel coordinate system are determined. In the embodiment of the present invention, the pixel coordinates of the first vanishing point VP1 are reconstructed in the pixel coordinate system as: vp1=(x1, y1, 0), and the pixel coordinates of the second vanishing point VP2 are reconstructed as: vp2=(x2, y2, 0).
[0086] In step S43, the camera focal length is determined according to the pixel coordinates of the camera optical center, the first vanishing point and the second vanishing point, and the intrinsic parameter matrix of the monitoring camera is determined according to the focal length. In the embodiment of the present invention, according to the camera imaging model, the lines connecting the first vanishing point VP1 and the second vanishing point VP2 with the camera optical center are perpendicular to each other, that is, the dot product of the two vectors is equal to 0, as shown in the following formula:
[0087]
[0088] Substituting the coordinates of the first vanishing point VP1, the second vanishing point VP2 and the camera optical center, we have:
[0089]
[0090] Thus, the focal length of the surveillance camera can be calculated .
[0091] In the embodiment of the present invention, according to the focal length of the surveillance camera The internal parameter matrix of the surveillance camera can be determined as:
[0092]
[0093] in, and They represent the focal length of the surveillance camera in the x and y axis directions respectively. Assuming that the optical center of the camera is the exact center of the image, the focal length in the x and y directions is equal, that is, , and Indicates the pixel coordinates of the camera's optical center in the image.
[0094] In step S44, the coordinates of the third vanishing point are determined according to the coordinates of the camera optical center, the first vanishing point, the second vanishing point in the world coordinate system and the camera focal length. In the embodiment of the present invention, according to the orthogonal characteristics of the vanishing points, the third vanishing point VP3 can be directly obtained by the vector cross product of the line connecting the first two vanishing points and the camera optical center (because they are orthogonal, the three vanishing points are perpendicular to each other). In the world coordinate system, the coordinates of the first vanishing point VP1, the second vanishing point VP2 and the camera optical center are respectively:
[0095]
[0096] Then the coordinates of the third vanishing point VP3 in the world coordinate system are:
[0097]
[0098] It can be understood that the coordinates of the third vanishing point VP3 can be obtained from the coordinates of the world coordinate system Convert the coordinates to pixel coordinates:
[0099]
[0100] Figure 8The figure is a schematic diagram of automatically determining vanishing points of a surveillance video according to an embodiment of the present invention. By selecting pixel points at a certain distance interval in the surveillance image and drawing vectors starting from the pixel points and pointing to the three vanishing points, it can be seen that the pixel points all have three vectors that are parallel to the road, perpendicular to the road on the ground plane, and perpendicular to the ground plane and pointing to the sky, which illustrates the effectiveness and accuracy of automatically determining the three vanishing points of a surveillance video according to the embodiment of the present invention.
[0101] In step S5, the rotation matrix of the monitoring camera is determined according to the coordinates of the three vanishing points and the intrinsic parameter matrix. Fig. 9 It is a schematic diagram of a process of determining a rotation matrix according to an embodiment of the present invention, including steps S51 to S52.
[0102] In step S51, the linear relationship between the three vanishing points and the rotation matrix is determined according to the coordinates of the three vanishing points and the corresponding scale parameters. In the embodiment of the present invention, the relationship between the pixel points in the pixel coordinate system and the world coordinate system can be expressed as:
[0103]
[0104] in, represents the magnification scale value, (u, v) represents the coordinates of the point in the pixel coordinate system, K represents the intrinsic parameter matrix, R represents the rotation matrix of the surveillance camera, t represents the translation vector of the surveillance camera, and (X, Y, Z) represents the coordinates of the point in the world coordinate system.
[0105] Since the three vanishing points are located at infinity in three orthogonal directions, the following equations can derive simple constraints on the projection matrix elements:
[0106]
[0107]
[0108] The simultaneous formula can eliminate t and establish a linear relationship between the vanishing point and the rotation matrix:
[0109]
[0110] in, Represent the coordinates of the three vanishing points on the image, They represent three scale parameters respectively, K represents the intrinsic parameter matrix, and R represents the rotation matrix.
[0111] In step S52, the rotation matrix is solved according to the linear relationship. In the embodiment of the present invention, since the rotation matrix is an orthogonal matrix, its determinant is 1, and each column vector is a unit vector and is orthogonal to each other, the inverse of the rotation matrix is equal to its transpose, that is, it satisfies , so by The above formula can be converted into the following formula:
[0112]
[0113] In the embodiment of the present invention, when three vanishing points and the intrinsic parameter matrix K are known, three parameters can be obtained: , thus obtaining the rotation matrix R.
[0114] In step S6, the translation vector of the surveillance camera is determined according to the center point of the image in the surveillance video, the intrinsic parameter matrix and the rotation matrix. Fig.10 It is a schematic diagram of a process of determining a translation vector according to an embodiment of the present invention, including steps S101 to S102.
[0115] In step S101, the ground plane position of the center point of the image in the surveillance video is determined as the origin of the world coordinate system, and a conversion equation between the camera coordinate system and the world coordinate system is constructed. In an embodiment of the present invention, the ground plane position of the center of the image is used as the origin of the world coordinate system, and the conversion equation between the camera coordinate system and the world coordinate system is determined as:
[0116]
[0117] in, Indicates the current zoom scale value, which is different at different coordinate points. Indicates the x coordinate of the surveillance camera in the world coordinate system, Indicates the y coordinate of the surveillance camera in the world coordinate system, h It represents the height of the surveillance camera in the world coordinate system. It can be understood that the equation on the left side of the above formula is:
[0118]
[0119] Representing the image center point under the basic camera imaging formula In the world coordinate system, it is the origin.
[0120] In step S102, the translation vector is solved according to the conversion equation. In the embodiment of the present invention, the translation vector can be calibrated according to the following three different schemes.
[0121] Solution 1: Substitute the installation height of the surveillance camera into the conversion equation to solve the translation vector. In this embodiment of the present invention, the installation height of the surveillance camera is known. h , then directly substitute into the above transformation equation to solve the three unknowns in the linear equation system , and .
[0122] According to the solution , we can get the following formula about t 1. t 2. t 3 linear equations:
[0123]
[0124] Solving the linear equations, we can obtain the translation vector t .
[0125] Solution 2: Substitute the coordinates of any two points on the ground plane in the monitoring video in the pixel coordinate system and the actual horizontal distance difference between the two points into the conversion equation to solve the translation vector. In this embodiment of the present invention, the coordinates of two points on the road plane in the pixel coordinate system are obtained as follows: and , and the actual distance difference between the two points is , the actual coordinates of the two unknown points are and , then there is the formula:
[0126]
[0127]
[0128] By combining the formulas, we can get:
[0129]
[0130] Make an appointment The item can be obtained:
[0131]
[0132] Combined with the following formula:
[0133]
[0134] We can get a quaternary nonlinear equation system with four unknown variables: , , and The system of equations has two sets of solutions, positive and negative. Choose according to the actual situation.
[0135] Known Then the following formula can be used to establish a three-variable linear equation system and solve it. :
[0136]
[0137] According to the solution , then the following formula is used to obtain t 1. t 2. t 3 linear equations:
[0138]
[0139] Solving the linear equations, we can obtain the translation vector t .
[0140] Solution 3: Substitute the coordinates of any two points on the vertical plane in the monitoring video in the pixel coordinate system and the actual vertical height difference between the two points into the transformation equation to solve the translation vector. and ,but and H is known but not known and , then the coordinate system conversion formula is as follows:
[0141]
[0142] Subtract the above two equations and eliminate t get:
[0143]
[0144] Then solving the linear equation gives and According to the solution Substitute the following formula to solve the linear equations:
[0145]
[0146] The solution is , and According to the solution , we can get the following formula about t 1. t 2. t 3 linear equations:
[0147]
[0148] Solving the linear equations, we can obtain the translation vector t .
[0149] Therefore, the camera's external parameters can be flexibly calibrated according to a variety of optional methods, with little restriction on environmental references, and can be adapted to more application scenarios.
[0150] In the embodiment of the present invention, based on the mutual conversion of coordinate systems, it is also possible to realize functions such as inferring the world coordinates from the pixel coordinates of the object and estimating the position and height of the object on the road surface.
[0151] When converting from world coordinates to pixel coordinates, only objects at points close to the ground plane in pixel coordinates are , use the formula:
[0152]
[0153] Solving linear equations , and , you can get the coordinates in the world coordinate system .
[0154] Get the lower endpoint of the object perpendicular to the ground in the pixel coordinate system , and the upper endpoint When , we have the formula:
[0155]
[0156] Solving a System of Linear Equations , and , get the coordinates in the world coordinate system , and by the formula:
[0157]
[0158] Solution and H, H is the actual height of the object.
[0159] When converting from pixel coordinates to world coordinates, the coordinates in the world coordinate system are known. , solve The value of can be obtained as a vector , then the image coordinates are .
[0160] This enables the calibration of camera parameters in traffic monitoring camera images based on side or front shots, and can infer the world coordinates from the pixel coordinates of the object, as well as estimate the position and height of objects on the road.
[0161] Fig.11The figure shows a schematic diagram of a ground plane distance grid drawn in units of 2 meters according to an embodiment of the present invention. The center point of the image is taken as the origin, and other points are selected around it at intervals of 2 meters in the world coordinate system. After conversion from the world coordinate system to the pixel coordinate system, a ground scale is formed in the picture. It is known that the road width is 3.75 meters. The ground scale drawn by the embodiment of the present invention can basically reflect the coordinate information of the real road surface, and can be used for applications such as road object size calculation, speed calculation, and relative position restoration.
[0162] The monitoring camera parameter calibration method of the embodiment of the present invention is adopted to detect the vehicle target in the monitoring video, determine the vehicle detection frame therein, track the vehicle detection frame to determine the trajectory segment, extract the key points of the vehicle detection frame, determine the light line segment, effectively improve the accuracy of the line segment extraction pointing to the vanishing point, determine the two vanishing points in the monitoring video image along the lane direction and parallel to the ground plane and perpendicular to the lane direction according to the trajectory segment and the light line segment, thereby effectively reducing the deviation of the vanishing point calculation result, and then determine the third vanishing point according to the two vanishing points and the camera optical center, realizing a universal calibration method for the vanishing point in the monitoring camera image of the side shot or the front shot, based on the feature extraction and analysis of deep learning, without the steps of artificial features and parameter adjustment, by automatically determining the three vanishing points in the monitoring camera image, and calibrating the camera internal parameters according to the three vanishing points, and flexibly calibrating the camera external parameters through a variety of optional methods, with little restriction on environmental reference, adapting to more application scenarios, and without manual intervention, easy to achieve large-scale deployment. And according to the calibration results and coordinate conversion, the present invention can estimate the position, height and movement speed of any point on the ground plane in the image.
[0163] The embodiment of the second aspect of the present invention also provides a monitoring camera parameter calibration device. Fig.12 1 is a schematic diagram of the structure of a surveillance camera parameter calibration device 1200 according to an embodiment of the present invention, which includes an acquisition module 1201 , a vanishing point determination module 1202 and a parameter calibration module 1203 .
[0164] The acquisition module 1201 is used to acquire the surveillance video of the surveillance camera.
[0165] The vanishing point determination module 1202 is used to perform target recognition based on the surveillance video, determine a first line segment set pointing to a first vanishing point and a second line segment set pointing to a second vanishing point; and is also used to determine the coordinates of the first vanishing point based on the first line segment set, and determine the coordinates of the second vanishing point based on the second line segment set.
[0166] The parameter calibration module 1203 is used to determine the focal length of the monitoring camera according to the coordinates of the first vanishing point and the coordinates of the second vanishing point, and to determine the intrinsic parameter matrix of the monitoring camera according to the focal length, and to determine the coordinates of the third vanishing point according to the coordinates of the first vanishing point and the coordinates of the second vanishing point, wherein the three vanishing points are orthogonal to each other; it is also used to determine the rotation matrix of the monitoring camera according to the coordinates of the three vanishing points and the intrinsic parameter matrix; it is also used to determine the translation vector of the monitoring camera according to the center point of the image in the monitoring video, the intrinsic parameter matrix and the rotation matrix.
[0167] In an embodiment of the present invention, the vanishing point determination module 1202 is further used to perform frame-by-frame analysis based on the surveillance video to determine the vehicle detection frame of each frame; determine the trajectory segment of the vehicle based on the vehicle detection frame; extract the headlights based on the vehicle detection frame, and determine the headlight line segment based on the result of the headlight extraction.
[0168] In the embodiment of the present invention, the parameter calibration module 1203 is also used to determine the ground plane position of the center point of the image in the monitoring video as the origin of the world coordinate system, construct a conversion equation between the camera coordinate system and the world coordinate system; solve the translation vector according to the conversion equation
[0169] The more specific implementation methods of the various modules of the surveillance camera parameter calibration device 1200 can refer to the description of the surveillance camera parameter calibration method of the present invention, and have similar beneficial effects, which will not be repeated here.
[0170] An embodiment of the third aspect of the present invention provides a non-transitory computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the monitoring camera parameter calibration method according to the first aspect of the present invention is implemented.
[0171] Generally speaking, the computer instructions for implementing the method of the present invention can be carried by any combination of one or more computer-readable storage media. Non-transitory computer-readable storage media can include any computer-readable media except for the signal itself that is temporarily propagating.
[0172] Computer readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.
[0173] Computer program code for performing the operation of the present invention can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages, in particular, Python suitable for neural network computing and platform frameworks based on TensorFlow, PyTorch, etc. can be used. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).
[0174] The fourth aspect of the present invention provides a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the monitoring camera parameter calibration method according to the first aspect of the present invention is implemented. It can be understood that the computing device of the present invention can be a server or a terminal device with limited computing power.
[0175] The non-temporary computer-readable storage medium and computing device according to the third and fourth aspects of the present invention can be implemented with reference to the contents specifically described in the embodiment according to the first aspect of the present invention, and have similar beneficial effects as the monitoring camera parameter calibration method according to the embodiment of the first aspect of the present invention, which will not be repeated here.
[0176] Although the embodiments of the present invention have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. A monitoring camera parameter calibration method, characterized in that: include: Get surveillance video from surveillance cameras; Performing target recognition according to the surveillance video to determine a first line segment set pointing to a first vanishing point and a second line segment set pointing to a second vanishing point; determining the coordinates of the first vanishing point according to the first line segment set, and determining the coordinates of the second vanishing point according to the second line segment set; Determine the focal length of the monitoring camera according to the coordinates of the first vanishing point and the second vanishing point, determine the intrinsic parameter matrix of the monitoring camera according to the focal length, and determine the coordinates of a third vanishing point according to the coordinates of the first vanishing point and the second vanishing point, wherein the first vanishing point, the second vanishing point, and the third vanishing point are orthogonal to each other; Determine a rotation matrix of the surveillance camera according to the coordinates of the first vanishing point, the second vanishing point, and the third vanishing point and the intrinsic parameter matrix; Determine the translation vector of the surveillance camera according to the center point of the image in the surveillance video, the intrinsic parameter matrix and the rotation matrix; The step of performing target recognition according to the surveillance video and determining a first line segment set pointing to the first vanishing point and a second line segment set pointing to the second vanishing point comprises: performing vehicle recognition according to the surveillance video and determining a trajectory segment and a headlight connecting segment of the vehicle; determining the first line segment set according to the trajectory segment, and determining the second line segment set according to the headlight connecting segment; Determining the focal length of the monitoring camera according to the coordinates of the first vanishing point and the second vanishing point, determining the intrinsic parameter matrix of the monitoring camera according to the focal length, and determining the coordinates of the third vanishing point according to the coordinates of the first vanishing point and the second vanishing point includes: constructing a pixel coordinate system according to the image in the monitoring video, and determining the pixel coordinates of the camera optical center of the monitoring camera in the pixel coordinate system; determining the pixel coordinates of the first vanishing point and the second vanishing point in the pixel coordinate system; determining the camera focal length according to the pixel coordinates of the camera optical center, the first vanishing point and the second vanishing point, and determining the intrinsic parameter matrix of the monitoring camera according to the focal length; determining the coordinates of the third vanishing point according to the coordinates of the camera optical center, the first vanishing point, the second vanishing point in the world coordinate system and the camera focal length.
2. The monitoring camera parameter calibration method according to claim 1, characterized in that: The step of identifying the vehicle according to the monitoring video and determining the trajectory segment and the headlight connection segment of the vehicle comprises: Analyze the surveillance video frame by frame to determine the vehicle detection frame of each frame; Determining the trajectory segment of the vehicle according to the vehicle detection frame; Car lights are extracted according to the vehicle detection frame, and the car lights connecting line segment is determined according to the result of the car lights extraction.
3. The monitoring camera parameter calibration method according to claim 2, characterized in that: The determining the trajectory segment of the vehicle according to the vehicle detection frame comprises: Determining the center of mass of the vehicle according to the vehicle detection frame; The trajectory segment of the vehicle is determined based on the center of mass.
4. The monitoring camera parameter calibration method according to claim 1, characterized in that: Determining a first line segment set according to the trajectory line segments comprises: Determining whether the trajectory segment meets a preset condition; The first line segment set is determined according to all trajectory line segments that meet the preset condition.
5. The monitoring camera parameter calibration method according to any one of claims 1 to 4, characterized in that: The step of determining the rotation matrix of the surveillance camera according to the coordinates of the first vanishing point, the second vanishing point, and the third vanishing point and the intrinsic parameter matrix comprises: determining a linear relationship between the first vanishing point, the second vanishing point, the third vanishing point and the rotation matrix according to the coordinates and corresponding scale parameters of the first vanishing point, the second vanishing point, and the third vanishing point; The rotation matrix is solved according to its linear relationship.
6. The monitoring camera parameter calibration method according to any one of claims 1 to 4, characterized in that: The determining the translation vector of the surveillance camera according to the center point of the image in the surveillance video, the intrinsic parameter matrix and the rotation matrix comprises: The ground plane position of the center point of the image in the monitoring video is determined as the origin of the world coordinate system, and a conversion equation between the camera coordinate system and the world coordinate system is constructed; The translation vector is solved according to the transformation equation.
7. The monitoring camera parameter calibration method according to claim 6, characterized in that: Solving the translation vector according to the conversion equation comprises: Substituting the installation height of the monitoring camera into the conversion equation to solve the translation vector; or Substituting the coordinates of any two points on the ground plane in the monitoring video in the pixel coordinate system and the actual horizontal distance difference between the two points into the conversion equation to solve the translation vector; or The translation vector is solved by substituting the coordinates of any two points on the vertical ground plane in the monitoring video in the pixel coordinate system and the actual vertical height difference between the two points into the conversion equation.
8. A monitoring camera parameter calibration device, characterized in that: include: An acquisition module is used to acquire surveillance videos from surveillance cameras; a vanishing point determination module, configured to perform target recognition according to the surveillance video, determine a first line segment set pointing to a first vanishing point and a second line segment set pointing to a second vanishing point; and further configured to determine the coordinates of the first vanishing point according to the first line segment set, and determine the coordinates of the second vanishing point according to the second line segment set; The step of performing target recognition according to the surveillance video and determining a first line segment set pointing to the first vanishing point and a second line segment set pointing to the second vanishing point comprises: performing vehicle recognition according to the surveillance video and determining a trajectory segment and a headlight connection segment of the vehicle; determining the first line segment set according to the trajectory segment, and determining the second line segment set according to the headlight connection segment; A parameter calibration module, for determining the focal length of the surveillance camera according to the coordinates of the first vanishing point and the second vanishing point, determining the internal parameter matrix of the surveillance camera according to the focal length, and determining the coordinates of a third vanishing point according to the coordinates of the first vanishing point and the second vanishing point, wherein the first vanishing point, the second vanishing point, and the third vanishing point are orthogonal to each other, and determining the focal length of the surveillance camera according to the coordinates of the first vanishing point and the second vanishing point, determining the internal parameter matrix of the surveillance camera according to the focal length, and determining the coordinates of the third vanishing point according to the coordinates of the first vanishing point and the second vanishing point comprises: constructing a pixel coordinate system according to the image in the surveillance video, determining the camera coordinates of the surveillance camera The method comprises the following steps: the pixel coordinates of the optical center in the pixel coordinate system; determining the pixel coordinates of the first vanishing point and the second vanishing point in the pixel coordinate system; determining the focal length of the camera according to the pixel coordinates of the optical center of the camera, the first vanishing point and the second vanishing point, and determining the intrinsic parameter matrix of the monitoring camera according to the focal length; determining the coordinates of the third vanishing point according to the coordinates of the optical center of the camera, the first vanishing point and the second vanishing point in the world coordinate system and the focal length of the camera; and determining the rotation matrix of the monitoring camera according to the coordinates of the first vanishing point, the second vanishing point and the third vanishing point and the intrinsic parameter matrix; and determining the translation vector of the monitoring camera according to the center point of the image in the monitoring video, the intrinsic parameter matrix and the rotation matrix.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the monitoring camera parameter calibration method according to any one of claims 1 to 7 is implemented.
10. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the monitoring camera parameter calibration method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Camera self-calibration method based on movement target image and movement information
CN101488222A
Method for camera self-calibration based on orthogonal vanishing points
CN105046691A
Road monitoring camera self-calibration method of utilizing road surface information
CN107492123A
Urban rail transit gate passing control method based on binocular camera behavior detection
CN109657581A
A camera calibration optimization method under a road cloud monitoring platform
CN109685855A
