Dynamic calibration and visual distance measurement method of vehicle-mounted camera

Abstract

The invention discloses a vehicle-mounted camera dynamic calibration and visual distance measurement method, which comprises a vehicle-mounted camera and a vehicle-mounted electronic terminal, and is characterized in that the vehicle-mounted camera is fixed on a vehicle, and after the vehicle-mounted camera is installed and fixed, the vehicle-mounted electronic terminal is connected with the vehicle-mounted electronic terminal under the condition that the length specification of a road marking line is known or unknown; a road marking vertex in a road picture shot by a vehicle-mounted camera is used as a calibration template for calibration, so that external parameters of the vehicle-mounted camera are calculated through a vehicle-mounted electronic terminal; under the condition that the length specifications of the road marking lines are known, four road marking line vertexes with known mutual distances are found to serve as calibration templates for calibration; under the condition that the length specification of the road marking is unknown, two transversely arranged road marking vertexes are found, the two vertexes are tracked in a subsequent video frame, and four position values of the two marking vertexes in a front frame and a rear frame are used as calibration templates for calibration; the method does not need a special template, is simple and convenient to implement through road marking, reduces errors and is widely applied to multiple fields.

Description

Technical Field

[0001] The invention belongs to the technical field of vehicle-mounted camera calibration, and more specifically, relates to a method for dynamic calibration and visual distance measurement of a vehicle-mounted camera. Background Art

[0002] With the continuous evolution of automotive electronics and intelligent technology, various sensors are widely used in automobiles in the form of pre-installed or post-installed. Cameras, as one of the most important sensors on vehicles, play a core role in many application scenarios such as assisted driving. However, before using the video images captured by the camera to execute various computer vision algorithms, the camera usually needs to be calibrated first. Automakers usually perform a detailed calibration process on the pre-installed cameras before the vehicle is produced, which involves using a calibration board to shoot and repeatedly adjust the internal and external parameters of the camera until the ideal calibration effect is achieved. This process is not only time-consuming and complicated to operate, but the cost of setting up the calibration environment is also quite high.

[0003] In patent CN105163065A, the vertex of the road marking is used as the calibration template of the camera, and the single vanishing point calibration method and the double vanishing point calibration method are used to calculate the external parameters of the camera. However, this patent is mainly aimed at roadside fixed cameras. For rear-mounted cameras such as dashcams, in addition to the pixel focal length of the camera, which can be predetermined, its external parameters such as installation height and orientation angle will vary depending on the installation method and location.

[0004] In order to efficiently and accurately calibrate such cameras to obtain their external parameters, the present invention proposes a vehicle-mounted camera dynamic calibration and visual distance measurement method that is simple to implement and highly feasible. Summary of the invention

[0005] In view of the above problems existing in the prior art, the present invention provides a method for dynamic calibration and visual distance measurement of a vehicle-mounted camera, which does not require a special template and can be easily implemented using road markings, thereby reducing errors and being widely applicable to multiple fields.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A dynamic calibration method for a vehicle-mounted camera comprises a vehicle-mounted camera and a vehicle-mounted electronic terminal. The vehicle-mounted camera is fixed on a vehicle. After the vehicle-mounted camera is installed and fixed, in the case where the road marking length specification is known or unknown, the road marking vertices in the road picture taken by the vehicle-mounted camera are used as a calibration template for calibration, so as to calculate the external parameters of the vehicle-mounted camera through the vehicle-mounted electronic terminal.

[0008] When the length specification of the road marking is known, four road marking vertices with known mutual distances are found as calibration templates for calibration;

[0009] When the length of the road marking is unknown, two vertices of the road marking are found and tracked in subsequent video frames. The four position values ​​of the two vertices in the previous and next frames are used as calibration templates for calibration.

[0010] The vehicle-mounted electronic terminal includes a high-precision positioning module, a vehicle-mounted inertial navigation unit (IMU) module, an on-board automatic diagnostic system (OBD) interface and an algorithm module. The high-precision positioning module integrates a global navigation satellite system (GNSS) unit and a geographic information system (GIS) unit.

[0011] As a further preferred technical solution of the present invention, when the length specification of the road marking is known, the dynamic calibration method of the vehicle-mounted camera specifically includes the following steps:

[0012] S1: Detect road markings from video frames captured by the vehicle camera;

[0013] S2: Find four road marking vertices from the detected road markings, and these four road marking vertices need to roughly form a rectangle in the ground plane;

[0014] S3: Determine the distances between the four road marking vertices in step S2;

[0015] S4: Obtain the image coordinates of the four road marking vertices in step S2 in the image captured by the vehicle-mounted camera, calibrate the vehicle-mounted camera using a single vanishing point calibration method, and finally calculate the external parameters of the vehicle-mounted camera.

[0016] As a further preferred technical solution of the present invention, when the length specification of the road marking is unknown, the dynamic calibration method of the vehicle-mounted camera specifically includes the following steps:

[0017] S1: Detect road markings from the video frame captured by the vehicle camera, and assume that the frame number is M;

[0018] S2: Find two road marking vertices from the detected road markings, and these two road marking vertices must be on the same horizontal line;

[0019] S3: continuously tracking the two road marking vertices in step S2 in subsequent video frames;

[0020] S4: obtaining the vehicle speed v from the vehicle electronic terminal, and calculating the distance from the tracked vertex to the original road marking vertex in the Mth frame using the vehicle speed v, the video frame rate, and the image frame difference;

[0021] S5: Obtain the image coordinates of the two road marking vertices in the Mth frame of the image captured by the vehicle camera, and then obtain the image coordinates of the two road marking vertices tracked in the M+Nth frame, a total of four points, to form a calibration template, and use the single vanishing point calibration method to calibrate the vehicle camera, and finally calculate the external parameters of the vehicle camera.

[0022] As a further preferred technical solution of the present invention, the algorithm module of the vehicle-mounted electronic terminal adopts lane line detection technology based on deep learning to achieve preliminary detection of lane lines. Subsequently, in order to perform virtual and real line segmentation and road marking vertex detection on the detected lane lines, the algorithm module of the vehicle-mounted electronic terminal further integrates computer vision algorithms of edge detection, color space segmentation and corner point detection.

[0023] As a further preferred technical solution of the present invention, the global navigation satellite system (GNSS) unit of the high-precision positioning module performs GNSS positioning on the calibration location, and based on the GNSS positioning information, in combination with the geographic information system (GIS) unit of the high-precision positioning module, queries the lane spacing, marking length and spacing between virtual and real lines of the calibration location to determine the distances between the four road marking vertices serving as the calibration template.

[0024] As a further preferred technical solution of the present invention, when the on-board electronic terminal continuously tracks two road marking vertices, the algorithm module adopts a faster and less computationally intensive optical flow algorithm for vertex tracking, or a hierarchical block matching algorithm (HBMA) based on a better tracking effect, or a tracking algorithm based on deep learning MDNet or TCNN.

[0025] As a further preferred technical solution of the present invention, the vehicle speed v is obtained from a high-precision positioning module, or from a vehicle-mounted inertial navigation unit (IMU) module, or from an on-board automatic diagnostic system (OBD) interface;

[0026] Then the distance that the tracked road marking vertex moves with the vehicle as the reference is calculated using the following formula:

[0027] d=v×N÷fps

[0028] As a further preferred technical solution of the present invention, the visual distance measurement method of the vehicle-mounted camera specifically includes the following steps:

[0029] S1: Obtain the image coordinates of the visual distance measurement target;

[0030] S2: When the target is at ground height, the image coordinates are converted into the actual scene coordinates through the coordinate conversion formula;

[0031] S3: Calculate the actual coordinates of the scene where the vehicle camera is located;

[0032] S4: The actual scene coordinates of the ranging target minus the actual scene coordinates of the vehicle camera are the distance of the ranging target relative to the vehicle camera, and the actual scene coordinates of the ranging target 1 minus the actual scene coordinates of the ranging target 2 are the relative distance between the two ranging targets;

[0033] S5: When the distance measurement target is not at the ground height, the distance measured by steps S2 to S4 is the projection distance. If the height of the distance measurement target is known, the actual distance of the distance measurement target can be obtained by the conversion formula of the projection distance and the actual distance.

[0034] As a further preferred technical solution of the present invention, based on the dynamic calibration method of a vehicle-mounted camera, the calculated external parameters of the vehicle-mounted camera, including the pitch angle t, rotation angle p, rotation angle s, pixel focal length f and camera height h, define the relationship between the image plane and the actual three-dimensional world coordinates.

[0035] As a further preferred technical solution of the present invention, after calculating the external parameters of the vehicle-mounted camera, the image coordinates are converted to the actual scene coordinates using the following coordinate conversion formula:

[0036]

[0037] Next, the actual coordinates of the scene where the vehicle camera is located are calculated using the following formula:

[0038]

[0039] When the distance measurement target is not at ground height, the projection distance and actual distance are converted using the following projection distance and actual distance conversion formula:

[0040]

[0041] As described above, the method for dynamic calibration and visual distance measurement of a vehicle-mounted camera provided by the present invention has the following beneficial effects:

[0042] 1. No special calibration templates and calibration workshops are required. Dynamic calibration can be performed using road markings, which is easy to implement and highly feasible.

[0043] 2. Using the single vanishing point calibration method to calibrate the external parameters of the vehicle camera can effectively avoid the pathological phenomenon when the vehicle camera is installed properly and reduce the calibration error.

[0044] 3. It can measure the height of all ground objects within the field of view of the vehicle camera or the distance between the target with known height and the vehicle camera, and can also measure the distance between targets. It has a wide range of applicability and can be expanded to multiple application fields based on vehicle electronic terminals.

[0045] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 A calibration template for a single vanishing point calibration method in a dynamic calibration and visual distance measurement method for a vehicle-mounted camera applied for by the present invention;

[0048] Figure 2 The vehicle-mounted camera imaging model in the vehicle-mounted camera dynamic calibration and visual distance measurement method applied for by the present invention;

[0049] Figure 3 This is one of the implementation schematic diagrams of a dynamic calibration method for a vehicle-mounted camera in the method for dynamic calibration and visual distance measurement of a vehicle-mounted camera applied for by the present invention;

[0050] Figure 4 The second schematic diagram of the implementation of a dynamic calibration method for a vehicle-mounted camera in the method for dynamic calibration and visual distance measurement of a vehicle-mounted camera applied for by the present invention;

[0051] Figure 5 The present invention is a schematic diagram of an implementation of a method for measuring visual distance of a vehicle-mounted camera in accordance with the method for measuring dynamic calibration and visual distance of a vehicle-mounted camera applied for by the present invention. DETAILED DESCRIPTION

[0052] The following is a description of the implementation of the present invention by means of specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification.

[0053] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions that the present invention can be implemented, so they have no substantial technical significance. Any modification of the structure, change in the proportional relationship, or adjustment of the size should still fall within the scope of the technical content disclosed by the present invention without affecting the effects that the present invention can produce and the purposes that can be achieved. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" quoted in this specification are only for the convenience of description, and are not used to limit the scope of the implementation of the present invention. The change or adjustment of the relative relationship should also be regarded as the scope of the implementation of the present invention without substantial changes in the technical content. The specific structure can be described with reference to the drawings of the patent application.

[0054] The present invention provides a method for dynamic calibration and visual distance measurement of a vehicle-mounted camera. Figures 1 to 5 shown.

[0055] A dynamic calibration method for a vehicle-mounted camera comprises a vehicle-mounted camera and a vehicle-mounted electronic terminal. The vehicle-mounted camera is fixed on a vehicle. After the vehicle-mounted camera is installed and fixed, in the case where the road marking length specification is known or unknown, the road marking vertices in the road picture taken by the vehicle-mounted camera are used as a calibration template for calibration, so as to calculate the external parameters of the vehicle-mounted camera through the vehicle-mounted electronic terminal.

[0056] When the length specification of the road marking is known, four road marking vertices with known mutual distances are found as calibration templates for calibration;

[0057] When the length of the road marking is unknown, two vertices of the road marking are found and tracked in subsequent video frames. The four position values ​​of the two vertices in the previous and next frames are used as calibration templates for calibration.

[0058] The vehicle-mounted electronic terminal includes a high-precision positioning module, a vehicle-mounted inertial navigation unit (IMU) module, an on-board automatic diagnostic system (OBD) interface and an algorithm module. The high-precision positioning module integrates a global navigation satellite system (GNSS) unit and a geographic information system (GIS) unit.

[0059] When the length specifications of the road markings are known, the dynamic calibration method of the vehicle-mounted camera specifically includes the following steps:

[0060] S1: Detect road markings from video frames captured by the vehicle camera;

[0061] S2: Find four road marking vertices from the detected road markings, and these four road marking vertices need to roughly form a rectangle in the ground plane;

[0062] S3: Determine the distances between the four road marking vertices in step S2;

[0063] S4: Obtain the image coordinates of the four road marking vertices in step S2 in the image captured by the vehicle-mounted camera, calibrate the vehicle-mounted camera using a single vanishing point calibration method, and finally calculate the external parameters of the vehicle-mounted camera.

[0064] The global navigation satellite system (GNSS) unit of the high-precision positioning module performs GNSS positioning on the calibration location, and inquires about the lane spacing, marking length and spacing between virtual and real lines of the markings at the calibration location based on the GNSS positioning information in combination with the geographic information system (GIS) unit of the high-precision positioning module to determine the distances between the four road marking vertices serving as the calibration template.

[0065] Combination Figure 1 As shown, using four road marking vertices as calibration templates and adopting the single vanishing point calibration method for calibration can effectively avoid the occurrence of pathological phenomena when the rotation angle is close to an integer multiple of 90°. It can adapt well to the side-mounted and front-mounted vehicle cameras, and is especially suitable for the front-mounted vehicle camera scene.

[0066] When the length specification of the road marking is unknown, the dynamic calibration method of the vehicle-mounted camera specifically includes the following steps:

[0067] S1: Detect road markings from the video frame captured by the vehicle camera, and assume that the frame number is M;

[0068] S2: Find two road marking vertices from the detected road markings, and these two road marking vertices must be on the same horizontal line;

[0069] S3: continuously tracking the two road marking vertices in step S2 in subsequent video frames;

[0070] S4: obtaining the vehicle speed v from the vehicle electronic terminal, and calculating the distance from the tracked vertex to the original road marking vertex in the Mth frame using the vehicle speed v, the video frame rate, and the image frame difference;

[0071] S5: Obtain the image coordinates of the two road marking vertices in the Mth frame of the image captured by the vehicle camera, and then obtain the image coordinates of the two road marking vertices tracked in the M+Nth frame, a total of four points, to form a calibration template, and use the single vanishing point calibration method to calibrate the vehicle camera, and finally calculate the external parameters of the vehicle camera.

[0072] When the on-vehicle electronic terminal continuously tracks two road marking vertices, the algorithm module adopts a faster and less computationally intensive optical flow algorithm, or a hierarchical block matching algorithm (HBMA) with better tracking effect, or a tracking algorithm based on deep learning such as MDNet or TCNN.

[0073] The vehicle speed v is obtained from a high-precision positioning module, an on-board inertial navigation unit (IMU) module, or an on-board automatic diagnostic system (OBD) interface;

[0074] Then the distance that the tracked road marking vertex moves with the vehicle as the reference is calculated using the following formula:

[0075] d=v×N÷fps

[0076] Where d is the distance the road marking vertex moves, v is the vehicle speed, N is the image frame difference, and fps is the video frame rate.

[0077] It should be noted that: the four road marking vertices in the Mth frame and the M+Nth frame of the vehicle camera are taken as the calibration template for camera calibration. The length of the calibration template can be calculated from the vehicle speed v. In the case of missing calibration template width, the double vanishing point calibration method can also be used for calibration. However, this calibration method will have a pathological phenomenon when the rotation angle is close to an integer multiple of 90°, that is, one of the two vanishing points is close to infinity, making the calculated vehicle camera parameters particularly sensitive to the error of the calibration point. This method is only applicable to the side-mounted camera environment;

[0078] For the upright angle of the vehicle-mounted camera, the single vanishing point calibration method is used to calibrate the vehicle-mounted camera. The one of the two vanishing points closer to the image origin is selected to effectively avoid the occurrence of pathological phenomena. The calibration template width required by the single vanishing point calibration method, that is, the lane width, can be replaced by a priori value when it is impossible to query from the geographic information system (GIS) unit based on the positioning information. The lane width of urban roads is usually 3.0 meters to 3.5 meters, and that of highways is usually 3.75 meters. The error ratio caused by using the priori value will not be too large.

[0079] When the length specification of the road marking is known or unknown, the algorithm module of the vehicle-mounted electronic terminal uses a lane line detection technology based on deep learning to achieve preliminary detection of the lane line. Subsequently, in order to perform virtual and real line segmentation and road marking vertex detection on the detected lane line, the algorithm module of the vehicle-mounted electronic terminal further integrates computer vision algorithms of edge detection, color space segmentation and corner point detection;

[0080] Specifically, the algorithm module of the vehicle-mounted electronic terminal adopts a lane line detection technology based on deep learning as a lane line detection model based on deep learning. The model can accurately identify lane lines in a complex road environment by training a large amount of labeled data;

[0081] After the lane lines are initially detected, the algorithm module introduces an edge detection algorithm (such as Canny edge detection) to capture the fine edges of the lane lines. At the same time, combined with the color space segmentation algorithm (such as threshold segmentation in HSV color space), the solid and dashed lines are further distinguished according to the distinctive features of the lane line colors (such as white or yellow). In addition, the judgment of the solid and dashed lines may also need to rely on the continuity or discontinuity characteristics of the lane line edges.

[0082] In order to detect the vertices of road markings, the algorithm module uses a corner detection algorithm (such as Harris corner detection or Shi-Tomasi corner detection) to identify significant corners in the image. These corners often correspond to the vertex positions of road markings. In addition, it is necessary to combine the geometry and direction of the lane lines to more accurately determine the vertex positions by fitting straight lines or curves.

[0083] A method for measuring the visual distance of a vehicle-mounted camera, the method for measuring the visual distance of a vehicle-mounted camera specifically comprises the following steps:

[0084] S1: Obtain the image coordinates of the visual distance measurement target;

[0085] S2: When the target is at ground height, the image coordinates are converted into the actual scene coordinates through the coordinate conversion formula;

[0086] S3: Calculate the actual coordinates of the scene where the vehicle camera is located;

[0087] S4: The actual scene coordinates of the ranging target minus the actual scene coordinates of the vehicle camera are the distance of the ranging target relative to the vehicle camera, and the actual scene coordinates of the ranging target 1 minus the actual scene coordinates of the ranging target 2 are the relative distance between the two ranging targets;

[0088] S5: When the distance measurement target is not at the ground height, the distance measured by steps S2 to S4 is the projection distance. If the height of the distance measurement target is known, the actual distance of the distance measurement target can be obtained by the conversion formula of the projection distance and the actual distance.

[0089] Based on the dynamic calibration method of a vehicle-mounted camera, the pitch angle t, the rotation angle p, the rotation angle s, the pixel focal length f and the camera height h in the calculated external parameters of the vehicle-mounted camera define the relationship between the image plane and the actual three-dimensional world coordinates, combined with Figure 2 As shown, where:

[0090] The pitch angle t is the vertical angle between the camera optical axis and the XY plane in the three-dimensional world coordinates;

[0091] The rotation angle p is the horizontal angle from the Y axis in the three-dimensional coordinate system in the counterclockwise direction to the projection line of the camera optical axis on the XY plane;

[0092] The rotation angle s refers to the rotation angle of the camera along its optical axis;

[0093] The pixel focal length f refers to the number of pixels corresponding to the focal length on the image plane, that is, the distance from the optical center to the image plane in pixels;

[0094] The camera height h refers to the vertical height from the center of the camera lens to the XY plane.

[0095] After calculating the external parameters of the vehicle camera, the image coordinates are converted to the actual scene coordinates using the following coordinate conversion formula:

[0096]

[0097] Among them, (X Q ,Y Q ) is the actual coordinate of the scene, (x q ,y q ) are image coordinates;

[0098] Next, the actual coordinates of the scene where the vehicle camera is located are calculated using the following formula:

[0099]

[0100] When the distance measurement target is not at ground height, the projection distance and actual distance are converted using the following projection distance and actual distance conversion formula:

[0101]

[0102] Among them, d a is the actual distance, d p is the projection distance, h cam is the camera height, h B is the target height.

[0103] The following is a specific embodiment of a dynamic calibration method for a vehicle-mounted camera when the road marking length specification is known or unknown:

[0104] Embodiment 1:

[0105] When the road marking length specifications are known, combined with Figure 3As shown, it is a video frame taken by the vehicle camera, and the image width and height of the video frame are 1920 and 1080 respectively. In the video frame, the algorithm module of the vehicle electronic terminal detects four road marking vertices using the lane line detection technology based on deep learning and the computer vision algorithm of corner point detection;

[0106] like Figure 3 As shown, the ABCD points marked in the figure roughly form a rectangle in the geodesic plane, and their image coordinates are A(748,613), B(625,717), C(1082,612), and D(1237,712);

[0107] The global navigation satellite system (GNSS) unit of the high-precision positioning module of the vehicle-mounted electronic terminal performs GNSS positioning on the calibration location. According to the positioning information, combined with the geographic information system (GIS) unit of the high-precision positioning module, it is found that the road markings at the calibration location are in a "2+4" mode, that is, each dotted line is 2 meters long, 4 meters apart, and the lane width is 3.25 meters, that is, the length of the calibration template (A to B) is 6 meters, and the width of the calibration template (A to C) is 3.25 meters. Based on this set of calibration data, the vehicle-mounted camera is calibrated using the single vanishing point calibration method, and the camera external parameters are calculated using the following formula:

[0108]

[0109] where α PQ =x q -x p , β PQ =y q -y p , c PQ =x p y q -x q y p ,

[0110]

[0111] Here

[0112]

[0113] in,

[0114] U Q =x q sins+y q coss,V Q =x q coss-y q sins

[0115] f=F / tant

[0116] If f<0, let f=-f, and s=s+p

[0117]

[0118] Wherein, L is the length of the calibration template, which is 6 meters in this embodiment, and W is the width of the calibration template, which is 3.25 meters in this embodiment;

[0119] (x q ,y q ) is the image coordinate of point Q. It should be pointed out that in the imaging model, the origin of the image coordinate is the position where the optical axis passes through the image, that is, the center point of the image. This is different from the upper left corner in traditional image processing, so coordinate conversion is required;

[0120] Take point A as an example. When the upper left corner is the origin, the image coordinates are (748,613). The following formula is used for coordinate conversion:

[0121] x q =x Q -Iw / 2,y q =Ih / 2-y Q

[0122] Among them, (x Q ,y Q ) is the image coordinate of point Q when the upper left corner is the origin, (x q ,y q ) is the image coordinate of point Q when the center point is the origin, Iw is the width of the image, which is 1920 in this embodiment, and Ih is the height of the image, which is 1080 in this embodiment. After conversion, the image coordinate of point A (x a ,y a ) is (-212, -73).

[0123] In this embodiment, the external parameters of the vehicle-mounted camera are obtained by the above formula as follows:

[0124] s=0.183351°, t=-2.09788°, p=87.1705°, f=1353.41(pixel), h=1.18852(m);

[0125] By using the above external parameters of the vehicle-mounted camera, the visual distance can be measured by the visual distance measurement method of the vehicle-mounted camera.

[0126] Embodiment 2:

[0127] When the length specification of road marking is unknown, Figure 4As shown in the figure, two frames of images are taken by the vehicle camera. The width and height of the video frames are 1920 and 1080 respectively. The upper picture is the Mth frame, and the lower picture is the M+30th frame. The frame rate of the video is 30, that is, the time interval between the upper and lower frames is one second. In the video frame in the upper picture, the algorithm module of the vehicle electronic terminal detects two road marking vertices using the lane line detection technology based on deep learning and the computer vision algorithm of corner point detection;

[0128] like Figure 4 As shown in the figure, the two road marking vertices are on the same horizontal line, namely point A and point C, and their image coordinates are A(748,613) and C(1082,612) respectively. The two road marking vertices are continuously tracked in subsequent video frames. In the M+30th frame, that is, in the figure below, the coordinates of the two tracking points are A~(557,778) and C~(1320,771) respectively.

[0129] Using the four points AA to CC as the calibration template, assuming that the vehicle is traveling in the forward direction of the road, the length AA to of the calibration template is approximately equal to the distance traveled by the vehicle in one second, and the vehicle speed v obtained from the high-precision positioning module or the vehicle-mounted inertial navigation unit (IMU) module or the vehicle-mounted automatic diagnostic system (OBD) interface is 27 kilometers per hour, and the distance traveled in one second is 7.5 meters, that is, the length of the calibration template is L = 7.5 meters, and the width of the calibration template is the middle value W = 3.25 meters of the lane width of the urban road. Based on this set of calibration data, the vehicle camera is calibrated using the single vanishing point calibration method, similar to the first embodiment, and the external parameters of the vehicle camera are calculated using the formula as follows:

[0130] s=0.103186°, t=-2.23208°, p=87.2463°, f=1366.78(pixel), h=1.22409(m);

[0131] By using the above external parameters of the vehicle-mounted camera, the visual distance can be measured by the visual distance measurement method of the vehicle-mounted camera.

[0132] In the above two embodiments, the external parameters of the vehicle-mounted camera obtained by calibration are relatively consistent with little difference.

[0133] Embodiment three:

[0134] Based on the external parameters of the vehicle-mounted camera calibration in Example 2, the visual distance is measured, combined with Figure 5 As shown, the figure shows a video frame taken by a vehicle-mounted camera, and the image width and height of the video frame are 1920 and 1080 respectively;

[0135] In this embodiment, a specific method and steps for measuring the distance between two motor vehicles in front of the video frame using the visual distance measurement method of the vehicle-mounted camera are provided:

[0136] First, the algorithm module uses the target detection algorithm to detect the motor vehicle target in the video frame. The target detection algorithm is the YOLO algorithm based on deep learning and detects the target to obtain the target detection frame, such as Figure 5 As shown in the figure, the midpoint of the lower bottom edge of the target detection box is used first for position and distance calculation, because the midpoint of the lower bottom edge of the target is generally close to the ground height, and the calculation process is relatively simple;

[0137] like Figure 5 As shown in the figure, the image coordinates of the midpoint of the bottom edge of the detection frame of the SUV vehicle in front are (912,604), and the image coordinates are converted using the following formula:

[0138] x q =x Q -Iw / 2,y q =Ih / 2-y Q

[0139] Then calculate the actual scene coordinates of the point using the following formula:

[0140]

[0141] The actual scene coordinates of this point are (17.13, -1.32). The actual scene coordinates use the ground position corresponding to the center point of the image (the position where the optical axis passes through the image plane) as the origin coordinates, so it is a relative coordinate. Then the actual scene coordinates of the intersection point of the vehicle camera position vertically to the ground are calculated using the following formula:

[0142]

[0143] The actual scene coordinates of the intersection of the vehicle camera position and the ground are (31.37, -1.51), from which the coordinate offset of the midpoint of the bottom edge of the vehicle detection frame relative to the vehicle camera position is (-14.24, 0.19), and the Euclidean distance is calculated to be approximately 14.24 meters;

[0144] like Figure 5As shown in the figure, the image coordinates of the midpoint of the bottom edge of the detection frame of the new energy taxi in the front left are (746,553). Similar to the calculation process of the SUV vehicle in front, the actual scene coordinates of this point are calculated to be (6.12, -4.23), while the coordinates of the onboard camera are still (31.37, -1.51). Therefore, the coordinate offset of the midpoint of the bottom edge of the vehicle detection frame relative to the onboard camera position is (-25.25, -2.72), and the Euclidean distance is calculated to be approximately 25.40 meters.

[0145] In addition to calculating the distance from a target to the vehicle camera position, the present invention can also calculate the distance between multiple targets. Figure 5 As shown in the figure, the actual scene coordinates of the midpoint of the bottom edge of the detection box of the SUV vehicle in the front of the figure are (17.13, -1.32), and the actual scene coordinates of the midpoint of the bottom edge of the detection box of the new energy taxi in the front left of the figure are (6.12, -4.23), the coordinate offset is (11.01, 2.91), and the Euclidean distance is 11.39 meters;

[0146] When the distance measurement target is not at ground height, if the distance measurement target height is known, the actual distance of the distance measurement target can be obtained by the following projection distance and actual distance conversion formula;

[0147]

[0148] In this embodiment, use Figure 5 The visual distance is measured at the midpoint of the detection frame of the new energy taxi in the middle left front. Since this point is not on the ground, its height needs to be estimated. According to the vehicle height of 1.6 meters, the height of this point is estimated to be 0.8 meters. The image coordinates of the midpoint of the detection frame are (746,509). The following formula is used for image coordinate conversion:

[0149] x q =x Q -Iw / 2,y q =Ih / 2-y Q

[0150] Then calculate the actual scene coordinates of the point using the following formula:

[0151]

[0152] The actual scene coordinates of the point projected on the ground are (-43.00, -9.52), while the camera coordinates are still (31.37, -1.51). Therefore, the coordinate offset of the projection point relative to the camera position is (-74.37, -8.01). The Euclidean distance is calculated to be approximately 74.80 meters, which is the projection distance. The projection distance is then converted to the actual distance using the following projection distance and actual distance conversion formula:

[0153]

[0154] It should be pointed out that the calibrated external parameters of the vehicle-mounted camera in Example 1 and Example 2 can be used not only in the scenario of Example 3. As long as the installation height angle of the vehicle-mounted camera does not change and the vehicle is traveling on a flat road, the above-mentioned vehicle-mounted camera visual distance measurement method can still maintain a high accuracy.

[0155] The dynamic calibration method of the vehicle-mounted camera provided by the present invention does not require a special calibration template and a calibration workshop, and uses road markings to perform dynamic calibration, which is simple to implement and has high feasibility.

[0156] The present invention uses a single vanishing point calibration method to calibrate the external parameters of the vehicle-mounted camera, which can effectively avoid pathological phenomena when the vehicle-mounted camera is installed normally and reduce calibration errors.

[0157] The visual distance measurement method of the vehicle camera provided by the present invention can measure the distance between all ground heights or known height targets and the vehicle camera within the field of view of the vehicle camera, and can also measure the distance between targets, which can be extended to multiple application fields based on vehicle electronic terminals. For example, the distance between the front vehicle can be measured in assisted driving to prevent rear-end collisions; the precise location of road diseases and road traffic incidents can also be given in combination with a high-precision positioning module; and crowdsourcing data collection of high-precision maps can also be performed in combination with a visual structured detection SDK. It has a wide range of applicability and has many beneficial effects.

[0158] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A dynamic calibration method for a vehicle-mounted camera, characterized in that: The invention comprises a vehicle-mounted camera and a vehicle-mounted electronic terminal, wherein the vehicle-mounted camera is fixed on the vehicle. After the vehicle-mounted camera is installed and fixed, when the road marking length specification is known or unknown, the road marking vertices in the road picture taken by the vehicle-mounted camera are used as a calibration template for calibration, so as to calculate the external parameters of the vehicle-mounted camera through the vehicle-mounted electronic terminal; When the length specification of the road marking is known, four road marking vertices with known mutual distances are found as calibration templates for calibration; When the length of the road marking is unknown, two vertices of the road marking are found and tracked in subsequent video frames, and the four position values ​​of the two vertices in the previous and next frames are used as calibration templates for calibration; The vehicle-mounted electronic terminal includes a high-precision positioning module, a vehicle-mounted inertial navigation unit (IMU) module, an on-board automatic diagnostic system (OBD) interface and an algorithm module. The high-precision positioning module integrates a global navigation satellite system (GNSS) unit and a geographic information system (GIS) unit.

2. The dynamic calibration method of a vehicle-mounted camera according to claim 1, characterized in that: When the length specifications of the road markings are known, the dynamic calibration method of the vehicle-mounted camera specifically includes the following steps: S1: Detect road markings from video frames captured by the vehicle camera; S2: Find four road marking vertices from the detected road markings, and these four road marking vertices need to roughly form a rectangle in the ground plane; S3: Determine the distances between the four road marking vertices in step S2; S4: Obtain the image coordinates of the four road marking vertices in step S2 in the image captured by the vehicle-mounted camera, calibrate the vehicle-mounted camera using a single vanishing point calibration method, and finally calculate the external parameters of the vehicle-mounted camera.

3. The dynamic calibration method of a vehicle-mounted camera according to claim 1, characterized in that: When the length specification of the road marking is unknown, the dynamic calibration method of the vehicle-mounted camera specifically includes the following steps: S1: Detect road markings from the video frame captured by the vehicle camera, and assume that the frame number is M; S2: Find two road marking vertices from the detected road markings, and these two road marking vertices must be on the same horizontal line; S3: continuously tracking the two road marking vertices in step S2 in subsequent video frames; S4: obtaining the vehicle speed v from the vehicle electronic terminal, and calculating the distance from the tracked vertex to the original road marking vertex in the Mth frame using the vehicle speed v, the video frame rate, and the image frame difference; S5: Obtain the image coordinates of the two road marking vertices in the Mth frame of the image captured by the vehicle camera, and then obtain the image coordinates of the two road marking vertices tracked in the M+Nth frame, a total of four points, to form a calibration template, and use the single vanishing point calibration method to calibrate the vehicle camera, and finally calculate the external parameters of the vehicle camera.

4. A dynamic calibration method for a vehicle-mounted camera according to claim 2 or 3, characterized in that: The algorithm module of the vehicle-mounted electronic terminal adopts the lane line detection technology based on deep learning to realize the preliminary detection of the lane line. Subsequently, in order to perform virtual and real line segmentation and road marking vertex detection on the detected lane line, the algorithm module of the vehicle-mounted electronic terminal further integrates the computer vision algorithms of edge detection, color space segmentation and corner point detection.

5. The dynamic calibration method of a vehicle-mounted camera according to claim 2, characterized in that: The global navigation satellite system (GNSS) unit of the high-precision positioning module performs GNSS positioning on the calibration location, and inquires about the lane spacing, marking length and spacing between virtual and real lines of the markings at the calibration location based on the GNSS positioning information in combination with the geographic information system (GIS) unit of the high-precision positioning module to determine the distances between the four road marking vertices serving as the calibration template.

6. The dynamic calibration method of a vehicle-mounted camera according to claim 3, characterized in that: When the on-vehicle electronic terminal continuously tracks two road marking vertices, the algorithm module adopts a faster and less computationally intensive optical flow algorithm, or a hierarchical block matching algorithm (HBMA) with better tracking effect, or a tracking algorithm based on deep learning such as MDNet or TCNN.

7. The dynamic calibration method of a vehicle-mounted camera according to claim 3, characterized in that: The vehicle speed v is obtained from a high-precision positioning module, an on-board inertial navigation unit (IMU) module, or an on-board automatic diagnostic system (OBD) interface; Then the distance that the tracked road marking vertex moves with the vehicle as the reference is calculated using the following formula: d=v×N÷fps 8. A method for measuring visual distance of a vehicle-mounted camera, characterized in that: The visual distance measurement method of the vehicle-mounted camera specifically comprises the following steps: S1: Obtain the image coordinates of the visual distance measurement target; S2: When the target is at ground height, the image coordinates are converted into the actual scene coordinates through the coordinate conversion formula; S3: Calculate the actual coordinates of the scene where the vehicle camera is located; S4: The actual scene coordinates of the ranging target minus the actual scene coordinates of the vehicle camera are the distance of the ranging target relative to the vehicle camera, and the actual scene coordinates of the ranging target 1 minus the actual scene coordinates of the ranging target 2 are the relative distance between the two ranging targets; S5: When the distance measurement target is not at the ground height, the distance measured by steps S2 to S4 is the projection distance. If the height of the distance measurement target is known, the actual distance of the distance measurement target can be obtained by the conversion formula of the projection distance and the actual distance.

9. The method for measuring visual distance of a vehicle-mounted camera according to claim 8, characterized in that: Based on the dynamic calibration method of a vehicle-mounted camera as described in any one of claims 4 to 7, the calculated external parameters of the vehicle-mounted camera, including the pitch angle t, the rotation angle p, the rotation angle s, the pixel focal length f and the camera height h, define the relationship between the image plane and the actual three-dimensional world coordinates.

10. The method for measuring visual distance of a vehicle-mounted camera according to claim 9, characterized in that: After calculating the external parameters of the vehicle camera, the image coordinates are converted to the actual scene coordinates using the following coordinate conversion formula: Next, the actual coordinates of the scene where the vehicle camera is located are calculated using the following formula: When the distance measurement target is not at ground height, the projection distance and actual distance are converted using the following projection distance and actual distance conversion formula:

Images