A method for calculating the height of road facilities based on monocular camera images

Abstract

The present invention relates to a method for calculating the height of road facilities based on monocular camera images, including: calibrating a monocular camera to obtain preset parameters; using the calibrated monocular camera to collect the original image of the road facilities, and obtaining the ground point coordinates and vertex coordinates; using the mapping change matrix, the ground point coordinates and the vertex coordinates to calculate the pixel distance in the plan view image, and using the projection mapping matrix and the ground point coordinates to calculate the distance from the road facilities to the monocular camera; bringing the pixel distance of the road facilities in the plan view image, the distance from the road facilities to the monocular camera, and the distance from the imaging plane to the monocular camera position into a preset height model to calculate the height of the road facilities. The present invention can quickly calculate the height information of road facilities through monocular camera images, and fill in the gaps in the height of facilities currently realized in the intelligent detection of road facilities based on monocular camera images; and has the characteristics of low implementation cost, low calculation cost, high operation efficiency, etc.

Description

Technical Field

[0001] The invention relates to the technical field of road engineering, and in particular to a method for calculating the height of road facilities based on a monocular camera image. Background Art

[0002] With the acceleration of global urbanization, the functions and pressures borne by urban transportation systems are increasing, becoming a key indicator for measuring the efficiency of modern urban operations and the quality of life of residents. In such an environment, building an efficient, safe and sustainable road transportation network has become a top priority. To achieve this goal, it not only depends on the physical layout of road infrastructure and the progress of material science, but also requires the deep integration of intelligent and information technology to provide accurate data support and intelligent decision-making basis for road management, maintenance and optimization.

[0003] In the traditional mode, the inspection and maintenance of road facilities mainly rely on manual on-site surveys, which is not only inefficient and costly, but also difficult to achieve full coverage and immediate response due to the distribution and skill level of human resources. For example, the damage, tilt or missing of facilities such as traffic lights, signs, isolation belts, and curbs are often not repaired quickly due to untimely discovery, thus posing a potential threat to traffic safety. In addition, the safety risks of staff during manual inspections cannot be ignored, especially in bad weather or during busy traffic hours.

[0004] In recent years, with the rapid development of emerging technologies such as the Internet of Things, big data, and artificial intelligence, intelligent detection technology has begun to gradually replace the traditional road facility maintenance model. Intelligent road monitoring systems, relying on various sensors integrated in patrol vehicles, drones, or fixed sites, can achieve all-weather, all-time, and all-round facility status monitoring, greatly improving the efficiency and accuracy of maintenance. These systems can not only detect immediate problems with facilities by collecting, processing, and analyzing various types of data, but also predict possible problems in the future through data analysis, providing a scientific basis for preventive maintenance.

[0005] At present, intelligent perception technologies for road facilities can be roughly divided into two categories: vision-based detection technology and physical sensing-based three-dimensional information acquisition technology. LiDAR and stereo vision technology are valued for their ability to directly obtain three-dimensional information of road facilities. These two technologies can accurately measure the size, position and even surface features of facilities by emitting laser beams or using dual cameras to obtain parallax information, and are suitable for high-precision scene reconstruction. However, the high cost, complex data processing flow and sensitivity to environmental conditions (such as performance degradation in rainy and foggy days) of these technologies limit their application in large-scale, routine maintenance. Especially for urban and rural roads that are sensitive to economic costs, the deployment and maintenance costs of these high-precision equipment are unaffordable. In contrast, intelligent detection technology based on monocular cameras has gradually become the mainstream trend of road facility monitoring with its advantages of low cost, easy deployment and simple maintenance.

[0006] Although the intelligent detection technology based on monocular cameras has shown its unique advantages in many scenarios, it still has certain limitations in height measurement. At present, most road facility analysis systems based on monocular images focus on identifying the surface conditions of facilities (such as damage, occlusion, pollution, etc.), but are relatively weak in obtaining height information. The lack of accurate judgment of the height of the facility, on the one hand, affects the assessment of the integrity and safety of the facility, and on the other hand, limits the application potential in special scenarios (such as compliance detection of bridge clearance height and road sign installation height).

[0007] In summary, traditional facility height measurement relies on manual on-site operations, including direct measurement using tools such as rulers and distance meters. This method is direct and intuitive, but its disadvantages are also very obvious: First, high cost and low efficiency: manual measurement not only requires a lot of manpower, but is also limited by the physical strength and working time of operators. For widely distributed facilities, it is difficult to achieve rapid and comprehensive detection, resulting in high overall costs; second, safety risks: operators need to measure in traffic or at high altitudes, which increases safety risks, especially in complex environments such as highways and overpasses.

[0008] The use of laser radar (LiDAR) and stereo vision technology can improve the efficiency and accuracy of road facility detection. However, there are also the following disadvantages: First, the cost and complexity of LiDAR and stereo vision: Although these two technologies can directly obtain three-dimensional data and accurately measure the height of facilities, the high sensor cost, complex data processing process and high power consumption limit their economy and practicality in large-scale applications. For public road facilities that require large-scale and frequent inspections, this undoubtedly increases the financial burden; second, data processing and storage pressure: Especially LiDAR, the amount of point cloud data it generates is huge, and the high performance requirements of data processing hardware and algorithms are extremely high, which increases the system's operation and maintenance costs and the complexity of data management.

[0009] Intelligent perception technology based on monocular images has emerged in the field of road facility monitoring with its advantages of low cost and easy deployment. Monocular vision technology can identify damage, abnormal conditions or occlusion of facilities through intelligent recognition algorithms, providing basic information for facility maintenance. However, there is still a lack of systematic and accurate solutions for information acquisition and processing technology for the key dimension of facility height. Summary of the invention

[0010] The purpose of the present invention is to overcome the defects of the prior art in view of the low efficiency, high cost and associated safety hazards of traditional facility height measurement methods, as well as the high cost and high data processing complexity faced by advanced technologies such as lidar and stereo vision in their applications. Specifically, the present invention aims to solve the following key technical problems:

[0011] Improve measurement efficiency and reduce costs: A method for calculating the height of road facilities based on monocular camera images is proposed to replace high-cost manual operations and expensive lidar / stereo vision equipment, thereby achieving economical and efficient large-scale road facility detection and monitoring.

[0012] Simplify data processing and reduce computing requirements: The method should be able to run efficiently on conventional computing platforms, reduce dependence on high-performance hardware, and at the same time reduce the complexity of data processing and storage requirements, thereby improving the practicality and deployability of the system.

[0013] In summary, this embodiment invents a method for calculating the height of road facilities based on monocular camera images. Combined with the current intelligent detection system for road facilities based on monocular camera images, the method quickly calculates the height information of road facilities through the monocular camera image, thereby completing the gap in the height of facilities in the current intelligent detection system for road facilities based on monocular camera images. The method has low implementation cost and low calculation cost.

[0014] To achieve the above object, the present invention provides the following solutions:

[0015] A method for calculating the height of road facilities based on a monocular camera image, comprising:

[0016] Perform monocular camera calibration to obtain the projection mapping matrix Pmat from the original image to the top view image, the mapping change matrix Tmat from the original image to the horizontal view image, and the distance Db from the imaging plane to the monocular camera position;

[0017] Using the calibrated monocular camera, collect the original image of the road facilities, and obtain the ground point coordinates P1 and vertex coordinates P2 of the road facilities in the original image;

[0018] Calculate the pixel distance PH of the road facility in the plan view image using the mapping change matrix Tmat, the ground point coordinates P1 and the vertex coordinates P2 of the road facility, and calculate the distance Da from the road facility to the monocular camera using the projection mapping matrix Pmat and the ground point coordinates P1;

[0019] The pixel distance PH of the road facility in the plan view image, the distance Da from the road facility to the monocular camera, and the distance Db from the imaging plane to the monocular camera position are brought into the preset height model to calculate the road facility height RH.

[0020] Optionally, obtaining the projection mapping matrix Pmat from the original image to the overhead image includes:

[0021] Lay a chessboard of a preset size on the road surface, with the lower edge of the chessboard coinciding with the lower edge of the original image;

[0022] Using a monocular camera to capture a first original road surface image with a checkerboard, and obtaining pixel coordinates of four corner points of the checkerboard in the first original road surface image;

[0023] Based on the pixel coordinates of the upper left corner of the chessboard in the first original road surface image and the size of the chessboard, the pixel coordinates of the four corner points of the chessboard in the overhead image are obtained;

[0024] Input the pixel coordinates in the first original road surface image and the pixel coordinates in the overhead image into a preset first coordinate transformation model to obtain a projection mapping matrix Pmat from the original image to the overhead image;

[0025] The first coordinate transformation model is:

[0026]

[0027] Among them, X and Y are the pixel coordinates in the top-view image; are the pixel coordinates in the first original road image respectively.

[0028] Optionally, obtaining a mapping change matrix Tmat from the original image to the horizontal viewing angle image includes:

[0029] A chessboard of a preset size is placed vertically on the road surface, a second original road surface image with the chessboard is captured using a monocular camera, and pixel coordinates of four corner points of the chessboard in the second original road surface image are obtained;

[0030] The pixel coordinates of the upper left corner of the chessboard in the plan view are made consistent with the pixel coordinates of the upper left corner in the second original road image, and the width of the chessboard in the plan view image is made consistent with the width of the second original road image, and the pixel coordinates of the four corner points of the chessboard in the plan view image are obtained;

[0031] Input the pixel coordinates of the four corner points of the chessboard in the second original road surface image and the pixel coordinates in the horizontal view image into a preset second coordinate transformation model to obtain a mapping change matrix Tmat from the original image to the horizontal view image;

[0032] The second coordinate transformation model is:

[0033]

[0034] in, are the pixel coordinates of the four corner points of the chessboard in the plan view image, are the pixel coordinates in the second original road image respectively.

[0035] Optionally, obtaining the distance Db from the imaging plane to the monocular camera position includes:

[0036] A calibration pole of preset height is placed vertically at different horizontal distances from the monocular camera to collect original images;

[0037] Obtain the ground point coordinates and vertex coordinates of several calibration rods in the original image;

[0038] Using the mapping change matrix Tmat, the coordinates of the ground point and the vertex in the original image of the calibration pole, and the coordinates PP1 and PP2 in the plan view image of the calibration pole are calculated;

[0039] According to the coordinates PP1 and PP2 of the calibration rod in the plan view image, calculate the pixel distance of the calibration rod in the plan view image ;

[0040] Calculate the grounding point coordinates PF1 (X1, Y1) of the calibration rod in the top view by using the projection mapping matrix Pmat and the grounding point coordinates of the calibration rod in the original image;

[0041] Input the grounding point coordinates PF1 (X1, Y1) of the calibration rod in the top view into the preset horizontal distance model to obtain the distance from the calibration rod to the monocular camera. ;

[0042] The distance from the calibration rod to the monocular camera, the height of the calibration rod, and the pixel distance of the calibration rod in the horizontal image , input into the preset height model to obtain the distance Db from the imaging plane to the monocular camera position.

[0043] Optionally, the preset horizontal distance model is:

[0044]

[0045] in, is the vertical pixel distance from the upper boundary to the lower boundary in the overhead image, is the distance from the lower boundary of the monocular camera to the camera, is the ordinate of the target to be measured in the top view, is the distance from the target to the monocular camera.

[0046] Optionally, the preset height model is:

[0047]

[0048] in, is the distance from the imaging plane to the monocular camera position, is the pixel distance of the target to be measured in the horizontal image, is the height of the target to be measured.

[0049] Optionally, using the mapping change matrix Tmat, the grounding point coordinates P1 and the vertex coordinates P2 of the road facility, calculating the pixel distance PH of the road facility in the plan view image includes:

[0050] Utilizing the mapping change matrix Tmat, calculating the coordinates PP1 and PP2 of the grounding point coordinates P1 and the vertex coordinates P2 of the road facility in the plan view image of the road facility;

[0051] Based on the coordinates PP1 and PP2 of the road facility in the plan view image, the pixel distance PH of the road facility in the plan view image is calculated.

[0052] Optionally, using the projection mapping matrix Pmat and the grounding point coordinates P1 of the road facility, calculating the distance Da from the road facility to the monocular camera includes:

[0053] Using the projection mapping matrix Pmat, calculate the grounding point coordinates P1 of the road facility, the grounding point coordinates PF1 (X1, Y1) in the road facility overhead image;

[0054] Input the grounding point coordinates PF1 (X1, Y1) of the road facility in the top view into the preset horizontal distance model to obtain the distance from the target to be measured to the monocular camera. .

[0055] The beneficial effects of the present invention are:

[0056] The present invention can be applied to a road facility perception system based on monocular camera images. The height of the road facilities can be obtained through simple calibration and calculation, thus filling the gap of the current intelligent detection of road facilities based on monocular camera images, which lacks the height of the facilities.

[0057] In terms of execution efficiency, compared with the inefficiency, high cost and accompanying safety hazards of traditional facility height measurement methods, the road facility height calculation method based on monocular camera images of the present invention can basically achieve no manual operation through simple calibration combined with the road facility intelligent perception system, thereby reducing measurement costs and improving efficiency.

[0058] Compared with the road facility height calculation methods such as laser radar and stereo vision, the road facility height calculation method based on monocular camera images of the present invention is simple to operate, has a small amount of data, and has a low calculation cost, thus reducing the usage threshold and cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0060] Figure 1 A schematic diagram of a method for calculating the height of road facilities based on a monocular camera image according to an embodiment of the present invention;

[0061] Figure 2 This is a schematic diagram of imaging with a monocular camera according to an embodiment of the present invention;

[0062] Figure 3 The calibration original image for obtaining the top view image change matrix according to the embodiment of the present invention;

[0063] Figure 4 This is a schematic diagram of the system installation of an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] The imaging principle of a monocular camera can be compared to pinhole imaging, such as Figure 2 As shown, the ratio of the actual height RH of the object to the horizontal distance Da from the target to the camera is equal to the ratio of the pixel distance of the object in the imaging to the distance Db from the imaging plane to the camera position. The formula is as follows:

[0067]

[0068]

[0069] In the calibration stage, the actual height RH of the object and the horizontal distance Da from the target to the camera can be obtained through on-site measurement. PH can be calculated through the grounding point pixel coordinates and vertex pixel coordinates of the object in the image. The unknown parameter Db is the parameter that this method needs to obtain through calibration calculation. Therefore, during calibration, Db is calculated by collecting the values ​​of Da, PH, and RH. Usually, multiple groups of RH, Da, and PH values ​​are obtained, and multiple Db are calculated. The average value of multiple Db is taken as the calculation parameter of this method, and the calculation formula for the height of the object in the plan view image can be obtained:

[0070]

[0071] It can be seen from the above formula that the method for calculating the height of road facilities based on the monocular camera image of this embodiment has two stages, the first is calibration and calculation of the parameter Da, and the second is calculation of the actual height of the target to be measured.

[0072] like Figure 1 As shown, this embodiment proposes a method for calculating the height of road facilities based on a monocular camera image, including:

[0073] Perform monocular camera calibration to obtain the projection mapping matrix Pmat from the original image to the top view image, the mapping change matrix Tmat from the original image to the horizontal view image, and the distance Db from the imaging plane to the monocular camera position;

[0074] Furthermore, obtaining the projection mapping matrix Pmat from the original image to the overhead image includes:

[0075] Lay a chessboard of a preset size on the road surface, with the lower edge of the chessboard coinciding with the lower edge of the original image;

[0076] Using a monocular camera to capture a first original road surface image with a checkerboard, and obtaining pixel coordinates of four corner points of the checkerboard in the first original road surface image;

[0077] Based on the pixel coordinates of the upper left corner of the chessboard in the first original road surface image and the size of the chessboard, the pixel coordinates of the four corner points of the chessboard in the overhead image are obtained;

[0078] Input the pixel coordinates of the four vertices of the chessboard of known size in the first original road surface image and the pixel coordinates of the four vertices of the chessboard of known size in the overhead image into the preset first coordinate transformation model, i.e., Formula 5, to obtain the projection mapping matrix Pmat from the original image to the overhead image;

[0079] Specifically, in this embodiment, the projection mapping matrix Pmat from the original image to the overhead image is obtained by calibration. In actual use, the distance from the target to be measured to the camera can be calculated through the overhead image to obtain Db in Formula 3. Pmat is a 3x3 matrix as shown in Formula 4.

[0080]

[0081] (Pmat: projection mapping matrix from original image pixel space to top view image pixel space, a11, a12, a21, a22 represent linear transformation elements. a13, a23 represent translation change elements, a31, a32, a33 represent perspective transformation elements.)

[0082]

[0083] Among them, X and Y are the pixel coordinates in the top-view image, are the pixel coordinates in the original road image, respectively.

[0084] In the image coordinate system of this embodiment, the coordinate origin is the upper left corner of the image, from left to right is the x direction, and from top to bottom is the y direction.

[0085] The coordinate information of the known points in two pixel spaces (the original image pixel space and the overhead image pixel space) can be obtained; the specific method is: by laying a chessboard of known size on the road surface, the lower edge of the chessboard coincides with the lower edge of the original image, and the monocular camera imaging image and the original image are collected at this time. Figure 3As shown, the pixel coordinates of the four corner points of the chessboard in the original road image are [(xtl, ytl), (xtr, ytr), (xbl, RawImageH), (xbr, RawImageH)], where (xtl, ytl) represents the coordinates of the upper left corner of the chessboard in the original image; (xtr, ytr) represents the coordinates of the upper right corner of the chessboard in the original image; (xbl, RawImageH) represents the coordinates of the lower left corner of the chessboard in the original image, and RawImageH is the height of the original image (the lower edge of the chessboard is flush with the lower edge of the imaging plane); (xbr, RawImageH) represents the coordinates of the lower right corner of the chessboard in the original image; since the chessboard does not deform in the overhead image, the length and width of the chessboard are set to 2 meters, and the distance between each two pixels in the overhead image represents 1 mm, thereby obtaining the coordinates of the four corner points of the chessboard in the overhead image [(Xtl, IMGH-2000), (Xtl+2000, IMGH- 2000), (Xtl, IMGH), (Xtl+2000, IMGH)], Xtl represents the pixel coordinates of the left border of the chessboard in the overhead image, which can be set as needed. Generally, the chessboard is placed in the center of the overhead image, that is, Xtl=IMGW / 2-1000. Among them, IMGW is the width of the overhead image; IMGH is the height of the overhead image. Substitute the above known coordinate information of the chessboard in the original image and the coordinate information in the overhead image into formula 5. The transformation matrix Pmat from the original image to the overhead image can be calculated; in addition, the getPerspectiveTransform method in opencv can also be used to quickly obtain the projection mapping matrix Pmat: input the coordinates of the corner points of the chessboard in the original road image and the coordinates of the corner points of the chessboard in the overhead image, and it can be quickly calculated.

[0086] During calibration: The first original image refers to the image obtained by taking a photo with a camera after placing the chessboard horizontally at a specified position, such as Figure 3 ; In the first original image, there is a whole chessboard, and the lower edge of the original image coincides with the lower edge of the chessboard. After Pmat is calculated through the first original image and the corresponding overhead image, the original images taken by subsequent cameras can be obtained through Pmat; The corresponding overhead image;

[0087] In the top view image, the shape of the chessboard is consistent with the actual shape. If the actual shape is a square, it is also a square in the top view. Assuming that one pixel in the top view represents 1 mm, as long as the coordinates of one corner point of the chessboard in the top view are determined, the position of the 2m x 2m square chessboard in the top view is known; In this example, for the convenience of calculation, the position of the chessboard in the top view is set below the center, so the coordinates of the upper left corner of the chessboard in the top view are set to (IMGW / 2-1000, IMGH-2000), and the coordinates of other corner points are calculated according to the actual side length of the chessboard.

[0088] Specifically, in this embodiment, the horizontal distance Dc from the actual position of the lower edge of the original image to the camera is measured;

[0089] The horizontal distance Dc from the actual position of the lower edge of the test original image to the camera is measured in mm. This is because the lower edge of the top-view image is flush with the lower edge of the imaging plane, and one pixel in the top-view image represents 1 mm.

[0090] Assuming that the grounding point coordinates of the target to be measured in the overhead image are (x, y), the horizontal distance Da from the target to be measured to the camera can be calculated as follows:

[0091]

[0092] in, is the vertical pixel distance from the upper boundary to the lower boundary in the overhead image, is the distance from the lower boundary of the monocular camera to the camera, is the ordinate of the target to be measured in the top view, is the distance from the target to be measured to the monocular camera. ) is the pixel distance from the ground point of the target to be measured to the lower boundary of the top view in the top view, in pixels. In the calibration stage, when calculating the transformation rectangle Pmat of the top view image, the ground distance between every two pixels in the top view image is set to 1 mm. Therefore, the ground distance between every two pixels of the top view image obtained by Pmat is also 1 mm, so (IMGH-y) 1 mm / pixel can get the distance from the ground point of the target to be measured to the position of the lower boundary of the overhead image on the ground, in millimeters. Dc is the distance from the position of the lower boundary of the monocular camera image on the ground to the camera. When obtaining the overhead image conversion matrix, it is set that the positions of the lower boundaries of the monocular camera image and the overhead image on the ground are consistent. Therefore, Dc is also the distance from the position of the lower boundary of the overhead image on the ground to the camera. This value is measured using a measuring tool during the calibration phase, in millimeters. Therefore, the ground distance from the target to be measured to the camera is equal to the distance from the target to be measured to the position of the lower boundary of the overhead view on the ground plus the distance from the position of the lower boundary of the overhead view on the ground to the camera. In the image coordinate system in this embodiment, the coordinate origin is the upper left corner point of the image, the x direction is from left to right, and the y direction is from top to bottom.

[0093] Furthermore, obtaining the mapping change matrix Tmat from the original image to the horizontal viewing angle image includes:

[0094] A chessboard of a preset size is placed vertically on the road surface, a second original road surface image with the chessboard is captured using a monocular camera, and pixel coordinates of four corner points of the chessboard in the second original road surface image are obtained;

[0095] The pixel coordinates of the upper left corner of the chessboard in the plan view are made consistent with the pixel coordinates of the upper left corner in the second original road image, and the width of the chessboard in the plan view image is made consistent with the width of the second original road image, and the pixel coordinates of the four corner points of the chessboard in the plan view image are obtained;

[0096] Input the pixel coordinates of the four corner points of the chessboard in the second original road surface image and the pixel coordinates in the horizontal view image into a preset second coordinate transformation model to obtain a mapping change matrix Tmat from the original image to the horizontal view image;

[0097] The second coordinate transformation model is:

[0098]

[0099] in, are the pixel coordinates of the four corner points of the chessboard in the plan view image, are the pixel coordinates in the second original road image respectively.

[0100] Specifically, in this embodiment, a mapping change matrix Tmat from the original image to the horizontal viewing angle image is obtained, and Tmat is a 3x3 matrix like Pmat;

[0101] Since the camera is installed at an angle to the road surface, the original image collected is not a horizontal perspective image, so it is necessary to convert the original image into a horizontal perspective image. The conversion matrix Tmat from the original image to the horizontal perspective image can be obtained by the actual coordinate information of the known points in two pixel spaces (the original image pixel space and the horizontal perspective pixel space), and the original image can be converted into a horizontal perspective image through Tmat. The specific method is as follows: a chessboard of known size is placed vertically on the road surface, and the original image at this time is collected; the pixel coordinates of the four corner points of the chessboard placed vertically on the ground in the original road surface image are [(xtl1, ytl1), (xtr1, ytr1), (xbl1, ybl1), (xbr11, ybr11)], where (xtl1, ytl1) represents the coordinates of the upper left corner of the chessboard placed vertically on the ground in the original image; (xtr1, ytr1) represents the coordinates of the upper right corner of the chessboard in the original image; (xbl1, ybl1) represents the coordinates of the lower left corner of the chessboard in the original image; (xbr1, ybr1) represents the coordinates of the lower right corner of the chessboard in the original image; because the chessboard is not Deformation occurs, and the coordinates of the upper left corner of the chessboard in the plan view are consistent with the coordinates of the upper left corner in the original image, and the width of the chessboard in the plan view image is consistent with the width in the original image. The coordinates of the four corner points of the chessboard in the plan view image can be obtained [(xtl1, ytl1), (xtr1, ytl1), (xtl1, ybl1), (xtr1, ybl1)], where (xtl1, ytl1) represents the coordinates of the upper left corner of the chessboard in the plan view; (xtr1, ytl1) represents the coordinates of the upper right corner of the chessboard in the plan view; (xtl1, ybl1) represents the coordinates of the lower left corner of the chessboard in the original image; (xtr1, ybl1) represents the coordinates of the lower right corner of the chessboard in the original image. Substitute the above known coordinate information of the chessboard perpendicular to the ground in the original image and the coordinate information in the plan view image into formula 5 to calculate the transformation matrix Tmat from the original image to the plan view image.

[0102] The second original image is also shot in a similar way to the first original image, except that the chessboard is placed in a different way and position. When shooting the second original image, the chessboard is placed perpendicular to the ground, and the camera's imaging screen includes the entire chessboard. After Tmat is calculated from the second original image and the corresponding plan view image, the original images shot by subsequent cameras can all be obtained through Tmat; corresponding plan view images;

[0103] Furthermore, obtaining the distance Db from the imaging plane to the monocular camera position includes:

[0104] A calibration pole of preset height is placed vertically at different horizontal distances from the monocular camera to collect original images;

[0105] Obtain the ground point coordinates and vertex coordinates of several calibration rods in the original image;

[0106] Using the mapping change matrix Tmat, the coordinates of the ground point and the vertex in the original image of the calibration pole are calculated, and the coordinates PP1 and PP2 in the plan view image of the calibration pole are calculated;

[0107] According to the coordinates PP1 and PP2 of the calibration rod in the plan view image, calculate the pixel distance of the calibration rod in the plan view image ;

[0108] The coordinates of the grounding point of the calibration rod in the top view are calculated using the projection mapping matrix Pmat and the grounding point coordinates of the calibration rod in the original image PF1 (X1, Y1);

[0109] Input the coordinates of the grounding point PF1 (X1, Y1) of the calibration rod in the top view into the preset horizontal distance model to obtain the distance from the calibration rod to the monocular camera. ;

[0110] The distance from the calibration rod to the monocular camera, the height of the calibration rod, and the pixel distance of the calibration rod in the horizontal image , input into the preset height model to obtain the distance Db from the imaging plane to the monocular camera position.

[0111] Specifically, in this embodiment, a calibration rod with a known height is placed vertically at positions with different horizontal distances from the camera, multiple sets of data are collected, and the parameter Db is calculated.

[0112] Place a calibration pole of known height vertically at different horizontal distances from the camera to collect the original image; obtain multiple sets of ground point coordinates P1 (x1, y1) and vertex coordinates P2 (x2, y2) of the calibration pole in the original image; use the original image to the plan view image change matrix Tmat to calculate the coordinates PP1 and PP2 of P1 and P2 in the plan view image, and use the cv2.perspectiveTransform method in opencv to pass in the original image coordinates and the image transformation matrix Tmat to quickly obtain the coordinates of the original image in the plan view image; according to PP1 and PP2, the pixel distance PH of the calibration pole in the plan view image can be calculated; using the original image to the top view image change matrix Pmat and the ground point coordinates P1 of the calibration pole in the original image, calculate the ground point coordinates PF1 (X1, Y1) of the calibration pole in the top view, and substitute them into formula 6 to calculate the distance Da from the calibration pole to the camera; given the height RH of the calibration pole, the required parameter Db can be calculated according to formula 2; multiple sets of Db can be calculated, and Db can be obtained by taking the average value to reduce measurement errors.

[0113] Using the calibrated monocular camera, collect the original image of the road facilities, and obtain the ground point coordinates P1 and vertex coordinates P2 of the road facilities in the original image;

[0114] Specifically, in this embodiment, the grounding point coordinates P1 (x1, y1) and the vertex coordinates P2 (x2, y2) of the target to be detected in the original image are obtained according to the intelligent detection algorithm.

[0115] Calculate the pixel distance PH of the road facility in the plan view image by using the mapping change matrix Tmat, the ground point coordinates P1 and the vertex coordinates P2 of the road facility; including:

[0116] Using the mapping change matrix Tmat, the coordinates PP1 and PP2 of the grounding point coordinates P1 and the vertex coordinates P2 of the road facility in the plan view image of the road facility are calculated;

[0117] Based on the coordinates PP1 and PP2 of the road facility in the plan view image, the pixel distance PH of the road facility in the plan view image is calculated.

[0118] Specifically, in this embodiment, the coordinates PP1 and PP2 of P1 and P2 in the plan view image are calculated using the original image to plan view image transformation matrix Tmat. The cv2.perspectiveTransform method in opencv can be used to pass in the original image coordinates and the image transformation matrix Tmat to quickly obtain the coordinates of the original image in the plan view image; the pixel distance PH of the target to be measured in the plan view image can be calculated based on PP1 and PP2;

[0119] Using the projection mapping matrix Pmat and the grounding point coordinates P1 of the road facilities, the distance Da from the road facilities to the monocular camera is calculated; including:

[0120] Using the projection mapping matrix Pmat, calculate the grounding point coordinates P1 of the road facility and the grounding point coordinates PF1 (X1, Y1) in the road facility overhead image;

[0121] Input the grounding point coordinates PF1 (X1, Y1) of the road facility in the top view into the preset horizontal distance model to obtain the distance from the road facility to the monocular camera. .

[0122] Specifically, in this embodiment, the coordinates PF1(X1, Y1) of the ground point P1 of the target to be measured in the original image in the top view image are calculated using the original image to top view image change matrix Pmat. The cv2.perspectiveTransform method in opencv can be used to pass in the original image coordinates and the image transformation matrix Pmat to quickly obtain the coordinates of the original image in the top view image; the ground point coordinates PF1(X1, Y1) of the target to be measured in the top view image are obtained and substituted into formula 6 to calculate the distance Da from the target to be measured to the camera;

[0123] The pixel distance PH of the road facilities in the plan view image, the distance Da from the road facilities to the monocular camera, and the distance Db from the imaging plane to the monocular camera position are brought into the preset height model to calculate the road facility height RH.

[0124] Specifically, in this embodiment, the pixel height PH of the target to be measured in the plan view image, the distance Da from the target to be measured to the camera, and the value of the distance Db from the calibrated parameter camera imaging surface to the camera are substituted into Formula 3 to obtain the real height RH of the target to be measured.

[0125] The method mentioned in this embodiment can be applied to a road facility perception system based on monocular camera images. The height of the road facilities can be obtained through simple calibration and calculation, thus filling the gap of the height of the facilities in the current intelligent detection of road facilities based on monocular camera images.

[0126] In terms of execution efficiency, compared with the inefficiency, high cost and accompanying safety hazards of traditional facility height measurement methods, the road facility height calculation method based on monocular camera images in this embodiment can basically achieve no manual operation through simple calibration combined with the road facility intelligent perception system, thereby reducing measurement costs and improving efficiency.

[0127] Compared with the road facility height calculation methods such as laser radar and stereo vision, the road facility height calculation method based on monocular camera images in this embodiment is simple to operate, has a small amount of data, and has a low calculation cost, thus reducing the usage threshold and cost.

[0128] In summary, the method and system for calculating the height of road facilities based on monocular camera images proposed in this embodiment, combined with the current intelligent detection system for road facilities based on monocular camera images, can quickly calculate the height information of road facilities through monocular camera images, and fill the gap of no facility height in the current intelligent detection of road facilities based on monocular camera images; and the present invention has the characteristics of low implementation cost, low calculation cost, and high operating efficiency.

[0129] This embodiment proposes a method for calculating the height of road facilities based on monocular camera images, which is used in combination with a road facility intelligent detection system based on a monocular camera. Figure 4 As shown in the figure, a vehicle-mounted monocular camera, a central industrial computer and other equipment are installed on the vehicle to collect road image information. The central industrial computer is powered by the vehicle, and the vehicle-mounted camera is connected to the central industrial computer. The central industrial computer sends image acquisition instructions and pushes image information, image acquisition time, GPS information, etc. to the cloud server. After receiving the data, the cloud server detects the road facilities in the picture through an intelligent detection algorithm, obtains the grounding point coordinates and vertex coordinates of the road facilities on the original image, and then calculates the height of the facilities.

[0130] The implementation process can be summarized as follows:

[0131] (1) If Figure 4 Install the equipment as shown and perform camera calibration;

[0132] (2) Start the equipment to collect the original images and GPS data of the road unit price camera;

[0133] (3) Use the intelligent detection algorithm to calculate the road facilities in the original image and obtain the ground point coordinates and vertex coordinates of the road facilities;

[0134] (4) Complete the calculation of road facility height and store the data in the database.

[0135] The above embodiments are only descriptions of the preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for calculating the height of road facilities based on monocular camera images, characterized in that: include: Perform monocular camera calibration to obtain the projection mapping matrix Pmat from the original image to the top view image, the mapping change matrix Tmat from the original image to the horizontal view image, and the distance Db from the imaging plane to the monocular camera position; Obtaining the distance Db from the imaging plane to the monocular camera position includes: A calibration pole of preset height is placed vertically at different horizontal distances from the monocular camera to collect original images; Obtain the ground point coordinates and vertex coordinates of several calibration rods in the original image; Using the mapping change matrix Tmat, the coordinates of the ground point and the vertex in the original image of the calibration pole, and the coordinates PP1 and PP2 in the plan view image of the calibration pole are calculated; According to the coordinates PP1 and PP2 of the calibration rod in the plan view image, calculate the pixel distance of the calibration rod in the plan view image ; Calculate the grounding point coordinates PF1 (X1, Y1) of the calibration rod in the top view by using the projection mapping matrix Pmat and the grounding point coordinates of the calibration rod in the original image; Input the coordinates of the grounding point PF1 (X1, Y1) of the calibration rod in the top view into the preset horizontal distance model to obtain the distance from the calibration rod to the monocular camera. ; The distance from the calibration rod to the monocular camera, the height of the calibration rod, and the pixel distance of the calibration rod in the horizontal image , input into the preset height model to obtain the distance Db from the imaging plane to the monocular camera position; Using the calibrated monocular camera, collect the original image of the road facilities, and obtain the ground point coordinates P1 and vertex coordinates P2 of the road facilities in the original image; Calculate the pixel distance PH of the road facility in the plan view image using the mapping change matrix Tmat, the ground point coordinates P1 and the vertex coordinates P2 of the road facility, and calculate the distance Da from the road facility to the monocular camera using the projection mapping matrix Pmat and the ground point coordinates P1; The pixel distance PH of the road facility in the plan view image, the distance Da from the road facility to the monocular camera, and the distance Db from the imaging plane to the monocular camera position are brought into a preset height model to calculate the road facility height RH; Obtaining the projection mapping matrix Pmat from the original image to the overhead image includes: Lay a chessboard of a preset size on the road surface, with the lower edge of the chessboard coinciding with the lower edge of the original image; Using a monocular camera to capture a first original road surface image with a checkerboard, and obtaining pixel coordinates of four corner points of the checkerboard in the first original road surface image; Based on the pixel coordinates of the upper left corner of the chessboard in the first original road surface image and the size of the chessboard, the pixel coordinates of the four corner points of the chessboard in the overhead image are obtained; Input the pixel coordinates in the first original road surface image and the pixel coordinates in the overhead image into a preset first coordinate transformation model to obtain a projection mapping matrix Pmat from the original image to the overhead image; The first coordinate transformation model is: in, are the pixel coordinates in the overhead image, are pixel coordinates in the first original road surface image respectively; Obtaining the mapping change matrix Tmat from the original image to the horizontal viewing angle image includes: A chessboard of a preset size is placed vertically on the road surface, a second original road surface image with the chessboard is captured using a monocular camera, and pixel coordinates of four corner points of the chessboard in the second original road surface image are obtained; The pixel coordinates of the upper left corner of the chessboard in the plan view are made consistent with the pixel coordinates of the upper left corner in the second original road image, and the width of the chessboard in the plan view image is made consistent with the width of the second original road image, and the pixel coordinates of the four corner points of the chessboard in the plan view image are obtained; Input the pixel coordinates of the four corner points of the chessboard in the second original road surface image and the pixel coordinates in the horizontal view image into a preset second coordinate transformation model to obtain a mapping change matrix Tmat from the original image to the horizontal view image; The second coordinate transformation model is: in, are the pixel coordinates of the four corner points of the chessboard in the plan view image, are pixel coordinates in the second original road surface image respectively; The preset horizontal distance model is: in, is the vertical pixel distance from the upper boundary to the lower boundary in the overhead image, is the distance from the lower boundary of the monocular camera to the camera, is the ordinate of the target to be measured in the top view, is the distance from the target to the monocular camera.

2. The method for calculating the height of road facilities based on monocular camera images according to claim 1, characterized in that: The preset height model is: in, is the distance from the imaging plane to the monocular camera position, is the pixel distance of the target to be measured in the horizontal image, is the height of the target to be measured.

3. The method for calculating the height of road facilities based on monocular camera images according to claim 2, characterized in that: Calculating the pixel distance PH of the road facility in the plan view image by using the mapping change matrix Tmat, the ground point coordinates P1 and the vertex coordinates P2 of the road facility includes: Utilizing the mapping change matrix Tmat, calculating the coordinates PP1 and PP2 of the grounding point coordinates P1 and the vertex coordinates P2 of the road facility in the plan view image of the road facility; Based on the coordinates PP1 and PP2 of the road facility in the plan view image, the pixel distance PH of the road facility in the plan view image is calculated.

4. The method for calculating the height of road facilities based on monocular camera images according to claim 3, characterized in that: Using the projection mapping matrix Pmat and the grounding point coordinates P1 of the road facility, calculating the distance Da from the road facility to the monocular camera includes: Using the projection mapping matrix Pmat, calculate the grounding point coordinates P1 of the road facility, the grounding point coordinates PF1 (X1, Y1) in the road facility overhead image; Input the grounding point coordinates PF1 (X1, Y1) of the road facility in the top view into the preset horizontal distance model to obtain the distance from the road facility to the monocular camera. .

Images