A mobile vehicle over-bridge positioning method and device based on unmanned aerial vehicle cruising

Abstract

A kind of mobile vehicle bridge positioning method and device based on unmanned aerial vehicle cruise, vehicle is shot using unmanned aerial vehicle, realize camera pose estimation and vehicle real-time positioning in combination with SLAM and YOLO algorithm, its method includes: A. mobile vehicle is shot using unmanned aerial vehicle crossing bridge;B. extract image ORB feature points, and adopt YOLOv4 to detect vehicle area in image, separate static feature points and vehicle dynamic feature points;C. camera pose is estimated from static feature points using SLAM algorithm initially, and chessboard is introduced to correct camera pose;D. the position of camera under world coordinate system is calculated in combination with the coordinates of vehicle in image and the corrected camera pose;The test method of the present application patent will be applied to vehicle positioning using unmanned aerial vehicle cruise, with the advantages of long-distance test, high precision.

Description

TECHNICAL FIELD

[0001] The application relates to a mobile vehicle bridge positioning method and device based on unmanned aerial vehicle cruising, and belongs to the field of bridge detection. BACKGROUND

[0002] Vehicle load is one of the main dynamic loads of bridge structures and plays an important role in various load combinations. The loading mode of mobile vehicles crossing bridges has the advantages of economy and efficiency, and is widely used in influence line extraction, damage detection, bearing capacity evaluation and the like in existing bridge structure research. Therefore, accurately obtaining vehicle load information is crucial for bridge service state evaluation.

[0003] Vehicle load information usually includes load size and load position. The load size usually adopts a dynamic weighing system or a vehicle-mounted weighing system to obtain vehicle axle load information. For the load position, some scholars keep the vehicle at a constant speed and use a laser range finder or a GPS to obtain the vehicle position. These methods are usually suitable for model tests or require pre-installation of sensors on vehicles, and are difficult to apply to actual vehicle bridge crossing scenarios.

[0004] In recent years, computer vision testing methods have been widely applied to the field of civil engineering, and vehicle positioning combined with video images has become a feasible idea. Most scholars set up cameras on bridges, use YOLOv4 to detect vehicles from monitoring videos, and realize vehicle positioning. Bridge structures are large in the span direction, and usually require the installation of multiple cameras, and the use of image stitching technology to realize vehicle positioning during the whole process of crossing the bridge. The unmanned aerial vehicle cruising technology based on SLAM uses a mobile shooting method, which can avoid the installation of cameras on the whole bridge and is a promising way to position mobile vehicles. However, for the shooting environment of vehicle movement across the bridge, the camera pose estimated by the SLAM algorithm will be disturbed by the moving vehicle and does not have a real size. Therefore, for the scenario of vehicle movement across the bridge, the application of unmanned aerial vehicle cruising technology to vehicle positioning still needs further research. SUMMARY

[0005] In view of the shortcomings of the existing methods, the application provides a mobile vehicle bridge positioning method and device based on unmanned aerial vehicle cruising. The technology uses an unmanned aerial vehicle to shoot a vehicle, combines SLAM and YOLO algorithms, and realizes camera pose estimation and real-time vehicle positioning.

[0006] The first aspect of the application relates to a mobile vehicle bridge positioning method based on unmanned aerial vehicle cruising, comprising the following steps:

[0007] A. using an unmanned aerial vehicle to shoot a mobile vehicle crossing a bridge video;

[0008] B. extracting image feature points and using YOLOv4 to detect the vehicle region in the image to separate static feature points and vehicle dynamic feature points;

[0009] C. Preliminary estimation of camera pose from static feature points using SLAM algorithm, and introduction of chessboard to correct camera pose;

[0010] D. Combined with the coordinates of the vehicle in the image and the corrected camera pose, the position of the camera in the world coordinate system is calculated.

[0011] Further, step A specifically includes:

[0012] A1. Arrange a chessboard at the starting point of the UAV.

[0013] A2. Control the UAV to hover and take pictures of the chessboard to provide data basis for camera calibration;

[0014] A3. During the process of the vehicle passing through the bridge, control the UAV to track and take pictures of the vehicle;

[0015] Further, step B specifically includes:

[0016] B1. Take several pictures of the target vehicle from different angles as YOLOv4 data set, and use them for model training and model testing according to the ratio of 8:2 respectively;

[0017] B2. Use the trained YOLOv4 model to identify the vehicle area in the video image.

[0018] B3. Extract the video ORB feature points, separate the dynamic feature points of the vehicle area and the static feature points of the surrounding environment.

[0019] Further, step C specifically includes:

[0020] C1. Preliminary estimation of camera pose using SLAM algorithm. As shown in Fig. Figure 2 , the camera pose estimated by ORB-SLAM is usually based on the initial camera coordinate system (CCS), and P n The pose matrix of the camera pose in CCS is:

[0021]

[0022] In the formula, A iest is the camera pose matrix estimated by ORB-SLAM, R pi is the rotation matrix in the pose matrix, t pi is the translation matrix in the pose matrix, and O is the zero vector.

[0023] C2. Partial camera pose calibration using chessboard. Place a chessboard in the field of view of the partial camera pose, and establish a world coordinate system (WCS) with the chessboard corner point as the center. P kThe camera poses for capturing the checkerboard are denoted as P1~P k The poses in the world coordinate system are denoted as P1~P

[0024]

[0025] where A ireal is the camera pose matrix for the checkerboard calibration, R qi is the rotation matrix in the pose matrix, and t qi is the translation matrix in the pose matrix.

[0026] C3. The initial full camera poses are corrected according to the partial calibration camera poses. The points of the camera center in the CCS and WCS are respectively:

[0027] {P i} = t pi , {Q i} = t qi (3) The transformation matrix of the point sets {P i} to {Q i} is calculated, and the point sets are centralized:

[0028]

[0029] where and are the center points of the point sets {P i} and {Q i}, respectively, and are the centralized coordinates.

[0030] The covariance matrix H is calculated:

[0031]

[0032] where H is the covariance matrix. The U and V of the matrix are obtained by SVD decomposition, and the rotation matrix R can be calculated as:

[0033]

[0034] The scaling factor and the translation matrix can be calculated as:

[0035]

[0036] Then the transformation matrix of the point sets {P i} to {Q i} is:

[0037]

[0038] The T1 transformation satisfies the rotation and displacement of the camera center point from the CCS and WCS, but changes the scale. To restore the scale relationship, T is further used. 2i Transformation:

[0039]

[0040] Where, T 2i is the transformation matrix to restore the scaling relationship, E is the identity matrix, t wpi For the point set {P i}Coordinates under T1 transformation:

[0041] t wpi =λRt pi +t(12)

[0042] Then the estimated posture matrix A at each location iest The following transformation can be used for correction:

[0043] A imod =T 2i T1A iest (13)

[0044] Where A imod is the corrected camera pose matrix, which can represent the pose of the camera in a specific world coordinate system.

[0045] Further, step D specifically includes:

[0046] D1. Calculate the center of the vehicle area in the image. The coordinates of vehicle i at time in the pixel coordinate system are:

[0047] V i =[u i v i ](14)

[0048] Where u i and v i is the coordinate of the vehicle in the pixel coordinate system.

[0049] D2. Locate the vehicle's position in the world coordinate system based on the camera's posture. The transformation from the image coordinate system to the world coordinate system can be expressed as:

[0050]

[0051] Where, X wi 、Y wi and Z wi is the coordinate in the world coordinate system, K I K is the camera internal parameter matrix, which can be obtained by camera calibration. Ei is the camera extrinsic matrix under different camera postures, which can be obtained by the corrected camera posture matrix Aimod Obtained:

[0052]

[0053] It is considered that the vehicle is always on the road plane, that is, Z wi = 0. By equation (15), the coordinates of the vehicle in the world coordinate system can be calculated to realize the positioning of the moving vehicle.

[0054] The second aspect of the present application relates to a mobile vehicle bridge positioning device based on unmanned aerial vehicle cruising, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the mobile vehicle bridge positioning method based on unmanned aerial vehicle cruising of the present application.

[0055] The third aspect of the present application relates to a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the mobile vehicle bridge positioning method based on unmanned aerial vehicle cruising of the present application.

[0056] The present application adopts unmanned aerial vehicle to shoot the vehicle, combines SLAM and YOLO algorithm, realizes camera pose estimation and real-time positioning of the vehicle, and the test method of the present application applies unmanned aerial vehicle cruising to vehicle positioning, has the advantages of long-distance test and high precision, and provides a new idea for mobile vehicle bridge positioning.

[0057] The present application has the following advantages:

[0058] (1) The method disclosed in the present application adopts YOLOv4 to identify the vehicle area in the image, and removes the feature points in the area as dynamic feature points, and adopts static feature points to improve the accuracy of camera pose estimation.

[0059] (2) The method disclosed in the present application adopts the chessboard to correct the camera pose estimated by SLAM initially, so that it has the size of the real world and can be directly used to establish the relationship between the camera coordinate system and the world coordinate system.

[0060] (3) The method disclosed in the present application adopts unmanned aerial vehicle to shoot the vehicle, which can locate the vehicle from the video image, without the need to install sensors on the vehicle or erect multiple cameras on the bridge, and has the advantage of convenience. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is the mobile vehicle positioning method flowchart of the present application;

[0062] Figure 2 is the schematic diagram of the unmanned aerial vehicle shooting the vehicle of the present application;

[0063] Figure 3is a vehicle moving over bridge test layout of the present application;

[0064] Figure 4 is a YOLOv4 flow chart of the present application;

[0065] Figure 5 is a static feature point extraction diagram of the present application;

[0066] Fig. 6(a) and Fig. 6(b) are camera trajectory diagrams of the present application, wherein Fig. 6(a) is a slow working condition, and Fig. 6(b) is a fast working condition;

[0067] Figure 7 is a vehicle trajectory diagram of the present application;

[0068] Fig. 8(a) and Fig. 8(b) are vehicle displacement real-time curve diagrams of the present application, wherein Fig. 8(a) is a slow working condition, and Fig. 8(b) is a fast working condition; DETAILED DESCRIPTION

[0069] The present application is further described in detail below in combination with the drawings.

[0070] Example 1

[0071] As shown in the drawings, the present embodiment relates to a mobile vehicle over bridge positioning method based on unmanned aerial vehicle cruising, which specifically comprises the following steps: Figure 1

[0072] A. Using an unmanned aerial vehicle to shoot a mobile vehicle over bridge video; specifically including:

[0073] A1. Arranging a checkerboard at the starting point of the unmanned aerial vehicle. The mobile vehicle over bridge model and the test site are as shown in the drawings. The test beam is a simply supported steel beam, and the calculated span is 4.85m. A model vehicle is used to simulate the movement of the vehicle over the bridge. Figure 3

[0074] A2. Controlling the unmanned aerial vehicle to hover and shoot the checkerboard to provide a data basis for camera calibration;

[0075] A3. In the process of vehicle driving, controlling the unmanned aerial vehicle to move and shoot the vehicle, and recording the whole process of the vehicle over the bridge. The unmanned aerial vehicle uses DJI Mavic3E unmanned aerial vehicle, the camera resolution is 3840×2160, and the frame rate is 30fps.

[0076] At the same time, a vehicle speed test system is used to verify the method of the present application. The speed test system connects the rear end of the model vehicle with a winding shaft by a nylon rope, and an angle sensor is installed at the center of rotation of the winding shaft to measure the rotation angle of the winding shaft. By converting the rotation of the winding shaft into the length of the rope released, the displacement of the vehicle advancing can be accurately calculated.

[0077] ​​B. Extract image feature points and use YOLOv4 to detect vehicle regions in the image, separate static feature points and vehicle dynamic feature points; specifically including:

[0078] B1. Take 200 photos of the model car as a data set, according to the proportion of 8:2, 160 for model training, 40 for model testing, the flow chart is as shown in Figure 4 ;

[0079] B2. Use the trained YOLOv4 model to identify the vehicle region in the video image.

[0080] B3. Extract video ORB feature points, separate dynamic feature points in the vehicle region and static feature points in the surrounding environment, the separated feature points are as shown in Figure 5 . White is the static feature point, which is used for camera pose estimation, and black is the removed dynamic feature point.

[0081] C. Use SLAM algorithm to preliminarily estimate the camera pose, and introduce the chessboard to correct the camera pose; specifically including:

[0082] C1. Use SLAM algorithm to preliminarily estimate the camera pose. P1-P n The pose matrix of the camera pose in CCS is:

[0083]

[0084] In the formula, A iest is the camera pose matrix estimated by ORB-SLAM, R pi is the rotation matrix in the pose matrix, t pi is the translation matrix in the pose matrix, and O is the zero vector.

[0085] C2. Use the chessboard to calibrate part of the camera pose. P1-P k is the part of the camera pose that is photographed to the chessboard, and the chessboard is used to calibrate P1-P k in the world coordinate system:

[0086]

[0087] In the formula, A ireal is the camera pose matrix calibrated by the chessboard, R qi is the rotation matrix in the pose matrix, and t qi is the translation matrix in the pose matrix.

[0088] C3. According to the part of the calibrated camera pose, correct the initial all camera poses. The transformation matrix of the camera center point in CCS and WCS is:

[0089]

[0090] T1 transformation satisfies the rotation and displacement pose of camera center point from CCS and WCS, but changes the scale. To restore the scale relationship, further adopt T 2i transformation:

[0091]

[0092] where T 2i is the transformation matrix to restore the scale relationship, E is the unit matrix, t wpi is the coordinates of the point set {P i} under T1 transformation:

[0093] t wpi = λRt pi + t (5)

[0094] Then the estimated pose matrix A iest at each point can be corrected by the following transformation:

[0095] A imod = T 2i T1A iest (6)

[0096] where A imod is the corrected camera pose matrix, which can represent the pose of the camera in a specific world coordinate system. The corrected camera pose is shown in FIG. 6.

[0097] D. Calculate the position of the camera in the world coordinate system in combination with the coordinates of the vehicle in the image and the corrected camera pose; specifically including:

[0098] D1. Calculate the center of the vehicle region in the image, then the coordinates of the vehicle i in the pixel coordinate system at the moment are:

[0099] V i = [u i v i ] (7)

[0100] where u i and v i are the coordinates of the vehicle in the pixel coordinate system.

[0101] D2. Position the vehicle in the world coordinate system in combination with the camera pose. The transformation from the image coordinate system to the world coordinate system can be represented as:

[0102]

[0103] where X wi , Y wi and Z wi are the coordinates in the world coordinate system, K IThe camera intrinsic matrix can be obtained by camera calibration. Ei The camera extrinsic matrix under different camera poses can be obtained by modifying the camera pose matrix A imod obtained:

[0104]

[0105] It is considered that the vehicle is always on the road plane, i.e. Z wi = 0. According to equation (15), the coordinates of the vehicle in the world coordinate system can be calculated to realize the positioning of the moving vehicle. The vehicle trajectory is shown in Figure 7 Fig. 6, and the real-time curve of the vehicle displacement is shown in Fig. 8.

[0106] Embodiment 2

[0107] This embodiment relates to a mobile vehicle bridge positioning device based on UAV cruising, comprising a memory and one or more processors, the memory storing executable code, and the one or more processors executing the executable code to implement the mobile vehicle bridge positioning method based on UAV cruising of embodiment 1.

[0108] Embodiment 3

[0109] This embodiment relates to a computer-readable storage medium, which stores a program that is executed by a processor to implement the mobile vehicle bridge positioning method based on UAV cruising of embodiment 1.

[0110] The content described in the specification of the embodiments of the present application is only a list of implementation forms of the inventive concept, and the protection scope of the present application should not be regarded as being limited to the specific forms stated in the embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A method for positioning a mobile vehicle crossing a bridge based on drone cruising, characterized in that: The following steps are involved: A. Use a drone to capture video of a moving vehicle crossing a bridge; specifically: A1. Arrange a checkerboard pattern at the starting point of the drone. A2. Control the drone to hover and capture the checkerboard pattern, providing data for camera calibration. A3. Control the drone to track and film the vehicle as it crosses the bridge. B. Extract image feature points and use YOLOv4 to detect vehicle areas in the image, separating static feature points from dynamic feature points of the vehicle; C. Use the SLAM algorithm to preliminarily estimate the camera pose from static feature points and introduce a checkerboard to correct the camera pose; specifically, C1. Use SLAM algorithm to preliminarily estimate the camera pose; the camera pose estimated by ORB-SLAM is based on the initial camera coordinate system (CCS), P1~P n is the initial full camera pose; C2. Use a checkerboard to calibrate some camera poses; place a checkerboard in the field of view of some camera poses and establish a world coordinate system (WCS) with the checkerboard corners as the center; P1~P k Part of the camera pose that captures the checkerboard; C3. Correct the initial camera pose based on the partially calibrated camera pose; the camera center point in the CCS and WCS are: Point set {P i }、{Q i Decentralize and calculate the covariance matrix; use SVD decomposition to obtain the covariance matrix U and V, and obtain the rotation matrix R; Then the point set {P i } to {Q i The transformation matrix of} is: The T1 transformation satisfies the rotation and displacement of the camera center point from the CCS and WCS, but changes the scale. To restore the scale relationship, T is further used. 2i Transformation: Where, T 2i is the transformation matrix to restore the scaling relationship, E is the identity matrix, t wpi For the point set {P i }Coordinates under T1 transformation: Then the estimated posture matrix A at each location iest The following transformation can be used for correction: Where A imod is the corrected camera pose matrix, which can represent the pose of the camera in a specific world coordinate system; D. Combine the vehicle's coordinates in the image and the corrected camera pose to calculate the camera's position in the world coordinate system.

2. The method for positioning a mobile vehicle crossing a bridge based on a UAV cruise according to claim 1, characterized in that: Step B specifically includes: B1. Take several images of the target vehicle from different angles as the YOLOv4 dataset, using an 8:2 ratio for model training and testing. B2. Use the trained YOLOv4 model to identify vehicle regions in the video image. B3. Extract video ORB feature points and separate the dynamic feature points of the vehicle area and the static feature points of the surrounding environment.

3. The method for positioning a mobile vehicle crossing a bridge based on a UAV cruise according to claim 1, characterized in that: In step C, P1~P n The posture matrix of the camera posture under CCS is: Where A iest is the camera pose matrix estimated by ORB-SLAM, R pi is the rotation matrix in the attitude matrix, t pi is the translation matrix in the attitude matrix, O is the zero vector; The P1~P k Pose in the world coordinate system: Where A ireal is the camera pose matrix calibrated by the chessboard, R qi is the rotation matrix in the attitude matrix, t qi is the translation matrix in the attitude matrix; The point set {P i }、{Q i Decentralization is represented by: Where, and They are point sets {P i } and {Q i }, and is the coordinate of decentralization; Compute the covariance matrix: Where H is the covariance matrix; SVD decomposition is used to obtain the matrices U and V, and the rotation matrix R can be calculated as: The scaling factor and translation matrix can be calculated as: 。 4. The method for positioning a mobile vehicle crossing a bridge based on a UAV cruise according to claim 1, characterized in that: Step D specifically includes: D1. Calculate the center of the vehicle area in the image, then the vehicle i The coordinates of the moment in the pixel coordinate system are: Where, u i and v i is the coordinate of the vehicle in the pixel coordinate system; D2. Locate the vehicle's position in the world coordinate system based on the camera's pose. The transformation from the image coordinate system to the world coordinate system can be expressed as: Where, X wi 、 Y wi and Z wi is the coordinate in the world coordinate system, K I is the camera internal parameter matrix, which can be obtained by camera calibration, K Ei is the camera extrinsic matrix under different camera postures, which can be obtained by the corrected camera posture matrix A imod get: Assuming that the vehicle is always on the road surface, that is Z wi =0, and the coordinates of the vehicle in the world coordinate system can be calculated by formula (15), thus realizing the positioning of the moving vehicle.

5. A mobile vehicle bridge positioning device based on drone cruising, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a mobile vehicle crossing bridge positioning method based on drone cruising as described in any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by the processor, a method for positioning a mobile vehicle crossing a bridge based on drone cruising as described in any one of claims 1 to 4 is implemented.

Images