Mobile vehicle bridge-crossing positioning method and device based on unmanned aerial vehicle cruise

Abstract

The invention relates to a mobile vehicle bridge-crossing positioning method and device based on unmanned aerial vehicle cruise, an unmanned aerial vehicle is adopted to shoot a vehicle, and SLAM and YOLO algorithms are combined to realize camera attitude estimation and vehicle real-time positioning, and the method comprises the following steps: A, the unmanned aerial vehicle is adopted to shoot a mobile vehicle bridge-crossing; b, extracting ORB feature points of the image, detecting a vehicle area in the image by adopting YOLOv4, and separating static feature points and vehicle dynamic feature points; c, preliminarily estimating the camera attitude from the static feature points by adopting an SLAM algorithm, and introducing a checkerboard to correct the camera attitude; d, calculating the position of the camera in the world coordinate system according to the coordinates of the vehicle in the image and the corrected camera attitude; according to the test method, unmanned aerial vehicle cruise is applied to vehicle positioning, and the test method has the advantages of long-distance test and high precision.

Description

Technical Field

[0001] The invention relates to a method and a device for positioning a mobile vehicle crossing a bridge based on unmanned aerial vehicle cruising, and belongs to the field of bridge detection. Background Art

[0002] Vehicle load is one of the main dynamic loads on bridge structures and occupies an important position in various load combinations. The loading method of moving vehicles across bridges has advantages such as economy and efficiency. In existing bridge structure research, it is widely used for influence line extraction, damage detection, and bearing capacity assessment. Therefore, accurate acquisition of vehicle load information is crucial for bridge service status assessment.

[0003] Vehicle load information usually includes two aspects: load size and load location. The load size is usually obtained by a dynamic weighing system or an on-board weighing system to obtain vehicle axle weight information. Regarding the load location, some scholars obtain the vehicle position by maintaining a constant speed of the vehicle, using laser rangefinders or GPS. These methods are usually suitable for model tests, or require sensors to be installed on the vehicle in advance, and are difficult to apply to actual vehicle bridge crossing scenarios.

[0004] In recent years, computer vision testing methods have been widely used in the field of civil engineering, and combining video images to achieve vehicle positioning has become a feasible idea. Most scholars set up cameras on bridges and use YOLOv4 to detect vehicles from surveillance videos to achieve vehicle positioning. The bridge structure is large in the span direction, and it is usually necessary to set up multiple cameras and combine image stitching technology to achieve the whole process positioning of the vehicle crossing the bridge. The SLAM-based drone cruising technology uses mobile shooting to avoid the installation of cameras on the entire bridge, which is a promising way to locate moving vehicles. However, for the shooting environment of vehicles moving across the bridge, the camera posture estimated by the SLAM algorithm will be disturbed by the moving vehicle and does not have the real size. Therefore, for the scene of vehicles moving across the bridge, the application of drone cruising technology to vehicle positioning needs further research. Summary of the invention

[0005] In view of the shortcomings of the existing methods, the present invention proposes a method and device for positioning a mobile vehicle crossing a bridge based on drone cruising. This technology uses a drone to shoot the vehicle and combines SLAM and YOLO algorithms to achieve camera posture estimation and real-time vehicle positioning.

[0006] The first aspect of the present invention relates to a method for positioning a mobile vehicle crossing a bridge based on drone cruising, comprising the following steps:

[0007] A. Use drones to shoot videos of moving vehicles crossing bridges;

[0008] B. Extract image feature points, use YOLOv4 to detect the vehicle area in the image, and separate static feature points and vehicle dynamic feature points;

[0009] C. Use SLAM algorithm to preliminarily estimate the camera pose from static feature points, and introduce chessboard to correct the camera pose;

[0010] D. Combine the vehicle's coordinates in the image and the corrected camera pose to calculate the camera's position in the world coordinate system.

[0011] Further, step A specifically includes:

[0012] A1. Arrange a checkerboard grid at the starting point of the drone.

[0013] A2. Control the drone to hover and shoot the chessboard to provide data basis for camera calibration;

[0014] A3. When the vehicle is crossing the bridge, control the drone to track and photograph the vehicle;

[0015] Further, step B specifically includes:

[0016] B1. Take several pictures of the target vehicle at different angles as the YOLOv4 dataset, which are used for model training and model testing in an 8:2 ratio;

[0017] B2. Use the trained YOLOv4 model to identify the vehicle area in the video image.

[0018] B3. Extract video ORB feature points and separate the dynamic feature points of the vehicle area and the static feature points of the surrounding environment.

[0019] Further, step C specifically includes:

[0020] C1. Use SLAM algorithm to preliminarily estimate the camera posture. Figure 2 As shown in the figure, the camera pose estimated by ORB-SLAM is usually based on the initial camera coordinate system (CCS), P1~P n The attitude matrix of the camera attitude under CCS is:

[0021]

[0022] In the formula, A iest is the camera pose matrix estimated by ORB-SLAM, R pi is the rotation matrix in the attitude matrix, t pi is the translation matrix in the attitude matrix, and O is the zero vector.

[0023] C2. Use a chessboard to calibrate some camera poses. Put a chessboard in the field of view of some camera poses, and establish a world coordinate system (WCS) with the corners of the chessboard as the center. kIn order to capture the partial camera posture of the chessboard, the chessboard is used to calibrate P1~P k Posture in the world coordinate system:

[0024]

[0025] In the formula, A ireal is the camera pose matrix calibrated by the chessboard, R qi is the rotation matrix in the attitude matrix, t qi is the translation matrix in the attitude matrix.

[0026] C3. Correct the initial camera pose according to the partially calibrated camera pose. The camera center point under CCS and WCS are:

[0027] {P i}=t pi , {Q i}=t qi (3) Calculate the point set {P i} to {Q i}, to decentralize the point set:

[0028]

[0029] In the formula, and They are point sets {P i} and {Q i} center point, and For decentralized coordinates.

[0030] Compute the covariance matrix:

[0031]

[0032] Where H is the covariance matrix. Using SVD decomposition to obtain the matrices U and V, the rotation matrix R can be calculated as:

[0033]

[0034] The scaling factor and translation matrix can be calculated as:

[0035]

[0036] Then the point set {P i} to {Q i The transformation matrix of} is:

[0037]

[0038] The T1 transformation satisfies the rotation and displacement of the camera center point from the CCS and WCS, but changes the scale. To restore the scale relationship, T 2i Transformation:

[0039]

[0040] Where, T 2i is the transformation matrix to restore the scaling relationship, E is the identity matrix, t wpi For the point set {P i}Coordinates under T1 transformation:

[0041] t wpi =λRt pi +t(12)

[0042] Then the estimated posture matrix A at each location iest The following transformation can be used for correction:

[0043] A imod =T 2i T1A iest (13)

[0044] In the formula, A imod is the corrected camera attitude matrix, which can represent the attitude of the camera in a specific world coordinate system.

[0045] Further, step D specifically includes:

[0046] D1. Calculate the center of the vehicle area in the image, then the coordinates of vehicle i at time in the pixel coordinate system are:

[0047] V i =[u i v i ](14)

[0048] In the formula, u i and v i is the coordinate of the vehicle in the pixel coordinate system.

[0049] D2. Locate the vehicle's position in the world coordinate system based on the camera's posture. The transformation from the image coordinate system to the world coordinate system can be expressed as:

[0050]

[0051] Where, X wi , Y wi and Z wi is the coordinate in the world coordinate system, K I K is the camera internal parameter matrix, which can be obtained by camera calibration. Ei is the camera extrinsic matrix under different camera postures, which can be obtained by the corrected camera posture matrix Aimod get:

[0052]

[0053] Assume that the vehicle is always on the road surface, that is, Z wi = 0. According to formula (15), the coordinates of the vehicle in the world coordinate system can be calculated to achieve the positioning of the mobile vehicle.

[0054] The second aspect of the present invention relates to a mobile vehicle crossing a bridge positioning device based on drone cruising, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a mobile vehicle crossing a bridge positioning method based on drone cruising of the present invention.

[0055] The third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a method for positioning a mobile vehicle crossing a bridge based on drone cruising of the present invention.

[0056] The present invention uses drones to shoot vehicles, combined with SLAM and YOLO algorithms, to achieve camera attitude estimation and real-time vehicle positioning. The patented test method of the present invention applies drone cruising to vehicle positioning, which has the advantages of long-distance testing and high precision, and provides a new idea for the positioning of mobile vehicles crossing bridges.

[0057] The present invention has the following advantages:

[0058] (1) The method disclosed in the present invention uses YOLOv4 to identify the vehicle area in the image, and removes the feature points of the area as dynamic feature points. The static feature points are used to improve the accuracy of camera posture estimation.

[0059] (2) The method disclosed in the present invention uses a checkerboard to correct the camera pose initially estimated by SLAM, so that it has the size of the real world and can be directly used to establish the relationship between the camera coordinate system and the world coordinate system.

[0060] (3) The method disclosed in the present invention uses a drone to move and shoot vehicles, and can locate the vehicle from the video image. There is no need to install sensors on the vehicle or set up multiple cameras on the bridge deck, which has the advantage of convenience. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flow chart of the mobile vehicle positioning method of the present invention;

[0062] Figure 2 is a schematic diagram of a drone photographing a vehicle of the present invention;

[0063] Figure 3It is a layout diagram of a vehicle moving over a bridge test of the present invention;

[0064] Figure 4 It is the YOLOv4 flow chart of the present invention;

[0065] Figure 5 is a static feature point extraction graph of the present invention;

[0066] FIG6(a) and FIG6(b) are schematic diagrams of camera trajectories of the present invention, wherein FIG6(a) is a slow working condition, and FIG6(b) is a fast working condition;

[0067] Figure 7 is a vehicle trajectory diagram of the present invention;

[0068] FIG8(a) and FIG8(b) are real-time curve diagrams of vehicle displacement of the present invention, wherein FIG8(a) is a slow working condition, and FIG8(b) is a fast working condition; DETAILED DESCRIPTION

[0069] The following is a further detailed description of a method for positioning a mobile vehicle crossing a bridge based on UAV cruising according to the present invention in conjunction with the accompanying drawings.

[0070] Example 1

[0071] like Figure 1 As shown, this embodiment relates to a method for positioning a mobile vehicle crossing a bridge based on drone cruising, which specifically includes the following steps:

[0072] A. Use drones to shoot videos of moving vehicles crossing bridges; specifically:

[0073] A1. Arrange a checkerboard grid at the starting point of the drone. The mobile vehicle crossing the bridge model and the test site are as follows Figure 3 As shown in the figure, the test beam is a simply supported steel beam with a calculated span of 4.85m. A model car is used to simulate the movement of vehicles across the bridge.

[0074] A2. Control the drone to hover and shoot the chessboard to provide data basis for camera calibration;

[0075] A3. While the vehicle is driving, control the drone to move and shoot the vehicle, recording the entire process of the vehicle crossing the bridge. The drone uses a DJI Mavic3E drone with a camera resolution of 3840×2160 and a frame rate of 30fps.

[0076] At the same time, a vehicle speed test system is used to verify the method of the present invention. The speed test system uses a nylon rope to connect the rear end of the model vehicle to the winding shaft, and an angle sensor is installed at the rotation center of the winding shaft to measure the rotation angle of the winding shaft. By converting the rotation of the winding shaft into the length of the rope released, the displacement of the vehicle can be accurately calculated.

[0077] B. Extract image feature points, use YOLOv4 to detect the vehicle area in the image, and separate static feature points and vehicle dynamic feature points; specifically include:

[0078] B1. Take 200 photos of the model car as a data set. According to the ratio of 8:2, 160 photos are used for model training and 40 photos are used for model testing. The flow chart is as follows: Figure 4 As shown;

[0079] B2. Use the trained YOLOv4 model to identify the vehicle area in the video image.

[0080] B3. Extract video ORB feature points, separate the dynamic feature points of the vehicle area and the static feature points of the surrounding environment. The separated feature points are as follows: Figure 5 As shown in Figure 2, white is the static feature points used for camera pose estimation, and black is the dynamic feature points that are removed.

[0081] C. Use the SLAM algorithm to preliminarily estimate the camera pose, and introduce the checkerboard to correct the camera pose; specifically include:

[0082] C1. Use SLAM algorithm to preliminarily estimate the camera posture. n The attitude matrix of the camera attitude under CCS is:

[0083]

[0084] In the formula, A iest is the camera pose matrix estimated by ORB-SLAM, R pi is the rotation matrix in the attitude matrix, t pi is the translation matrix in the attitude matrix, and O is the zero vector.

[0085] C2. Use chessboard to calibrate some camera poses. k In order to capture the partial camera posture of the chessboard, the chessboard is used to calibrate P1~P k Posture in the world coordinate system:

[0086]

[0087] In the formula, A ireal is the camera pose matrix calibrated by the chessboard, R qi is the rotation matrix in the attitude matrix, t qi is the translation matrix in the attitude matrix.

[0088] C3. Correct the initial camera posture according to the partially calibrated camera posture. The transformation matrix of the camera center point in CCS and WCS is:

[0089]

[0090] The T1 transformation satisfies the rotation and displacement of the camera center point from the CCS and WCS, but changes the scale. To restore the scale relationship, T 2i Transformation:

[0091]

[0092] Where, T 2i is the transformation matrix to restore the scaling relationship, E is the identity matrix, t wpi For the point set {P i}Coordinates under T1 transformation:

[0093] t wpi =λRt pi +t(5)

[0094] Then the estimated posture matrix A at each location iest The following transformation can be used for correction:

[0095] A imod =T 2i T1A iest (6)

[0096] In the formula, A imod is the corrected camera attitude matrix, which can represent the attitude of the camera in a specific world coordinate system. The corrected camera attitude is shown in Figure 6.

[0097] D. Combine the coordinates of the vehicle in the image and the corrected camera pose to calculate the position of the camera in the world coordinate system; specifically:

[0098] D1. Calculate the center of the vehicle area in the image, then the coordinates of vehicle i at time in the pixel coordinate system are:

[0099] V i =[u i v i ](7)

[0100] In the formula, u i and v i is the coordinate of the vehicle in the pixel coordinate system.

[0101] D2. Locate the vehicle's position in the world coordinate system based on the camera's posture. The transformation from the image coordinate system to the world coordinate system can be expressed as:

[0102]

[0103] Where, X wi , Y wi and Z wi is the coordinate in the world coordinate system, K IK is the camera internal parameter matrix, which can be obtained by camera calibration. Ei is the camera extrinsic matrix under different camera postures, which can be obtained by the corrected camera posture matrix A imod get:

[0104]

[0105] Assume that the vehicle is always on the road surface, that is, Z wi = 0. By formula (15), the coordinates of the vehicle in the world coordinate system can be calculated to achieve the positioning of the moving vehicle. The vehicle trajectory is as follows Figure 7 As shown, the real-time curve of vehicle displacement is shown in Figure 8.

[0106] Example 2

[0107] The present embodiment relates to a mobile vehicle bridge crossing positioning device based on drone cruising, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a mobile vehicle bridge crossing positioning method based on drone cruising of embodiment 1.

[0108] Example 3

[0109] This embodiment relates to a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a method for positioning a mobile vehicle crossing a bridge based on drone cruising according to embodiment 1 is implemented.

[0110] The contents described in the implementation cases of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be deemed to be limited to the specific forms described in the implementation cases. The protection scope of the present invention also extends to the equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for positioning a mobile vehicle crossing a bridge based on UAV cruising, characterized in that: The following steps are involved: A. Use drones to shoot videos of moving vehicles crossing bridges; B. Extract image feature points, use YOLOv4 to detect the vehicle area in the image, and separate static feature points and vehicle dynamic feature points; C. Use SLAM algorithm to preliminarily estimate the camera pose from static feature points, and introduce chessboard to correct the camera pose; D. Combine the vehicle's coordinates in the image and the corrected camera pose to calculate the camera's position in the world coordinate system.

2. The method for positioning a mobile vehicle crossing a bridge based on UAV cruising according to claim 1 is characterized in that: Step A specifically includes: A1. Arrange a chessboard at the starting point of the drone; A2. Control the drone to hover and shoot the chessboard to provide data basis for camera calibration; A3. When a vehicle crosses the bridge, control the drone to track and photograph the vehicle.

3. The method for positioning a mobile vehicle crossing a bridge based on UAV cruising according to claim 1 is characterized in that: Step B specifically includes: B1. Take several pictures of the target vehicle at different angles as the YOLOv4 dataset, which are used for model training and model testing in an 8:2 ratio; B2. Use the trained YOLOv4 model to identify the vehicle area in the video image; B3. Extract video ORB feature points and separate the dynamic feature points of the vehicle area and the static feature points of the surrounding environment.

4. The method for positioning a mobile vehicle crossing a bridge based on UAV cruising according to claim 1 is characterized in that: Step C specifically includes: C1. Use SLAM algorithm to preliminarily estimate the camera pose. The camera pose estimated by ORB-SLAM is usually based on the initial camera coordinate system (CCS). n The attitude matrix of the camera attitude under CCS is: In the formula, A iest is the camera pose matrix estimated by ORB-SLAM, R pi is the rotation matrix in the attitude matrix, t pi is the translation matrix in the attitude matrix, O is the zero vector; C2. Use a chessboard to calibrate some camera poses. Put a chessboard in the field of view of some camera poses, and establish a world coordinate system (WCS) with the corners of the chessboard as the center. k In order to capture the partial camera posture of the chessboard, the chessboard is used to calibrate P1~P k Posture in the world coordinate system: In the formula, A ireal is the camera pose matrix calibrated by the chessboard, R qi is the rotation matrix in the attitude matrix, t qi is the translation matrix in the attitude matrix; C3. Correct the initial camera pose according to the partially calibrated camera pose. The camera center point under CCS and WCS are: {P i }=t pi ,{Q i }=t qi (3) Calculate the point set {P i } to {Q i }, to decentralize the point set: In the formula, and They are point sets {P i } and {Q i } center point, and For decentralized coordinates. Compute the covariance matrix: Where H is the covariance matrix. Using SVD decomposition to obtain the matrices U and V, the rotation matrix R can be calculated as: The scaling factor and translation matrix can be calculated as: Then the point set {P i } to {Q i The transformation matrix of} is: The T1 transformation satisfies the rotation and displacement of the camera center point from the CCS and WCS, but changes the scale. To restore the scale relationship, T 2i Transformation: Where, T 2i is the transformation matrix to restore the scaling relationship, E is the identity matrix, t wpi For the point set {P i }Coordinates under T1 transformation: t wpi =λRt pi +t (12) Then the estimated posture matrix A at each location iest The following transformation can be used for correction: A imod =T 2i T1A iest (13) In the formula, A imod is the corrected camera attitude matrix, which can represent the attitude of the camera in a specific world coordinate system.

5. The method for positioning a mobile vehicle crossing a bridge based on UAV cruising according to claim 1 is characterized in that: Step D specifically includes: D1. Calculate the center of the vehicle area in the image, then the coordinates of vehicle i at time in the pixel coordinate system are: V i =[u i v i ] (14) In the formula, u i and v i is the coordinate of the vehicle in the pixel coordinate system; D2. Locate the vehicle's position in the world coordinate system based on the camera's posture. The transformation from the image coordinate system to the world coordinate system can be expressed as: Where, X wi , Y wi and Z wi is the coordinate in the world coordinate system, K I K is the camera internal parameter matrix, which can be obtained by camera calibration. Ei is the camera extrinsic matrix under different camera postures, which can be obtained by the corrected camera posture matrix A imod get: Assume that the vehicle is always on the road surface, that is, Z wi = 0. According to formula (15), the coordinates of the vehicle in the world coordinate system can be calculated to achieve the positioning of the mobile vehicle.

6. A mobile vehicle bridge positioning device based on drone cruising, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement a method for positioning a mobile vehicle crossing a bridge based on drone cruising as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by the processor, a method for positioning a mobile vehicle crossing a bridge based on drone cruising as described in any one of claims 1 to 5 is implemented.

Images