Expressway disaster monitoring and early warning method
Through the fixed-wing drone and YOLOv5 algorithm, efficient and accurate monitoring and early warning of road emergencies are achieved, and the problems of single monitoring methods, insufficient coverage and high costs in the existing technology are solved, and disaster warning capabilities are improved.
Patent Information
- Application Number
- CN202410997056.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing road emergency monitoring methods are single, insufficient coverage, inaccurate data, high cost and unpredictable, resulting in blind spots and inefficient monitoring and ineffective disaster prevention.
The fixed-wing drone is equipped with a camera for image acquisition and three-dimensional reconstruction, and the potential disasters are identified through path planning and data comparison, and the YOLOv5 algorithm is used to monitor the road surface conditions in real time to achieve intelligent disaster warning.
It improves the accuracy and coverage of road emergencies monitoring, realizes prediction and real-time monitoring of potential disasters, reduces costs, and improves monitoring efficiency and early warning capabilities.
Smart Images

Figure CN118967095A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of disaster early warning, and in particular relates to a highway disaster monitoring and early warning method. Background Art
[0002] Road emergency monitoring is a vital part of traffic management. By monitoring traffic conditions and emergencies such as landslides and predicting accidents before they occur, countermeasures can be taken to improve road safety and traffic efficiency. However, the current road emergency monitoring is not popular and the means are single, which has great limitations.
[0003] 1. The monitoring methods are extremely simple: Traditional monitoring of road emergencies mainly relies on traffic cameras on the road or regular manual inspections. The methods are simple and the selection space is small, which often cannot meet the needs of on-site monitoring.
[0004] 2. Insufficient equipment coverage: The distribution of monitoring traffic cameras is limited, and some sections of the road are often not covered. Manual monitoring has great geographical limitations and cannot conduct large-scale inspections. It can only detect small areas. Moreover, roads in remote areas such as mountainous areas are usually not equipped with traffic cameras or manual monitoring. This leads to blind spots and insufficient monitoring.
[0005] 3. Inaccurate data: Existing traffic cameras have poor accuracy in identifying sudden situations such as car accidents and landslides. They can usually only identify data such as vehicle speed and traffic volume, and cannot detect sudden situations well.
[0006] 4. High monitoring cost: Because cameras can only monitor a very short section of road, a large number of devices are required. The installation and maintenance of a large number of devices also require extremely high costs. The cost of manual patrols is also very high. Therefore, the government needs to invest a lot of funds and human resources, which is a great challenge to the sustainability and long-term operation of road monitoring.
[0007] 5. Poor data timeliness: Traditional monitoring methods often take a long time to transmit data to the driver after discovering an emergency. This is slow and inefficient.
[0008] 6. Unpredictable: Unable to predict emergencies that have not yet occurred, lack of foresight, unable to prevent, and unable to effectively avoid casualties.
[0009] Therefore, with the discovery of these problems, in order to efficiently and accurately detect road emergencies, a new and intelligent method is needed for monitoring. Summary of the invention
[0010] The purpose of the present invention is to provide a highway disaster monitoring and early warning method to solve the problems existing in the background technology.
[0011] In order to achieve the above technical objectives, the technical solution adopted by the present invention is as follows:
[0012] A highway disaster monitoring and early warning method comprises the following steps:
[0013] Step 100: Camera calibration, installing the camera at the lower end of the fixed-wing UAV, and then calibrating the camera carried by the fixed-wing UAV to obtain internal and external parameters to ensure the accuracy of the image data;
[0014] Obtain the camera's intrinsic parameters, extrinsic parameters, world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system, and establish the camera's imaging model based on the obtained content:
[0015] Model building includes:
[0016] Step 110: Rigid body transformation: that is, the transformation from the world coordinate system to the camera coordinate system. The coordinates of the object in the world coordinate system (x w ,y w , z w ) and the coordinates in the camera coordinate system (x c ,y c , z c ) satisfies the following formula:
[0017]
[0018] Step 120: Perspective projection: that is, the conversion from the camera coordinate system to the image coordinate system. The coordinates of the object in the camera coordinate system (x c ,y c , z c ) and the coordinates of the corresponding object image in the image coordinate system (x p ,y p ) satisfies the following formula:
[0019]
[0020] Step 130: Distortion model: Since the object is distorted when it is imaged through a lens, a distortion model needs to be introduced. The image coordinates before distortion removal are (x, y), and the image coordinates after distortion removal are (x crt ,y crt ); the distance r from a point (x, y) in the image to the center of the image satisfies the following formula:
[0021]
[0022] Distortion includes radial distortion and tangential distortion. The coordinates after distortion elimination (x crt ,y crt ) satisfies the following formula:
[0023]
[0024] Among them, x rcrt and rcrt is the radial distortion term, satisfying the following formula:
[0025]
[0026] x rcrt and rcrt is the tangential distortion term, satisfying the following formula:
[0027]
[0028] Among them, k1, k2, k3, p1, and p2 are all distortion parameters;
[0029] Step 140: secondary conversion: conversion from the image coordinate system to the pixel coordinate system, and conversion from the pixel coordinate system to the world coordinate system;
[0030] The coordinates of the object in the image coordinate system (x p ,y p ) and the coordinates (u, v) of the corresponding pixel in the digital image satisfy the following formula:
[0031]
[0032] Among them, (u0, v0) is the coordinate of the image center in the pixel coordinate system;
[0033] The transformation from pixel coordinate system to world coordinate system satisfies the following formula:
[0034]
[0035] in, is the camera’s intrinsic parameter matrix, is the extrinsic parameter matrix of the camera.
[0036] Step 200: Route planning, by setting a number of road checkpoints on the console, the drone automatically plans the path and flies autonomously, traversing all the checkpoints;
[0037] The steps of automatically planning the path of the drone are as follows:
[0038] Step 210: Initialization, creating openList to store explorable points and closeList to store explored points to prevent repeated searches, gridding the three-dimensional space, treating each grid as a node; marking obstacles in the grid, determining the start node and the target node, and adding the start node to the openList;
[0039] Step 220: define a node evaluation function f(n)=g(n)+h(n), and use Manhattan distance to calculate the cost from the current node to the target node and the start node; the Manhattan distance calculation formula is as follows:
[0040] d(i, j) = |x i -x j |-|y i -y j |
[0041] Where i and j are two target points, and d(i, j) represents the Manhattan distance between the two points;
[0042] Step 230: Starting from the starting node, search other nodes, first perform a straight line search, and then perform an oblique search: if an obstacle, a jump point or a boundary is found in a certain direction, the search ends. If a jump point is found, it is added to the openList;
[0043] The specific method for searching nodes in step 230 is as follows:
[0044] Step 231: Set the g values of all other nodes in the graph to infinity;
[0045] Step 232: loop (until the queue is empty or the target node is found), if the queue is originally empty, return false;
[0046] Step 234: For all unexpanded neighbors of node n, first search in all straight line directions. If the boundary is touched, stop the exploration in this direction; if the jump point is touched, mark the touched jump point as node m and stop searching in this direction;
[0047] Step 235: Perform a single-step search in all oblique directions. If the boundary is touched, stop the exploration in this direction; if the jump point is touched, mark the touched jump point as node m and stop the search in this direction;
[0048] If the boundary or jump point is not touched, search in the horizontal decomposition direction of the current direction of travel. If the boundary is touched, stop exploring in this direction. If the jump point is touched, mark the touched jump point as node m and stop searching in this direction.
[0049] Among them: if n has a parent node, the search direction only needs to be the direction of the line connecting the parent node and n, the horizontal decomposition direction and the direction of the line connecting n and the mandatory neighbor;
[0050] Step 236: If the g value of node m is infinite, it means that node m is not expanded, so let g(m) = g(n) + cost nm , where cost nmIndicates the distance cost between node n and node m, and adds node m to openList, removes n from openList and adds it to closeList
[0051] Step 237: If g(m)>g(n)+cost nm , then let g(m)=g(n)+cost nm , do not add it to openList, delete n from openList and add it to closeList;
[0052] Step 237: End the loop.
[0053] Step 240: If the search in the oblique direction is not completed in a single round, take a step forward in the oblique direction and repeat step 230 until the search is completed. If all directions starting from this node have been searched, remove this node from the openList and add it to the closeList;
[0054] Step 250: Take the node with the lowest cost in openList, that is, the node with the smallest f value, as the new parent node, search all straight and diagonal directions, and repeat steps 230 and 240 until openList is empty or the target node is found;
[0055] If openList is empty and the target node has not been found, false is returned;
[0056] If the target node is found, connect all the points in closeList to get the desired path.
[0057] Step 300: Data collection and data transmission. During the flight, the drone uses the camera it carries to collect images of the ground and the surrounding environment, and transmits the collected images back to the ground station through the wireless communication module for subsequent processing;
[0058] Step 400: 3D reconstruction: the ground station performs 3D reconstruction of the road surface and surrounding environment along the expressway based on the images collected by the fixed-wing UAV and saves the data;
[0059] The specific steps of three-dimensional reconstruction in step 400 include:
[0060] Step 410: Image preprocessing. In the image preprocessing stage, a bilateral filter is used to denoise the image. The expression is as follows:
[0061]
[0062] Among them, (k, l) is the coordinate of the center point of the template window, (i, j) is any point in the neighborhood of point (k, l), and ω(i, j, k, l) is the weighting coefficient of the filter, which satisfies the following formula:
[0063] ω(i,j,k,l)=d(i,j,k,l)*r(i,j,k,l)
[0064] Among them, d(i, j, k, l) and r(i, j, k, l) are the spatial domain kernel and range kernel of the filter respectively, satisfying the following formula:
[0065]
[0066] Among them, σ d and σ r are the standard deviations of Gaussian functions;
[0067] Step 420: feature extraction, using SIFT algorithm to extract features of the image;
[0068] The key steps of the SIFT algorithm are as follows:
[0069] Step 421: scale space extreme value detection;
[0070] Key points are detected by constructing a Gaussian scale space. Gaussian filters of different scales are applied to the image to obtain a series of images at different scales. Then, the algorithm detects local extreme points in the scale space.
[0071] Step 422: Accurate positioning of key points;
[0072] Once the extreme points are detected, the algorithm further refines the position and scale of the key points by fitting a three-dimensional quadratic function to eliminate the noise of edge responses and improve the accuracy of key point localization;
[0073] Step 423: key point main direction assignment;
[0074] It is implemented based on the gradient direction histogram in the neighborhood of the key point, and the gradient direction is quantized into an interval from 0 to 360 degrees;
[0075] Step 424: generating key point descriptors;
[0076] The descriptor is generated based on the local image features around the key point. The key point neighborhood is divided into 16 small sub-regions, and the magnitude and direction histogram of the gradient are calculated for each sub-region.
[0077] These histograms are normalized and merged to form a 128-dimensional feature vector (descriptor), which is invariant to the scale, rotation and brightness changes of the image;
[0078] Step 430: Image matching, including the following steps:
[0079] Step 431: feature point matching;
[0080] For the feature points in the two images, feature descriptors are used for matching, which is accomplished by calculating the distance between the descriptors of different feature points. At the same time, the nearest neighbor distance ratio (Ratio Test) is used to improve the matching accuracy. For each feature point, the two nearest matching points are found. If the ratio of the nearest distance to the second nearest distance is less than the set threshold, it is considered a good match.
[0081] Step 432: Geometric constraint verification;
[0082] Since feature point matching may produce mismatches, a random sample consensus (RANSAC) algorithm is needed to filter these errors.
[0083] The RANSAC algorithm iteratively selects matching point pairs and calculates the homography matrix and fundamental matrix between them. It then classifies all matching points according to these geometric relationships and removes point pairs that do not meet the geometric constraints as mismatches.
[0084] Step 433: screening matching point pairs;
[0085] Set a threshold to remove matching points that do not meet specific conditions to ensure the quality and quantity of matching point pairs;
[0086] Step 434: optimizing matching point pairs;
[0087] Use Bundle Adjustment (BA method) to minimize the reprojection error and optimize the matching point pairs to improve the matching accuracy;
[0088] Step 440: generating a three-dimensional point cloud;
[0089] After finding the correspondence between the matching point pairs, the three-dimensional coordinates of these points are calculated by the triangulation principle;
[0090] Step 450: point cloud processing: After the point cloud model is generated, the point cloud data needs to be further processed, which specifically includes the following steps:
[0091] Step 451: point cloud denoising, using a mean filter method to perform denoising, the specific steps include:
[0092] Step 4511: Determine a neighborhood. For each point in the point cloud, determine a neighborhood range.
[0093] Step 4512: Calculate the average position: For each point, calculate the average coordinates of all points in its neighborhood;
[0094] Assume that there are N points in the neighborhood (x1, y1, z1), (x2, y2, z2), ..., (x N ,y N , z N ), then the average position It can be calculated by the following formula:
[0095]
[0096]
[0097] Step 4513: Replace the original position: replace the position of the original point with the calculated average position. If the number of points in the neighborhood is less than a certain threshold, the point can be considered an isolated point or noise, and can be deleted or retained;
[0098] Step 4514: Iterative processing: repeat the above steps 4511-4513 until the positions of all points are processed by the mean filter or the preset number of iterations is reached;
[0099] Step 452: Point cloud segmentation, the process of dividing the point cloud data set into different regions or objects, each region or object represents an independent entity.
[0100] Step 500: Data comparison and disaster warning. After the ground station completes the 3D reconstruction, the obtained data is compared with the 3D reconstruction data established during the last patrol to identify potential disasters.
[0101] The step 500, data comparison and disaster warning step, includes:
[0102] Step 510: Point cloud registration, the process of converting two or more point cloud data sets to the same coordinate system so that they can be compared and analyzed in the same reference frame. Point cloud registration includes the following steps:
[0103] Step 511: feature extraction, extracting features from each point cloud model, including points, edges, surfaces, etc., to provide reference points in the registration process;
[0104] Step 512: feature matching, calculating the Euclidean distance between the descriptors of each feature point in the source point cloud model P and all feature points in the target point cloud model Q, and selecting the point with the closest distance as the matching point pair;
[0105] Step 513: optimizing the matching results, using the RANSAC algorithm to optimize the matching results to eliminate erroneous matches that do not meet the geometric constraints;
[0106] Step 514: Calculate the transformation matrix T using the optimized matching point pairs;
[0107] Step 515: point cloud conversion, applying the transformation matrix T to the source point cloud model P to obtain a converted point cloud model P′;
[0108] Step 520: Difference calculation: after registering the point cloud models reconstructed at the same location at different times, calculate the difference value. The calculation formula is as follows:
[0109] Δx=x1-x2
[0110] Δy=y1-y2
[0111] Δz=z1-z
[0112] Among them, (x1, y1, z1) and (x2, y2, z2) are the coordinates of the same pair of matching feature points in different point cloud models;
[0113] Step 530: terrain change detection, after calculating the difference values between the matching feature point pairs, statistical analysis is performed. If the difference values of a certain area are generally greater than the set threshold, it is determined that the terrain of the area has changed significantly;
[0114] Step 540: Disaster warning. When a potential geological disaster is identified, the analysis results are compiled into a report, including information such as the location, size, and trend of terrain changes.
[0115] Step 600: Road condition identification. In addition to identifying potential disasters, the ground station also monitors road conditions in real time to identify unexpected conditions such as disasters or traffic accidents that have already occurred.
[0116] The road condition recognition method in step 600 adopts the YOLOv5 algorithm, and the specific steps include:
[0117] Step 610: pre-process the captured image, and then transmit it to the neural network via the input and Mosaic data enhancement, and pre-process the incoming image to facilitate subsequent processing of the image;
[0118] Step 620: Send the image passed in by the input end to the network backbone;
[0119] Step 621: In the network skeleton, the image will first pass through the Conv module, and the Conv module will first perform a convolution operation on the input image;
[0120] Step 622: Then, the convolved image is subjected to a Batch Normalization operation to keep the input of each layer of the neural network in the same distribution, thereby accelerating convergence and preventing overfitting, and reducing the gradient vanishing of the lower layer neural network during back propagation:;
[0121] Step 623: The Conv module finally imports the image after convolution and BN operation into the SiLu activation function, whose formula is:
[0122] silu(x)=x*6(x)
[0123] Step 624: After the Conv module is finished, the data is sent to the C3 module, which contains 3 standard convolutional layers, the number of which is determined by the product of the n and depth_multiple parameters in the configuration file yaml;
[0124] The module structure is divided into two branches, one uses a specified number of Bottleneck stacks, and the other only passes through a basic convolution module and performs a concat operation on the two branches;
[0125] Step 625: The data is fed into the SPPF module, where the input channels are halved through a standard convolution module, and then max pooling is performed with kernel-sizes of 5, 9, and 13 (the padding is adaptive for different kernel sizes).
[0126] Step 630: Import the data transformed by the Conv module into the Neck module for feature fusion;
[0127] Step 640: Put the basically processed data into the Head module to detect the target;
[0128] Step 641: After the prediction is completed, the Non-maximum suppression method is used to calculate the IOU of different bounding boxes and the bounding box with the highest confidence. If the IOU threshold between a certain bounding box and the bounding box with the highest confidence is greater than the IOU threshold we set, it means that the same object is predicted by two repeated bounding boxes, then this bounding box is removed:
[0129] Step 650: Calculate the loss function. The total loss is composed of four types of losses: position loss, size loss, confidence loss, and prediction category loss. The calculation function formulas of the four losses are as follows:
[0130]
[0131]
[0132] If the loss value is less than the specified threshold, step 660 is executed; if the loss value is greater than the specified threshold, the algorithm is back-propagated, and steps 610 to 640 are repeated to optimize the parameters;
[0133] Step 660: Output the result.
[0134] Step 700: Information transmission and response measures. The ground station presents the potential disaster analysis and warning report generated in step 5, the alarm information of the accident that has occurred, as well as the collected original image data and the reconstructed point cloud model to the control console as a decision support, and can be used to assist in the formulation of defense measures, so that relevant personnel can respond quickly according to the situation.
[0135] The present invention plans the shortest path for the operation of the drone through path planning, realizes automatic intelligent path generation in different scenarios, identifies and classifies different road emergencies through machine recognition of emergencies, thereby improving monitoring accuracy, and compares 3D models of different time periods to find differences, thereby predicting disasters that will occur, thereby realizing intelligent drone monitoring of road emergencies. BRIEF DESCRIPTION OF THE DRAWINGS
[0136] The present invention is further illustrated by means of the non-limiting examples given in the accompanying drawings.
[0137] Figure 1 A schematic diagram of the route planning process in step 200 of the present invention;
[0138] Figure 2 Schematic diagram of the process of identifying the road condition in step 600 of the present invention. DETAILED DESCRIPTION
[0139] In order to enable those skilled in the art to better understand the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0140] Example 1: Figure 1-2 As shown, a highway disaster monitoring and early warning method of the present invention,
[0141] Here are the steps:
[0142] Camera calibration: calibrate the camera on the fixed-wing drone to obtain internal and external parameters to ensure the accuracy of image data;
[0143] Route planning: by setting several road checkpoints on the console, the drone automatically plans the path and flies autonomously, traversing all checkpoints;
[0144] Data collection and data transmission: During the flight, the drone uses the camera on board to collect images of the ground and surrounding environment, and transmits the collected images back to the ground station through the wireless communication module for subsequent processing;
[0145] 3D reconstruction: The ground station reconstructs the road surface and surrounding environment along the highway in 3D based on the images collected by the fixed-wing drone and saves the data;
[0146] Data comparison and disaster warning: After the ground station completes the 3D reconstruction, it compares the obtained data with the 3D reconstruction data established during the previous inspection to identify potential disasters;
[0147] Road condition identification: In addition to identifying potential disasters, the ground station also monitors road conditions in real time to identify unexpected situations such as disasters or traffic accidents that have already occurred;
[0148] Information transmission and response measures: The ground station transmits early warning information of potential disasters, alarm information of unexpected situations, and corresponding image data to the control console, and relevant personnel respond quickly according to the situation.
[0149] Step 100: Camera calibration. Since the present invention obtains the required information from the image data, it is necessary to know where the points in the image correspond to in the real three-dimensional world. In order to determine the correspondence between the pixel points in the image and the three-dimensional world coordinates, it is necessary to establish an imaging model of the camera. In order to establish the imaging model, the camera needs to be calibrated.
[0150] Key concepts include:
[0151] (1) Intrinsic parameters: These are the internal parameters of the camera, which are the inherent characteristics of the camera lens, including focal length, pixel size, and the coordinates of the principal point (center of the image). These parameters determine how the camera maps the three-dimensional world onto a two-dimensional image.
[0152] (2) External parameters: refers to the position and orientation of the camera in space, including the rotation matrix and translation vector. These parameters describe the spatial relationship of the camera relative to a given reference coordinate system;
[0153] (3) World coordinate system: a three-dimensional coordinate system that describes the position of an object in the three-dimensional world;
[0154] (4) Camera coordinate system: a three-dimensional coordinate system that describes the position of an object relative to the camera;
[0155] (5) Image coordinate system: a two-dimensional coordinate system that describes the position of the object image on the camera imaging plane;
[0156] (6) Pixel coordinate system: a two-dimensional coordinate system that describes the position of image points on a digital image.
[0157] Model building:
[0158] (1) Rigid body transformation: the transformation from the world coordinate system to the camera coordinate system. The coordinates of the object in the world coordinate system (x w ,y w , z w ) and the coordinates in the camera coordinate system (x c ,y c , z c ) satisfies the following formula:
[0159]
[0160] (2) Perspective projection: the transformation from the camera coordinate system to the image coordinate system. The coordinates of the object in the camera coordinate system (x c ,y c , z c ) and the coordinates of the corresponding object image in the image coordinate system (x p ,y p ) satisfies the following formula:
[0161]
[0162] (3) Distortion model: Since the object is distorted when it is imaged through a lens, a distortion model needs to be introduced. The image coordinates before distortion removal are (x, y), and the image coordinates after distortion removal are (x crt ,y crt ). The distance r from a point (x, y) in the image to the center of the image satisfies the following formula:
[0163]
[0164] Distortion includes radial distortion and tangential distortion. The coordinates after distortion elimination (x crt ,y crt ) satisfies the following formula:
[0165]
[0166] Among them, x rcrt and rcrt is the radial distortion term, satisfying the following formula:
[0167]
[0168] x tcrt and tcrt is the tangential distortion term, satisfying the following formula:
[0169]
[0170] Among them, k1, k2, k3, p1, and p2 are all distortion parameters.
[0171] (4) Secondary transformation: that is, the transformation from the image coordinate system to the pixel coordinate system. The coordinates of the object in the image coordinate system (x p ,y p ) and the coordinates (u, v) of the corresponding pixel in the digital image satisfy the following formula:
[0172]
[0173] Among them, (u0, v0) is the coordinate of the image center in the pixel coordinate system.
[0174] In summary, we have established the imaging model of the camera. The transformation from the pixel coordinate system to the world coordinate system satisfies the following formula:
[0175]
[0176] in, is the camera’s intrinsic parameter matrix, is the extrinsic parameter matrix of the camera.
[0177] In order to obtain the camera's intrinsic matrix, extrinsic matrix, and distortion parameters, it is necessary to use the camera to shoot the calibration object and find a large number of points (x w ,y w , z w ) and its corresponding point (u, v), and solve the unknown parameters in the model. Through camera calibration, we can obtain an accurate imaging model that can map each pixel in the image back to the corresponding 3D world coordinates. This is crucial for subsequent image processing tasks such as 3D reconstruction, terrain change detection, and disaster warning, because they all require accurate depth information and geometric relationships.
[0178] Step 200: Route planning.
[0179] In this step, the supervisor sets several road checkpoints through the console, expressed in the form of longitude and latitude coordinates. The drone automatically plans the path and flies autonomously based on the location coordinates of the next checkpoint, traversing all checkpoints to collect the required image data.
[0180] Path planning is the core part of the drone navigation system, which is responsible for generating the optimal path from the starting point to the end point. In the present invention, the controller sets several road checkpoints on the console, and the drone automatically plans the path and flies autonomously based on the checkpoints, and finally traverses all the checkpoints;
[0181] In terms of path planning, we use the JPS algorithm, which is a path planning algorithm suitable for quickly calculating the optimal path. It is optimized based on the A* algorithm to more efficiently find the optimal path.
[0182] The core idea of the JPS algorithm is to maintain a dynamic open list that contains all known nodes with the lowest cost. The algorithm searches and identifies forced neighbors and jump points, and uses the heuristic function h(n) to estimate the cost from the current node n to the target point, so as to find the path with the lowest cost and the fastest speed more efficiently.
[0183] Key concepts include:
[0184] 1.openList: A collection of valid points needs to be saved during the pathfinding process. This list stores the points that can be explored.
[0185] 2.closeList: A collection of valid points needs to be saved during the pathfinding process. This list stores the points that have been explored.
[0186] 3.g(n): represents the cost from the starting node of the current record to node n, also known as the g value of node n.
[0187] 4.h(n): represents the estimated cost from the currently recorded node n to the target point, also known as the h value of node n.
[0188] 5.f(n): represents the sum of the cost from the starting node to node n and from node n to the end point of the current record, also known as the f value of node n, that is:
[0189] f(n)=g(n)+h(n)
[0190] 6. Forced neighbor: If there is an obstacle among the adjacent nodes of node n, and the distance cost of node n's parent node p to reach node m through node n is smaller than the distance cost of any path to node m without passing through node n, then node m is called a forced neighbor of node n.
[0191] 7. Jump point:
[0192] If node n is a starting point or a destination point, then n is a hop point.
[0193] If node n has neighbors and they are forced neighbors, then n is a hop point.
[0194] If the parent node to node n is a diagonal movement and node n can reach the jump point by moving horizontally or vertically, then n is a jump point.
[0195] The algorithm steps are as follows:
[0196] (1) Initialization: Create openList to store explorable points and closeList to store explored points to prevent repeated searches. Grid the three-dimensional space and treat each grid as a node. Mark the obstacles in the grid. Determine the starting node and the target node, and add the starting node to the openList.
[0197] (2) Define the node evaluation function f(n) = g(n) + h(n), and use Manhattan distance to calculate the cost from the current node to the target node and the starting node. The Manhattan distance calculation formula is as follows:
[0198] d(i, j) = |x i -x j |-|y i -y j |
[0199] Where i and j are two target points, and d(i, j) represents the Manhattan distance between the two points.
[0200] (3) Starting from the starting node, search other nodes, first in a straight line, then in an oblique direction. If an obstacle, jump point or boundary is found in a certain direction, the search ends. If a jump point is found, it is added to the openList.
[0201] (4) If the search in the diagonal direction is not completed in a single round, move forward in the diagonal direction and repeat step 3 until the search is completed. If all directions starting from this node have been searched, remove this node from the openList and add it to the closeList. The specific method is as follows:
[0202] ① Set the g value of all other nodes in the graph to infinity
[0203] ②Loop (until the queue is empty or the target node is found), if the queue is originally empty, return false
[0204] ③ For all unexpanded neighbors of node n, first search in all straight line directions. If the boundary is touched, stop exploring in this direction; if the jump point is touched, mark the touched jump point as node m and stop searching in this direction.
[0205] ④ Then, perform a single-step search in all oblique directions. If the boundary is touched, stop the exploration in this direction; if the jump point is touched, mark the touched jump point as node m and stop the search in this direction. If the boundary or jump point is not touched, search in the horizontal decomposition direction of the current direction of travel. If the boundary is touched, stop the exploration in this direction; if the jump point is touched, mark the touched jump point as node m and stop the search in this direction.
[0206] (If n has a parent node, the search direction only needs to be the direction of the line connecting the parent node and n, the horizontal decomposition direction, and the direction of the line connecting n and the mandatory neighbor)
[0207] ⑤If the g value of node m is infinite, it means that node m is not expanded, let g(m)=g(n)+cost nm , where cost n Indicates the distance cost between node n and node m. Add node m to openList, delete node n from openList and add it to closeList
[0208] ⑥If g(m)>g(n)+cost nm , then let g(m)=g(n)+cost nm , do not add it to the openList, and delete n from the openList and add it to the closeList. This operation greatly reduces the number of nodes in the openlist of the JPS algorithm, greatly improving the efficiency of the algorithm.
[0209] ⑦End the loop.
[0210] (5) Take the node with the lowest cost in openList, that is, the node with the smallest f value, as the new parent node, search all straight and diagonal directions, and repeat steps 300 and 400. Until openList is empty or the target node is found. If openList is empty and the target node has not been found, return false. If the target node is found, connect all the points in closeList to get the desired path.
[0211] After obtaining the desired path, the drone will fly along the desired path by adjusting the input of the flight control system.
[0212] Step 300: Data collection and data transmission
[0213] When flying along the planned path, the drone uses its oblique camera to collect images of the ground and surrounding environment at each checkpoint.
[0214] Oblique photography is a 3D modeling method that combines drones and photogrammetry. It uses the oblique photography system carried by drones to shoot from different angles, directions, and multiple perspectives to obtain a large amount of image data. After these data are processed by image processing, registration, splicing and other technologies, they are finally used to generate a 3D model. By applying oblique photography technology, a large amount of image data can be quickly collected, and the texture information of the objects can be obtained from multiple angles, which improves the accuracy and authenticity of modeling.
[0215] After each frame of image is acquired, the drone transmits the image data to the console through the wireless transmission module it carries for further processing and identification.
[0216] Step 400: 3D reconstruction
[0217] The control console uses the image data collected by the drone through oblique photography technology to perform a three-dimensional reconstruction of the highway and surrounding terrain at the checkpoint. The specific steps include:
[0218] (1) Image preprocessing.
[0219] In the image preprocessing stage, bilateral filtering is used to denoise the image. Bilateral filtering is a nonlinear filtering technology that combines the spatial domain information and pixel intensity domain information of the image to reduce image noise while retaining edge information. Its expression is as follows:
[0220]
[0221] Among them, (k, l) is the coordinate of the center point of the template window, (i, j) is any point in the neighborhood of point (k, l), and ω(i, j, k, l) is the weighting coefficient of the filter, which satisfies the following formula:
[0222] ω(i,j,k,l)=d(i,j,k,l)*r(i,j,k,l)
[0223] Among them, d(i, j, k, l) and r(i, j, k, l) are the spatial domain kernel and range kernel of the filter respectively, satisfying the following formula:
[0224]
[0225] Among them, σ d and σ r are the standard deviations of the Gaussian functions.
[0226] (2) Feature extraction
[0227] The SIFT algorithm is used to extract features from images. The SIFT (Scale-Invariant Feature Transform) algorithm is a widely used computer vision algorithm for extracting key points from images and generating feature descriptors. Its key feature is that it is very robust to image scaling, rotation, brightness changes, etc. The key steps of the SIFT algorithm are as follows:
[0228] 1. Scale space extremum detection. Key points are detected by constructing a Gaussian scale space. By applying Gaussian filters of different scales to the image, a series of images at different scales are obtained. The algorithm then detects local extreme points in the scale space. This involves comparing each pixel with other points in its scale neighborhood, including different locations at the same scale and corresponding points at upper and lower scales.
[0229] 2. Accurate positioning of key points. Once the extreme points are detected, the algorithm further refines the position and scale of the key points. This is achieved by fitting a three-dimensional quadratic function to eliminate the noise of edge responses and improve the accuracy of key point positioning.
[0230] 3. Key point main direction assignment. In order to make the feature descriptor invariant to rotation, the SIFT algorithm assigns a main direction to each key point. This is achieved based on the gradient direction histogram in the key point neighborhood. The gradient direction is quantized into an interval from 0 to 360 degrees.
[0231] 4. Keypoint descriptor generation. The descriptor is generated based on the local image features around the keypoint. The keypoint neighborhood is divided into 16 small sub-regions, and the magnitude and direction histograms of the gradient are calculated for each sub-region. These histograms are normalized and merged to form a 128-dimensional feature vector (descriptor) that is invariant to the scale, rotation, and brightness changes of the image.
[0232] (3) Image matching.
[0233] After feature extraction, image matching is required to find the same feature points in different images, which includes the following steps:
[0234] 1. Feature point matching. For the feature points in the two images, feature descriptors are used for matching, which is done by calculating the distance between different feature point descriptors. At the same time, the nearest neighbor distance ratio (Ratio Test) is used to improve the accuracy of matching. For each feature point, find its two nearest matching points. If the ratio of the nearest distance to the second nearest distance is less than the set threshold, it is considered a good match.
[0235] 2. Geometric constraint verification. Since feature point matching may produce false matches, the Random Sample Consensus (RANSAC) algorithm is needed to filter these errors. The RANSAC algorithm iteratively selects matching point pairs and calculates the homography matrix and fundamental matrix between them. It then classifies all matching points according to these geometric relationships and treats point pairs that do not meet the geometric constraints as false matches and removes them.
[0236] 3. Matching point pair screening. After geometric constraint verification, in order to further screen the matching point pairs, the matching points that do not meet specific conditions are removed by setting a threshold to ensure the quality and quantity of the matching point pairs.
[0237] 4. Optimize matching point pairs. Use Bundle Adjustment (BA method) to minimize the reprojection error and optimize the matching point pairs to improve the matching accuracy.
[0238] (4) Three-dimensional point cloud generation.
[0239] After finding the correspondence between the matching point pairs, the three-dimensional coordinates of these points are calculated by triangulation. Triangulation is based on the observation of the same object from two or more viewpoints, and the distance of the object is calculated by measuring the disparity (i.e. the difference in the position of the same object at two different viewpoints).
[0240] Through triangulation, the three-dimensional coordinates of each point in the scene can be obtained, and these coordinates form a point cloud model. A point cloud is a data set composed of a large number of discrete points, each of which represents a position in the scene and contains the coordinates and color information of the position.
[0241] (5) Point cloud processing.
[0242] After generating the point cloud model, the point cloud data needs to be further processed, which includes the following steps:
[0243] 1. Point cloud denoising. Point cloud data often contains noise and outliers, which may be caused by errors in the measurement equipment, environmental factors, or errors in the data processing process. Mean filtering is a commonly used denoising method. Its basic principle is to select a point in the point cloud, then calculate the average position of all points in its neighborhood, and use this average position to replace the position of the original point, thereby smoothing out the noise. The basic steps of mean filtering are as follows:
[0244] ① Determine the neighborhood: For each point in the point cloud, determine a neighborhood range. The size of the neighborhood depends on the density and noise level of the point cloud. Generally speaking, the neighborhood should be large enough to contain multiple points, but not so large that it contains too many irrelevant points.
[0245] ②Calculate the average position: For each point, calculate the average coordinates of all points in its neighborhood. Suppose there are N points in the neighborhood (x1, y1, z1), (x2, y2, z2), ..., (x N ,y N , z N ), then the average position It can be calculated by the following formula:
[0246]
[0247] ③Replace the original position: Replace the position of the original point with the calculated average position. If the number of points in the neighborhood is less than a certain threshold, the point can be considered an isolated point or noise, and you can choose to delete or keep it.
[0248] ④Iterative processing: Repeat the above steps until the positions of all points have been processed by mean filtering, or the preset number of iterations is reached.
[0249] 2. Point cloud segmentation. Point cloud segmentation is the process of dividing a point cloud dataset into different regions or objects, each of which represents an independent entity.
[0250] Step 500: Data comparison and disaster warning. After the ground station completes the 3D reconstruction, it saves the point cloud model of the current batch of reconstruction and performs a time series analysis with the point cloud model of the previous batch of reconstruction at the same location to identify potential geological disasters. The following steps are included:
[0251] (1) Point cloud registration. Point cloud registration is the process of converting two or more point cloud data sets to the same coordinate system so that they can be compared and analyzed in the same reference frame. Point cloud registration is a key step in applications such as 3D reconstruction, model merging, and data fusion. The basic principle of point cloud registration is to solve the transformation matrix T, which transforms a point cloud model P into the coordinate system of another point cloud model Q, that is, P′=T·P. The transformation matrix T usually contains rotation and translation parts, which can be expressed as:
[0252]
[0253] Where R is the rotation matrix used to rotate the point cloud P to align it with Q; t is the translation vector used to translate the point cloud P to align it with Q. Point cloud registration includes the following steps:
[0254] 1. Feature extraction: extract features from each point cloud model, including points, edges, surfaces, etc., to provide reference points in the registration process.
[0255] 2. Feature matching: calculate the Euclidean distance between the descriptors of each feature point in the source point cloud model P and all feature points in the target point cloud model Q, and select the points with the closest distance as matching point pairs.
[0256] 3. Matching result optimization: Use the RANSAC algorithm to optimize the matching results to eliminate false matches that do not meet the geometric constraints.
[0257] 4. Use the optimized matching point pairs to calculate the transformation matrix T
[0258] 5. Point cloud conversion, apply the transformation matrix T to the source point cloud model P to obtain the converted point cloud model P′.
[0259] (2) Difference calculation. After registering the point cloud models reconstructed at the same location at different times, calculate the difference value. Specifically, the difference is calculated for each pair of matched feature points. The calculation formula is as follows:
[0260] Δx=x1-x2
[0261] Δy=y1-y2
[0262] Δz=z1-z
[0263] Among them, (x1, y1, z1) and (x2, y2, z2) are the coordinates of the same pair of matching feature points in different point cloud models.
[0264] (3) Terrain change detection. After calculating the difference values between the matching feature point pairs, statistical analysis is performed. If the difference values in a certain area are generally greater than the set threshold, it is judged that the terrain in that area has changed significantly.
[0265] (4) Disaster warning: When potential geological disasters are identified, the analysis results are compiled into a report, including information such as the location, size, and trend of terrain changes.
[0266] Step 600: Road surface condition identification
[0267] In addition to identifying potential geological disasters from terrain changes, the present invention will also identify the collected raw image data to find accidents such as geological disasters and car accidents that have occurred in the area where the flight passes. In the present invention, we use the YOLOv5 algorithm to identify accidents such as geological disasters and car accidents. This is an algorithm suitable for high-precision and fast target detection. It is optimized on the basis of the original YOLO algorithm, improving the accuracy and speed of recognition to achieve better applications.
[0268] The core idea of the YOLO algorithm is to use the whole image as input and regard the target detection task (in this invention, it is the real-time picture of the drone operation scene) as a single regression problem. It detects the target category, position and confidence through single-stage detection, grid division, multi-scale feature fusion, loss function design and other technical means.
[0269] YOLOv5 is the fifth version of the YOLO (You Only Look Once) target detection model series. It improves the detection accuracy and speed based on the previous four versions of the algorithm. First, YOLOv5 adopts an improved network structure, which improves the network representation and learning capabilities by introducing deeper network layers and more convolution modules. Secondly, YOLOv5 uses feature fusion technology to fuse feature maps of different levels and scales, greatly improving the accuracy of target detection. In addition, YOLOv5 uses Mosaic data enhancement at the input end, referring to the CutMix data enhancement method. Mosaic data enhancement increases the number of images from the original two to four for splicing, and randomly scales, randomly crops, and randomly arranges the images. Using data enhancement can improve the problem of imbalanced data among small, medium, and large targets in the data set.
[0270] Key concepts include:
[0271] Grid cell: The input image is divided into a fixed number of cells, and each grid cell is responsible for detecting an object in the image.
[0272] Convolutional layer: An important hierarchical structure in convolutional neural networks (CNNs), which extracts local features of the input image through convolution operations and generates feature maps for subsequent processing and analysis. The formula is as follows:
[0273]
[0274] Pooling layer: A special convolutional layer used to downsample the input feature map, reduce the number of model parameters, and improve the generalization ability of the model. It imitates the human visual system to reduce the dimension of the data and represents the image with higher-level features. The purpose of implementing pooling includes reducing information redundancy, improving the scale invariance of the model, and preventing overfitting.
[0275] Anchor box: used to predict the shape and position of the target box. The YOLO algorithm uses multiple anchor boxes to predict target boxes of different sizes and length ratios, which improves the accuracy of detection.
[0276] Feature extraction: Extract image features through convolutional neural networks (such as Darknet) for subsequent target prediction.
[0277] Object prediction: For each grid cell, the category of the object contained in the grid, the coordinates of the center point, and the width and height of the target box are predicted to achieve object detection.
[0278] Feature fusion: Feature maps of different levels and scales are fused together to adapt to objects of different scales and improve detection performance.
[0279] Loss function design: Design a comprehensive loss function that takes into account the accuracy of target location, confidence, and category prediction, and optimizes network parameters by minimizing the loss function. The specific loss function formula is as follows:
[0280] 8.1: Position loss formula:
[0281]
[0282] 8.2: Size loss formula:
[0283]
[0284] 8.3: Confidence loss formula:
[0285]
[0286] 8.4: Prediction category loss:
[0287]
[0288] The algorithm steps are as follows:
[0289] (1) The captured images are preprocessed and then transmitted to the neural network through the input and Mosaic data enhancement. The incoming images are preprocessed to facilitate subsequent image processing.
[0290] (2) The image passed in from the input end is sent to the network backbone. In the network backbone, the image will first pass through the Conv module. The Conv module will first perform a convolution operation on the input image, and then perform a BatchNormalization operation on the convolved image to keep the input of each layer of the neural network in the same distribution, thereby accelerating convergence to prevent overfitting and reduce the gradient disappearance of the low-level neural network during back propagation. The Conv module finally imports the image that has undergone convolution and BN operations into the SiLu activation function, whose formula is: silu(x) = x*6(x). After the Conv module ends, the data will be sent to the C3 module, which contains 3 standard convolution layers, and the number is determined by the product of the n and depth_multiple parameters in the configuration file yaml. This module is the main module for learning residual features. Its structure is divided into two branches, one uses a specified number of Bottleneck stacks, and the other only passes through a basic convolution module. Finally, the two branches are concat operated. Finally, the data is sent to the SPPF module, where a standard convolution module is used to halve the input channels, and then max pooling is performed with kernel-sizes of 5, 9, and 13 (padding is adaptive for different kernel sizes). The results of the three max pooling operations are concat- ed with the data without pooling operations, and the number of channels after the final merging is twice the original number, which greatly improves the accuracy of complex multi-target detection.
[0291] (3) Import the data transformed by the Conv module into the Neck module. YOLOv5 adopts the PANet structure in the Neck module, extracts the feature hierarchy within the network on the FPN (feature pyramid network), and introduces a bottom-up path. After the top-down feature fusion, the bottom-up feature fusion is performed again, so that the bottom-level location information can also be transmitted to the deep layer, thereby enhancing the positioning capability at multiple scales.
[0292] (4) Next, the basically processed data is put into the Head module for target detection. The Head expands the number of channels of the feature maps of different scales obtained in the Neck through 1×1 convolution. The number of feature channels after expansion is (number of categories + 5) × number of anchors on each detection layer. Among them, 5 corresponds to the horizontal coordinate, vertical coordinate, width, height and confidence of the center point of the prediction box. The confidence here represents the credibility of the prediction box. The value range is (0, 1). The larger the value, the more likely the target is to exist in the prediction box. YOLOv5 uses a matching strategy based on the aspect ratio to screen the prior box and divide it into positive samples and negative samples. After the prediction is completed, a large number of redundant bounding boxes will be generated. The Non-maximum suppression method is used to calculate and compare the IOU (intersection of Union) of different bounding boxes with the bounding box with the highest confidence. If the IOU threshold of a bounding box and the bounding box with the highest confidence is greater than the IOU threshold we set, it means that the same object is predicted by two repeated bounding boxes. Then, this bounding box is removed to reduce redundancy.
[0293] (5) Calculate the loss function. The total loss is composed of four types of losses: position loss, size loss, confidence loss, and prediction category loss. The calculation function formulas of the four losses are as follows:
[0294]
[0295] By calculating and comparing the loss function, if the loss value is less than the specified threshold, step 600 is executed; if the loss value is greater than the specified threshold, the algorithm is back-propagated, and steps 100 to 400 are repeated to optimize the parameters and improve the detection accuracy of the YOLOv5 algorithm.
[0296] (6) Output results. The identified target object (in this invention, the geological disaster, car accident or other accident that has occurred) is finally output, that is, its location information is presented in the image in the form of a bounding box, which is convenient for the subsequent UAV flight route planning.
[0297] Step 700: Information transmission and response measures. The ground station presents the potential disaster analysis and warning report generated in step 500, the alarm information of the accident that has occurred, as well as the collected original image data and the reconstructed point cloud model to the control console as a decision support, which can also be used to assist in formulating defense measures.
[0298] In the specific embodiment, step 200 ( Figure 1 )
[0299] 1. Divide the 3D space map into a grid, where each grid point represents a node. The connection between nodes is represented by an edge, and the weight w of the edge is based on the distance between nodes.
[0300] 2. Start point and target point sequence: Generate the position sequence of the start point and target point according to the previous steps, and mark these points as special nodes in the graph.
[0301] 3. Dynamic obstacle detection: Detect obstacles in the environment in real time and add obstacle nodes to the corresponding positions in the graph.
[0302] 4. Heuristic function: The present invention relates to the distance cost from the current point to the starting point and the target point in the grid map, so Manhattan distance is used as the heuristic function. Its formula is as follows:
[0303] h(n)=|x n -x g |+|y n -y g |
[0304] Among them, (x n ,y n ) is the coordinate of node n, (x g ,y g ) are the coordinates of the target point.
[0305] 5. Path search: Starting from the starting node, execute the JPS algorithm to calculate and generate the optimal path through jump points.
[0306] 6. Real-time adjustment: During the execution of the path, the control system of the drone monitors environmental changes in real time, such as the appearance of obstacles, and adjusts the motion trajectory according to the output of the JPS algorithm.
[0307] In the specific embodiment, step 600 ( Figure 2 )
[0308] The YOLO model is trained using a large amount of scene image data with accident annotations to train the YOLOv5 model suitable for identifying accidents in the area. Through supervised learning, the algorithm can identify a large number of accidents of different types and sizes.
[0309] Test the YOLOv5 model, back-propagate the neural network through the returned loss value, continuously debug and optimize the parameters in Backbone, Neck and Head, and improve the accuracy of the YOLOv5 model. This will facilitate rapid detection of emergencies in the field in subsequent applications.
[0310] Take a large number of comprehensive images of the area where the drone is flying. Make sure the images can clearly reflect the road conditions during the flight.
[0311] Import these images into the YOLOv5 algorithm, use the YOLO algorithm to identify accidents and output location information. By analyzing a large number of images, determine the location type and size of all accidents in the flight area. Record and mark their coordinates.
[0312] Based on the accident information provided by the YOLOv5 algorithm, a three-dimensional map is drawn to facilitate subsequent human repair, crowd evacuation and accident prevention.
[0313] The present invention can realize the monitoring of road emergencies by intelligent drones, and it is necessary to combine multiple technologies, such as path planning, machine recognition of emergencies, 3D model comparison, etc. to improve the monitoring and early warning effect. For example, path planning uses algorithms to plan the shortest path for drone operation, and realizes automatic intelligent path generation in different scenarios. Machine recognition of emergencies uses deep learning to realize the robot's recognition and classification of different road emergencies, thereby improving monitoring accuracy.
[0314] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed by the present invention should still be covered by the claims of the present invention.
Claims
1. A highway disaster monitoring and early warning method, It is characterized in that The steps include: Step 100: Camera calibration, installing the camera at the lower end of the fixed-wing UAV, and then calibrating the camera carried by the fixed-wing UAV to obtain internal and external parameters to ensure the accuracy of the image data; Step 200: Route planning, by setting a number of road checkpoints on the console, the drone automatically plans the path and flies autonomously, traversing all the checkpoints; Step 300: Data collection and data transmission. During the flight, the drone uses the camera it carries to collect images of the ground and the surrounding environment, and transmits the collected images back to the ground station through the wireless communication module for subsequent processing; Step 400: 3D reconstruction: the ground station performs 3D reconstruction of the road surface and surrounding environment along the expressway based on the images collected by the fixed-wing UAV and saves the data; Step 500: Data comparison and disaster warning. After the ground station completes the 3D reconstruction, the obtained data is compared with the 3D reconstruction data established during the last patrol to identify potential disasters. Step 600: Road condition identification. In addition to identifying potential disasters, the ground station also monitors road conditions in real time to identify unexpected conditions such as disasters or traffic accidents that have already occurred. Step 700: Information transmission and response measures. The ground station presents the potential disaster analysis and warning report generated in step 500, the alarm information of the accident that has occurred, as well as the collected original image data and the reconstructed point cloud model to the control console as a decision support, and can be used to assist in the formulation of defense measures, so that relevant personnel can respond quickly according to the situation.
2. A highway disaster monitoring and early warning method according to claim 1, Features: Obtain the camera's intrinsic parameters, extrinsic parameters, world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system, and establish the camera's imaging model based on the obtained content: Model building includes: Step 110: Rigid body transformation: that is, the transformation from the world coordinate system to the camera coordinate system. The coordinates of the object in the world coordinate system (x w ,y w ,z w ) and the coordinates in the camera coordinate system (x c ,y c ,z c ) satisfies the following formula: Step 120: Perspective projection: that is, the conversion from the camera coordinate system to the image coordinate system. The coordinates of the object in the camera coordinate system (x c ,y c , z c ) and the coordinates of the corresponding object image in the image coordinate system (x p ,y p ) satisfies the following formula: Step 130: Distortion model: Since the object is distorted when it is imaged through a lens, a distortion model needs to be introduced. The image coordinates before distortion removal are (x, y), and the image coordinates after distortion removal are (x crt ,y cry ); the distance r from a point (x, y) in the image to the center of the image satisfies the following formula: Distortion includes radial distortion and tangential distortion. The coordinates after distortion elimination (x crt ,y crt ) satisfies the following formula: Among them, x rcrt and rcrt is the radial distortion term, satisfying the following formula: x rcrt and rcrt is the tangential distortion term, satisfying the following formula: Among them, k 1 , k 2 , k 3 , p 1 , p 2 All are distortion parameters; Step 140: secondary conversion: conversion from the image coordinate system to the pixel coordinate system, and conversion from the pixel coordinate system to the world coordinate system; The coordinates of the object in the image coordinate system (x p ,y p ) and the coordinates (u, v) of the corresponding pixel in the digital image satisfy the following formula: Among them, (u 0 , v 0 ) is the coordinate of the image center in the pixel coordinate system; The transformation from pixel coordinate system to world coordinate system satisfies the following formula: in, is the camera’s intrinsic parameter matrix, is the extrinsic parameter matrix of the camera.
3. A highway disaster monitoring and early warning method according to claim 1, Features: The steps of automatically planning the path of the drone are as follows: Step 210: Initialization, creating openList to store explorable points and closeList to store explored points to prevent repeated searches, gridding the three-dimensional space, treating each grid as a node; marking obstacles in the grid, determining the start node and the target node, and adding the start node to the openList; Step 220: define a node evaluation function f(n)=g(n)+h(n), and use Manhattan distance to calculate the cost from the current node to the target node and the starting node; the Manhattan distance calculation formula is as follows: d(i,j)=|x i -x j |-|y i -y j | Where i and j are two target points, and d(i, j) represents the Manhattan distance between the two points; Step 230: Starting from the starting node, search other nodes, first perform a straight line search, and then perform an oblique search: if an obstacle, a jump point or a boundary is found in a certain direction, the search ends. If a jump point is found, it is added to the openList; Step 240: If the search in the oblique direction is not completed in a single round, take a step forward in the oblique direction and repeat step 230 until the search is completed. If all directions starting from this node have been searched, remove this node from the openList and add it to the closeList; Step 250: Take the node with the lowest cost in openList, that is, the node with the smallest f value, as the new parent node, search all straight and diagonal directions, and repeat steps 230 and 240 until openList is empty or the target node is found; If openList is empty and the target node has not been found, false is returned; If the target node is found, connect all the points in closeList to get the desired path.
4. A highway disaster monitoring and early warning method according to claim 3, Features: The specific method for searching nodes in step 230 is as follows: Step 231: Set the g values of all other nodes in the graph to infinity; Step 232: loop (until the queue is empty or the target node is found), if the queue is originally empty, return false; Step 234: For all unexpanded neighbors of node n, first search in all straight line directions. If the boundary is touched, stop searching in this direction. If the jump point is touched, the touched jump point is marked as node m and the search in this direction is stopped; Step 235: Perform a single-step search in all oblique directions. If the boundary is touched, stop the search in this direction. If the jump point is touched, the touched jump point is marked as node m and the search in this direction is stopped; If the boundary or jump point is not touched, the search is continued in the horizontal decomposition direction of the current direction of travel. If the boundary is touched, the exploration in this direction is stopped. If the jump point is touched, the touched jump point is marked as node m and the search in this direction is stopped; Among them: if n has a parent node, the search direction only needs to be the direction of the line connecting the parent node and n, the horizontal decomposition direction and the direction of the line connecting n and the mandatory neighbor; Step 236: If the g value of node m is infinite, it means that node m is not expanded, so let g(m) = g(n) + cost nm , where cost nm Indicates the distance cost between node n and node m, and adds node m to openList, deletes n from openList and adds it to closeList Step 237: If g(m)>g(n)+cost nm , then let g(m)=g(n)+cost nm , do not add it to openList, delete n from openList and add it to closeList; Step 237: End the loop.
5. A highway disaster monitoring and early warning method according to claim 1, Features: The specific steps of three-dimensional reconstruction in step 400 include: Step 410: Image preprocessing. In the image preprocessing stage, a bilateral filter is used to denoise the image. The expression is as follows: Among them, (k, l) is the coordinate of the center point of the template window, (i, j) is any point in the neighborhood of point (k, l), and ω(i, j, k, l) is the weighting coefficient of the filter, which satisfies the following formula: ω(i,j,k,l)=d(i,j,k,l)*r(i,j,k,l) Among them, d(i, j, k, l) and r(i, j, k, l) are the spatial domain kernel and range kernel of the filter respectively, satisfying the following formula: Among them, σ d and σ r are the standard deviations of Gaussian functions; Step 420: feature extraction, using SIFT algorithm to extract features of the image; The key steps of the SIFT algorithm are as follows: Step 421: scale space extreme value detection; Key points are detected by constructing a Gaussian scale space. Gaussian filters of different scales are applied to the image to obtain a series of images at different scales. Then, the algorithm detects local extreme points in the scale space. Step 422: Accurate positioning of key points; Once the extreme points are detected, the algorithm further refines the position and scale of the key points by fitting a three-dimensional quadratic function to eliminate the noise of edge responses and improve the accuracy of key point localization; Step 423: key point main direction assignment; It is implemented based on the gradient direction histogram in the neighborhood of the key point, and the gradient direction is quantized into an interval from 0 to 360 degrees; Step 424: generating key point descriptors; The descriptor is generated based on the local image features around the key point. The key point neighborhood is divided into 16 small sub-regions, and the magnitude and direction histogram of the gradient are calculated for each sub-region. These histograms are normalized and merged to form a 128-dimensional feature vector (descriptor), which is invariant to the scale, rotation and brightness changes of the image; Step 430: Image matching, including the following steps: Step 431: feature point matching; For the feature points in the two images, feature descriptors are used for matching, which is accomplished by calculating the distance between the descriptors of different feature points. At the same time, the nearest neighbor distance ratio (Ratio Test) is used to improve the matching accuracy. For each feature point, the two nearest matching points are found. If the ratio of the nearest distance to the second nearest distance is less than the set threshold, it is considered a good match. Step 432: Geometric constraint verification; Since feature point matching may produce mismatches, a random sample consensus (RANSAC) algorithm is needed to filter these errors. The RANSAC algorithm iteratively selects matching point pairs and calculates the homography matrix and fundamental matrix between them. It then classifies all matching points according to these geometric relationships and removes point pairs that do not meet the geometric constraints as mismatches. Step 433: screening matching point pairs; Set a threshold to remove matching points that do not meet specific conditions to ensure the quality and quantity of matching point pairs; Step 434: optimizing matching point pairs; Use Bundle Adjustment (BA method) to minimize the reprojection error and optimize the matching point pairs to improve the matching accuracy; Step 440: generating a three-dimensional point cloud; After finding the correspondence between the matching point pairs, the three-dimensional coordinates of these points are calculated by the triangulation principle; Step 450: point cloud processing: After the point cloud model is generated, the point cloud data needs to be further processed, which specifically includes the following steps: Step 451: point cloud denoising, using a mean filter method to perform denoising, the specific steps include: Step 4511: Determine a neighborhood. For each point in the point cloud, determine a neighborhood range. Step 4512: Calculate the average position: For each point, calculate the average coordinates of all points in its neighborhood; Assume that there are N points (x 1 ,y 1 , z 1 ), (x 2 ,y 2 , z 2 ),...,(x N ,y N , z N ), then the average position It can be calculated by the following formula: Step 4513: Replace the original position: replace the position of the original point with the calculated average position. If the number of points in the neighborhood is less than a certain threshold, the point can be considered an isolated point or noise, and can be deleted or retained; Step 4514: Iterative processing: repeat the above steps 4511-4513 until the positions of all points are processed by the mean filter or the preset number of iterations is reached; Step 452: Point cloud segmentation, the process of dividing the point cloud data set into different regions or objects, each region or object represents an independent entity.
6. A highway disaster monitoring and early warning method according to claim 1, Features: The step 500, data comparison and disaster warning step includes: Step 510: Point cloud registration, the process of converting two or more point cloud data sets to the same coordinate system so that they can be compared and analyzed in the same reference frame. Point cloud registration includes the following steps: Step 511: feature extraction, extracting features from each point cloud model, including points, edges, surfaces, etc., to provide reference points in the registration process; Step 512: feature matching, calculating the Euclidean distance between the descriptors of each feature point in the source point cloud model P and all feature points in the target point cloud model Q, and selecting the point with the closest distance as the matching point pair; Step 513: optimizing the matching results, using the RANSAC algorithm to optimize the matching results to eliminate erroneous matches that do not meet the geometric constraints; Step 514: Calculate the transformation matrix T using the optimized matching point pairs; Step 515: point cloud conversion, applying the transformation matrix T to the source point cloud model P to obtain a converted point cloud model P′; Step 520: Difference calculation: after registering the point cloud models reconstructed at the same location at different times, calculate the difference value. The calculation formula is as follows: Δx=x 1 -x 2 Δy=y 1 -y 2 Δz=z 1 -with Among them, (x 1 ,y 1 , z 1 ) and (x 2 ,y 2 , z 2 ) are the coordinates of the same pair of matching feature points in different point cloud models; Step 530: terrain change detection, after calculating the difference values between the matching feature point pairs, statistical analysis is performed. If the difference values of a certain area are generally greater than the set threshold, it is determined that the terrain of the area has changed significantly; Step 540: Disaster warning. When a potential geological disaster is identified, the analysis results are compiled into a report, including information such as the location, size, and trend of terrain changes.
7. A highway disaster monitoring and early warning method according to claim 1, Features: The road condition recognition method in step 600 adopts the YOLOv5 algorithm, and the specific steps include: Step 610: pre-process the captured image, and then transmit it to the neural network via the input and Mosaic data enhancement, and pre-process the incoming image to facilitate subsequent processing of the image; Step 620: Send the image passed in by the input end to the network backbone; Step 621: In the network skeleton, the image will first pass through the Conv module, and the Conv module will first perform a convolution operation on the input image; Step 622: Then, the convolved image is subjected to a Batch Normalization operation to keep the input of each layer of the neural network in the same distribution, thereby accelerating convergence and preventing overfitting, and reducing the gradient vanishing of the lower layer neural network during back propagation:; Step 623: The Conv module finally imports the image after convolution and BN operation into the SiLu activation function, whose formula is: silu(x)=x*6(x) Step 624: After the Conv module is finished, the data is sent to the C3 module, which contains 3 standard convolutional layers, the number of which is determined by the product of the n and depth_multiple parameters in the configuration file yaml; The module structure is divided into two branches, one uses a specified number of Bottleneck stacks, and the other only passes through a basic convolution module and performs a concat operation on the two branches; Step 625: The data is fed into the SPPF module, where the input channels are halved through a standard convolution module, and then max pooling is performed with kernel-sizes of 5, 9, and 13 (the padding is adaptive for different kernel sizes). Step 630: Import the data transformed by the Conv module into the Neck module for feature fusion; Step 640: Put the basically processed data into the Head module to detect the target; Step 641: After the prediction is completed, the Non-maximum suppression method is used to calculate the IOU of different bounding boxes and the bounding box with the highest confidence. If the IOU threshold between a certain bounding box and the bounding box with the highest confidence is greater than the IOU threshold we set, it means that the same object is predicted by two repeated bounding boxes, then this bounding box is removed: Step 650: Calculate the loss function. The total loss is composed of four types of losses: position loss, size loss, confidence loss, and prediction category loss. The calculation function formulas of the four losses are as follows: If the loss value is less than the specified threshold, step 660 is executed; if the loss value is greater than the specified threshold, the algorithm is back-propagated, and steps 610 to 640 are repeated to optimize the parameters; Step 660: Output the result.
Citation Information
Patent Citations
Logistics distribution vehicle adaptive motion planning method and device and storage medium
CN117537825A
Urban rail transit station fire intelligent evacuation method based on monitoring video
CN117952248A
Method for processing BIM three-dimensional model special state through unmanned aerial vehicle detection technology
CN118113061A
Cited By
Multi-feature fused variable-scale moving target positioning and tracking method and device
CN119380044A
