Water accumulation positioning method, device, equipment and medium based on unmanned aerial vehicle three-dimensional oblique photography

Abstract

A method, device, equipment and medium for locating water accumulation based on three-dimensional oblique photography of unmanned aerial vehicles. A three-dimensional virtual scene graph is constructed based on three-dimensional terrain data obtained by oblique photography of unmanned aerial vehicles, and a target fitting plane is generated by fitting its point cloud data through a random sample consistency algorithm. The uneven area set is determined according to the height difference between each point cloud and the target fitting plane; the road surface image obtained by a vehicle-mounted camera is input into a water accumulation detection model generated by training an adaptive dynamic channel pruning strategy to obtain a detection frame set, and the strategy is to calculate the square of the channel weight amplitude and normalize it to the sum of the weight squares of all retained channels in the same layer to obtain an adaptive dynamic channel pruning score and prune the channel according to its size; the uneven area set and the detection frame set are weightedly fused to obtain a fused detection area; the fused detection area with a confidence greater than a confidence threshold is taken as a water accumulation area and its coordinates are output, which can improve detection efficiency and reduce costs.

Description

Technical Field

[0001] The present application relates to the technical field of water accumulation detection, and in particular to a method, device, equipment and medium for locating water accumulation based on three-dimensional oblique photography of unmanned aerial vehicles. Background Art

[0002] In the measurement of vehicle tire-road rolling resistance and the detection of road performance parameters, water accumulation on the road surface is a key factor affecting the measurement results. Local depressions in the road surface, deviations in the transverse and longitudinal slopes, and geometric features such as rutting are key factors that lead to water accumulation on the road surface. Therefore, before carrying out the measurement work, accurate prediction and positioning of the waterlogged area is an important means to improve the measurement accuracy. The existing technology mainly realizes the detection and positioning of water accumulation on the road surface by combining road detection vehicles with manual detection methods. However, due to the need for manual participation, some roads usually need to be closed during the detection process, which not only interferes with the normal use of the road, but also leads to high consumption of manpower, material and financial resources, and the detection efficiency is low. Summary of the invention

[0003] The present application provides a method, device, equipment and medium for locating water accumulation based on three-dimensional oblique photography by unmanned aerial vehicles, which can solve the problems of low efficiency and high cost in the prior art due to the need for manual participation in water accumulation detection and positioning.

[0004] In a first aspect, an embodiment of the present application provides a method for locating accumulated water based on three-dimensional oblique photography by an unmanned aerial vehicle, comprising:

[0005] Construct a three-dimensional virtual scene map based on the three-dimensional terrain data obtained by the drone taking oblique photos along a preset route according to preset flight parameters;

[0006] Fitting the point cloud data in the three-dimensional virtual scene graph by a random sample consensus algorithm to generate a target fitting plane;

[0007] Determine a set of uneven regions according to the height difference between each point cloud and the target fitting plane;

[0008] The road surface images captured by the on-board camera along the preset route are input into the preset water accumulation detection model based on the Transformer architecture for water accumulation recognition, so as to output a set of detection frames;

[0009] Performing weighted fusion on the uneven regions in the uneven region set and the detection frames in the detection frame set to obtain at least one fused detection region;

[0010] For each fused detection area, when the confidence of the fused detection area is greater than a preset confidence threshold, the fused detection area is used as a water accumulation area and the coordinates of the fused detection area are output;

[0011] Among them, the water accumulation detection model is generated by training an adaptive dynamic channel pruning strategy, and the adaptive dynamic channel pruning strategy is: calculate the square of the channel weight amplitude, and normalize the square of the channel weight amplitude to the sum of the square weights of all retained channels in the same layer to obtain an adaptive dynamic channel pruning score, and perform channel pruning according to the size of the adaptive dynamic channel pruning score.

[0012] In combination with the first aspect, in one implementation, the calculation formula of the adaptive dynamic channel pruning score is:

[0013]

[0014] Where Adcp-score represents the adaptive dynamic channel pruning score, W is the target weight tensor in the form of a one-dimensional vector, and the weight values ​​in the target weight tensor are sorted in ascending order, (W[u]) 2 Represents the square of the u-th weight value in the target weight tensor, (W[v]) 2 represents the square of the vth weight value in the target weight tensor, θ represents the adaptive threshold, γ represents the adjustment factor used to balance the importance of the weight value, and δ represents the weight decay factor.

[0015] In combination with the first aspect, in one embodiment, the water accumulation detection model includes a backbone network, a hybrid encoder and a detection head connected in sequence, the backbone network is a fast network FasterNet including a partial convolution PConv, the hybrid encoder includes an attention-based intra-scale feature interaction module AIFI and a cross-scale feature fusion module CCFF based on a convolutional neural network, and the CCFF module includes a reparameterized C3 module RepC3 based on a reparameterized convolution RepConv.

[0016] In combination with the first aspect, in one implementation, weighted fusion of the uneven regions in the uneven region set and the detection frames in the detection frame set to obtain at least one fused detection region includes:

[0017] For the detection box set B car Each detection box B in i and the set of uneven regions C tilt Each uneven area C j , calculate the detection box B i and the flat area C j The target intersection and union ratio between

[0018] If the target IoU is greater than the preset IoU threshold, the detection box B is determined i and the uneven area C iBelong to the same cluster, where one cluster corresponds to one fusion detection area.

[0019] In combination with the first aspect, in one implementation, the calculation formula for the coordinates of the fusion detection area is:

[0020]

[0021] Where n represents the number of detection areas in the fusion detection area, and the detection area includes the detection frame and the uneven area, s i represents the confidence of the i-th detection area, and Respectively represent the upper left corner coordinates and lower right corner coordinates of the i-th detection area, and Respectively represent the upper left corner coordinates and lower right corner coordinates of the fusion detection area.

[0022] In combination with the first aspect, in one embodiment, the three-dimensional virtual scene map is constructed based on the three-dimensional terrain data obtained by the drone according to preset flight parameters and along a preset route, including:

[0023] Control the drone to perform tilt photography along a preset route according to preset flight parameters to obtain three-dimensional terrain data;

[0024] Distortion correction and contrast enhancement are performed on the three-dimensional topographic data to obtain target three-dimensional data;

[0025] Constructing a three-dimensional virtual scene graph based on the target three-dimensional data;

[0026] Wherein, the calculation formula of the contrast enhancement is:

[0027]

[0028] Where T(l) represents the pixel brightness value after enhancement, l represents the pixel brightness value before enhancement, and l max Represents the maximum value of all pixel brightness values, l min Represents the minimum value of all pixel brightness values, CDF s (l) represents the cumulative distribution function, p represents the adaptive parameter, and its value range is [0.5,1].

[0029] In combination with the first aspect, in one embodiment, in the process of fitting the point cloud data in the three-dimensional virtual scene graph through the random sample consistency algorithm, the weight of the point cloud data is adjusted through the double square weighting method so that the weight of the first point cloud is higher than the weight of the second point cloud, and the distance between the first point cloud and the fitting plane is smaller than the distance between the second point cloud and the fitting plane.

[0030] In a second aspect, the embodiment of the present application provides a water accumulation positioning device based on three-dimensional oblique photography of a drone, comprising:

[0031] A model building unit, which is used to build a three-dimensional virtual scene map based on three-dimensional terrain data obtained by oblique photography of the drone according to preset flight parameters and along a preset route;

[0032] A plane fitting unit, which is used to fit the point cloud data in the three-dimensional virtual scene graph through a random sample consistency algorithm to generate a target fitting plane;

[0033] An uneven region determination unit, which is used to determine an uneven region set according to a height difference between each point cloud and the target fitting plane;

[0034] A detection frame determination unit, which is used to input a road surface image obtained by a vehicle-mounted camera along a preset route into a preset water accumulation detection model based on a Transformer architecture to perform water accumulation recognition, so as to output a detection frame set, wherein the water accumulation detection model is generated by training an adaptive dynamic channel pruning strategy, and the adaptive dynamic channel pruning strategy is: calculating the square of the channel weight amplitude, and normalizing the square of the channel weight amplitude to the sum of the squares of the weights of all retained channels in the same layer to obtain an adaptive dynamic channel pruning score, and performing channel pruning according to the size of the adaptive dynamic channel pruning score;

[0035] A region fusion unit, configured to perform weighted fusion on the uneven regions in the uneven region set and the detection frames in the detection frame set to obtain at least one fused detection region;

[0036] The water accumulation positioning unit is used to, for each fused detection area, when the confidence of the fused detection area is greater than a preset confidence threshold, regard the fused detection area as a water accumulation area and output the coordinates of the fused detection area.

[0037] In the third aspect, an embodiment of the present application provides a water accumulation positioning device based on drone three-dimensional oblique photography, wherein the water accumulation positioning device based on drone three-dimensional oblique photography comprises a processor, a memory, and a water accumulation positioning program based on drone three-dimensional oblique photography stored in the memory and executable by the processor, wherein when the water accumulation positioning program based on drone three-dimensional oblique photography is executed by the processor, the steps of the water accumulation positioning method based on drone three-dimensional oblique photography as mentioned above are implemented.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which is stored a water accumulation positioning program based on drone three-dimensional oblique photography. When the water accumulation positioning program based on drone three-dimensional oblique photography is executed by a processor, the steps of the water accumulation positioning method based on drone three-dimensional oblique photography as described above are implemented.

[0039] The beneficial effects brought by the technical solution provided in the embodiments of the present application include:

[0040] A 3D virtual scene graph is constructed by using the 3D terrain data obtained by oblique photography of the drone according to preset flight parameters and along a preset route, and the point cloud data in the 3D virtual scene graph is fitted based on a random sample consistency algorithm to generate a target fitting plane, and then the set of uneven areas is determined according to the height difference between each point cloud and the target fitting plane; at the same time, the road surface image taken by the on-board camera along the preset route is input into the water accumulation detection model for water accumulation recognition to output a detection frame set, and then the uneven areas in the uneven area set and the detection frames in the detection frame set are weightedly fused to obtain a fused detection area, and finally the water accumulation area and the detection frame set are determined according to the confidence of the fused detection area. its coordinates; it can be seen that the present application automatically realizes accurate detection of waterlogged areas by combining image data from different angles and conditions provided by drone oblique photography and a more realistic and accurate three-dimensional environment as well as the rich details provided by the on-board camera. It does not require road closures and excessive manual participation, which not only improves efficiency but also reduces costs. In addition, an adaptive dynamic channel pruning strategy is used to prune the number of redundant channels to generate a waterlogging detection model, which improves the frame rate of the waterlogging detection model while maintaining accuracy, so as to reduce the impact of image blur caused by excessive vehicle speed or jitter on waterlogging detection, and reduce the complexity of the model, thereby speeding up the calculation speed and improving the efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of an embodiment of a method for locating accumulated water based on three-dimensional oblique photography by a drone of the present application;

[0042] Figure 2 This is a schematic diagram of the water accumulation detection model architecture involved in the embodiment of the present application;

[0043] Figure 3 This is a schematic diagram of the FasterNet module structure involved in the embodiment of the present application;

[0044] Figure 4 This is a schematic diagram of the PConv module structure involved in the embodiment of the present application;

[0045] Figure 5This is a schematic diagram of the RepC3 module structure involved in the embodiment of the present application;

[0046] Figure 6 This is a schematic diagram of the hardware structure of the water accumulation positioning device based on drone three-dimensional oblique photography involved in the embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0049] In a first aspect, an embodiment of the present application provides a method for locating accumulated water based on three-dimensional oblique photography by an unmanned aerial vehicle.

[0050] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of the method for locating water accumulation based on three-dimensional oblique photography of a drone. Figure 1 As shown, the method for locating accumulated water based on UAV three-dimensional oblique photography includes:

[0051] Step S10: constructing a three-dimensional virtual scene map based on the three-dimensional terrain data obtained by the drone performing oblique photography along a preset route according to preset flight parameters.

[0052] For example, in this embodiment, it is first necessary to control the drone to perform the task of collecting and processing oblique photography data. Among them, for the collection task, it involves pre-flight preparation, that is, planning the flight path and setting related flight parameters according to specific business needs. Specifically, the flight planning software can be used to formulate preset flight routes and preset flight parameters according to specific task requirements (such as the scope of the project area, terrain and geomorphic features, engineering accuracy requirements, etc.). The preset flight parameters include but are not limited to flight altitude, speed, direction and overlap; then, the drone-mounted sensor system performs automatic or semi-automatic data collection tasks in the specified area according to the established flight plan, that is, controlling the drone to perform oblique photography of the ground to focus on capturing image data of terrain and landforms to obtain three-dimensional terrain and geomorphic data; at the same time, the system will record key data such as the real-time position, attitude and time of the drone.

[0053] After data collection is completed, the data processing stage will begin; in which the collected three-dimensional terrain data can be subjected to image preprocessing such as distortion correction, contrast enhancement, spatial position and attitude correction to generate target three-dimensional data. It should be understood that when correcting the spatial position and attitude in the three-dimensional terrain data, the GPS (Global Positioning System) and PPK (Post-Processed Kinematic) technologies can be used to accurately determine the spatial position of the image data, and the IMU (Inertial Measurement Unit) data and image feature points can be combined for attitude correction.

[0054] It is understandable that in an urban environment, high-rise buildings may block or reflect the GPS signal, thereby affecting the positioning accuracy; therefore, this embodiment will use the PPK technology to correct the error of the GPS signal, and the position calculation formula after the PPK positioning is improved is:

[0055]

[0056] In the formula, is the position vector corrected by PPK technology, is the position vector measured directly by a standard GPS receiver, It is a differential correction provided by the PPK technology; it can be seen that this embodiment provides a higher-precision position correction through differential GPS based on the PPK technology, thereby significantly improving the positioning accuracy.

[0057] In this embodiment, real-time acceleration and angular velocity data are also obtained from the IMU data, and the image obtained by photography is detected and image feature points are tracked; wherein, the image feature points can establish a connection between images taken at different times, and the combination of IMU data and image feature points can improve the accuracy and robustness of posture estimation. Therefore, the image feature points in this embodiment can more accurately track and correct posture changes by introducing a time series transformation matrix, and the calculation formula for the improved posture correction is:

[0058]

[0059] In the formula, n represents the navigation system, b represents the body coordinate system, i represents the inertial coordinate system, m represents the time step or moment, and C b (m) n Refers to the transformation matrix from the navigation system to the body coordinate system at the mth moment, C n (m-1) n Refers to the transformation matrix from the navigation coordinate system to the body coordinate system at the m-1th moment, Refers to the transformation matrix from the inertial coordinate system to the navigation coordinate system at the m-1th moment, C b (m-1) i Refers to the transformation matrix from the body coordinate system to the inertial coordinate system at the m-1th moment.

[0060] After completing the data preprocessing to obtain the target three-dimensional data, modeling will be performed based on the target three-dimensional data to generate a high-resolution three-dimensional virtual scene graph; specifically, in this embodiment, the target three-dimensional data can be preferably organized into a json format (a lightweight data exchange format), and then the existing three-dimensional modeling tool (such as DJI Zhitu) is used to reconstruct the json file storing the target three-dimensional data (that is, the json file contains the key information required to build a three-dimensional model, such as point cloud data, camera parameters, image shooting angle, etc.) to construct a three-dimensional virtual scene graph of drone oblique photography. Among them, the three-dimensional modeling tool will perform modeling by parsing the point cloud information in the file. It can be understood that the point cloud is a data set composed of a large number of points, each point contains its position information in three-dimensional space, and these points together constitute the geometric shape of the scene.

[0061] It should be noted that the UAV oblique photogrammetry technology uses the WGS-84 global reference coordinate system, while the Chinese highway reconstruction and expansion project uses the CGCS2000 national coordinate system; and in the route design stage, the Gauss-Kruger projection is used as the projection coordinate system. Therefore, in order to effectively integrate the UAV oblique photogrammetry data, the WGS-84 coordinate system needs to be converted to the CGCS2000 coordinate system, and this conversion involves two geographic reference frames (ITRF2008 and ITRF97) and two time bases (2005.0 and 2000.0); in this embodiment, the seven-parameter model and the time change model can be applied to realize the coordinate conversion, and the specific formula is as follows: X CGCS2000 =X WGS84 +ΔX+ΔX′(t-t0)+m(X WGS84 cosacosβ-Y WGS84 sinacosβ

[0062] +Z WGS84 sinβ)+m′&(t-t0)(X WGS84 cosαcosβ-Y WGS84 sinαcosβ

[0063] +Z WGS84 sinβ)

[0064] Y CGCS2000 =Y WGS84 +ΔY+ΔY′(t-t0)+m(X WGS84 sinαcosβ+YWGS84 cosαcosβ

[0065] -Z WGS84 sinα)+m′(t-t0)(X WGS84 sinαcosβ+Y WGS84 cosαcosβ

[0066] -Z WGS84 sinα)

[0067] Z CGCS2000 =Z WGS84 +ΔZ+ΔZ′(t-t0)+m(-X WGS84 sinβ+Z WGS84 cosβ)

[0068] +m′(t-t0)(-X WGS84 sinβ+Z WGS84 cosβ)

[0069] Where, X WGS84 , Y WGS84 , Z WGS84 They represent the coordinate values ​​of the WGS-84 global reference coordinate system, X CGCS2000 , Y CGCS2000 , Z CGCS2000 They respectively represent the coordinate values ​​of the CGCS2000 national coordinate system, ΔX, ΔY, ΔZ respectively represent the translation parameters in three directions, ΔX′, ΔY′, ΔZ′ respectively represent the translation rates in three directions, m represents the scale parameter, m′ represents the scale rate, α and β represent the rotation angles around the X-axis and Y-axis respectively, t represents the current epoch, and t0 represents the reference epoch.

[0070] Therefore, after completing the construction of the three-dimensional virtual scene graph, this embodiment will convert its coordinate system from WGS-84 to CGCS2000 coordinate system and apply it to subsequent water accumulation detection.

[0071] Furthermore, in one embodiment, the three-dimensional virtual scene map is constructed based on the three-dimensional terrain data obtained by the drone according to preset flight parameters and along a preset route, including:

[0072] Control the drone to perform tilt photography along a preset route according to preset flight parameters to obtain three-dimensional terrain data;

[0073] Distortion correction and contrast enhancement are performed on the three-dimensional topographic data to obtain target three-dimensional data;

[0074] Constructing a three-dimensional virtual scene graph based on the target three-dimensional data;

[0075] Wherein, the calculation formula of the contrast enhancement is:

[0076]

[0077] Where T(l) represents the pixel brightness value after enhancement, l represents the pixel brightness value before enhancement, and l max Represents the maximum value of all pixel brightness values, l min Represents the minimum value of all pixel brightness values, CDF s (l) represents the cumulative distribution function, P represents the adaptive parameter, and its value range is [0.5,1].

[0078] For example, in this embodiment, after controlling the drone to perform tilt photography along a preset route according to preset flight parameters to obtain three-dimensional terrain data, when preprocessing the three-dimensional terrain data to generate target three-dimensional data, for distortion correction, more distortion coefficients can be introduced to more finely adjust the effect of distortion correction to improve image quality, wherein the calculation formula for the improved distortion correction (radial distortion) in this embodiment is:

[0079]

[0080] In the formula, (x dr ,y dr ) represents the corrected pixel coordinates, and r represents the distance from the pixel to the center of the image, which can be obtained by calculating the distance between the pixel coordinates (x, y) and the center of the image, that is, r 2 =x 2 +y 2 ; k1, k2 and k3 are distortion coefficients, and Represent higher-order radial distortions, which are related to r 2 is proportional to the distortion correction effect, and is used to adjust the distortion correction effect more finely; The term can more accurately correct high-order radial distortion, especially at the edges of the image, where this high-order distortion is more obvious.

[0081] For contrast enhancement, the contrast can be adjusted according to the brightness distribution of different regions by introducing the cumulative distribution function (CDF) and the adaptive parameter P, thereby improving the overall visual effect of the image; wherein, the calculation formula for the improved contrast enhancement in this embodiment is:

[0082]

[0083] Where T(l) represents the pixel brightness value after enhancement, l represents the pixel brightness value before enhancement, and l max and l minRepresents the maximum and minimum values ​​of all pixel brightness values, so that the converted brightness value T(l) can be adjusted within this range; CDF s (l) represents the cumulative distribution function, and P represents the adaptive parameter, whose value range is [0.5, 1]. It can be seen that this embodiment can dynamically adjust the contrast according to the local features of the image through adaptive contrast enhancement, so that the local details of the image are clearer, thereby making the three-dimensional virtual scene graph constructed based on the target three-dimensional data clearer and more accurate.

[0084] Step S20: fitting the point cloud data in the three-dimensional virtual scene graph by using a random sample consensus algorithm to generate a target fitting plane.

[0085] Exemplarily, in this embodiment, the random sample consensus algorithm RANSAC can be preferably used to estimate an optimal fitting plane from the point cloud data in the three-dimensional virtual scene graph to generate a target fitting plane. It should be noted that the method and principle of plane fitting by the RANSAC algorithm are common knowledge in the art, so for the sake of brevity of description, it will not be repeated here.

[0086] Furthermore, in one embodiment, in the process of fitting the point cloud data in the three-dimensional virtual scene graph by the random sample consistency algorithm, the weight of the point cloud data is adjusted by the double square weighting method so that the weight of the first point cloud is higher than the weight of the second point cloud, and the distance between the first point cloud and the fitting plane is smaller than the distance between the second point cloud and the fitting plane.

[0087] For example, in this embodiment, based on the RANSAC algorithm, bi-square weighting and Huber function are introduced to improve the robustness of quadratic surface fitting, so as to more accurately perform road point cloud modeling.

[0088] Among them, the robust least squares approximation formula corresponding to the double square weighted method is:

[0089]

[0090] Where f(w) is a function of weight w1, which combines the residual u1 and robustness parameter M1 in the RANSAC algorithm, σ represents the model parameter, σ T represents the transpose of the parameter vector and the eigenvector x i The dot product of i represents the feature vector of the i-th data point, y i Represents the i-th data point x iRelated observations. The f(w) function is designed to use square terms for accurate fitting when the residual is small (that is, when the point cloud is close to the fitting plane). In other words, a higher weight is given to point clouds that are closer to the fitting plane; and when the residual is large (that is, the point cloud is far from the fitting plane, that is, the point cloud may be an outlier), a linear term is used to reduce its influence. In other words, a lower weight is given to point clouds that are farther from the fitting plane. It should be noted that the extreme points of f(w) can help determine the weight w1 of each point cloud data, so as to adjust the model during the iteration process to better fit the data while reducing the impact of outliers; among them, when the residual |u1|<M1, the function f(w) is monotonically increasing, and the extreme value is obtained at w1=0; when |u1|>M1, the extreme value is at w1 * The formula for the extreme value is:

[0091] In addition, this embodiment will also use the Huber function to replace the traditional square loss function to improve the robustness to outliers; wherein the Huber function formula is:

[0092]

[0093] It can be understood that the Huber function helps to use square loss when the residual is small to maintain the smoothness of the model, and to use linear loss when the residual is large to reduce the impact of outliers on the model.

[0094] Step S30: determining a set of uneven regions according to the height differences between each point cloud and the target fitting plane.

[0095] Exemplarily, in this embodiment, the best target fitting plane estimated by the optimized RANSAC algorithm is used as a benchmark to evaluate the height changes of each point in the point cloud data, that is, the possible pothole area is identified by calculating the height difference between each point cloud in the three-dimensional virtual scene graph and the target fitting plane, that is, a height difference threshold is set, and the point cloud exceeding the threshold will be considered to be part of the uneven road surface area.

[0096] Specifically, a ray can be defined as a series of points extending from the laser radar's emission point along a certain direction (usually horizontal), which can be regarded as projections on the target fitting plane; it should be understood that on a ray, adjacent points refer to two points that are adjacent in space and have the smallest distance in the ray direction; therefore, the horizontal distance (point_distance) can be obtained by calculating the projection distance of these adjacent points in the ray direction. For each point on the ray, the slope difference threshold (height_threshold) can be calculated based on the local maximum slope (local_max_slope) and the horizontal distance (point_distance); the local maximum slope can be determined by calculating the ratio of the height difference between two adjacent points on the ray to the horizontal distance:

[0097]

[0098] Where (a1, b1, c1) and (a2, b2, c2) are the coordinates of two adjacent points on the ray.

[0099] It can be understood that the slope difference threshold is used to determine whether the height change between the current point and its neighboring points is within a normal range, that is, whether it is consistent with the local slope of the target fitting plane.

[0100] Then, the maximum slope of the entire ground (general_max_slope) is calculated based on the target fitting plane. The maximum slope of the entire ground can be determined by traversing all points in the point cloud to calculate the distance from each point to the target fitting plane to find the maximum value of these distances:

[0101]

[0102] In the formula, (a i ,b i ,c i ) is the i-th point in the point cloud, A, B, C and D are the parameters of the target fitting plane, that is, the coefficients in the plane equation Aa+Bb+Cc+D=0.

[0103] The height difference threshold (general_height_threshold) of the entire ground is determined according to the maximum slope of the ground. It is used to evaluate whether the height change between any two points in the point cloud data exceeds the natural slope of the ground, thereby helping to identify those points that have significant height differences from the target fitting plane. Specifically, for each point on the ray, if its z coordinate (height) is within the target range of the z coordinate of the previous point plus height_threshold and minus height_threshold, it means that the height difference between this point and the target fitting plane is within an acceptable range, then this point is considered to be a ground point; and if the z coordinate of a point exceeds the target range or exceeds the general_height_threshold threshold, then this point is considered to be part of the uneven road surface area, indicating that it has a large height difference with the target fitting plane, and is therefore identified as an uneven road surface area. Based on this, after obtaining all uneven areas, a set of uneven areas can be obtained.

[0104] Step S40: input the road surface image captured by the on-board camera along the preset route into the preset water accumulation detection model based on the Transformer architecture for water accumulation recognition, so as to output a set of detection frames; wherein the water accumulation detection model is generated by training the adaptive dynamic channel pruning strategy, and the adaptive dynamic channel pruning strategy is: calculate the square of the channel weight amplitude, and normalize the square of the channel weight amplitude to the sum of the squares of the weights of all retained channels in the same layer to obtain an adaptive dynamic channel pruning score, and perform channel pruning according to the size of the adaptive dynamic channel pruning score; the calculation formula of the adaptive dynamic channel pruning score is:

[0105]

[0106] Where Adcp-score represents the adaptive dynamic channel pruning score, W is the target weight tensor in the form of a one-dimensional vector, and the weight values ​​in the target weight tensor are sorted in ascending order, (W[u]) 2 Represents the square of the u-th weight value in the target weight tensor, (W[v]) 2 represents the square of the vth weight value in the target weight tensor, θ represents the adaptive threshold, γ represents the adjustment factor used to balance the importance of the weight value, and δ represents the weight decay factor.

[0107] For example, in this embodiment, a water accumulation detection model is constructed based on the Transformer (deep learning model based on self-attention mechanism) architecture to realize water accumulation detection of road surface images taken by vehicle-mounted cameras through the water accumulation detection model. However, since the vehicle-mounted camera is moving, its excessive speed or shaking will cause blurred images, so it is necessary to improve the frame rate of the detection model. In order to adapt to resource-constrained devices, this embodiment will prune the initial water accumulation detection model constructed based on the Transformer architecture to obtain the final water accumulation detection model.

[0108] Specifically, this embodiment will implement the pruning of the water accumulation detection model based on adaptive dynamic channel pruning (Adcp); it should be understood that in a feedforward neural network architecture with a depth of d, each layer (whether a fully connected layer or a 2D convolutional layer) is associated with a corresponding weight tensor W(i) (i=1,2,...,d); for a fully connected layer, the weight tensor is in the form of a two-dimensional matrix, while for a 2D convolutional layer, the weight tensor is in the form of a four-dimensional tensor. Therefore, in order to achieve a unified measure of the Adcp scores of the fully connected layer and the convolutional layer weight tensors, this embodiment expands all weight tensors into a one-dimensional vector form; and during the expansion process, the weight tensors are preset to be sorted in ascending order according to the weight value, that is, for any two different elements u and v in the weight tensor, if u is less than v, the corresponding weight absolute value satisfies |W[u]|≤|W[v]|, and this sorting mechanism ensures the orderliness of the weight value, laying the foundation for the subsequent calculation of the Adcp score.

[0109] Based on this, the calculation formula for the Adcp score of the target weight tensor W in the form of a one-dimensional vector at a specific element u is:

[0110]

[0111] Where Adcp-score represents the adaptive dynamic channel pruning score corresponding to the u-th weight value in the target weight tensor W, (W[u]) 2 Represents the square of the u-th weight value in the target weight tensor, (W[v]) 2 represents the square of the vth weight value in the target weight tensor, θ represents the adaptive threshold, which is dynamically adjusted according to the performance of the model on the validation set, γ represents the adjustment factor used to balance the importance of the weight value, and δ represents the weight decay factor, which is used to control the degree of weight decay; among them, the update rule of the adaptive threshold θ, the adjustment factor γ, and the weight decay factor δ can be expressed as:

[0112] θ new =θ oldf(validationperformance)

[0113] γ new =γ old g (validation performance)

[0114] δ new =δ old h (validation performance)

[0115] In the formula, f is preferably an Adadelta threshold update function for updating the adaptive threshold θ, g is preferably an Adadelta adjustment factor update function for updating the adjustment factor γ, and h is preferably an Adam weight decay factor update function for adjusting the weight decay factor δ; in this way, the calculation of the Adcp score can be adaptively adjusted according to the actual performance of the model, thereby effectively reducing the complexity of the model while maintaining the model performance, and this adaptability can help the model generalize better and make more appropriate pruning decisions at different training stages.

[0116] It should be understood that the above Adcp score calculation formula comprehensively considers the distribution of weight values ​​and their relative positions in the expanded one-dimensional vector; therefore, by calculating the Adcp score of each weight, the weight structure of the neural network can be quantitatively analyzed, which provides a theoretical basis for network optimization and performance evaluation, that is, the Adcp score evaluates the relative significance of the target channel and other retained channels in a specific neural network layer. In this framework, those channels with lower weights have been preferentially pruned; thus, even two channels with equal weights may show different Adcp scores due to different mappings, which reflects the key role of element order in the scoring mechanism:

[0117]

[0118] Therefore, after the Adcp scores are determined, global pruning is performed to remove those channels with the lowest Adcp scores to achieve the preset global sparsity requirement.

[0119] In general, the Adcp score measures the relative importance of different channels in the same layer of the neural network, and prunes the weights with the lowest Adcp score in each layer until the pruning requirements are met; among them, whether to prune and the degree of pruning can be decided based on the model's verification error or other performance indicators. The main process of Adcp score pruning is given below: the initial water accumulation detection model network built based on the Transformer architecture is trained and the weight file is initialized; then the square of the channel weight amplitude is calculated and normalized to the sum of the squares of the weights of all retained channels in the same layer (that is, the sum of the squares of all "surviving weights") to obtain the Adcp score; then according to the pre-set global sparsity requirements and the size of the Adcp score, a corresponding number of channels with smaller scores are selected for pruning, where the pruning process refers to removing the selected channels from the model and setting their weights to zero; then, the pruned model is retrained (fine-tune) to obtain the final water accumulation detection model; it should be understood that fine-tuning is to restore the performance that may be lost during the pruning process, and through such a process design, Adcp can adaptively select hierarchical sparsity to perform global pruning and ensure that at least one link is retained in each layer.

[0120] It can be seen that the adaptive dynamic channel pruning method provided in this embodiment will not change the structure of the original model, that is, the pruning process will not damage the basic framework of the model, ensuring the integrity and stability of the model, and reducing the complexity of the model by removing unnecessary details, thereby speeding up the calculation speed and improving the efficiency of data processing. Therefore, after controlling the on-board camera to capture the road image along a preset route (which can be determined according to actual needs and is not limited here), the road image is input into the water accumulation detection model trained by the adaptive dynamic channel pruning method, and the water accumulation area can be identified to output a detection frame set; wherein each detection frame in the detection frame set represents a water accumulation area.

[0121] Furthermore, in one embodiment, the water accumulation detection model includes a backbone network, a hybrid encoder and a detection head connected in sequence, the backbone network is a fast network FasterNet including a partial convolution PConv, the hybrid encoder includes an attention-based intra-scale feature interaction module AIFI and a cross-scale feature fusion module CCFF based on a convolutional neural network, and the CCFF module includes a reparameterized C3 module RepC3 based on a reparameterized convolution RepConv.

[0122] For example, in this embodiment, the water accumulation detection model is built based on the Transformer architecture, which reduces the computational burden by processing high-dimensional features and decoupling the interaction between scales to meet the needs of real-time detection, and during the training process, the IoU-aware query mechanism can be used to optimize the initial detection and loss function of the object. Figure 2 As shown in the figure, the water accumulation detection model specifically includes a backbone network, a hybrid encoder and a detection head connected in sequence; wherein, the backbone network is a fast network FasterNet including a partial convolution PConv, the hybrid encoder includes an attention-based intra-scale feature interaction module AIFI and a cross-scale feature fusion module CCFF based on a convolutional neural network, and the CCFF module includes a reparameterized C3 module RepC3 based on the reparameterized convolution RepConv.

[0123] It should be understood that the traditional backbone network often uses the ResNet18 residual neural network to extract key features in the image, but the number of convolutions is large, which affects the efficiency of information transmission. In this embodiment, the fast network FasterNet, which is faster and has less computational complexity, is preferred to replace ResNet18. In addition, although the traditional lightweight method improves efficiency and reduces resource consumption by simplifying the model, it may over-simplify the model, resulting in the loss of important information and performance degradation, and lacks flexibility and is difficult to adapt to the complexity and diversity of different data. In this embodiment, the partial convolution PConv (Partial Convolution) introduced in FasterNet is used to perform convolution processing on only certain channels of the input feature map, while the remaining channels remain unchanged, so as to reduce the computational load while ensuring that the spatial feature information is retained.

[0124] For details, see Figure 3 As shown in the figure, FasterNet includes sequentially connected partial convolution PConv, convolution with a convolution kernel of 1×1, batch normalization BN, activation function RELU and convolution with a convolution kernel of 1×1. The output of the last convolution will be residually connected to the input. Figure 4 As shown, the input of the partial convolution PConv is a network with height (h), width (w) and number of channels (c p ) feature map, after the input feature map passes through the partial convolution layer, the partial convolution PConv will only process part of the channels (c p) is convolved, that is, the PConv with a convolution kernel (Filters) of k×k performs dot multiplication on only a part of the channels, that is, it is only applied to the convolved channels, and the channels that are not convolved are directly passed to the output through the identity mapping (i.e., Identity), which means that the feature maps of these channels remain unchanged in the convolution operation; and the output of the partial convolution and the output of the identity mapping are integrated in the channel dimension to form the final output feature map; It should be noted that the feature map size of the partial convolution PConv output is the same as the input feature map, that is, the height (h), width (w) and number of channels (c p ) are the same. It can be seen that the design of partial convolution in this embodiment allows the network to perform convolution operations on some channels while maintaining the size and number of channels of the feature map, while other channels are not convolved, thereby reducing the amount of calculation and improving the computing efficiency of the network.

[0125] Therefore, see Figure 2 As shown, the backbone network in this embodiment is preferably composed of an embedding module Embedding, four sequentially connected FasterNets (i.e., FasterNet-Blocks) with faster speed and smaller computational complexity, wherein the Embedding module is used to initialize the input embedding, and input the embedding result to FasterNet-Block for feature extraction; the features output by the last three FasterNet-Blocks (i.e., S3, S4, and S5) are respectively input to the corresponding convolution modules Conv for sequential convolution Con2d, batch normalization BN, and activation function SiLU processing to obtain S3′, S4′, and S5′ features; finally, the three processing results are respectively used as inputs of the hybrid encoder. Among them, the AIFI module in the hybrid encoder only learns and extracts the deep features of S5, that is, the AIFI module uses the Transformer encoder and the highest features on the backbone network to perform same-scale feature interaction, which reduces the computational cost, and uses the self-attention mechanism to process high-level features in the image, so as to allow the model to consider other related parts of the data while processing a specific part of the data, so as to process high-level image features with rich semantic information; then the output result of the AIFI module is input into the CCFF module, so as to perform paired interactive fusion of multi-scale features through the CCFF module, and then convert them into serialized image features with rich semantic information.

[0126] It should be noted that the reparameterization C3 module RepC3 in the CCFF module is built based on the reparameterization convolution RepConv (Reparameterization Convolution); see Figure 5As shown in the figure, RepC3 forms a residual connection by adding two convolutions to the middle multi-layer RepConv module, which helps gradient propagation, speeds up training and improves model convergence; among them, through multiple consecutive RepConv modules, stronger expressive power can be achieved with fewer computing resources, and the two branches inside each RepConv module provide diversity during training, and the fused structure remains efficient during reasoning.

[0127] See also Figure 2 As shown, for the hybrid encoder, the AIFI module is first used to extract the feature S5′ to output the feature S5″; then the feature S5″ is processed by the convolution module Conv in CCFF to obtain the feature S5″′, which is then input to the upsampling module Upsample for scale enlargement and fused with the feature S4′, and then the fused and spliced ​​result is processed by the RepC3 module to output the feature S4″; similarly, the feature S4″ is processed by the convolution module Conv to obtain the feature S4″′, which is then input to the Upsample module for scale enlargement and fused with the feature S3′; then based on the RepC3 module, the feature S4″ is processed by the convolution module Conv to obtain the feature S4″′, which is then input to the Upsample module for scale enlargement and fused with the feature S3′; The fused splicing results are extracted and fused to output feature S3″; wherein, feature S3″ is output to the detection head in one way and is output to the convolution module Conv for processing in the other way, and the processing result is fused with feature S4″′, so that the RepC3 module extracts and fuses the fused splicing result again to obtain feature P1; similarly, feature P1 is output to the detection head in one way and is output to the convolution module Conv for processing in the other way, and the processing result is fused with feature S5″′, so that the RepC3 module extracts and fuses the fused splicing result again, and outputs feature P2 to the detection head, so that the detection head can perform water accumulation detection based on feature S3″, feature P1 and feature P2.

[0128] After completing the construction of the water accumulation detection module, it still needs to be trained before the final water accumulation detection model can be produced; wherein, firstly, the web crawler technology of Google Browser and the use of vehicle-mounted cameras are used for field shooting to collect image data of possible water accumulation areas such as urban roads and low-lying areas, which will cover various weather conditions and water accumulation environments of varying degrees; the collected image data are randomly allocated to the training set and the validation set in a ratio of 8:2; it should be understood that compared with many single background image sets used, the image set of this embodiment contains more complex and diverse backgrounds, which undoubtedly increases the challenge of data processing, but this complexity is extremely beneficial for enhancing the robustness of the training model and improving its generalization ability in different environments.

[0129] Before the model training phase begins, the water accumulation image dataset must be accurately labeled. This embodiment uses a labeling tool (such as LabelImg) to create labels. During the labeling process, the image can be marked according to the minimum rectangular area around the water accumulation, in order to ensure that the background area contained in the selected rectangle is as small as possible. After the labeling is completed, the data is saved as a txt file for subsequent training and model optimization.

[0130] It should be understood that in the field of deep learning, the performance of the model significantly depends on large-scale and high-quality image datasets. Given that it takes a lot of time and resources to obtain and annotate data, this embodiment will use data enhancement technology and generate diverse samples based on existing data to reduce the risk of overfitting of the model and improve its generalization performance. Among them, in order to strike a balance between sample diversity and training efficiency, this embodiment will apply a variety of data enhancement strategies including Mosaic to the images in the image dataset to effectively double the size of the dataset, such as rotation (random rotation of 90°, 180° or 270°), mirroring (left and right or up and down mirroring), random HSV enhancement, etc., which can not only enhance the diversity of the dataset, but also improve the robustness of the model for target recognition in complex scenes.

[0131] Finally, the enhanced data is used to train the waterlogging detection model, and the algorithm parameters (such as learning rate, batch size, etc.) are continuously adjusted during the training process to optimize the performance of the model. Multiple iterations of training are performed until the model reaches the expected accuracy on the test set.

[0132] Step S50: performing weighted fusion on the uneven regions in the uneven region set and the detection frames in the detection frame set to obtain at least one fused detection region.

[0133] For example, in this embodiment, the uneven area and the detection frame are combined for spatial comparison analysis to determine the spatial distribution of the waterlogged area and improve the robustness of the system under different conditions. tilt All the uneven areas in also belong to the CGCS2000 coordinate system; it can be seen that the detection box set B output by the water accumulation detection model car and the set of uneven regions C tiltBelong to the same coordinate system to ensure that the two can directly compare the spatial layer water accumulation area. This embodiment can preferably use a weighted bounding box fusion method to superimpose the detection box output by the water accumulation detection model and the uneven area corresponding to the three-dimensional oblique photography to determine all fused detection areas. It should be noted that the method and principle of weighted fusion of bounding boxes by the weighted bounding box fusion method are common knowledge in the field, so for the sake of brevity of description, they will not be repeated here.

[0134] Furthermore, in one embodiment, the performing weighted fusion on the uneven regions in the uneven region set and the detection frames in the detection frame set to obtain at least one fused detection region includes:

[0135] For the detection box set B car Each detection box B in i and the set of uneven regions C tilt Each uneven area C j , calculate the detection box B i and the flat area C f The target intersection and union ratio between

[0136] If the target IoU is greater than the preset IoU threshold, the detection box B is determined i and the uneven area C i Belong to the same cluster, where one cluster corresponds to one fusion detection area.

[0137] For example, in this embodiment, the traditional weighted bounding box fusion method is improved, that is, for set B car Each detection box and set C in tilt For each uneven area in , calculate the IoU (intersection over union) between the two:

[0138]

[0139] In the formula, B i For set B car The i-th detection box in C j It is a set C tilt The jth uneven area in Area represents the area. If IoU(B i ,C j )>IoU threshold, then B i and C jIt is understood that since there may be multiple detection boxes and multiple uneven areas belonging to the same cluster, there are often multiple detection boxes and multiple uneven areas in the fused detection area. It should be noted that for the convenience of description, the subsequent embodiments refer to the detection boxes and uneven areas in the fused detection area as detection areas; the specific value of the IoU threshold can be determined according to actual needs and is not limited here.

[0140] Step S60: for each fused detection area, when the confidence of the fused detection area is greater than a preset confidence threshold, the fused detection area is used as a water accumulation area and the coordinates of the fused detection area are output; wherein the calculation formula of the coordinates of the fused detection area is:

[0141]

[0142]

[0143] Where n represents the number of detection areas in the fusion detection area, and the detection area includes the detection frame and the uneven area, s i represents the confidence of the i-th detection area, and Respectively represent the upper left corner coordinates and lower right corner coordinates of the i-th detection area, and Respectively represent the upper left corner coordinates and lower right corner coordinates of the fusion detection area.

[0144] Exemplarily, in this embodiment, a confidence calculation is performed on each fusion detection area to determine whether it is a water accumulation area. It should be understood that since the set C tilt The uneven area C j It is calculated from the height difference between the fitted plane and the point cloud, not predicted by the model, so its credibility is relatively high. j The confidence level of set B is set to 100%; car The detection box B in i The confidence level is the confidence level output by the water accumulation detection model; therefore, for each fusion detection area, the calculation formula for its average confidence level is:

[0145]

[0146] In the formula, F s Represents the average confidence of the fused detection area, s iIt represents the confidence of the i-th detection area in the fused detection area, and n represents the total number of detection areas in the fused detection area, that is, it is equal to the sum of the total number of detection boxes in the fused detection area and the total number of inequality areas.

[0147] It should be understood that a global minimum confidence threshold min_confidence can be set according to actual needs to determine whether the fused detection area is a waterlogged area. Specifically, the average confidence F of the fused detection area is s Compared with min_confidence, if F s <min_confidence, then the fusion detection area is judged not to be a water accumulation area; and if F s ≥min_confidence, then F s As the final confidence of the fused detection area, the fused detection area is determined to be a water accumulation area, and then the coordinates of the fused detection area are calculated according to the following calculation formula:

[0148]

[0149]

[0150] In the formula, and Respectively represent the upper left corner coordinates and lower right corner coordinates of the i-th detection area, and Respectively represent the upper left corner coordinates and lower right corner coordinates of the fusion detection area.

[0151] It can be seen that the coordinates of the fusion detection area and That is the coordinates of the corresponding waterlogged area, thereby achieving accurate positioning of the waterlogged area.

[0152] In summary, this embodiment obtains three-dimensional terrain information through oblique photography by drones to provide image data and a more realistic and accurate three-dimensional environment at different angles and under different conditions, and uses the point cloud information in the three-dimensional virtual scene map created based on it to identify possible potholes; at the same time, the on-board camera is used to collect image data of the waterlogged area with richer details, and the waterlogged area detection model based on deep learning is used for identification; the detection results of the two are combined to achieve accurate identification and positioning of the waterlogged area, which not only improves the efficiency and accuracy of waterlogging detection, but also reduces the impact on the normal use of the road. In addition, this embodiment also provides an adaptive dynamic channel pruning strategy to reduce the complexity of the model to speed up the calculation speed and improve data processing efficiency.

[0153] In a second aspect, an embodiment of the present application also provides a water accumulation positioning device based on three-dimensional oblique photography of an unmanned aerial vehicle.

[0154] In one embodiment, a water accumulation positioning device based on three-dimensional oblique photography of a drone includes:

[0155] A model building unit, which is used to build a three-dimensional virtual scene map based on three-dimensional terrain data obtained by oblique photography of the drone according to preset flight parameters and along a preset route;

[0156] A plane fitting unit, which is used to fit the point cloud data in the three-dimensional virtual scene graph through a random sample consistency algorithm to generate a target fitting plane;

[0157] An uneven region determination unit, which is used to determine an uneven region set according to a height difference between each point cloud and the target fitting plane;

[0158] A detection frame determination unit, which is used to input a road surface image obtained by a vehicle-mounted camera along a preset route into a preset water accumulation detection model based on a Transformer architecture to perform water accumulation recognition, so as to output a detection frame set, wherein the water accumulation detection model is generated by training an adaptive dynamic channel pruning strategy, and the adaptive dynamic channel pruning strategy is: calculating the square of the channel weight amplitude, and normalizing the square of the channel weight amplitude to the sum of the squares of the weights of all retained channels in the same layer to obtain an adaptive dynamic channel pruning score, and performing channel pruning according to the size of the adaptive dynamic channel pruning score;

[0159] A region fusion unit, configured to perform weighted fusion on the uneven regions in the uneven region set and the detection frames in the detection frame set to obtain at least one fused detection region;

[0160] The water accumulation positioning unit is used to, for each fused detection area, when the confidence of the fused detection area is greater than a preset confidence threshold, regard the fused detection area as a water accumulation area and output the coordinates of the fused detection area.

[0161] Furthermore, in one embodiment, the calculation formula of the adaptive dynamic channel pruning score is:

[0162]

[0163] Where Adcp-score represents the adaptive dynamic channel pruning score, W is the target weight tensor in the form of a one-dimensional vector, and the weight values ​​in the target weight tensor are sorted in ascending order, (W[u]) 2 Represents the square of the u-th weight value in the target weight tensor, (W[v]) 2represents the square of the vth weight value in the target weight tensor, θ represents the adaptive threshold, γ represents the adjustment factor used to balance the importance of the weight value, and δ represents the weight decay factor.

[0164] Furthermore, in one embodiment, the water accumulation detection model includes a backbone network, a hybrid encoder and a detection head connected in sequence, the backbone network is a fast network FasterNet including a partial convolution PConv, the hybrid encoder includes an attention-based intra-scale feature interaction module AIFI and a cross-scale feature fusion module CCFF based on a convolutional neural network, and the CCFF module includes a reparameterized C3 module RepC3 based on a reparameterized convolution RepConv.

[0165] Furthermore, in one embodiment, the region fusion unit is specifically used for:

[0166] For the detection box set B car Each detection box B in i and the set of uneven regions C tilt Each uneven area C j , calculate the detection box B i and the flat area C j The target intersection and union ratio between

[0167] If the target IoU is greater than the preset IoU threshold, the detection box B is determined i and the uneven area C i Belong to the same cluster, where one cluster corresponds to one fusion detection area.

[0168] Furthermore, in one embodiment, the calculation formula for the coordinates of the fusion detection area is:

[0169]

[0170]

[0171] Where n represents the number of detection areas in the fusion detection area, and the detection area includes the detection frame and the uneven area, s i represents the confidence of the i-th detection area, and Respectively represent the upper left corner coordinates and lower right corner coordinates of the i-th detection area, and Respectively represent the upper left corner coordinates and lower right corner coordinates of the fusion detection area.

[0172] Furthermore, in one embodiment, the model building unit is specifically used to:

[0173] Control the drone to perform tilt photography along a preset route according to preset flight parameters to obtain three-dimensional terrain data;

[0174] Distortion correction and contrast enhancement are performed on the three-dimensional topographic data to obtain target three-dimensional data;

[0175] Constructing a three-dimensional virtual scene graph based on the target three-dimensional data;

[0176] Wherein, the calculation formula of the contrast enhancement is:

[0177]

[0178] Where T(l) represents the pixel brightness value after enhancement, l represents the pixel brightness value before enhancement, and l max Represents the maximum value of all pixel brightness values, l min Represents the minimum value of all pixel brightness values, CDF s (l) represents the cumulative distribution function, P represents the adaptive parameter, and its value range is [0.5,1].

[0179] Furthermore, in one embodiment, in the process of fitting the point cloud data in the three-dimensional virtual scene graph by the random sample consistency algorithm, the weight of the point cloud data is adjusted by the double square weighting method so that the weight of the first point cloud is higher than the weight of the second point cloud, and the distance between the first point cloud and the fitting plane is smaller than the distance between the second point cloud and the fitting plane.

[0180] Among them, the functional implementation of each unit in the above-mentioned water accumulation positioning device based on drone three-dimensional oblique photography corresponds to the steps in the above-mentioned water accumulation positioning method embodiment based on drone three-dimensional oblique photography, and its functions and implementation process will not be repeated here one by one.

[0181] On the third aspect, an embodiment of the present application provides a water accumulation positioning device based on drone three-dimensional oblique photography. The water accumulation positioning device based on drone three-dimensional oblique photography can be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0182] Reference Figure 6 , Figure 6 The hardware structure diagram of the water accumulation positioning device based on drone 3D oblique photography involved in the embodiment of the present application is shown in FIG. In the embodiment of the present application, the water accumulation positioning device based on drone 3D oblique photography may include a processor, a memory, a communication interface and a communication bus.

[0183] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0184] The communication interface includes input / output (I / O) interface, physical interface and logical interface, etc., which are used to realize the interconnection of devices inside the water accumulation positioning device based on UAV 3D oblique photography, and the interface used to realize the interconnection between the water accumulation positioning device based on UAV 3D oblique photography and other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display (Display), a keyboard (Keyboard), etc.

[0185] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0186] The processor may be a general-purpose processor, which may call the water accumulation positioning program based on drone 3D oblique photography stored in the memory, and execute the water accumulation positioning method based on drone 3D oblique photography provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the water accumulation positioning program based on drone 3D oblique photography is called may refer to the various embodiments of the water accumulation positioning method based on drone 3D oblique photography of the present application, which will not be repeated here.

[0187] Those skilled in the art will understand that Figure 6 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0188] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.

[0189] The readable storage medium of the present application stores a water accumulation positioning program based on drone three-dimensional oblique photography, wherein when the water accumulation positioning program based on drone three-dimensional oblique photography is executed by a processor, the steps of the water accumulation positioning method based on drone three-dimensional oblique photography as described above are implemented.

[0190] Among them, the method implemented when the water accumulation positioning program based on drone three-dimensional oblique photography is executed can refer to the various embodiments of the water accumulation positioning method based on drone three-dimensional oblique photography in this application, and will not be repeated here.

[0191] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0192] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0193] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0194] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0195] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish between different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0196] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0197] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for locating water accumulation based on three-dimensional oblique photography of unmanned aerial vehicles, characterized in that: include: Construct a three-dimensional virtual scene map based on the three-dimensional terrain data obtained by the drone taking oblique photos along a preset route according to preset flight parameters; Fitting the point cloud data in the three-dimensional virtual scene graph by a random sample consensus algorithm to generate a target fitting plane; Determine a set of uneven regions according to the height difference between each point cloud and the target fitting plane; The road surface images captured by the on-board camera along the preset route are input into the preset water accumulation detection model based on the Transformer architecture for water accumulation recognition, so as to output a set of detection frames; Performing weighted fusion on the uneven regions in the uneven region set and the detection frames in the detection frame set to obtain at least one fused detection region; For each fused detection area, when the confidence of the fused detection area is greater than a preset confidence threshold, the fused detection area is used as a water accumulation area and the coordinates of the fused detection area are output; Among them, the water accumulation detection model is generated by training an adaptive dynamic channel pruning strategy, and the adaptive dynamic channel pruning strategy is: calculate the square of the channel weight amplitude, and normalize the square of the channel weight amplitude to the sum of the square weights of all retained channels in the same layer to obtain an adaptive dynamic channel pruning score, and perform channel pruning according to the size of the adaptive dynamic channel pruning score.

2. The method for locating accumulated water based on three-dimensional oblique photography of unmanned aerial vehicles according to claim 1, characterized in that: The calculation formula of the adaptive dynamic channel pruning score is: Where Adcp-score represents the adaptive dynamic channel pruning score, W is the target weight tensor in the form of a one-dimensional vector, and the weight values ​​in the target weight tensor are sorted in ascending order, (W[u]) 2 Represents the square of the u-th weight value in the target weight tensor, (W[v]) 2 represents the square of the vth weight value in the target weight tensor, θ represents the adaptive threshold, γ represents the adjustment factor used to balance the importance of the weight value, and δ represents the weight decay factor.

3. The method for locating accumulated water based on three-dimensional oblique photography of unmanned aerial vehicle according to claim 1, characterized in that: The water accumulation detection model includes a backbone network, a hybrid encoder and a detection head connected in sequence, the backbone network is a fast network FasterNet including a partial convolution PConv, the hybrid encoder includes an attention-based intra-scale feature interaction module AIFI and a cross-scale feature fusion module CCFF based on a convolutional neural network, and the CCFF module includes a reparameterized C3 module RepC3 based on a reparameterized convolution RepConv.

4. The method for locating accumulated water based on three-dimensional oblique photography of unmanned aerial vehicle according to claim 1, characterized in that: The performing weighted fusion on the uneven regions in the uneven region set and the detection frames in the detection frame set to obtain at least one fused detection region includes: For the detection box set B car Each detection box B in i and the set of uneven regions C tilt Each uneven area C j , calculate the detection box B i and the flat area C j The target intersection and union ratio between If the target IoU is greater than the preset IoU threshold, the detection box B is determined i and the uneven area C i Belong to the same cluster, where one cluster corresponds to one fusion detection area.

5. The method for locating accumulated water based on three-dimensional oblique photography of unmanned aerial vehicle according to claim 1, characterized in that: The calculation formula of the coordinates of the fusion detection area is: Where n represents the number of detection areas in the fusion detection area, and the detection area includes the detection frame and the uneven area, s i represents the confidence of the i-th detection area, and Respectively represent the upper left corner coordinates and lower right corner coordinates of the i-th detection area, and Respectively represent the upper left corner coordinates and lower right corner coordinates of the fusion detection area.

6. The method for locating accumulated water based on three-dimensional oblique photography of unmanned aerial vehicle according to claim 1, characterized in that: The method of constructing a three-dimensional virtual scene map based on three-dimensional terrain data obtained by photographing the drone along a preset route according to preset flight parameters includes: Control the drone to perform tilt photography along a preset route according to preset flight parameters to obtain three-dimensional terrain data; Distortion correction and contrast enhancement are performed on the three-dimensional topographic data to obtain target three-dimensional data; Constructing a three-dimensional virtual scene graph based on the target three-dimensional data; Wherein, the calculation formula of the contrast enhancement is: Where T(l) represents the pixel brightness value after enhancement, l represents the pixel brightness value before enhancement, and l max Represents the maximum value of all pixel brightness values, l min Represents the minimum value of all pixel brightness values, CDF s (l) represents the cumulative distribution function, P represents the adaptive parameter, and its value range is [0.5, 1].

7. The method for locating accumulated water based on three-dimensional oblique photography of unmanned aerial vehicle according to claim 1, characterized in that: In the process of fitting the point cloud data in the three-dimensional virtual scene graph by the random sample consistency algorithm, the weight of the point cloud data is adjusted by the double square weighting method so that the weight of the first point cloud is higher than the weight of the second point cloud, and the distance between the first point cloud and the fitting plane is smaller than the distance between the second point cloud and the fitting plane.

8. A water accumulation positioning device based on three-dimensional oblique photography of unmanned aerial vehicles, characterized in that: include: A model building unit, which is used to build a three-dimensional virtual scene map based on three-dimensional terrain data obtained by oblique photography of the drone according to preset flight parameters and along a preset route; A plane fitting unit, which is used to fit the point cloud data in the three-dimensional virtual scene graph through a random sample consistency algorithm to generate a target fitting plane; An uneven region determination unit, which is used to determine an uneven region set according to a height difference between each point cloud and the target fitting plane; A detection frame determination unit, which is used to input a road surface image obtained by a vehicle-mounted camera along a preset route into a preset water accumulation detection model based on a Transformer architecture to perform water accumulation recognition, so as to output a detection frame set, wherein the water accumulation detection model is generated by training an adaptive dynamic channel pruning strategy, and the adaptive dynamic channel pruning strategy is: calculating the square of the channel weight amplitude, and normalizing the square of the channel weight amplitude to the sum of the squares of the weights of all retained channels in the same layer to obtain an adaptive dynamic channel pruning score, and performing channel pruning according to the size of the adaptive dynamic channel pruning score; A region fusion unit, configured to perform weighted fusion on the uneven regions in the uneven region set and the detection frames in the detection frame set to obtain at least one fused detection region; The water accumulation positioning unit is used to, for each fused detection area, when the confidence of the fused detection area is greater than a preset confidence threshold, regard the fused detection area as a water accumulation area and output the coordinates of the fused detection area.

9. A water accumulation positioning device based on three-dimensional oblique photography of unmanned aerial vehicles, characterized in that: The water accumulation positioning device based on drone three-dimensional oblique photography includes a processor, a memory, and a water accumulation positioning program based on drone three-dimensional oblique photography stored in the memory and executable by the processor. When the water accumulation positioning program based on drone three-dimensional oblique photography is executed by the processor, the steps of the water accumulation positioning method based on drone three-dimensional oblique photography as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a water accumulation positioning program based on drone three-dimensional oblique photography. When the water accumulation positioning program based on drone three-dimensional oblique photography is executed by a processor, the steps of the water accumulation positioning method based on drone three-dimensional oblique photography as described in any one of claims 1 to 7 are implemented.

Images