Unmanned aerial vehicle three-dimensional point cloud-based lightweight semantic segmentation roadside sign recognition method

Abstract

The present application relates to a roadside sign recognition method based on unmanned aerial vehicle three-dimensional point cloud lightweight semantic segmentation, belonging to the technical field of intelligent transportation. The method comprises: optimizing the point cloud collection path through adaptive flight control; enhancing the local geometric features of the point cloud by using the RSPAE algorithm; converting the point cloud into a three-channel fusion image (depth map, intensity map, and local depth variance map); extracting multi-scale features by using a double-branch neural network, and fusing and aligning the features through GFM and CFM modules; restoring a high-precision semantic segmentation map by using a lightweight decoder; generating a final recognition result by combining geographic registration and multi-frame redundancy suppression; and realizing state evaluation and anomaly detection based on an MLP scorer and Mahalanobis distance. The present application significantly improves segmentation accuracy, inference speed, and positioning accuracy in complex scenarios, is suitable for embedded device deployment, and solves the problems of low efficiency and high omission rate in the prior art.

Description

TECHNICAL FIELD

[0001] The application relates to a recognition method, in particular to a light-weight semantic segmentation roadside sign recognition method based on a three-dimensional point cloud of a unmanned aerial vehicle. BACKGROUND

[0002] With the rapid development of new infrastructure construction and intelligent transportation system, the whole life cycle digital management of highway assets has become an industry demand. Traditional highway inspection mainly relies on manual visual inspection and two-dimensional image recording. The inspection personnel need to carry professional equipment and walk or take a car along the road, which has three core pain points: first, the operation efficiency is low, a single manual inspection needs to invest 3-5 people team, and the daily detection mileage is less than 15 kilometers, which is difficult to meet the periodic detection demand of tens of thousands of kilometers of road network; second, the coverage ability of high-risk areas is insufficient, and high-risk road sections such as sharp curves and steep slopes, tunnel entrances and exits are often excluded from the detection range due to safety restrictions, resulting in a facility missing rate of more than 20%; third, the data management dimension is extensive, and the two-dimensional image lacks spatial geometric information, which is difficult to support the quantitative analysis of key parameters such as guardrail inclination and sign spatial coordinates, seriously restricting the scientific nature of maintenance decision-making.

[0003] The rise of unmanned aerial vehicle three-dimensional point cloud technology provides an innovative solution to the above problems. By carrying laser radar, multispectral camera and inertial navigation system, the unmanned aerial vehicle can realize three-dimensional space data collection with centimeter-level precision, and can cover 10-15 kilometers of road section in a single flight, which is 8-10 times more efficient than traditional manual operation. The point cloud data generated not only contains spatial coordinates (X, Y, Z), reflection intensity and other basic attributes, but also can integrate high-resolution texture information, completely record the geometric shape, surface reflection characteristics and topological relationship with the surrounding environment of the sign, providing a multi-dimensional data basis for the digital filing and state evaluation of highway facilities. However, to realize the automatic recognition and accurate segmentation of roadside signs and other small targets, there are still three technical bottlenecks:

[0004] Adaptive defects in point cloud preprocessing. Existing algorithms generally use fixed threshold downsampling or uniform voxelization processing to reduce data size, but such methods will irreversibly lose the local features of small-sized targets (such as 0.5m x 0.5m signs). For example, random downsampling can reduce the point cloud density of the sign edge by 40%-60%, causing the key geometric details (such as speed limit number contour) to be blurred; while the adaptive sampling method based on curvature features can preserve details, it is difficult to deploy in engineering due to high computational complexity (O(N 2 ).

[0005] The efficiency and accuracy of the segmentation algorithm are imbalanced. Mainstream point cloud segmentation architectures such as PointNet++, KPConv, etc. rely on multi-level feature aggregation, which can capture global context, but its computational complexity grows exponentially with the size of the point cloud. Tests show that when processing point clouds of a single kilometer road section (about 5 million points), the inference delay of models based on Transformers exceeds 300 ms, which cannot meet the dual requirements of real-time processing and low-power operation of edge devices for inspection systems. In addition, noise points generated by vegetation cover and vehicle dynamics in complex road scenes can easily lead to a 5%-8% misclassification rate of traditional convolutional networks.

[0006] The multi-scale feature modeling capability is insufficient. Accurate segmentation of road signs requires simultaneous analysis of millimeter-level character edges (such as direction arrow zigzag structures) and meter-level spatial poses (such as tilt angles and installation heights), but existing single-scale feature extraction mechanisms cannot accommodate both. Typical problems include: small receptive field convolution kernels lose macro-topological information of support rods, large-scale feature maps blur micro-texture features of reflective films, ultimately leading to spatial positioning errors exceeding 0.5m and semantic information extraction completeness less than 70%, severely affecting the usability of digital models.

[0007] The above technical defects seriously hinder the intelligent management and maintenance process of highway assets. According to statistics, in the digital models generated by existing systems, the coordinate offset of the sign exceeds 20%, and the miss rate in complex scenes such as curves and tunnels exceeds 15%, forcing maintenance departments to invest a large amount of manpower in data verification. Therefore, developing a road sign segmentation method that balances high efficiency, high accuracy, and strong adaptability has become a core breakthrough in promoting highway inspection from "digital archiving" to "intelligent decision-making".

[0008] Existing technologies, such as the sign detection method, device, and vehicle disclosed in CN117671644A. The application discloses a sign detection method, device, and vehicle, which includes: obtaining target point cloud data corresponding to a target area through a laser radar of a vehicle, and obtaining camera visual data corresponding to the target area through a camera of the vehicle, the target area including at least one sign; performing point cloud segmentation on the target point cloud data to obtain at least one three-dimensional bounding box corresponding to the target point cloud data; obtaining at least one two-dimensional visual detection box corresponding to the camera visual data; and obtaining the position of at least one sign included in the target area according to the at least one three-dimensional bounding box and the at least one two-dimensional visual detection box. The application obtains the position of the sign included in the target area according to the three-dimensional point cloud data corresponding to the target area obtained through the laser radar and the two-dimensional camera visual data corresponding to the target area obtained through the camera, thereby improving the accuracy of determining the position of the sign.

[0009] The prior technical solutions basically meet the requirements, but still have certain deficiencies, one of which is that the existing fast segmentation method directly uses a segmentation algorithm for segmentation after completing point cloud data processing, which may cause incomplete sign information extraction; the second is that the existing fast segmentation method is limited to sign detection and mile marker sign segmentation, and has low segmentation efficiency and precision, thereby affecting the speed and precision of sign information extraction; and the third is that the existing fast segmentation method may have poor model performance in point cloud segmentation tasks, thereby affecting the efficiency and accuracy of the segmentation algorithm calculation. SUMMARY

[0010] In order to solve the defects in the prior art, the present application discloses a road sign recognition method based on unmanned aerial vehicle three-dimensional point cloud lightweight semantic segmentation, and the technical solution is as follows:

[0011] The road sign recognition method based on unmanned aerial vehicle three-dimensional point cloud lightweight semantic segmentation has the following steps:

[0012] Step 1: adaptive flight control and point cloud acquisition: a flight control model is constructed by fusing visual perception and target density feedback, the flight height, angle and speed are optimized according to the heat map and point cloud statistical information, the acquisition path is dynamically adjusted without increasing the total length of the task route, and the coverage integrity and edge target quality of the sign area are improved.

[0013] Step 2: point cloud adaptive enhancement: using RSPAE algorithm based on local density i and curvature c i to calculate the enhancement priority E i , fine-tune the local geometry of the key area and keep the point cloud size constant, improve the geometric clarity of the key structure points and suppress the low confidence area.

[0014] Step 3: multi-channel image generation: the enhanced point cloud is converted into a three-channel fusion image I 2 composed of depth map D, intensity map I and local depth variance map f to retain spatial depth, material reflectivity and local structure stability for subsequent neural network processing.

[0015] Step 4: image-level feature extraction: the fusion image I f is input into a double-branch neural network including a CNN branch and a Mamba branch to extract local geometric details and global semantic context respectively, and spatial distribution modeling is completed through four-stage feature extraction.

[0016] Step 5: Feature fusion and alignment enhancement: The features extracted from the two paths are input into the global fusion module (GFM) and the cross-feature fusion module (CFM). The former adjusts and enhances the channel weights, while the latter compensates for the context through a Transformer attention structure. The output is the final multi-scale aligned feature representation F'.

[0017] Step 6: Lightweight decoder restores semantic map: The fused features F' are input into a two-layer decoder to complete upsampling, convolution, and prediction, outputting the original resolution semantic segmentation map M, ensuring accuracy while meeting the deployment requirements of embedded platforms.

[0018] Step 7: Semantic back-projection and geographic registration: The semantic segmentation map M is back-projected to the original point cloud coordinates, combined with the UAV flight pose matrix T(t) to complete geographic registration, and based on the error minimization objective function to optimize registration accuracy and fuse neighboring frames to generate a redundancy suppression result

[0019] Step 8: State scoring and classification: Input into the scoring and identification module, and the category is determined by resulting in a state vector The score R is generated by the MLP scorer i , and the Mahalanobis distance D i is calculated to perform self-supervised anomaly identification.

[0020] Advantages

[0021] 1: RSPAE point cloud adaptive enhancement algorithm

[0022] o Corresponding bottleneck: Solving the problem of "fixed voxelization leading to loss of small structures" in the preprocessing stage.

[0023] o Innovation: Without introducing new points, only based on RSPAE selected structure points and local smoothing, maintaining the same size of point cloud to improve geometric discrimination.

[0024] 2: Mamba+CNN dual-path lightweight network structure

[0025] o Corresponding bottleneck: Segmentation model accuracy and efficiency imbalance problem.

[0026] o Innovation: Mamba introduces state space modeling to construct a linear complexity long dependency extraction structure, combined with CNN to complete multi-scale modeling, supporting Jetson-level device operation.

[0027] 3: State recognition module based on scoring and Mahalanobis distance

[0028] o Corresponding bottleneck: Insufficient integration of cross-scale information, missing fuzzy and abnormal point identification.

[0029] o Innovation: Combining images and point cloud attributes into a score vector, predicting states through MLP and introducing a complete anomaly label, achieving automatic identification of occlusion / fog / damage. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The method flowchart of the present application is shown in the figure;

[0031] Figure 2 The overall network architecture of the present application is shown in the figure;

[0032] Figure 3 The overall architecture of GFM and CFM of the present application is shown in the figure;

[0033] Figure 4 The decoder architecture of the present application is shown in the figure;

[0034] Figure 5 The test scene schematic diagram of embodiment 1 of the present application is shown in the figure;

[0035] Figure 6 The test scene schematic diagram of embodiment 2 of the present application is shown in the figure. DETAILED DESCRIPTION

[0036] Referring to Figures 1-4 The figure shows a rapid segmentation method for roadside signboards based on three-dimensional point cloud data of unmanned aerial vehicles, characterized by the following steps:

[0037] Step 1: Adaptive flight control and three-dimensional point cloud acquisition. A flight control model is constructed that integrates visual perception and target density feedback, and the flight height, angle and speed are optimized jointly according to the heat map and point cloud statistical information, dynamically adjusting the acquisition path without increasing the total length of the task route, and improving the coverage integrity and edge target quality of the signboard area.

[0038] This step constructs a road perception and flight control joint model, which enables the unmanned aerial vehicle to adjust the flight height, heading angle and speed according to the complexity of the road environment and the distribution of visual targets during flight, ensuring efficient perception and coverage of the signs on both sides of the road. This mechanism is different from traditional path planning with fixed height or trajectory, effectively improving the acquisition quality of edge targets.

[0039] The road perception and flight control joint model refers to the collaborative modeling mechanism of the visual perception module and the flight control module, and its core idea is to dynamically introduce the importance feedback of the visual attention area in flight control to realize task-driven trajectory adjustment. The model is composed of the following:

[0040] a. Visual perception module. Used for processing image data or projected point cloud data to generate a region detection heat map The construction of this module includes the following sub-processes:

[0041] • Input is depth image D(u, v) and intensity image I(u, v), respectively obtained by projecting three-dimensional point cloud P;

[0042] • Feature extraction network f feat Extracting edges, shapes, brightness, etc. from images;

[0043] • Attention estimation network f aat Map features to heat maps Where represents the visual saliency score at position (x, y).

[0044] b. Target density estimation module. Used to construct the target distribution function D(x, y) in the region based on historical point cloud statistical information. The construction of this module includes:

[0045] • Set the historical target set

[0046] • Use the density estimation function with Gaussian kernel to construct the spatial distribution: Where δ(·) represents a two-dimensional Gaussian kernel function, and σ is the kernel bandwidth.

[0047] c. Flight cost function. Defined as:

[0048] d. Joint optimization objective. Defined as: min h,θ,v J(h, θ, v) where:

[0049] • h: flight height;

[0050] • θ: flight heading angle;

[0051] • v: flight speed;

[0052] • Ω: predetermined flight coverage area;

[0053] • Region detection heat map;

[0054] • D(x, y): spatial target density function.

[0055] The unmanned aerial vehicle collects three-dimensional point cloud data of laser radar and depth camera to generate point cloud set: And record the flight pose matrix: Where:

[0056] • P i : the i-th point in the point cloud, containing its spatial coordinates;

[0057] • R(t): the pose rotation matrix of the drone at time t;

[0058] • t(t): the 3D translation position vector of the drone at time t;

[0059] • SE(3): special Euclidean group, representing the set of 3D space rigid body transformations (rotation and translation).

[0060] Step 2: Point cloud adaptive enhancement: using the RSPAE algorithm based on local density p i and curvature c i to calculate the enhancement priority E i , fine-tune the local geometry of key areas, and keep the point cloud size constant, improve the geometric clarity of key structure points, and suppress low-confidence areas.

[0061] The original point cloud is first preprocessed by the innovative lightweight adaptive geometric enhancement algorithm RSPAE to generate an enhanced point cloud set P' = {P'1, P'2,..., P'N}, which is used to improve the local structure clarity and keep the total point number unchanged, so as to realize the optimization of geometric features without introducing additional computational burden. n

[0062] The RSPAE algorithm includes the following steps:

[0063] a. Local density calculation: where:

[0064] • k represents the number of neighborhood points, and the recommended range is [10, 50] The k-neighborhood of point P i

[0065] • k represents the number of neighborhood points, and the recommended range is [10, 50]

[0066] i : represents the average neighborhood distance of point P i , that is, the local point cloud sparsity.

[0067] b. Curvature estimation: where:

[0068] • λ0, λ1, λ2: eigenvalues of the neighborhood covariance matrix of point P i , λ0 is the smallest eigenvalue;

[0069] • c i : represents the degree of geometric mutation of point P i , the value range is [0, 1], the closer to 1 indicates the stronger mutation, and the closer to 0 indicates local flatness.

[0070] c. Enhancement priority calculation: where:​​​

[0071] · E i : enhanced priority score of point P i ;

[0072] · ρ min , ρ max : global minimum and maximum density;

[0073] · c min , c max : global minimum and maximum curvature;

[0074] · α ∈ [0, 1]: weighting coefficient between density and curvature, taking 0.5 can achieve equal weight fusion.

[0075] d. Structure enhancement mechanism: sort all points according to E i , select the top m% of points P i in the neighborhood for local coordinate reconstruction, and use the following update method: wherein:

[0076] · w ij : normalized weight factor, reflecting the importance of neighborhood points;

[0077] · δ i : geometric enhancement offset, only applied to important structure points;

[0078] · m: enhancement point selection ratio, recommended range [10, 40].

[0079] RSPAE does not introduce new points, only fine-tunes the key point positions, so that the enhanced point cloud set P' is the same size as the original set P, i.e. |P'| = |P|. Since enhancement only occurs in high-scoring areas, and the neighborhood mean is calculated using a sparse matrix, the overall computational complexity is linearly related to the original point cloud. Therefore, it improves the geometric discrimination while maintaining the efficiency of the original network, making it suitable for resource-constrained embedded system deployment scenarios.

[0080] The enhanced point cloud P' is used to generate subsequent three-channel fusion images for further feature extraction and network processing.

[0081] Step 3: Multi-channel image generation: convert the enhanced point cloud into a three-channel fusion image I 2 composed of depth map D, intensity map I, and local depth variance map σ f to preserve spatial depth, material reflectivity, and local structure stability for subsequent neural network processing.

[0082] The point cloud is converted into a three-channel fusion projection image form to enhance the comprehensive perception ability of the neural network to the spatial structure and target texture.

[0083] The three-channel image Simultaneously contains:

[0084] · Depth map D(u, v): records the depth information of the minimum Euclidean distance in the point cloud;

[0085] · Intensity map I(u, v): records the reflection intensity information of the point cloud;

[0086] · Local depth variance map σ D (u, v): used to enhance the perception ability to geometric deformation, edge structure and object mutation.

[0087] The construction method of each channel is as follows:

[0088] a. Depth map construction: Wherein:

[0089] · π(·): projection function of point cloud to image pixel coordinates;

[0090] · ||P i ||: Euclidean distance of point P i to the origin.

[0091] b. Intensity map construction: Wherein:

[0092] -I i : reflection intensity value of point P i .

[0093] c. Local depth variance map: an adaptive estimated local depth variance σ D (u, v) is proposed to dynamically describe the complexity of local depth variation, and its calculation method is: Wherein:

[0094] · Local area centered at (u, v) with window size r x r;

[0095] · μ r (u, v): mean value of depth values in the area ;

[0096] · r: window scale is automatically adjusted according to image content, set according to image gradient or edge density, set as: Wherein represents the gradient response of the depth map, β is the amplification coefficient, and r0 is the basic window.

[0097] By this improved way, σ D (u, v) can more accurately reflect surface mutations, the complexity of edge junctions, significantly improve the network's ability to distinguish sensitive areas of structure, especially suitable for edge detection and small target extraction tasks.

[0098] The final generated fusion image: Where H, W are the height and width dimensions of the image, and the output is used as the input of the neural network to support multi-scale geometric feature learning and enhance perception ability.

[0099] The fusion image is input into a four-stage two-parallel feature extraction network, which includes a convolutional neural network (CNN) path and a Mamba path for state space modeling. The two paths are used to extract local geometric structure features and long-distance semantic dependency features in the image, respectively, so as to jointly model texture details and context information.

[0100] Wherein, the CNN path contains four stages (Stage 1-4):

[0101] Stage 1: Two-dimensional convolution module (Conv2D) followed by batch normalization (BN) and ReLU activation, input channel C, used to preliminarily extract local point cloud features;

[0102] Stage 2: Residual structure module, containing downsampling operation (Stride=2), realizing spatial dimension compression and enhancing feature expression ability;

[0103] Stage 3: Atrous convolution module, convolution kernel size 3x3x3, atrous rate r=2, used to expand the receptive field to capture the mesoscale spatial relationship;

[0104] Stage 4: Global average pooling and fully connected (FC) layer, output local feature representation:

[0105]

[0106] Wherein, F CNN represents the local feature, N the number of effective pixels in the image (after flattening), and d1 the number of channels of the CNN branch.

[0107] The Mamba path also contains four stages:

[0108] Stage 1: Patch Embedding module, grouping and embedding point cloud feature blocks to form a sequence structure input;

[0109] • Stage 2: Mamba Block, modeling the state space sequence in local regions with sliding window, to obtain cross-point feature dependency;

[0110] • Stage 3: Token Mixing module, to realize information mixing between cross-channels and cross-patches, to enhance global perception;

[0111] • Stage 4: Normalization and linear projection layer, to output the final semantic embedding feature:

[0112]

[0113] where: d2 represents the output dimension of the Mamba branch; F Mamba : feature matrix output by the Mamba path;

[0114] Finally, the output features of the two paths are concatenated in the channel dimension to form a fused representation:

[0115]

[0116] where Concat(·) represents the concatenation operation in the feature channel dimension, used to fuse local details and global context features. F represents the fused feature.

[0117] Step 4: Image-level feature extraction: fuse image I f Input the dual-branch neural network, including the CNN branch and the Mamba branch, to extract local geometric details and global semantic context, respectively, to complete spatial distribution modeling through four-stage feature extraction.

[0118] The fused feature F in this step 4 is input to the global feature fusion module (GFM) and the cross-feature fusion module (CFM), which respectively realize multi-scale alignment and context perception modeling at the channel level.

[0119] where:

[0120] The GFM module takes the outputs of the CNN 3rd stage and the Mamba 3rd stage as input, and the module structure includes:

[0121] a. Feature concatenation unit: receives output features from two paths and Fuse the features in the channel dimension to get the fused feature

[0122] b. Channel attention estimation submodule: for each F i Apply a fully connected layer Calculate the attention coefficient:

[0123] c. Reconfiguration output unit: use γ i F i Weighted:

[0124] Where:

[0125] · Represents the output feature of the third stage of CNN, the size is

[0126] · Represents the output feature of the third stage of Mamba, the size is

[0127] ·F i : represents the fusion feature vector after channel splicing;

[0128] ·W: full connection weight matrix;

[0129] ·γ i : represents the normalized weight coefficient of channel importance;

[0130] · Global enhancement features output by GFM module.

[0131] The CFM module takes the output of the second stage of CNN and the fourth stage of Mamba as input, and the module structure includes:

[0132] a. Feature splicing unit: input and Splicing gets

[0133] b. Linear projection layer: define projection matrix Generation: Q i = W Q F i ,K j = W K F j ,V j = W V F j

[0134] c. Attention calculation layer: use scaled dot-product attention mechanism to construct context features:

[0135]

[0136] Where:

[0137] · Represents the output feature of the second stage of CNN, the size is

[0138] · Mamba stage 4 output feature, size is

[0139] ·F i : concatenated fusion feature vector;

[0140] ·W Q ,W K ,W V : linear projection matrix of query, key, value respectively;

[0141] ·Q i ,K j ,V j : query, key, value vector at pixel position i, j respectively;

[0142] ·d a : low-dimensional mapping dimension of attention module, used to reduce computational complexity, can be set to 32, 64 or 128, etc.

[0143] · Context enhanced feature output by CFM module.

[0144] The final fusion output is: Wherein:

[0145] ·F' i : final fusion feature at each pixel position;

[0146] ·F': feature output after fusion of GFM and CFM, with channel attention and context information, as the input basis for subsequent semantic segmentation or detection modules;

[0147] ·d1+d2: represents the channel dimension of the final fusion feature.

[0148] Step 5: Feature fusion and alignment enhancement: input the features extracted by the two paths into the global fusion module (GFM) and the cross-feature fusion module (CFM), the former adjusts and enhances the channel weight, and the latter compensates the context through the Transformer attention structure, outputting the final multi-scale alignment feature representation F'.

[0149] The fused feature representation F' is input into a lightweight decoder structure suitable for embedded platforms to generate a semantic segmentation map consistent with the original image resolution.

[0150] The lightweight decoder includes two layers, which are:

[0151] 1. Decoding layer one (up-sampling + convolution module):

[0152] • fusing features Up-sampling operation (Upsample) to intermediate resolution H / 2 x W / 2;

[0153] • immediately followed by a two-dimensional convolution operation (Conv2D, kernel size 3 x 3, output channels C1), and applying batch normalization (BN) and ReLU activation function:

[0154]

[0155] 2. Decoding layer two (up-sampling + convolution + prediction layer):

[0156] • fusing F (up1) Further up-sampling to the original image size H x W;

[0157] • passing through a 1 x 1 convolution operation to reduce the number of channels to the number of classes C, outputting the final segmentation logits:

[0158] wherein:

[0159] • F': represents the fused image feature tensor, with dimensions H / 4 x W / 4 x (d1+d2), derived from the outputs of GFM and CFM;

[0160] • H, W: represent the height and width of the original input image, used to define the final output resolution;

[0161] • C1: the number of channels of the intermediate feature map, used to maintain spatial structure and feature integrity, usually set to 64 or 128;

[0162] • F (up1) : the intermediate feature map obtained after the first decoding layer, with dimensions H / 2 x W / 2 x C1;

[0163] • C: represents the number of semantic segmentation classes, i.e. the output channel number, for example C = 2 for binary classification;

[0164] • M: represents the final generated semantic segmentation prediction map, outputting a C-dimensional vector for each pixel position, corresponding to the prediction probability (Logits form) of each class.

[0165] The decoder structure is simple, only using two levels of up-sampling and low-computational convolution operations, which can effectively maintain decoding accuracy while greatly reducing model parameter size and computational overhead, suitable for deployment on resource-constrained embedded devices or unmanned aerial vehicle platforms, realizing real-time semantic segmentation.

[0166] Step 6: Lightweight decoder restores semantic map: input fused feature F' into the two-layer structure decoder, complete upsampling, convolution and prediction, output original resolution semantic segmentation map M, ensure accuracy while meeting embedded platform deployment.

[0167] The semantic segmentation image result M output by the decoder is mapped back into the original three-dimensional point cloud coordinate system to realize the fusion of semantic masks and point clouds, geographic registration, and multi-frame redundancy suppression.

[0168] a. Image-to-point cloud back-projection fusion:

[0169] According to the projection relationship π -1 (u,v)→P i , the predicted label of the pixel coordinate (u,v) in the image is mapped to its corresponding point cloud point P i , generating a set of semantic point cloud masks: Where:

[0170] · The predicted probability value of pixel point (u,v) in the image segmentation output belongs to class c;

[0171] ·l i : the final semantic class label assigned to point cloud point P i , taking the class corresponding to the maximum probability;

[0172] ·π -1 (u,v): mapping function that back-projects image coordinates to three-dimensional point cloud point P i ;

[0173] ·P i : the i-th point in the original three-dimensional point cloud;

[0174] ·S i : the semantic point cloud set of the current frame, containing P i and the corresponding label l i .

[0175] b. Geographic registration processing based on fusion error minimization strategy:

[0176] - Use the pose matrix T(t) ∈ SE(3) in the flight navigation information of the unmanned aerial vehicle to transform the local point cloud point P i to the global geographic coordinate system: Geo(P i ) = proj(T(t) · P i ). Introduce a fusion error minimization mechanism and design a registration loss function as follows:

[0177] Where:

[0178] • T(t): pose matrix of the drone at time t, representing rotation and translation, belonging to the special Euclidean group SE(3);

[0179] • proj(·): projection function to convert 3D coordinates to geographic coordinate system (e.g. longitude, latitude, height);

[0180] • Geo(P i ): geographic coordinates of the point cloud points after registration by T(t);

[0181] • P i : reference points matched in the map or previous frame with P j ; i • S j : semantic point cloud results of the neighboring frames of the current frame S i ;

[0182] • Ω: overlapping area point cloud index set for registration calculation;

[0183] • fusion error loss function for optimizing the accuracy of T(t).

[0184] c. Multi-frame redundancy suppression and completion mechanism:

[0185] - For the redundant recognition results that may occur in adjacent frames, a non-maximum suppression (NMS) strategy based on intersection over union is adopted:

[0186]

[0187] Fusion and completion using multi-frame results within a time window:

[0188] Wherein:

[0189] • S j : semantic point cloud results of the neighboring frames of the current frame S i ;

[0190] • ∪: represents "union set";

[0191] • IoU(S i ,S j ): calculates the voxel intersection over union index of two point cloud semantic regions;

[0192] • δ: loU threshold, determines whether to retain or fuse, implementation needs to be greater than 80%;

[0193] • max(S i ,Sj): the largest mask region selected by category confidence or area;

[0194] • temporal neighborhood set, containing frame indexes near the current frame i;

[0195] • The final output of high-quality semantic point cloud results after multi-frame fusion and redundancy suppression.

[0196] Step 7: Semantic back-projection and geographic registration: back-project the semantic segmentation map M back to the original point cloud coordinates, complete geographic registration combined with the UAV flight pose matrix T(t), and based on the error minimization objective function Optimize registration accuracy, fuse adjacent frames to generate redundancy suppression results

[0197] Fusion of the completed semantic point cloud results Input to the state recognition and type classification module to realize target category determination and integrity state evaluation of roadside signs.

[0198] a. Category prediction module:

[0199] -Statistical or aggregated semantic label results in each semantic point cloud region to obtain the final predicted category:

[0200] Where:

[0201] ·T i : represents the predicted category of the i-th target;

[0202] ·c: category index (such as speed limit sign, indicator, prohibition sign, etc.);

[0203] · The proportion of pixels or points belonging to category c in the region , weighted confidence value, or maximum response value;

[0204] ·argmax c : represents finding the c value that makes the largest among all category indices c, i.e., the category with the highest probability of prediction.

[0205] 2. State scoring and self-supervised anomaly detection module:

[0206] First, calculate the baseline score vector: Use a multi-target state scoring function (such as a lightweight neural network scorer) to predict state confidence: R i = MLP(v i ) ∈ [0, 1];

[0207] Introduce a self-supervised anomaly recognition mechanism based on Mahalanobis distance to detect abnormal signs from historical normal distribution: D i = (v i - μ) T ∑ -1 (v i- μ)A i = 1 (D ii > τ) where:

[0208] · v i : represents the state vector of the current i-th signboard;

[0209] · I i : image or point cloud intensity indicator, measuring the clarity or reflection characteristics;

[0210] · θ i : shooting angle, reflecting the recognition impact caused by the perspective shift;

[0211] · Current visible area (unit: m 2 );

[0212] · Theoretical complete area (unit: m 2 );

[0213] · R i : integrity score output by the scoring network, the larger the value, the more reliable;

[0214] · μ: mean of historical normal state vector;

[0215] ·∑: historical state covariance matrix;

[0216] ·D i : Mahalanobis distance, representing the difference between the current state and the historical distribution;

[0217] · τ: set abnormal detection threshold;

[0218] · A i : binary abnormality identifier, 1 indicates that the signboard may be abnormal.

[0219] Step 8: State scoring and classification: input into the scoring and recognition module, the category is derived from State vector is generated by the MLP scorer to generate score R i , and combined with Mahalanobis distance D i for self-supervised anomaly recognition.

[0220] Example 1

[0221] Referring to Figure 5 shown. Test scene: highway night scene, low point cloud density (average <50 points per square meter), dynamic acquisition blur and partial vegetation occlusion, target is speed limit signboard (size 0.6m x 0.6m).

[0222] Comparison results:

[0223]

[0224] Reference Figure 6 The test scene is shown in FIG. 6, which is: light rain accompanied by haze, point cloud noise increases (signal-to-noise ratio decreases by 40%), sign surface reflectivity decreases. Construction warning sign, size 1.2m x 0.8m, part of the area is disturbed by rain and fog scattering. The UAV is equipped with an NVIDIA Jetson Xavier edge computing module.

[0225] Comparison results:

[0226]

[0227]

[0228] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above examples, and the above examples and descriptions in the specification are only the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection required by the present application is defined by the appended claims and their equivalents.

Claims

1. A roadside signboard rapid identification method based on a lightweight semantic segmentation of a three-dimensional point cloud of a UAV, characterized in that: The method comprises the following steps: Step 1: adaptive flight control and point cloud acquisition: a flight control model is constructed by fusing a visual perception module and target density feedback, a spatial target density function is generated based on a heat map and point cloud statistical information, and a collection path is dynamically adjusted by optimizing flight height, heading angle and flight speed; Step 2: point cloud adaptive enhancement: an RSPAE algorithm is used to calculate an enhancement priority based on local density and curvature, and local geometry of a key area is fine-tuned; Step 3: multi-channel image generation: the enhanced point cloud is converted into a three-channel fused image composed of a depth map, an intensity map and a local depth variance map; Step 4: image-level feature extraction: a double-branch neural network is input with the fused image, local geometric features and global semantic context features are extracted through a CNN branch and a Mamba branch; Step 5: feature fusion and alignment enhancement: a global fusion module is used to adjust channel weights, and a Transformer attention structure of a cross-feature fusion module is used for context compensation to generate multi-scale alignment features; Step 6: light decoder restores semantic map: the fused features are input into a two-layer structure decoder to complete upsampling, convolution and prediction, and an original resolution semantic segmentation map is output; Step 7: semantic back projection and geographic registration: the semantic segmentation map is back projected to a point cloud coordinate system, a registration accuracy is optimized by combining a UAV pose matrix and an error minimization objective function, and a redundancy suppression result is generated by fusing multiple frames; Step 8: state scoring and classification: the semantic point cloud result is input into a scoring and recognition module to realize target category determination and integrity state evaluation of a roadside signboard.

2. The roadside signboard rapid identification method based on UAV three-dimensional point cloud lightweight semantic segmentation according to claim 1, characterized in that: The visual perception module generates a heat map H(x, y) = faat(f feat (D, I)); the target density function D(x, y) is calculated by Gaussian kernel density estimation based on historical point cloud data; the flight parameter optimization target is to minimize the cost function defined as: Where: θ: flight heading angle; v: flight speed; Ω: predetermined flight coverage area; Region detection heat map; D(x, y): spatial target density function.

3. The roadside signboard rapid identification method based on UAV three-dimensional point cloud lightweight semantic segmentation according to claim 2, characterized in that: The RSPAE algorithm comprises the following steps: Local density computation: where: k-neighborhood of point P i k represents the number of neighborhood points, range [10, 50]; p i : represents the average neighborhood distance of point P i , i.e. the local point cloud sparsity degree; Curvature estimation: where: λ0, λ1, λ2: eigenvalues of the neighborhood covariance matrix at point P i λ0is the smallest eigenvalue; c i : represents the degree of geometric abruptness of point P i , the value range is [0, 1]; Enhanced priority computation: where: E i : enhanced priority score for point P i ; p min , p max : global minimum and maximum density; c min , c max : global minimum and maximum curvature; a e [0, 1]: weighting coefficient between density and curvature; Structure enhancement mechanism: all points are sorted according to E i , and the top m% points P i are selected to perform local coordinate reconstruction in their neighborhood, using the following update method: where: w ij : normalized weight factor, reflecting the importance of the neighborhood points; δ i : geometric enhancement offset, only applied to important structure points; m: enhancement point selection ratio, range [10, 40].

4. The roadside signboard rapid identification method based on UAV three-dimensional point cloud lightweight semantic segmentation according to claim 2, characterized in that: The pre-processed point cloud is converted into a three-channel fused projected image form, the three-channel image Meanwhile, it contains: a depth map D(u, v) recording the minimum Euclidean distance depth information in the point cloud; an intensity map I(u, v) recording the reflection intensity information of the point cloud; a local depth variance map σ D (u, v) for enhancing the perception ability of geometric deformation, edge structure and object mutation; The construction method of each channel is as follows: Depth map construction: where: π(·): projection function from point cloud to image pixel coordinates; ||P i ||: Euclidean distance of point P i to the origin; Strength map construction: wherein: - I i : the reflection intensity value of the point P i . Local depth variance map: an adaptive estimation of local depth variance σ D (u, v) is proposed to dynamically describe the complexity of local depth variation, which is calculated as: where: Local region centered at (u, v) with window size r x r; μ r (u, v): region Mean of depth values within the region; r: window scale is automatically adjusted according to image content, set as: where denotes the gradient response of the depth map, β is the amplification coefficient, and r0 is the basic window.

5. The method of claim 2, wherein the method is characterized by: The fused image Input into a four-stage two-parallel feature extraction network, the network comprising a convolutional neural network (CNN) path and a Mamba path for state space modeling, the two paths being respectively used to extract local geometric structure features and long-distance semantic dependency features in the image, so as to jointly model texture details and context information; The CNN path comprises four stages: Stage 1: a two-dimensional convolution module Conv2D is followed by batch normalization BN and ReLU activation, the input channel is C, and is used for preliminary extraction of local point cloud features; Stage 2: a residual structure module, which contains a downsampling operation, realizes spatial dimension compression and enhances feature expression ability; Stage 3: a hole convolution module, the convolution kernel size is 3x3x3, the hole rate is r=2, and is used for expanding the receptive field to capture the mesoscale spatial relationship; Stage 4: global average pooling and fully connected FC layer, output local feature representation: where F CNN denotes the local feature, N denotes the number of valid pixels in the image, and d1 denotes the number of channels of the CNN branch.

6. The roadside signboard rapid identification method based on UAV three-dimensional point cloud lightweight semantic segmentation according to claim 2, characterized in that: The fused feature representation F' is input into a light decoder structure suitable for embedded platforms to generate a semantic segmentation map consistent with the original image resolution; The light decoder comprises two layers, which are: Decoding layer one: upsampling + convolution module fusing features up-sampling operations to an intermediate resolution H / 2 x W / 2; Immediately following the two-dimensional convolution operation and applying batch normalization and a ReLU activation function: Decoding layer two: upsampling + convolution + prediction layer F (up1) Further up-sampling to original picture size HxW; After 1x1 convolution operation, the number of channels is reduced to the number of categories C, and the final segmentation heat map is output: Wherein: F': represents the fused image feature tensor, the dimension is H / 4xW / 4x(d1+d2), and is derived from the output of GFM and CFM; H, W: represent the height and width of the original input image, used to define the final output resolution; C1: the number of channels of the intermediate feature map, used to maintain spatial structure and feature integrity, usually set to 64 or 128; F (up1) : the intermediate feature map obtained after decoding the first layer, with a size of H / 2xW / 2xC1; C: represents the number of semantic segmentation categories, i.e. the output channel number, for example, C=2 for binary classification; M: denotes the final generated semantic segmentation prediction map, each pixel position outputs a C-dimensional vector, corresponding to the predicted probability of each class.

7. The method of claim 2, wherein the method is characterized by: The semantic segmentation image result M decoded in step 6 is mapped back to the original three-dimensional point cloud coordinate system to realize the fusion of the semantic mask and the point cloud, geographical registration and multi-frame redundancy suppression; a. Image-to-point cloud back-projection fusion: According to the projection relationship π -1 (u,v)→P i mapping the prediction label of the pixel coordinate (u,v) in the image to its corresponding point cloud point P i The semantic point cloud mask set is generated as follows: wherein: The predicted probability value of the pixel point (u, v) belonging to the category c in the image segmentation output; l i : final semantic class label assigned to point cloud point P i , take the class corresponding to the maximum probability π -1 (u,v): mapping function to back-project image coordinates to 3D point cloud point P i (u,v): mapping function to back-project image coordinates to 3D point cloud point P P i : i-th point in the original 3D point cloud; S i : semantic point cloud set of the current frame, containing P i and the corresponding label l i ; b. Geographical registration processing based on fusion error minimization strategy: With the pose matrix T(t) ∈ SE(3) in the flight navigation information of the unmanned aerial vehicle, the local point cloud point P i is transformed to the global geographic coordinate system: Geo(P i ) = proj(T(t) · P i ); a fusion error minimization mechanism is introduced, and a registration loss function is designed as follows: wherein: T(t): the pose matrix of the UAV at time t, representing rotation and translation, belonging to the special Euclidean group SE(3); proj(·): a projection function that converts three-dimensional coordinates to a geographical coordinate system (such as latitude, longitude, and altitude); Geo(P i ): geographic coordinates of a point cloud point after T(t) registration; With P i Reference points matched in the map or previous frame; Ω: a set of overlapping area point cloud indexes for registration calculation; The fusion error loss function is used to optimize the accuracy of T(t); c. Multi-frame redundancy suppression and completion mechanism: For the redundant recognition results that may appear in adjacent frames, a non-maximum suppression (NMS) strategy based on intersection over union is used: Fusion and completion are performed on multiple frames within a time window: Where: S j : semantic point cloud results of the current frame S i adjacent frames; ∪: represents "set union"; IoU(S i ,S j ) : calculate the voxel intersection over union index of two point cloud semantic regions; δ: loU threshold, determines whether to retain or fuse, and the implementation needs to be greater than 80%; max(S i ,S j ) : maximum mask region selected by class confidence or area temporal neighborhood set, containing frame indices in the vicinity of the current frame i; High-quality semantic point cloud results after multi-frame fusion and several suppression are finally output.

8. The roadside signboard rapid identification method based on UAV three-dimensional point cloud lightweight semantic segmentation according to claim 2, characterized in that: The fused and completed semantic point cloud result in step 7 The state recognition and type classification module is input to realize the target category determination and integrity state evaluation of the roadside sign; the category prediction module: count or aggregate the semantic label results within each semantic point cloud region to obtain the final predicted class: Where: T i : represents the predicted class of the i-th target; c: class index; region proportion of pixels or points belonging to class c, weighted confidence values, or maximum response values; arg max c : denotes finding the value of c that maximizes the probability of being predicted as that class; State score and self-supervised anomaly detection module: First, the reference score vector is calculated: Predicting state confidences using a multi-objective state scoring function (such as a light neural network scorer): R i = MLP(v i ) e [0, 1]; A self-supervised anomaly detection mechanism based on Mahalanobis distance is introduced to detect abnormal signboards from the historical normal distribution: D i = (v i - μ) T ∑ -1 (v i - μ)A i = 1(D i > τ) where: v i : denotes the state vector of the current i-th sign; I i : image or point cloud intensity indicator, measuring sharpness or reflectance properties; θ i : angle of shooting, reflecting the recognition influence caused by the perspective offset; Current viewable area, unit: m 2 ; Theoretical complete area, unit: m 2 ; R i : the integrity score output by the scoring network, the larger the value the more reliable; μ: mean of historical normal state vector; ∑: historical state covariance matrix; D i : Mahalanobis distance, representing the difference between the current state and the historical distribution; τ: set anomaly detection threshold; A i : Binary exception flag, 1 if the signboard is likely to be abnormal.

9. A non-volatile storage medium, characterized by The non-volatile storage medium includes a stored program, wherein the program, when executed, controls the device in which the non-volatile storage medium is located to perform the method of claim 1.

10. An electronic device, comprising: A processor and a memory are included; the memory has computer readable instructions stored therein, and the processor is used to run the computer readable instructions, wherein the computer readable instructions, when executed, perform the method of claim 1.

Images