Unmanned aerial vehicle three-dimensional point cloud-based lightweight semantic segmentation roadside signboard identification method

Abstract

The invention relates to a roadside signboard identification method based on unmanned aerial vehicle three-dimensional point cloud lightweight semantic segmentation, and belongs to the technical field of intelligent traffic. The method comprises the following steps: optimizing a point cloud acquisition path through adaptive flight control; adopting an RSPAE algorithm to enhance local geometric features of the point cloud; converting the point cloud into a three-channel fusion image (a depth image, an intensity image and a local depth variance image); extracting multi-scale features by using a double-branch neural network, and fusing the aligned features through a GFM module and a CFM module; the lightweight decoder recovers a high-precision semantic segmentation map; and generating a final identification result by combining geographical registration and multi-frame redundancy suppression. And state evaluation and anomaly detection are realized based on an MLP scoring device and a mahalanobis distance. According to the method, the segmentation precision, the reasoning speed and the positioning precision are remarkably improved in a complex scene, the method is suitable for deployment of embedded equipment, and the problems of low efficiency and high omission ratio in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to a recognition method, and in particular to a roadside sign recognition method based on lightweight semantic segmentation of unmanned aerial vehicle three-dimensional point clouds. Background Art

[0002] With the rapid development of new infrastructure and smart transportation systems, digital management of highway assets throughout their lifecycle has become an industry imperative. Traditional highway inspections rely primarily on manual visual inspections and two-dimensional image recording. Inspectors must carry specialized equipment and conduct their inspections on foot or by vehicle. This presents three key pain points: First, operational efficiency is low. A single manual inspection requires a team of 3-5 people, and the average daily inspection mileage is less than 15 kilometers, making it difficult to meet the periodic inspection needs of a road network spanning tens of thousands of kilometers. Second, coverage of high-risk areas is insufficient. Due to safety restrictions, high-risk sections such as sharp bends, steep slopes, and tunnel entrances are often excluded from inspection, resulting in a missed inspection rate exceeding 20%. Third, data management is crude. 2D images lack spatial geometric information, making it difficult to quantitatively analyze key parameters such as guardrail inclination and signage coordinates, severely hindering the scientific nature of maintenance decisions.

[0003] The rise of drone 3D point cloud technology has provided an innovative solution to the above problems. Equipped with lidar, multispectral cameras and inertial navigation systems, drones can achieve centimeter-level accuracy in 3D spatial data collection. A single flight can cover a 10-15 km road section, which is 8-10 times more efficient than traditional manual operations. The point cloud data it generates not only contains basic attributes such as spatial coordinates (X, Y, Z) and reflection intensity, but also integrates high-resolution texture information to fully record the geometric shape of signs, surface reflective properties and topological relationships with the surrounding environment, providing a multi-dimensional data foundation for digital archiving and status assessment of highway facilities. However, to achieve the automated recognition and precise segmentation of small targets such as roadside signs, there are still three major technical bottlenecks:

[0004] Adaptability defects in point cloud preprocessing. Existing algorithms generally use fixed threshold downsampling or uniform voxelization to reduce data size, but such methods will irreversibly lose the local features of small-sized targets (such as 0.5m×0.5m signs). For example, random downsampling will reduce the point cloud density at the edge of the sign by 40%-60%, resulting in the blurring of key geometric details (such as the outline of the speed limit number); while the adaptive sampling method based on curvature features can retain details, it has high computational complexity (O(N 2 ))Difficult to deploy in an engineering manner.

[0005] There is an imbalance between the efficiency and accuracy of segmentation algorithms. Mainstream point cloud segmentation architectures such as PointNet++ and KPConv rely on multi-level feature aggregation. Although they can capture global context, their computational complexity increases exponentially with the scale of the point cloud. Actual measurements show that when processing a point cloud of a single kilometer of road section (approximately 5 million points), the inference delay of the Transformer-based model exceeds 300ms, which cannot meet the dual requirements of the inspection system for real-time processing and low-power operation of edge devices. In addition, the noisy point clouds generated by vegetation occlusion and vehicle dynamic interference in complex road scenes can easily lead to a 5%-8% missegmentation rate in traditional convolutional networks.

[0006] Insufficient multi-scale feature modeling capabilities. Accurate segmentation of roadside signs requires simultaneous analysis of millimeter-level character edges (such as the sawtooth structure of directional arrows) and meter-level spatial poses (such as tilt angle and installation height). Existing single-scale feature extraction mechanisms struggle to account for both. Typical issues include: small receptive field convolution kernels lose macroscopic topological information of supporting poles, and large-scale feature maps blur the microscopic texture features of reflective film. Ultimately, this results in spatial positioning errors exceeding 0.5m and semantic information extraction completeness less than 70%, seriously affecting the usability of digital models.

[0007] These technical deficiencies severely hamper the advancement of intelligent highway asset management and maintenance. Statistics show that in digital models generated by existing systems, sign coordinate deviations exceed 20%, and missed inspections exceed 15% in complex scenarios like curves and tunnels. This forces maintenance departments to invest significant manpower in data verification. Therefore, developing a roadside sign segmentation method that balances efficiency, precision, and adaptability has become a key breakthrough in advancing highway inspection from "digital archiving" to "intelligent decision-making."

[0008] Existing technologies, such as the method, device, and vehicle for detecting a signboard with publication number CN117671644A. The application discloses a method, device, and vehicle for detecting a signboard, the method comprising: obtaining target point cloud data corresponding to a target area through a laser radar of a vehicle, and obtaining camera visual data corresponding to the target area through a camera of the vehicle, the target area including at least one signboard; performing point cloud segmentation on the target point cloud data to obtain at least one three-dimensional bounding box corresponding to the target point cloud data; obtaining at least one two-dimensional visual detection box corresponding to the camera visual data; and obtaining the position of at least one signboard included in the target area based on the at least one three-dimensional bounding box and the at least one two-dimensional visual detection box. The present application obtains the position of the signboard included in the target area based on the three-dimensional point cloud data corresponding to the target area obtained by the laser radar and the two-dimensional camera visual data corresponding to the target area obtained by the camera, thereby improving the accuracy of determining the position of the signboard.

[0009] Existing technical solutions basically meet the requirements, but there are still certain shortcomings. First, the existing fast segmentation method directly uses the segmentation algorithm for segmentation after completing point cloud data processing, which may lead to incomplete sign information extraction; second, the existing fast segmentation method is limited to the detection of signboards and the segmentation of mileage signs, and the segmentation efficiency and accuracy are low, which affects the speed and accuracy of sign information extraction; third, the model performance of the existing fast segmentation method in the point cloud segmentation task may be poor, which affects the efficiency and accuracy of the segmentation algorithm calculation. Summary of the Invention

[0010] In order to solve the defects in the existing technology, the present invention discloses a roadside sign recognition method based on lightweight semantic segmentation of drone 3D point cloud. The technical solution is as follows:

[0011] A method for rapid recognition of roadside signs based on lightweight semantic segmentation of UAV 3D point clouds is characterized by comprising the following steps:

[0012] Step 1: Adaptive flight control and point cloud acquisition: Build a flight control model that integrates visual perception and target density feedback. Combined with thermal maps and point cloud statistics, the model optimizes flight altitude, angle, and speed. Dynamically adjust the acquisition path without increasing the total mission route length, improving coverage of the sign area and edge target quality.

[0013] Step 2: Point cloud adaptive enhancement: Use RSPAE algorithm based on local density ρ i With curvature c i Calculate the enhancement priority E i , fine-tuning the local geometry of key areas and keeping the point cloud scale constant, improving the geometric clarity of key structure points and suppressing low-confidence areas.

[0014] Step 3: Multi-channel image generation: Convert the enhanced point cloud into a depth map D, an intensity map I, and a local depth variance map σ 2 The three-channel fusion image I f , to preserve spatial depth, material reflectivity and local structural stability for subsequent neural network processing.

[0015] Step 4: Image-level feature extraction: Fusion image I f A two-branch neural network is input, including a CNN branch and a Mamba branch, which respectively extract local geometric details and global semantic context, and completes spatial distribution modeling through four-stage feature extraction.

[0016] Step 5: Feature fusion and alignment enhancement: The features extracted by the two paths are input into the global fusion module (GFM) and the cross-feature fusion module (CFM). The former adjusts and enhances the channel weights, and the latter performs context compensation through the Transformer attention structure and outputs the final multi-scale aligned feature representation F'.

[0017] Step 6: The lightweight decoder restores the semantic map: The fused feature F' is input into the two-layer decoder, which completes upsampling, convolution and prediction, and outputs the original resolution semantic segmentation map M, ensuring accuracy while meeting the requirements of embedding platform deployment.

[0018] Step 7: Semantic back-projection and geo-registration: Back-project the semantic segmentation map M back to the original point cloud coordinates, combine it with the UAV flight pose matrix T(t) to complete geo-registration, and minimize the error based on the objective function Optimize registration accuracy and fuse neighboring frames to generate redundant suppression results

[0019] Step 8: Status Scoring and Classification: Input Enter the scoring and identification module, the category is It turns out that the state vector The score R is generated by the MLP scorer i , and combined with the Mahalanobis distance D i Perform self-supervised anomaly recognition.

[0020] Beneficial effects

[0021] 1: RSPAE point cloud adaptive enhancement algorithm

[0022] oCorresponding bottleneck: Solve the problem of "fixed voxelization leading to loss of small structures" in the preprocessing stage.

[0023] Innovation: No new points are introduced. Instead, structural points are selected based on RSPAE and locally smoothed, improving geometric discrimination while maintaining the same scale as the point cloud.

[0024] 2: Mamba+CNN dual-path lightweight network structure

[0025] oCorresponding bottleneck: imbalance between segmentation model accuracy and efficiency.

[0026] Innovation: Mamba introduces state-space modeling, constructs a linearly complex long-dependency extraction structure, and combines it with CNN to achieve multi-scale modeling, supporting operation on Jetson-class devices.

[0027] 3: State recognition module based on score and Mahalanobis distance

[0028] oCorresponding bottlenecks: insufficient cross-scale information integration and lack of fuzzy and outlier identification.

[0029] Innovation: Combine image and point cloud attributes into a scoring vector, predict the state through MLP, and introduce complete abnormal marking to achieve automatic identification of occlusion / blur / damage. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the process of the present invention;

[0031] Figure 2 This is the overall network architecture diagram of the present invention;

[0032] Figure 3 The overall architecture of GFM and CFM of the present invention;

[0033] Figure 4 The decoder architecture of the present invention;

[0034] Figure 5 This is a schematic diagram of the test scenario of Example 1 of the present invention;

[0035] Figure 6 This is a schematic diagram of the test scenario of Example 2 of the present invention. DETAILED DESCRIPTION

[0036] See also Figure 1-4 A method for fast segmentation of roadside signs based on UAV 3D point cloud data, characterized by comprising the following steps:

[0037] Step 1: Adaptive Flight Control and 3D Point Cloud Capture. A flight control model is constructed that integrates visual perception and target density feedback. Flight altitude, angle, and speed are optimized based on heatmap and point cloud statistics. The acquisition path is dynamically adjusted without increasing the total mission length, improving coverage of the sign area and edge target quality.

[0038] This step builds a joint model for road perception and flight control, enabling the drone to autonomously adjust its altitude, heading, and speed based on the complexity of the road environment and the distribution of visual targets, ensuring efficient perception and coverage of road signs. This mechanism, unlike traditional path planning that uses a fixed altitude or trajectory, effectively improves the quality of capturing edge targets.

[0039] The joint road perception and flight control model is a collaborative modeling mechanism between the visual perception module and the flight control module. Its core concept is to dynamically introduce the importance feedback of the visual attention area in flight control to achieve task-driven trajectory adjustment. The model consists of the following components:

[0040] a. Visual perception module. Used to process image data or projected point cloud data and generate area detection heat maps. The construction of this module includes the following sub-processes:

[0041] The input is the depth image D(u,v) and the intensity image I(u,v), which are respectively obtained by projecting the 3D point cloud P;

[0042] Feature extraction network f feat Extract edge, shape, brightness and other information from images;

[0043] Attention estimation network f aat Mapping features to heatmaps in Represents the visual saliency score at position (x,y).

[0044] b. Target density estimation module. This module is used to construct the target distribution function D(x,y) within the region based on historical point cloud statistics. The construction of this module includes:

[0045] Set a historical target set

[0046] Construct spatial distribution using density estimation function with Gaussian kernel: where δ(·) represents the two-dimensional Gaussian kernel function and σ is the kernel bandwidth.

[0047] c. Flight cost function. Defined as:

[0048] d. Joint optimization objective. Defined as: min h,θ,v J(h,θ,v)where:

[0049] h: flight altitude;

[0050] θ: flight heading angle;

[0051] v: flight speed;

[0052] Ω: scheduled flight coverage area;

[0053] · Regional detection heat map;

[0054] D(x,y): spatial target density function.

[0055] The drone collects 3D point cloud data from the lidar and depth camera to generate a point cloud collection: And record the flight pose matrix: in:

[0056] ·P i : The i-th point in the point cloud, including its spatial coordinates;

[0057] R(t): The attitude rotation matrix of the drone at time t;

[0058] t(t): the three-dimensional translation position vector of the UAV at time t;

[0059] SE(3): Special Euclidean group, representing the set of rigid body transformations (rotations and translations) in three-dimensional space.

[0060] Step 2: Point cloud adaptive enhancement: Use RSPAE algorithm based on local density ρ i With curvature c i Calculate the enhancement priority E i , fine-tuning the local geometry of key areas and keeping the point cloud scale constant, improving the geometric clarity of key structure points and suppressing low-confidence areas.

[0061] The original point cloud is first preprocessed by the innovative lightweight adaptive geometric enhancement algorithm RSPAE to generate an enhanced point cloud set P'={P'1,P'2,…,P' n}, which is used to improve the clarity of local structures and keep the overall number of points unchanged, thereby achieving geometric feature optimization without introducing additional computational burden.

[0062] The RSPAE algorithm includes the following steps:

[0063] a. Local density calculation: in:

[0064] · Click P i k-neighborhood of ;

[0065] k represents the number of domain points, the recommended range is [10,50]

[0066] ·ρ i : represents point P i The average neighborhood distance of , that is, the sparsity of the local point cloud.

[0067] b. Curvature estimation: in:

[0068] λ0,λ1,λ2: point P i Eigenvalues of the neighborhood covariance matrix, λ0 is the minimum eigenvalue;

[0069] c i : represents point P i The degree of geometric mutation is in the range of [0,1]. The closer to 1, the stronger the mutation, and the closer to 0, the local flatness.

[0070] c. Enhanced priority calculation: in:

[0071] ·E i :Click P i Enhanced priority scoring;

[0072] ·ρ min ,ρ max : global minimum and maximum density;

[0073] c min ,c max : global minimum and maximum curvature;

[0074] ·α∈[0,1]: weighting coefficient between density and curvature. A value of 0.5 can achieve equal-weighted fusion.

[0075] d. Structural enhancement mechanism: All points are transformed according to E i Sort and select the top m% points P i The local coordinate reconstruction is performed in the neighborhood of , using the following update method: in:

[0076] w ij : Normalized weight factor, reflecting the importance of neighborhood points;

[0077] ·δ i : Geometric enhancement offset, applied only to important structural points;

[0078] m: The ratio of enhancement points to be selected. The recommended range is [10, 40].

[0079] RSPAE does not introduce new points; it only fine-tunes key point positions, ensuring that the enhanced point cloud set P' is of equal size to the original set P, i.e., |P'| = |P|. Because enhancement occurs only in high-scoring areas and a sparse matrix is used to calculate neighborhood means, the overall computational complexity is linearly related to the original point cloud. This improves geometric discrimination while maintaining the efficiency of the original network, making it suitable for deployment in resource-constrained embedded systems.

[0080] The enhanced point cloud P' is used to generate the subsequent three-channel fusion image Used for further feature extraction and network processing.

[0081] Step 3: Multi-channel image generation: Convert the enhanced point cloud into a depth map D, an intensity map I, and a local depth variance map σ 2 The three-channel fusion image I f , to preserve spatial depth, material reflectivity and local structural stability for subsequent neural network processing.

[0082] The point cloud is converted into a three-channel fused projection image to enhance the neural network's comprehensive perception of spatial structure and target texture.

[0083] The three-channel image Also includes:

[0084] Depth map D(u,v): records the depth information of the minimum Euclidean distance in the point cloud;

[0085] Intensity map I(u,v): records the reflection intensity information of the point cloud;

[0086] Local depth variance map σ D (u,v): used to enhance the perception of geometric deformation, edge structure and object mutation.

[0087] The construction method of each channel is as follows:

[0088] a. Depth map construction: in:

[0089] π(·): projection function from point cloud to image pixel coordinates;

[0090] ·||P i ||: Point P i Euclidean distance to the origin.

[0091] b. Intensity map construction: in:

[0092] -I i :Click P i The reflection intensity value.

[0093] c. Local depth variance map: Propose an adaptive estimation of local depth variance σ D (u,v), used to dynamically describe the complexity of local depth changes, and its calculation method is: in:

[0094] · A local area with a window size of r×r centered at (u,v);

[0095] μ r (u,v): area The mean of the inner depth values;

[0096] r: The window size is automatically adjusted based on the image content, set according to the image gradient or edge density, and is set to: in represents the gradient response of the depth map, β is the magnification coefficient, and r0 is the base window.

[0097] Through this improvement, σ D (u,v) can more accurately reflect surface mutations and the complexity of edge junctions, significantly improving the network's ability to discriminate structurally sensitive areas. It is especially suitable for edge detection and small target extraction tasks.

[0098] The final fused image is: Where H and W are the height and width of the image, and the output is used as the input of the neural network to support multi-scale geometric feature learning and enhance perception capabilities.

[0099] The fused image The input is fed into a four-stage, two-parallel feature extraction network, which includes a convolutional neural network (CNN) path and a Mamba path for state-space modeling. The two paths are used to extract local geometric structure features and long-range semantic dependency features in the image, respectively, thereby jointly modeling texture details and contextual information.

[0100] The CNN path consists of four stages (Stage 1-4):

[0101] Stage 1: A two-dimensional convolutional module (Conv2D) followed by batch normalization (BN) and ReLU activation with input channels C, used to initially extract local point cloud features.

[0102] Stage 2: Residual structure module, including downsampling operation (Stride = 2), to achieve spatial dimension compression and enhance feature expression capability;

[0103] Stage 3: Atrous convolution module with a kernel size of 3×3×3 and a dilation ratio of r=2, which is used to expand the receptive field to capture mid-scale spatial relationships.

[0104] Stage 4: Global average pooling and fully connected (FC) layers, outputting local feature representations:

[0105]

[0106] Among them, F CNN represents the local features, N represents the number of valid pixels in the image (after flattening), and d1 represents the number of channels of the CNN branch.

[0107] The Mamba Path also consists of four stages:

[0108] Stage 1: Patch Embedding module, which groups and embeds point cloud feature blocks to form a sequence structure input;

[0109] Stage 2: Mamba Block, which uses a sliding window approach to build state space sequence modeling in a local area and obtain cross-point feature dependencies;

[0110] Stage 3: Token Mixing module, which enables cross-channel and cross-patch information mixing to enhance global perception;

[0111] Stage 4: Normalization and linear projection layer, outputting the final semantic embedding features:

[0112]

[0113] Where: d2 represents the output dimension of the Mamba branch; F Mamba : Feature matrix output by Mamba path;

[0114] Finally, the output features of the two paths are concatenated in the channel dimension to form a fusion representation:

[0115]

[0116] Concat(·) represents the concatenation operation in the feature channel dimension, which is used to fuse local details with global context features. F represents the fused feature.

[0117] Step 4: Image-level feature extraction: Fusion image I f A two-branch neural network is input, including a CNN branch and a Mamba branch, which respectively extract local geometric details and global semantic context, and completes spatial distribution modeling through four-stage feature extraction.

[0118] In step 4, the fused feature F is input into the global feature fusion module (GFM) and the cross-feature fusion module (CFM) to achieve multi-scale alignment and context-aware modeling at the channel level, respectively.

[0119] in:

[0120] The GFM module takes the output of CNN stage 3 and Mamba stage 3 as input. The module structure includes:

[0121] a. Feature splicing unit: receives output features from two paths and Combine the channel dimensions to obtain fusion features

[0122] b. Channel attention estimation submodule: For each F i Applying a fully connected layer Calculate the attention coefficient:

[0123] c. Reconstruct the output unit: use γ i F i Weighted:

[0124] in:

[0125] · Represents the output features of the CNN 3rd stage, with a size of

[0126] · represents the output features of Mamba stage 3, with size

[0127] ·F i : represents the fused feature vector after channel splicing;

[0128] W: fully connected weight matrix;

[0129] γ i : The normalized weight coefficient indicating the importance of the channel;

[0130] · Global enhanced features output by the GFM module.

[0131] The CFM module takes the output of the CNN stage 2 and the Mamba stage 4 as input. The module structure includes:

[0132] a. Feature splicing unit: input and Splicing

[0133] b. Linear projection layer: define the projection matrix separately Generate: Q i =W Q F i ,K j =W K F j ,V j =W V F j

[0134] c. Attention calculation layer: Use the scaled dot-product attention mechanism to construct context features:

[0135]

[0136] in:

[0137] · Represents the output features of the CNN second stage, with a size of

[0138] · Represents the output features of the fourth stage of Mamba, with size

[0139] ·F i : The fused feature vector after splicing;

[0140] W Q ,W K ,W V : Linear projection matrices for query, key, and value respectively;

[0141] Q i ,K j ,V j : are the query, key, and value vectors at pixel i and j, respectively;

[0142] ·d a : is the low-dimensional mapping dimension of the attention module, used to reduce the amount of computation, and can be set to 32, 64 or 128, etc.

[0143] · Context-enhanced features output by the CFM module.

[0144] The final fusion output is: in:

[0145] ·F' i : The final fusion feature of each pixel position;

[0146] F': The feature output after the fusion of GFM and CFM, with channel attention and context information, serves as the input basis for subsequent semantic segmentation or detection modules;

[0147] d1+d2: represents the channel dimension of the final fused feature.

[0148] Step 5: Feature fusion and alignment enhancement: The features extracted by the two paths are input into the global fusion module (GFM) and the cross-feature fusion module (CFM). The former adjusts and enhances the channel weights, and the latter performs context compensation through the Transformer attention structure and outputs the final multi-scale aligned feature representation F'.

[0149] The fused feature representation F' is input into a lightweight decoder structure suitable for embedded platforms to generate a semantic segmentation map with the same resolution as the original image.

[0150] The lightweight decoder includes two layers of structure:

[0151] 1. Decoding layer 1 (upsampling + convolution module):

[0152] Fusion features Upsample to an intermediate resolution of H / 2×W / 2;

[0153] Followed by a 2D convolution operation (Conv2D, with a kernel size of 3×3 and output channels of C1), and applying batch normalization (BN) and ReLU activation function:

[0154]

[0155] 2. Decoding layer 2 (upsampling + convolution + prediction layer):

[0156] ·To F (up1) Further upsample to the original image size H×W;

[0157] After a 1×1 convolution operation, the number of channels is reduced to the number of categories C, and the final segmentation heat map (Logits) is output:

[0158] in:

[0159] F': represents the fused image feature tensor with dimensions H / 4 × W / 4 × (d1 + d2), derived from the outputs of GFM and CFM;

[0160] H, W: represents the height and width of the original input image, which are used to define the final output resolution;

[0161] C1: The number of channels in the intermediate feature map, used to maintain spatial structure and feature integrity, usually set to 64 or 128;

[0162] ·F (up1) : The intermediate feature map obtained after decoding the first layer has a size of H / 2×W / 2×C1;

[0163] C: represents the number of semantic segmentation categories, that is, the number of output channels, for example, C = 2 for binary classification;

[0164] M: represents the final semantic segmentation prediction map, which outputs a C-dimensional vector at each pixel position, corresponding to the predicted probability of each category (in the form of logits).

[0165] The decoder has a simple structure and only uses two-level upsampling and low-computation convolution operations. It can effectively maintain decoding accuracy while greatly reducing the model parameter scale and computational overhead. It is suitable for deployment in resource-constrained embedded devices or drone platforms to achieve real-time semantic segmentation.

[0166] Step 6: The lightweight decoder restores the semantic map: The fused feature F' is input into the two-layer decoder, which completes upsampling, convolution and prediction, and outputs the original resolution semantic segmentation map M, ensuring accuracy while meeting the requirements of embedding platform deployment.

[0167] The semantic segmentation image result M output by the decoder is mapped back to the original 3D point cloud coordinate system to achieve the fusion of semantic mask and point cloud, geo-registration and multi-frame redundancy suppression.

[0168] a. Back-projection fusion of image to point cloud:

[0169] - According to the projection relationship π -1 (u,v)→P i , maps the predicted label of the pixel coordinate (u,v) in the image to its corresponding point cloud point P i On top of , generate a semantic point cloud mask set: in:

[0170] · In the image segmentation output, the predicted probability value of the pixel (u, v) belonging to category c;

[0171] ·l i : Assigned to point cloud point P i The final semantic category label is the category with the maximum probability;

[0172] π -1 (u,v): Back-project the image coordinates to the 3D point cloud point P i The mapping function of

[0173] ·P i : the i-th point in the original 3D point cloud;

[0174] ·S i : The semantic point cloud set of the current frame, including P i With the corresponding label l i .

[0175] b. Geo-reference processing based on fusion error minimization strategy:

[0176] - Using the pose matrix T(t)∈SE(3) in the UAV flight navigation information, the local point cloud point P i Transform to global geographic coordinate system: Geo(P i )=proj(T(t)·P i ). The fusion error minimization mechanism is introduced and the registration loss function is designed as follows:

[0177] in:

[0178] T(t): The pose matrix of the drone at time t, representing rotation and translation, belonging to the special Euclidean group SE(3);

[0179] proj(·): projection function that converts three-dimensional coordinates into geographic coordinate systems (such as latitude, longitude, and height);

[0180] Geo(P i ): geographic coordinates of the point cloud points after T(t) registration;

[0181] · With P i Reference points matched in the map or previous frame;

[0182] Ω: The index set of overlapping area point clouds used for registration calculation;

[0183] · Fusion error loss function, used to optimize the accuracy of T(t).

[0184] c. Multi-frame redundancy suppression and completion mechanism:

[0185] - For redundant recognition results that may appear in adjacent frames, a non-maximum suppression (NMS) strategy based on intersection-over-union is adopted:

[0186]

[0187] Use the results of multiple frames within the time window for fusion and completion:

[0188] in:

[0189] ·S j : With the current frame S i Semantic point cloud results of adjacent frames;

[0190] ∪: represents "union";

[0191] IoU(S i ,S j ): Calculate the voxel intersection-over-union (IoU) index of the semantic regions of two point clouds;

[0192] δ: IoU threshold, which determines whether to retain or merge. It must be greater than 80% for implementation.

[0193] ·max(S i ,Sj): the maximum mask region selected by category confidence or area;

[0194] · The temporal neighborhood set contains the frame indices near the current frame i;

[0195] · The final output is a high-quality semantic point cloud result after multi-frame fusion and redundant suppression.

[0196] Step 7: Semantic back-projection and geo-registration: Back-project the semantic segmentation map M back to the original point cloud coordinates, combine it with the UAV flight pose matrix T(t) to complete geo-registration, and minimize the error based on the objective function Optimize registration accuracy and fuse neighboring frames to generate redundant suppression results

[0197] Semantic point cloud results after fusion and completion It is input into the status recognition and type classification module to realize the target category determination and integrity status assessment of the roadside sign.

[0198] a. Category prediction module:

[0199] - For each semantic point cloud region The semantic label results in the dataset are counted or aggregated to obtain the final predicted category:

[0200] in:

[0201] ·T i : represents the predicted category of the i-th target;

[0202] c: Category index (such as speed limit sign, direction sign, prohibition sign, etc.);

[0203] · area The proportion of pixels or points belonging to category c, weighted confidence value, or maximum response value;

[0204] argmax c :Indicates searching for all category indexes c such that The largest c value indicates that the prediction is the category with the highest probability.

[0205] 2. State scoring and self-supervised anomaly detection module:

[0206] First calculate the benchmark score vector: Use a multi-objective state scoring function (such as a lightweight neural network scorer) to predict state confidence: R i =MLP(v i )∈[0,1];

[0207] Introducing a self-supervised anomaly recognition mechanism based on Mahalanobis distance to detect anomaly signs from historical normal distribution: D i =(v i -μ) T ∑ -1 (v i-μ)A i =1(D ii >τ) where:

[0208] ·v i : represents the state vector of the current i-th sign;

[0209] I i : Image or point cloud intensity index, measuring clarity or reflective properties;

[0210] ·θ i : Shooting angle, reflecting the recognition impact caused by perspective offset;

[0211] · Current visible area (unit: m 2 );

[0212] · Theoretical complete area (unit: m 2 );

[0213] ·R i : The integrity score output by the scoring network. The larger the value, the more reliable it is.

[0214] μ: the mean of the historical normal state vector;

[0215] ∑: historical state covariance matrix;

[0216] ·D i : Mahalanobis distance, indicating the degree of difference between the current state and the historical distribution;

[0217] τ: the set anomaly detection threshold;

[0218] ·A i : Binary abnormality flag. If it is 1, it means the sign may be abnormal.

[0219] Step 8: Status Scoring and Classification: Input Enter the scoring and identification module, the category is It turns out that the state vector The score R is generated by the MLP scorer i , and combined with the Mahalanobis distance D i Perform self-supervised anomaly recognition.

[0220] Example 1

[0221] See also Figure 5 Test scenario: highway night scene, low point cloud density (average <50 points per square meter), dynamic acquisition blur and partial vegetation occlusion, the target is a speed limit sign (size 0.6m×0.6m).

[0222] Comparison results:

[0223]

[0224] See also Figure 6 As shown in the test scenario, light rain and haze increase point cloud noise (signal-to-noise ratio decreases by 40%) and reduces the sign's surface reflectivity. The construction warning sign, measuring 1.2m x 0.8m, is partially obscured by rain and fog scattering. The drone is equipped with an NVIDIA Jetson Xavier edge computing module.

[0225] Comparison results:

[0226]

[0227]

[0228] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the invention as claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A fast roadside sign recognition method based on lightweight semantic segmentation of UAV 3D point clouds, characterized by: The following steps are involved: Step 1: Adaptive flight control and point cloud acquisition: Build a flight control model that integrates a visual perception module with target density feedback. Based on the spatial target density function generated from the heat map and point cloud statistics, the acquisition path is dynamically adjusted by optimizing the flight altitude, heading angle, and flight speed. Step 2: Adaptive point cloud enhancement: Use the RSPAE algorithm to calculate the enhancement priority based on local density and curvature, and fine-tune the local geometry of key areas; Step 3: Multi-channel image generation: Convert the enhanced point cloud into a three-channel fused image consisting of a depth map, an intensity map, and a local depth variance map; Step 4: Image-level feature extraction: The fused image is input into a two-branch neural network, and local geometric features and global semantic context features are extracted through the CNN branch and the Mamba branch; Step 5: Feature fusion and alignment enhancement: The global fusion module adjusts the channel weights and performs context compensation through the Transformer attention structure across the feature fusion module to generate multi-scale aligned features. Step 6: The lightweight decoder restores the semantic map: the fused features are input into the two-layer decoder, which completes upsampling, convolution and prediction, and outputs the original resolution semantic segmentation map; Step 7: Semantic back-projection and geo-registration: Back-project the semantic segmentation map to the point cloud coordinate system, optimize the registration accuracy by combining the UAV pose matrix and the error minimization objective function, and fuse multiple frames to generate a redundant suppression result. Step 8: Status scoring and classification: Input the semantic point cloud results into the scoring and recognition module to determine the target category and assess the integrity status of the roadside sign.

2. The method for rapid roadside sign recognition based on lightweight semantic segmentation of drone-derived 3D point clouds according to claim 1 is characterized by: The visual perception module generates a heat map H(x,y)=faat(f feat (D, I)); the target density function D(x, y) is calculated based on the historical point cloud data by Gaussian kernel density estimation; the flight parameter optimization goal is to minimize the cost function defined as: Where: θ: flight heading angle; v: flight speed; Ω: planned flight coverage area; Regional detection heat map; D(x,y): spatial target density function.

3. The method for rapid roadside sign recognition based on lightweight semantic segmentation of drone-derived 3D point clouds according to claim 2 is characterized by: The RSPAE algorithm includes the following steps: Local density calculation: in: Click P i k neighborhood; k represents the number of points in the field, ranging from [10,50]; ρ i : represents point P i The average neighborhood distance of , that is, the sparsity of the local point cloud; Curvature estimation: Where: λ0,λ1,λ2: point P i The eigenvalues of the neighborhood covariance matrix, λ0 is the minimum eigenvalue; c i : represents point P i The degree of geometric mutation, the value range is [0,1]; Enhanced priority calculation: Where: E i :Click P i Enhanced priority score of min ,ρ max : global minimum and maximum density; c min ,c max : global minimum and maximum curvature; α∈[0,1]: weighting coefficient between density and curvature; Structural enhancement mechanism: All points are transformed according to E i Sort and select the top m% points P i The local coordinate reconstruction is performed in the neighborhood of , using the following update method: Where: w ij : Normalized weight factor, reflecting the importance of neighborhood points; δ i : Geometric enhancement offset, applied only to important structural points; m: enhancement point selection ratio, range [10,40].

4. The method for rapid roadside sign recognition based on lightweight semantic segmentation of drone-derived 3D point clouds according to claim 2 is characterized by: The pre-processed point cloud is converted into a three-channel fused projection image. It also includes: depth map D(u,v), which records the depth information of the minimum Euclidean distance in the point cloud; intensity map I(u,v), which records the reflection intensity information of the point cloud; local depth variance map σ D (u,v), used to enhance the perception of geometric deformation, edge structure and object mutation; The construction method of each channel is as follows: Depth map construction: Where: π(·): projection function from point cloud to image pixel coordinates; ||P i ||: Point P i Euclidean distance to the origin; Intensity map construction: Among them: -I i :Click P i The reflection intensity value; Local Depth Variance Map: Propose an adaptive estimation of local depth variance σ D (u,v), used to dynamically describe the complexity of local depth changes, and its calculation method is: in: A local area with a window size of r×r centered at (u,v); μ r (u,v): area The mean of the inner depth values; r: The window scale is automatically adjusted according to the image content and is set according to the image gradient or edge density. It is set to: in represents the gradient response of the depth map, β is the magnification coefficient, and r0 is the base window.

5. The method for rapid roadside sign recognition based on lightweight semantic segmentation of drone-derived 3D point clouds according to claim 2 is characterized by: The fused image The input is fed into a four-stage, two-parallel feature extraction network, which includes a convolutional neural network (CNN) pathway and a state-space modeling Mamba pathway. The two pathways are used to extract local geometric structure features and long-range semantic dependency features in the image, respectively, thereby jointly modeling texture details and contextual information. Among them, the CNN path consists of four stages: Stage 1: The two-dimensional convolution module Conv2D is followed by batch normalization (BN) and ReLU activation, with an input channel of C, for preliminary extraction of local point cloud features; Stage 2: Residual structure module, including downsampling operations, to achieve spatial dimension compression and enhance feature expression capabilities; Stage 3: Dilated convolution module with a kernel size of 3×3×3 and a dilation ratio of r=2, which is used to expand the receptive field to capture mid-scale spatial relationships; Stage 4: Global average pooling and fully connected FC layer, outputting local feature representation: Among them, F CNN represents the local features, N represents the number of valid pixels in the image, and d1 represents the number of channels of the CNN branch.

6. The method for rapid roadside sign recognition based on lightweight semantic segmentation of drone-derived 3D point clouds according to claim 2 is characterized by: The fused feature representation F' is input into a lightweight decoder structure suitable for embedded platforms to generate a semantic segmentation map consistent with the original image resolution; The lightweight decoder includes two layers of structure: Decoding layer 1: upsampling + convolution module The fusion features Upsampling is performed to an intermediate resolution of H / 2×W / 2; Followed by a 2D convolution operation and applying batch normalization and ReLU activation function: Decoding layer 2: upsampling + convolution + prediction layer F (up1) Further upsample to the original image size H×W; After a 1×1 convolution operation, the number of channels is reduced to the number of categories C, and the final segmentation heat map is output: in: F': represents the fused image feature tensor with dimensions of H / 4×W / 4×(d1+d2), which comes from the output of GFM and CFM; H, W: represents the height and width of the original input image, which is used to define the final output resolution; C1: The number of channels of the intermediate feature map, used to maintain spatial structure and feature integrity, usually set to 64 or 128; F (up1) : The intermediate feature map obtained after decoding the first layer has a size of H / 2×W / 2×C1; C: represents the number of semantic segmentation categories, that is, the number of output channels, for example, C = 2 for binary classification; M: represents the final generated semantic segmentation prediction map, and each pixel position outputs a C-dimensional vector corresponding to the predicted probability of each category.

7. The method for rapid roadside sign recognition based on lightweight semantic segmentation of drone 3D point clouds according to claim 2 is characterized by: The semantic segmentation image result M output by the decoder in step 6 is mapped back to the original 3D point cloud coordinate system to achieve the fusion of semantic mask and point cloud, geo-registration and multi-frame redundancy suppression; a. Back-projection fusion from image to point cloud: According to the projection relationship π -1 (u,v)→P i , maps the predicted label of the pixel coordinate (u,v) in the image to its corresponding point cloud point P i On top of , generate a semantic point cloud mask set: in: In the image segmentation output, the predicted probability value of the pixel (u, v) belonging to category c; l i : Assigned to point cloud point P i The final semantic category label is the category with the maximum probability; π -1 (u,v): Back-project the image coordinates to the 3D point cloud point P i The mapping function of P i : the i-th point in the original 3D point cloud; S i : The semantic point cloud set of the current frame, including P i With the corresponding label l i ; b. Geo-reference processing based on fusion error minimization strategy: Using the pose matrix T(t)∈SE(3) in the UAV flight navigation information, the local point cloud point P i Transform to global geographic coordinate system: Geo(P i )=proj(T(t)·P i ); introduce the fusion error minimization mechanism and design the registration loss function as follows: in: T(t): The pose matrix of the drone at time t, representing rotation and translation, belonging to the special Euclidean group SE(3); proj(·): projection function that converts three-dimensional coordinates into geographic coordinate systems (such as latitude, longitude, and height); Geo(P i ): geographic coordinates of the point cloud points after T(t) registration; With P i Reference points matched in the map or previous frame; Ω: The index set of overlapping area point clouds used for registration calculation; Fusion error loss function, used to optimize the accuracy of T(t); c. Multi-frame redundancy suppression and completion mechanism: For the redundant recognition results that may appear in adjacent frames, the non-maximum suppression (NMS) strategy based on intersection-over-union is adopted: Use the results of multiple frames within the time window for fusion and completion: in: S j : With the current frame S i Semantic point cloud results of adjacent frames; ∪: means "take the union"; IoU(S i ,S j ): Calculate the voxel intersection-over-union (IoU) index of the semantic regions of two point clouds; δ: loU threshold, which determines whether to retain or merge, and must be greater than 80% for implementation; max(S i ,S j ): the maximum mask region selected by category confidence or area; The temporal neighborhood set contains the frame indices near the current frame i; The final output is a high-quality semantic point cloud result after multi-frame fusion and redundant suppression.

8. The method for rapid roadside sign recognition based on lightweight semantic segmentation of drone 3D point clouds according to claim 2 is characterized by: The semantic point cloud result after fusion and completion in step 7 The data is input into the state recognition and type classification module to determine the target category and evaluate the integrity of the roadside sign; the category prediction module: For each semantic point cloud region The semantic label results in the dataset are counted or aggregated to obtain the final predicted category: in: T i : represents the predicted category of the i-th target; c: category index; area The proportion of pixels or points belonging to category c, weighted confidence value, or maximum response value; arg max c :Indicates searching for all category indexes c such that The largest c value is the class with the highest probability of being predicted; State scoring and self-supervised anomaly detection module: First calculate the benchmark score vector: Use a multi-objective state scoring function (such as a lightweight neural network scorer) to predict state confidence: R i =MLP(v i )∈[0,1]; Introducing a self-supervised anomaly recognition mechanism based on Mahalanobis distance to detect anomaly signs from historical normal distribution: D i =(v i -μ) T ∑ -1 (v i -μ)A i =1(D i >τ) where: v i : represents the state vector of the current i-th sign; I i : Image or point cloud intensity index, measuring clarity or reflective properties; θ i : Shooting angle, reflecting the recognition impact caused by perspective offset; Current visible area, unit: m 2 ; Theoretical complete area, unit: m 2 ; R i : The integrity score output by the scoring network. The larger the value, the more reliable it is. μ: mean of the historical normal state vector; ∑: historical state covariance matrix; D i : Mahalanobis distance, indicating the degree of difference between the current state and the historical distribution; τ: set anomaly detection threshold; A i : Binary abnormality flag. If it is 1, it means the sign may be abnormal.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the method according to claim 1.

10. An electronic device, characterized in that: The method comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is configured to execute the computer-readable instructions, wherein the computer-readable instructions execute the method according to claim 1 when executed.

Images