Collected image recognition method and system for surveying and mapping unmanned aerial vehicle

Abstract

The invention discloses a collected image recognition method and system for a surveying and mapping unmanned aerial vehicle, surveying and mapping image data and surveying and mapping point cloud data of a target surveying and mapping area are obtained through a multi-mode sensor carried by the unmanned aerial vehicle, and meanwhile, a time camera is used for capturing a motion signal of a dynamic target to generate event stream data; histogram equalization and multi-scale enhancement are carried out on the surveying and mapping image data, and voxel filtering and curved surface reconstruction are carried out on the surveying and mapping point cloud data; wavelet packet decomposition is utilized to extract frequency domain features of an image in the preprocessed acquisition data, and the frequency domain features are fused with motion trail features in event stream data; and inputting the mixed feature vector into a lightweight adaptive model, dynamically adjusting weights of different modes through an attention mechanism, and outputting a surveying and mapping result of the target surveying and mapping area. The information of the image in different frequencies and directions can be described more comprehensively and meticulously, and the surveying and mapping efficiency is effectively improved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of surveying and mapping data processing, in particular to a surveying and mapping unmanned aerial vehicle image acquisition and recognition method and system. BACKGROUND

[0002] With the wide application of unmanned aerial vehicles in the field of surveying and mapping, the traditional image recognition method faces the following challenges: poor adaptability to complex environments: traditional algorithms are sensitive to problems such as light changes, terrain occlusions and dynamic target interference, resulting in a decrease in recognition accuracy. Insufficient multi-view data fusion: single-view images easily lose key information, making it difficult to meet the high-precision surveying and mapping demand. Real-time performance and energy consumption conflict: existing methods rely on high-performance hardware, resulting in short endurance time and inability to meet long-term task requirements. Weak dynamic target recognition capability: insufficient trajectory prediction and feature extraction capability for moving targets (vehicles, crowds). The processing scheme in the prior art generally has weak model generalization capability, large calculation overhead and low multi-source data fusion efficiency. SUMMARY

[0003] The purpose of the present application is to solve the above problems, and a surveying and mapping unmanned aerial vehicle image acquisition and recognition method and system are designed.

[0004] To achieve the above purpose, the technical scheme of the present application is as follows: further, in the surveying and mapping unmanned aerial vehicle image acquisition and recognition method, the surveying and mapping unmanned aerial vehicle image acquisition and recognition method comprises the following steps: acquiring surveying and mapping image data and surveying and mapping point cloud data of a target surveying and mapping area by a multi-modal sensor carried by an unmanned aerial vehicle, and simultaneously capturing motion signals of dynamic targets by a time camera to generate event stream data; performing histogram equalization and multi-scale Retinex enhancement on the surveying and mapping image data, and performing voxel filtering and surface reconstruction on the surveying and mapping point cloud data to obtain preprocessed acquisition data; extracting frequency domain features of the images in the preprocessed acquisition data by wavelet packet decomposition, and fusing the motion trajectory features in the event stream data to generate a hybrid feature vector; inputting the hybrid feature vector into a MobileNetV3+Transformer lightweight adaptive model, dynamically adjusting the weights of different modalities by an attention mechanism, outputting a surveying and mapping result of the target surveying and mapping area, and generating a surface change heat map based on the surveying and mapping result.

[0005] Further, in the surveying and mapping unmanned aerial vehicle image acquisition and recognition method, the histogram equalization and multi-scale Retinex enhancement of the surveying and mapping image data comprises: performing channel histogram equalization on the RGB images in the surveying and mapping image data by adaptive histogram equalization to obtain equalized images; By separating the illumination component and the reflection component of the equalized image, and enhancing details and suppressing noise, a component image is obtained; A Gaussian blur of three different scales is performed on the component image to generate a multi-scale illumination estimation, and an illumination estimation image is obtained; The original image is divided by the illumination estimation image to obtain a reflection component, and a logarithmic transformation is performed on the enhanced image to obtain preprocessed image data.

[0006] Further, in the above-mentioned image acquisition recognition method for surveying and mapping unmanned aerial vehicles, the voxel filtering and surface reconstruction of the surveying and mapping point cloud data to obtain the preprocessed acquisition data further comprises: Based on the VoxelGridFiltering voxel filtering algorithm, the voxel edge length is set according to the task; relatively coarse voxels are used in flat terrain areas, and relatively fine voxels are used in complex areas, for adaptive downsampling; The 3σ criterion of statistical filtering is combined to remove outliers, and the average or median of points in each voxel is used to replace the original points to obtain filtered surveying and mapping data.

[0007] Further, in the above-mentioned image acquisition recognition method for surveying and mapping unmanned aerial vehicles, the voxel filtering and surface reconstruction of the surveying and mapping point cloud data to obtain the preprocessed acquisition data further comprises: According to the MLS moving least squares method, neighboring points within a spherical neighborhood with a radius r=0.5m are extracted for each point cloud point to construct a local plane equation; Based on the Gaussian weight function, the optimal fitting plane of the neighborhood points is calculated to correct the coordinates of the current point, gradually smooth the point cloud surface, and eliminate error data; The PCA principal component analysis is used to calculate the covariance matrix of the neighborhood point cloud, extract the minimum feature vector as the normal vector, and obtain the normal vector surveying and mapping data.

[0008] Further, in the above-mentioned image acquisition recognition method for surveying and mapping unmanned aerial vehicles, the frequency domain features of the preprocessed acquisition data are extracted using wavelet packet decomposition, and the motion trajectory features in the event stream data are fused to generate a hybrid feature vector, comprising: The Symlets wavelet function is used to perform wavelet packet decomposition on each channel of the RGB enhanced image respectively; According to the image resolution, the decomposition layer number is selected, the sub-band energy vectors of the RGB channels are spliced according to the weight, the feature dimension is reduced through LDA linear discriminant analysis, and is normalized to the range of [0, 1] to obtain the image frequency domain feature vector.

[0009] Further, in the above-mentioned image acquisition and recognition method for surveying and mapping unmanned aerial vehicles, the frequency domain features of the image in the preprocessed acquisition data are extracted by wavelet packet decomposition, and the motion trajectory features in the event stream data are fused to generate a hybrid feature vector, including: The event stream is divided into a dynamic target area by using a DBSCAN clustering algorithm, the target trajectory is reconstructed based on a time window sliding method, and a motion feature vector is calculated; The image frequency domain feature vector and the motion feature vector are spliced, the weights of the two features are dynamically adjusted by an attention mechanism, the motion features are mapped to the image space to generate a hybrid feature vector.

[0010] Further, in the above-mentioned image acquisition and recognition method for surveying and mapping unmanned aerial vehicles, the surveying and mapping result generation module is configured to input the hybrid feature vector into a MobileNetV3+Transformer lightweight adaptive model, dynamically adjust the weights of different modalities by an attention mechanism, output the surveying and mapping result of the target surveying and mapping area, and generate a ground surface change heat map based on the surveying and mapping result, including: The hybrid feature vector is divided into 8 heads, each head independently calculates a query, key and value matrix, and an attention weight is generated by Softmax normalization; A bidirectional attention mechanism is introduced in the Transformer encoder, so that the image features and the motion trajectory features influence each other, and the surveying and mapping result of the target surveying and mapping area is output.

[0011] Further, in an image acquisition and recognition system for surveying and mapping unmanned aerial vehicles, the image acquisition and recognition system for surveying and mapping unmanned aerial vehicles comprises the following modules: A surveying and mapping data acquisition module is configured to acquire surveying and mapping image data and surveying and mapping point cloud data of a target surveying and mapping area by a multi-modal sensor carried by an unmanned aerial vehicle, and generate event stream data by capturing motion signals of dynamic targets using a time camera; A surveying and mapping data processing module is configured to perform histogram equalization and multi-scale Retinex enhancement on the surveying and mapping image data, and perform voxel filtering and surface reconstruction on the surveying and mapping point cloud data to obtain preprocessed acquisition data; A hybrid feature generation module is configured to extract frequency domain features of the image in the preprocessed acquisition data by wavelet packet decomposition, and fuse the motion trajectory features in the event stream data to generate a hybrid feature vector; A surveying and mapping result generation module is configured to input the hybrid feature vector into a MobileNetV3+Transformer lightweight adaptive model, dynamically adjust the weights of different modalities by an attention mechanism, output the surveying and mapping result of the target surveying and mapping area, and generate a ground surface change heat map based on the surveying and mapping result.

[0012] Further, in a kind of surveying and mapping unmanned plane's acquisition image recognition system, the surveying and mapping data acquisition module includes following submodule: Setting submodule is used to set voxel length according to task based on VoxelGridFiltering voxel filtering algorithm;Adaptive downsampling is carried out using coarser voxel in flat terrain area and finer voxel in complex area; Filtering submodule is used to remove outlier by combining 3 sigma criterion of statistical filtering, and the average value or median of each voxel point is replaced by original point, to obtain filtered surveying and mapping data.

[0013] Further, in a kind of surveying and mapping unmanned plane's acquisition image recognition system, the surveying and mapping data acquisition module further includes following submodule: Construction submodule is used to extract adjacent points in spherical neighborhood with radius r=0.5m according to MLS mobile least square method for each point cloud point, to construct local plane equation; Smoothing submodule is used to calculate optimal fitting plane of neighborhood points based on Gaussian weight function, to correct the coordinates of current point, and to gradually smooth point cloud surface to eliminate error data; Obtaining submodule is used to calculate covariance matrix of neighborhood point cloud by PCA principal component analysis, to extract minimum feature vector as normal vector, and to obtain normal vector surveying and mapping data.

[0014] The beneficial effects are that 1, the multi-modal data of unmanned plane can be adaptively fused according to scene requirements, the adaptability of model to complex scene and the recognition accuracy are improved. The final output surveying and mapping result covers rich information such as three-dimensional terrain model and feature recognition, and generates intuitive surface change heat map based on this, which provides accurate, reliable and high-visualization degree decision basis for geological monitoring, urban planning and other fields, and effectively improves the efficiency of surveying and mapping. 2, can more comprehensively and meticulously describe the information of image in different frequencies and directions;Motion trajectory features are extracted from event stream data, and motion parameters of dynamic target are accurately obtained by combining Kalman filtering algorithm. 3, effectively improve the contrast of image, suppress the influence of uneven illumination, make the image details clearer, and enhance the readability and analyzability of image. BRIEF DESCRIPTION OF DRAWINGS

[0015] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments with reference made to the accompanying drawings. The drawings are for purposes of illustration only and are not considered as limiting the application.

[0016] Figure 1 It is a first embodiment schematic diagram of a kind of surveying and mapping unmanned plane's acquisition image recognition method in the embodiment of the application; Figure 2It is a second embodiment schematic view of a kind of collection image recognition method for surveying and mapping unmanned aerial vehicle in the embodiment of the present application; Figure 3 It is a first embodiment schematic view of a kind of collection image recognition system for surveying and mapping unmanned aerial vehicle in the embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical scheme and advantage of the present application more clear, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0018] Those skilled in the art can understand that, unless specifically stated, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the use of the word "comprise" in the specification of the present application means that the features, integers, steps, operations, elements and / or components exist, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0019] The present application will be described in detail below with reference to the drawings, Figure 1 As shown, a collection image recognition method for surveying and mapping unmanned aerial vehicle, the collection image recognition method for surveying and mapping unmanned aerial vehicle includes the following steps: Step 101, obtain the surveying and mapping image data and surveying and mapping point cloud data of the target surveying and mapping area by the multi-modal sensor carried by the unmanned aerial vehicle, and simultaneously capture the motion signal of the dynamic target by using the time camera to generate event stream data; Specifically, in the present embodiment, the RGB image in the surveying and mapping image data is subjected to channel histogram equalization by using adaptive histogram equalization, to obtain an equalized image; By separating the illumination component and the reflection component of the equalized image, and enhancing details and suppressing noise, a component image is obtained; The component image is subjected to Gaussian blur of three different scales to generate multi-scale illumination estimation, to obtain an illumination estimation image; The original image is divided by the illumination estimation image to obtain the reflection component, and the enhanced image is subjected to logarithmic transformation to obtain preprocessed image data.

[0020] According to the task, the VoxelGridFiltering voxel filtering algorithm sets the voxel edge length; in the flat terrain area, relatively coarse voxel is used, and in the complex area, relatively fine voxel is used, to perform adaptive downsampling; Combined with the 3σ criterion of statistical filtering, outliers are removed, and the average value or median of points in each voxel is used to replace the original points to obtain filtered surveying and mapping data.

[0021] According to the MLS moving least squares method, the neighboring points within a spherical neighborhood with a radius r = 0.5 m are extracted for each point cloud point, and a local plane equation is constructed; Based on the Gaussian weight function, the optimal fitting plane of the neighborhood points is calculated, and the coordinates of the current point are corrected to gradually smooth the point cloud surface and eliminate error data. Using PCA principal component analysis, the covariance matrix of the neighborhood point cloud is calculated, the minimum feature vector is extracted as the normal vector, and the normal vector surveying data is obtained.

[0022] Specifically: 1. Multi-modal sensor configuration and collaborative work; Sensor types and functions: The unmanned aerial vehicle carries the following multi-modal sensors to achieve complementary data acquisition: Visible light camera: high-resolution RGB camera (4K / 30fps), used to collect high-precision two-dimensional images of the target area, covering a wide spectrum of spectral range.

[0023] Infrared thermal imager: used to capture the distribution of ground surface temperature, assist in identifying vegetation cover, water body boundaries and hidden targets (underground pipelines).

[0024] LiDAR point cloud sensor: 32-line or 64-line laser radar (scanning frequency ≥10Hz), generates high-density three-dimensional point cloud data (accuracy up to ±1cm), supports terrain modeling and obstacle detection.

[0025] Event camera (EventCamera): uses dynamic vision sensor (DAVIS346) to capture the motion trajectory of dynamic targets with microsecond-level response time, generating event stream data (event stream frequency ≥10,000Hz).

[0026] Sensor collaboration and synchronization mechanism: Time synchronization: through the global clock of the unmanned aerial vehicle flight control system (Pixhawk), all sensors are timestamped and calibrated (error ≤1ms) to ensure that the data is aligned in the time dimension.

[0027] Spatial registration: use IMU (inertial measurement unit) and GPS data to compensate for changes in the attitude of the unmanned aerial vehicle, and combine calibration algorithms (checkerboard calibration method) to align the spatial coordinate systems of multiple sensors, eliminating deviations caused by unmanned aerial vehicle motion.

[0028] 2. Dynamic target event stream data generation; Event camera working principle: The event camera is based on an asynchronous pixel-level response mechanism, which only outputs events when the pixel brightness changes beyond a threshold. Each event contains the following information: Timestamp (microsecond-level precision); Pixel coordinates (x, y); Brightness change direction (up / down); Dynamic target trajectory feature (reconstructing motion path through continuous event stream).

[0029] Event stream data processing: Event stream compression and denoising: Convert event stream into sparse time-space grid using time window sliding algorithm (TimeSurface), reducing data redundancy.

[0030] Remove noise events through morphological filtering (opening, closing), preserving real motion trajectories.

[0031] Dynamic target trajectory reconstruction: Based on the spatiotemporal distribution of event stream, use optical flow estimation algorithm (Horn-Schunck) or deep learning trajectory prediction model (LSTM) to extract the motion speed, direction, and trajectory prediction of dynamic targets (vehicles, crowds).

[0032] 3. Preliminary integration of multi-source data; Data hierarchical storage and transmission: Visible light images and thermal imaging data: real-time return to ground station through UAV's high-speed SD card (UHS-II) or 5G module, storage format is JPEG2000 (lossless compression).

[0033] LiDAR point cloud data: stored in PCD (PointCloudData) format, preprocessed by VoxelGridFiltering through edge computing module to reduce point cloud density (from 1 million points / second to 100,000 points / second).

[0034] Event stream data: transmitted in binary format (.csv or custom protocol), dynamically target areas are identified through event stream clustering algorithm (DBSCAN), marked as key objects for subsequent processing.

[0035] Data quality assurance: Visible light image correction: eliminate barrel distortion through lens distortion correction algorithm (OpenCV's fisheye correction), ensure image geometric accuracy.

[0036] LiDAR point cloud correction: combined with IMU data, use MotionCompensation algorithm to correct point cloud drift caused by UAV motion.

[0037] Event stream alignment with images: Through a spatio-temporal registration algorithm (ICP, Iterative Closest Point), the dynamic target trajectory in the event stream is mapped to the visible light image coordinate system, realizing pixel-level alignment of multi-modal data.

[0038] Step 102, histogram equalization and multi-scale Retinex enhancement are performed on the survey image data, and voxel filtering and surface reconstruction are performed on the survey point cloud data to obtain preprocessed acquisition data; Specifically, in the embodiment, the Symlets wavelet function is used to perform wavelet packet decomposition on each channel of the RGB enhanced image. According to the image resolution, the decomposition layer is selected, the sub-band energy vector of the RGB channel is spliced according to the weight, the feature dimension is reduced through LDA linear discriminant analysis, and is normalized to the range of [0, 1], to obtain the image frequency domain feature vector.

[0039] Specifically: 1. Preprocessing of survey image data; Histogram equalization; Principle: By adjusting the gray scale distribution of the image, the dynamic range of the low contrast area is expanded, and the overall contrast of the image is enhanced.

[0040] Implementation: Perform channel-by-channel histogram equalization (R, G, and B channels are processed separately) on the original RGB image to avoid color distortion.

[0041] Adaptive histogram equalization (CLAHE) is used to divide the image into multiple local regions and perform equalization on each region to avoid over-enhancement or noise amplification caused by global equalization.

[0042] Parameter setting: The size of the local region is 8x8 pixels, and the contrast limit threshold is set to 0.01 to prevent overstretching of the highlight area.

[0043] Multi-scale Retinex enhancement; Principle: Simulate the multi-scale illumination compensation mechanism of the human visual system, separate the illumination component and the reflection component of the image, enhance the details and suppress the noise.

[0044] Implementation: Multi-scale Gaussian filtering: Perform Gaussian blur on the image with three different scales (σ=15, σ=80, σ=200) to generate multi-scale illumination estimation.

[0045] Reflection component calculation: Divide the original image by the Gaussian blurred image to obtain the reflection component (i.e. the image after detail enhancement).

[0046] Dynamic range compression: Perform logarithmic transformation on the enhanced image to avoid oversaturation caused by high dynamic range.

[0047] Parameter setting: weight coefficients are set as {0.4, 0.4, 0.2}, corresponding to the contribution proportion of 3 scales respectively.

[0048] Comprehensive optimization: Image fusion: fuse the histogram equalization result and the multi-scale Retinex enhancement result according to the weight (0.7:0.3), balance the contrast enhancement and detail preservation.

[0049] Noise suppression: introduce non-local mean denoising (NLM) after enhancement, use image redundancy information to smooth noise and preserve edge details.

[0050] 2. Preprocessing of point cloud data; Voxel filtering (VoxelGridFiltering); Principle: divide the point cloud space into a regular three-dimensional voxel grid, replace the original points with the average or median value of the points in each voxel, reduce the point cloud density and remove noise.

[0051] Implementation: voxel size selection: set the voxel edge length (0.1m) according to the task accuracy requirement, reduce the number of point clouds (from 1 million points / second to 100,000 points / second) while ensuring accuracy.

[0052] Dynamic density adjustment: use coarse voxels (0.2m) in flat terrain areas and fine voxels (0.05m) in complex areas (building edges) to achieve adaptive downsampling.

[0053] Outlier rejection: remove outliers using statistical filtering (3σ criterion) to avoid isolated noise points after voxel filtering.

[0054] Surface reconstruction (SurfaceReconstruction); Core algorithm: use the moving least squares method (MLS) to generate a smooth surface by local fitting, repairing holes and discontinuous areas in the point cloud.

[0055] Implementation: local neighborhood search: for each point cloud point, extract neighboring points (KNN=30) within a spherical neighborhood with a radius r=0.5m to construct a local plane equation.

[0056] Weighted least squares fitting: based on the Gaussian weight function (w(x)=e^(-d² / (2σ²)), σ=0.1m), calculate the optimal fitting plane of the neighborhood points and correct the coordinates of the current point.

[0057] Iterative optimization: repeat the above steps 3 times to gradually smooth the point cloud surface and eliminate the unevenness caused by occlusion or sensor errors.

[0058] Normal vector estimation and feature extraction; Normal Vector Calculation: Compute the covariance matrix of the neighborhood point cloud using PCA (Principal Component Analysis), extract the smallest eigenvector as the normal vector, which is used for subsequent surface reconstruction.

[0059] Feature Point Detection: Identify key points (building corners, vegetation edges) through the ISS (Intrinsic Shape Signatures) algorithm to assist in surface target classification.

[0060] 3. Integration of pre-processed data; Multi-modal data alignment; Temporal and spatial synchronization correction: Compensate for changes in UAV flight attitude using IMU (Inertial Measurement Unit) and GPS data, and perform 6DoF (6 Degrees of Freedom) registration of image and point cloud data.

[0061] Eliminate sensor installation errors by calibration board calibration (checkerboard calibration method) to ensure the mapping relationship between image pixel coordinates and point cloud spatial coordinates.

[0062] Feature consistency verification: Project the point cloud onto the image plane and check the geometric consistency (building contour alignment) of the enhanced image and point cloud surface.

[0063] If there is a deviation (±5cm), fine-tune through the Iterative Closest Point algorithm (ICP) to optimize registration accuracy.

[0064] Data format standardization: Image storage: Use JPEG2000 lossless compression to preserve enhanced high-frequency details, reducing file size by 40% compared to traditional JPEG.

[0065] Point cloud storage: Use PCD (PointCloudData) format, including fields such as coordinates (x, y, z), intensity, and normal vector (nx, ny, nz), supporting fast reading and visualization.

[0066] Step 103, extract the frequency domain features of the image in the pre-processed collected data using wavelet packet decomposition, and fuse them with the motion trajectory features in the event stream data to generate a hybrid feature vector; Specifically, in this embodiment, the event stream is divided into dynamic target regions using the DBSCAN clustering algorithm, the target trajectory is reconstructed based on the time window sliding method, and the motion feature vector is calculated; Concatenate the image frequency domain feature vector and the motion feature vector, dynamically adjust the weights of the two features through the attention mechanism, map the motion features to the image space to generate a hybrid feature vector.

[0067] Specifically: 1. Extract image frequency domain features using wavelet packet decomposition; Principles and objectives: Wavelet Packet Decomposition (WPD) is a multi-scale time-frequency analysis method that extracts local frequency domain features of images by recursively decomposing high and low frequency components of signals. Compared with traditional wavelet transform, wavelet packet decomposition can more finely divide frequency bands and is suitable for extracting complex texture and edge features.

[0068] Objective: Extract multi-scale frequency energy features (medium-high frequency energy distribution) from pre-processed images to capture surface details (vegetation texture, road cracks) and structural features.

[0069] Specific implementation steps: Image channel processing; Perform wavelet packet decomposition on each channel (R, G, B) of the RGB enhanced image to avoid color distortion.

[0070] Wavelet base selection: Select wavelet base functions (Haar, Daubechies, Symlets) according to task requirements. For example, Daubechies base (db4) is suitable for smooth edge extraction, and Haar base is suitable for fast calculation.

[0071] Multi-level decomposition and sub-band division: Decomposition level: Select the decomposition level (3 layers) according to the image resolution. For example, a 1024x1024 image decomposed into 3 layers can generate 8 sub-bands (LL1, LH1, HL1, HH1,..., HH3).

[0072] Sub-band selection strategy: Medium-high frequency sub-band: Focus on high frequency sub-bands (LH, HL, HH) to extract texture and edge information.

[0073] Low frequency sub-band: Preserve low frequency components (LL) to maintain the overall structure of the image.

[0074] Energy calculation: Calculate the energy value (Energy = Σ|coefficients|²) for each sub-band to form an energy vector.

[0075] Feature vector generation: Multi-channel fusion: Concatenate the sub-band energy vectors of R, G, B channels according to the weight (1:1:1) to generate the final image frequency domain feature vector.

[0076] Dimensionality reduction and normalization: Reduce the feature dimension through LDA (Linear Discriminant Analysis) and normalize to the range [0, 1] to facilitate subsequent fusion.

[0077] Event stream data motion trajectory feature extraction target; Extract motion features (velocity, direction, trajectory pattern) of dynamic targets from event stream data to assist in identifying the impact of moving objects (vehicles, crowds) on surface changes.

[0078] Specific implementation steps: event stream preprocessing; Event stream clustering: use DBSCAN or K-means algorithm to divide event stream into dynamic target regions.

[0079] Trajectory reconstruction: reconstruct target trajectory based on time window sliding method (TimeSurface) or optical flow estimation algorithm (Horn-Schunck).

[0080] Motion feature extraction: Velocity feature: calculate the displacement speed of the target in consecutive event streams.

[0081] Direction feature: represent the direction of motion through the tangent direction of the trajectory or the polar angle.

[0082] Trajectory pattern feature: use LSTM or GRU model to predict the periodicity (back-and-forth motion) or suddenness (sudden stop) of the trajectory.

[0083] Feature vector generation: Statistical features: extract the mean, variance of velocity, and histogram distribution (8-bin histogram) of direction.

[0084] Time series features: calculate acceleration, curvature, and other dynamic properties of the trajectory through sliding window.

[0085] 2. Fusion of frequency domain features and motion trajectory features; Fusion strategy: Combine image frequency domain features (static texture) and motion trajectory features (dynamic behavior) to generate hybrid feature vectors. According to task requirements, choose one of the following methods: Channel concatenation (ChannelConcatenation); Principle: directly concatenate frequency domain feature vectors and motion feature vectors, preserving the independence of the two modalities.

[0086] Advantages: simple and efficient, suitable for low-dimensional feature scenarios.

[0087] Weighted fusion (WeightedFusion); Principle: dynamically adjust the weights of the two features through attention mechanism (SENet channel attention).

[0088] Advantages: suppress noise and enhance key features, suitable for complex scenarios (dynamic target interference).

[0089] FeatureMapping Principle: Map the motion features to the image space (through the spatio-temporal registration of the event stream and the image), and generate a joint feature map.

[0090] Step 104, input the mixed feature vector into the MobileNetV3+Transformer lightweight adaptive model, dynamically adjust the weights of different modalities through attention mechanism, output the surveying and mapping result of the target surveying and mapping area, and generate the ground surface change heat map based on the surveying and mapping result.

[0091] Specifically, in this embodiment, the mixed feature vector is divided into 8 heads, each head independently calculates the query, key and value matrix, and generates attention weight through Softmax normalization; Introduce bidirectional attention mechanism in the Transformer encoder, let the image features and motion trajectory features influence each other, and output the surveying and mapping result of the target surveying and mapping area.

[0092] Specifically: 1. Mixed feature vector input and model architecture; Model input: Feature standardization: Normalize the mixed feature vector (image frequency domain feature + event stream motion trajectory feature) generated in step 3 (Z-score standardization) to ensure that the features of each modality are within the range of [0, 1], and avoid the influence of numerical scale difference on model training.

[0093] Feature dimension alignment: If the image frequency domain feature and the motion trajectory feature have different dimensions, use a fully connected layer (DenseLayer) or PCA dimensionality reduction to map them to the same dimension (256 dimensions).

[0094] Model architecture design: (1) MobileNetV3 as the backbone network; Function: Responsible for extracting local semantic information (texture, edge, motion pattern) of mixed feature vector.

[0095] Improvements: Depthwise separable convolution optimization: Replace some standard convolutions with depthwise separable convolutions (DepthwiseSeparableConvolution) to reduce computational complexity (reduce parameter size by 30%).

[0096] Lightweight SE module: Embed an improved Squeeze-and-Excitation (SE) attention mechanism in the Bottleneck module of MobileNetV3, and only retain key channel features (compression ratio adjusted from 16:1 to 8:1).

[0097] (2) Transformer module realizes cross-modal attention; Function: dynamically adjust the weight of image frequency domain features and event stream motion trajectory features through self-attention mechanism.

[0098] Specific implementation: Multi-Head Self-Attention: Divide the mixed feature vector into multiple subspaces (8 heads), independently calculate the query (Q), key (K), and value (V) matrices for each head, and generate attention weights through Softmax normalization.

[0099] (3) Lightweight adaptive model design; Model compression: Knowledge Distillation: Use a more complex teacher model (ResNet-50+Transformer); generate soft labels to guide the training of the student model MobileNetV3+Transformer, reduce the number of parameters while maintaining accuracy.

[0100] Channel Pruning: Evaluate channel importance through the γ coefficient of the BN layer, and prune low-importance channels (retain 80% of the original channels).

[0101] Hardware adaptation: Quantization-Aware Training: Compress model weights from 32-bit floating-point numbers to 8-bit integers, reduce memory usage (model size from 150MB to 40MB), and adapt to UAV end-side computing.

[0102] 2. Generation of mapping results; Task definition: Objective: Output the ground classification map (vegetation, buildings, water) and the ground change detection map (new roads, collapse areas) of the target mapping area.

[0103] Output format: Classification map: Each pixel point corresponds to a class label (1-5 classes).

[0104] Change detection map: Binary mask (0=no change, 1=change).

[0105] Model inference process; Feature fusion and classification: Input the mixed feature vector into the MobileNetV3+Transformer model, and output the classification probability through the fully connected layer (Softmax classification).

[0106] Multi-task learning: add two branches in the last layer of Transformer: Classification branch: predict land cover classes (loss function: CrossEntropy).

[0107] Change detection branch: predict change regions (loss function: DiceLoss).

[0108] Post-processing optimization: CRF (Conditional Random Field) smoothing: graph optimization on the classification map to remove small noisy patches (isolated pixels).

[0109] Morphological operations: dilation and erosion on the change detection map to fix boundary breaks.

[0110] 3. Land cover change heat map generation; Heat map principle; Definition: represent the intensity of land cover change through color depth (red = strong change, blue = stable area).

[0111] Data source: based on the confidence (model output probability) and spatial density (number of adjacent change pixels) of the change detection map.

[0112] Generation steps: Confidence mapping; Get the confidence of each pixel (0.0~1.0) from the change detection branch and map it to the color gradient (use Viridis or Jet color spectrum).

[0113] Spatial density calculation; Use kernel density estimation (KDE) to calculate the spatial distribution density of the change region: Take each change pixel as the center, use Gaussian kernel (σ=2) to calculate the number of change pixels within the 10×10 window around it.

[0114] Its beneficial effects are that 1, the multi-modal data of the unmanned aerial vehicle can be adaptively fused according to the scene requirements, and the adaptability and recognition accuracy of the model to the complex scene are improved. The final output mapping result covers rich information such as three-dimensional terrain model and ground object recognition, and generates an intuitive surface change heat map, which provides accurate, reliable and high-visualization degree decision basis for the fields of geological monitoring and urban planning, and effectively improves the efficiency of mapping. 2, can more comprehensively and meticulously describe the information of the image in different frequencies and directions; the motion trajectory features are extracted from the event stream data, and the motion parameters of the dynamic target are accurately obtained by combining Kalman filtering algorithm. 3, effectively improves the contrast of the image, suppresses the influence of uneven illumination, makes the image details clearer, and enhances the readability and analyzability of the image.

[0115] Please refer to Figure 2 In a kind of surveying and mapping unmanned aerial vehicle's acquisition image recognition method, histogram equalization and multi-scale Retinex enhancement are carried out to surveying and mapping image data, comprising the following steps: Step 201, using adaptive histogram equalization to carry out pass-through histogram equalization to RGB image in surveying and mapping image data, to obtain equalization image; Step 202, by separating the illumination component and the reflection component of the equalization image, and enhancing details and suppressing noise, to obtain component image; Step 203, Gaussian blur is carried out to component image in 3 different scales, to generate multi-scale illumination estimation, to obtain illumination estimation image; Step 204, divide the original image by the illumination estimation image, to obtain the reflection component, and carry out logarithmic transformation to the enhanced image, to obtain preprocessed image data.

[0116] The above describes the embodiment of the surveying and mapping unmanned aerial vehicle's acquisition image recognition method, please refer to Figure 3 In a kind of surveying and mapping unmanned aerial vehicle's acquisition image recognition system, surveying and mapping unmanned aerial vehicle's acquisition image recognition system includes the following modules: Surveying and mapping data acquisition module, for obtaining surveying and mapping image data and surveying and mapping point cloud data of target surveying and mapping area by multi-modal sensor carried by unmanned aerial vehicle, and simultaneously using time camera to capture motion signal of dynamic target, to generate event stream data; Surveying and mapping data processing module, for histogram equalization and multi-scale Retinex enhancement to surveying and mapping image data, voxel filtering and surface reconstruction to surveying and mapping point cloud data, to obtain preprocessed acquisition data; Hybrid feature generation module, for extracting frequency domain features of image in preprocessed acquisition data by wavelet packet decomposition, and fusing with motion trajectory features in event stream data, to generate hybrid feature vector; The survey result generation module is configured to input the mixed feature vector into a MobileNetV3+Transformer lightweight adaptive model, dynamically adjust the weights of different modalities through an attention mechanism, and output a survey result of a target survey area, and generate a surface change heat map based on the survey result.

[0117] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only preferred examples of the present application and are not intended to limit the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for identifying images collected by a surveying drone, characterized in that, The image acquisition and recognition method for the surveying and mapping unmanned aerial vehicle comprises the following steps: The multi-modal sensor carried by the unmanned aerial vehicle acquires surveying and mapping image data and surveying and mapping point cloud data of a target surveying and mapping area, and simultaneously uses a time camera to capture a motion signal of a dynamic target to generate event stream data; The surveying and mapping image data is subjected to histogram equalization and multi-scale Retinex enhancement, and the surveying and mapping point cloud data is subjected to voxel filtering and surface reconstruction to obtain preprocessed acquisition data; Wavelet packet decomposition is used to extract frequency domain features of the image in the preprocessed acquisition data, and the motion trajectory features in the event stream data are fused to generate a hybrid feature vector; The hybrid feature vector is input into a MobileNetV3+Transformer lightweight adaptive model, the weights of different modalities are dynamically adjusted through an attention mechanism, and a surveying and mapping result of the target surveying and mapping area is output, and a surface change heat map is generated based on the surveying and mapping result.

2. The method of claim 1, wherein the method further comprises: determining a location of the UAV based on the image data; and determining a location of the object based on the image data. The histogram equalization and multi-scale Retinex enhancement of the surveying and mapping image data comprises: Adaptive histogram equalization is used to perform channel histogram equalization on the RGB image in the surveying and mapping image data to obtain an equalized image; The illumination component and the reflection component of the equalized image are separated, and the details are enhanced and the noise is suppressed to obtain a component image; Gaussian blur is performed on the component image in three different scales to generate multi-scale illumination estimation, and an illumination estimation image is obtained; The original image is divided by the illumination estimation image to obtain a reflection component, and a logarithmic transformation is performed on the enhanced image to obtain preprocessed image data.

3. The method of claim 1, wherein the method further comprises: determining a location of the UAV based on the image data; and determining a location of the object based on the image data. The voxel filtering and surface reconstruction of the surveying and mapping point cloud data to obtain preprocessed acquisition data further comprises: Based on the VoxelGridFiltering voxel filtering algorithm, the voxel edge length is set according to the task; relatively coarse voxels are used in flat terrain areas, and relatively fine voxels are used in complex areas for adaptive down-sampling; The 3σ criterion of statistical filtering is combined to remove outliers, and the average value or median value of points in each voxel is used to replace the original points to obtain filtered surveying and mapping data.

4. The method of claim 1, wherein the method further comprises: determining a location of the UAV based on the image data; and determining a location of the object based on the image data. The voxel filtering and surface reconstruction of the surveying and mapping point cloud data to obtain preprocessed acquisition data further comprises: According to the MLS moving least squares method, adjacent points within a spherical neighborhood with a radius r=0.5m are extracted for each point cloud point to construct a local plane equation; Based on the Gaussian weight function, the optimal fitting plane of the neighborhood points is calculated to correct the coordinates of the current point, and the point cloud surface is gradually smoothed to eliminate error data; PCA principal component analysis is used to calculate the covariance matrix of the neighborhood point cloud, the minimum feature vector is extracted as a normal vector, and normal vector surveying and mapping data are obtained.

5. The method of claim 1, wherein the method further comprises: determining a location of the UAV based on the image data; and determining a location of the object based on the image data. The wavelet packet decomposition is used to extract the frequency domain features of the image in the preprocessed acquisition data, and the motion trajectory features in the event stream data are fused to generate a hybrid feature vector, which comprises: Symlets wavelet function is used to perform wavelet packet decomposition on each channel of the RGB of the RGB enhanced image respectively; According to the image resolution selection decomposition layer number, the sub-band energy vectors of the RGB channels are spliced by weight, the feature dimension is reduced through LDA linear discriminant analysis, and is normalized to the range of [0, 1] to obtain the image frequency domain feature vector.

6. The method of claim 1, wherein the method further comprises: determining a location of the UAV based on the image data; and determining a location of the object based on the image data. The frequency domain features of the image in the preprocessed collected data are extracted by wavelet packet decomposition, and are fused with the motion trajectory features in the event stream data to generate a hybrid feature vector, including: The event stream is divided into a dynamic target area by using a DBSCAN clustering algorithm, target trajectories are reconstructed based on a time window sliding method, and a motion feature vector is calculated; The image frequency domain feature vector and the motion feature vector are spliced, the weights of the two kinds of features are dynamically adjusted through an attention mechanism, the motion features are mapped to the image space to generate a hybrid feature vector.

7. The method of claim 1, wherein the method further comprises: determining a location of the UAV based on the image data; and determining a location of the object based on the image data. The hybrid feature vector is divided into 8 heads, each head independently calculates a query, key and value matrix, and attention weights are generated through Softmax normalization; A bidirectional attention mechanism is introduced in the Transformer encoder, so that the image features and the motion trajectory features influence each other, and the mapping result of the target mapping area is output. The image recognition system for the surveying and mapping unmanned aerial vehicle includes the following modules:

8. A surveying unmanned aerial vehicle image acquisition and recognition system, characterized by, The surveying and mapping data acquisition module is used to acquire surveying and mapping image data and surveying and mapping point cloud data of a target mapping area through a multi-modal sensor carried by an unmanned aerial vehicle, and to capture motion signals of dynamic targets through a time camera to generate event stream data; The surveying and mapping data processing module is used to perform histogram equalization and multi-scale Retinex enhancement on the surveying and mapping image data, and to perform voxel filtering and surface reconstruction on the surveying and mapping point cloud data to obtain preprocessed collected data; The hybrid feature generation module is used to extract frequency domain features of images in the preprocessed collected data by wavelet packet decomposition, and to fuse the features with motion trajectory features in the event stream data to generate a hybrid feature vector; The mapping result generation module is used to input the hybrid feature vector into a MobileNetV3+Transformer lightweight adaptive model, dynamically adjust the weights of different modalities through an attention mechanism, output a mapping result of a target mapping area, and generate a ground surface change heat map based on the mapping result. The surveying and mapping data acquisition module includes the following sub-modules:

9. The image acquisition and recognition system for mapping drones of claim 8, wherein, The setting sub-module is used to set a voxel length according to a task based on a VoxelGridFiltering voxel filtering algorithm; a relatively coarse voxel is used in a flat terrain area, and a relatively fine voxel is used in a complex area for adaptive down-sampling; The filtering sub-module is used to remove outliers by combining a 3σ criterion of statistical filtering, and to replace original points with an average value or a median value of points in each voxel to obtain filtered surveying and mapping data. The surveying and mapping data acquisition module further includes the following sub-modules:

10. The image acquisition and recognition system for mapping drones of claim 8, wherein, ​ The constructing submodule is configured to extract neighboring points in a spherical neighborhood with a radius r=0.5m for each point cloud point according to the MLS (moving least squares) method, and construct a local plane equation; The smoothing submodule is configured to calculate an optimal fitting plane of the neighborhood points based on a Gaussian weight function, correct the coordinates of the current point, and gradually smooth the point cloud surface to eliminate error data; The obtaining submodule is configured to calculate a covariance matrix of the neighborhood point cloud by using PCA (principal component analysis), extract a minimum feature vector as a normal vector, and obtain normal vector surveying data.

Images