Geographic mapping data processing method and system based on big data

Abstract

The application discloses a geographic surveying and mapping data processing method and system based on big data, which comprises the following steps: collecting geographic surveying and mapping data in real time, performing intelligent format conversion, self-adaptive coordinate system unification and dynamic noise filtering preprocessing on the collected original data based on a big data processing platform to obtain pretreated surveying and mapping data; inputting the pretreated multi-source data into a shared backbone network based on an improved ConvNeXt architecture, combining the shared backbone network with a private branch network, and extracting shared features and private features of the data; fusing the shared features and the private features by using a spatial attention mechanism and a channel attention mechanism to generate a self-adaptive weighted fusion feature map; and inputting the fusion feature map into a Transformer encoder to identify an environmental state and output a geographic information recognition result. The recognition efficiency and accuracy of the geographic surveying and mapping data are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of geographic information processing, in particular to a geographic surveying and mapping data processing method and system based on big data. BACKGROUND

[0002] With the deepening of the construction of Digital China, geographic surveying and mapping data is growing explosively, including satellite remote sensing, unmanned aerial photography, ground sensor networks, mobile device location information and other multi-source heterogeneous data. Traditional geographic surveying and mapping data processing methods mainly rely on manual processing and simple automated tools, which are inefficient in data processing and difficult to cope with large-scale and high-frequency geographic surveying and mapping data, resulting in low surveying efficiency. SUMMARY

[0003] The present application relates to the technical field of geographic information processing, in particular to a geographic surveying and mapping data processing method and system based on big data.

[0004] To achieve the above-mentioned purpose, the technical solution of the present application is as follows: further, in the above-mentioned geographic surveying and mapping data processing method based on big data, the geographic surveying and mapping data processing method comprises the following steps:

[0005] A big data processing platform is established based on Hadoop and Spark, geographic surveying and mapping data is collected in real time, and the collected raw data is preprocessed by intelligent format conversion, adaptive coordinate system unification and dynamic noise filtering based on the big data processing platform, to obtain preprocessed surveying and mapping data;

[0006] The preprocessed multi-source data is input into a shared backbone network based on an improved ConvNeXt architecture, the shared backbone network is combined with a private branch network, and the shared features and private features of the data are extracted;

[0007] The shared features and private features are fused by using a spatial attention mechanism and a channel attention mechanism to generate a self-adaptive weighted fusion feature map;

[0008] The fusion feature map is input into a Transformer encoder to identify the environmental state and output a geographic information recognition result.

[0009] Further, in the above-mentioned geographic surveying and mapping data processing method based on big data, the big data processing platform is established based on Hadoop and Spark, geographic surveying and mapping data is collected in real time, and the collected raw data is preprocessed by intelligent format conversion, adaptive coordinate system unification and dynamic noise filtering based on the big data processing platform, to obtain preprocessed surveying and mapping data, comprising:

[0010] Deploy core components based on the Hadoop ecosystem, including the HDFS distributed file system, the YARN resource scheduling framework, integrate the Spark framework to deploy the SparkCore basic computing engine, the SparkStreaming real-time data processing module and the SparkSQL structured data query module, and establish a big data processing platform;

[0011] Real-time acquisition of geographic mapping data, including at least point cloud data, image stream data, and remote sensing satellite image data;

[0012] Read the file information of the geographic mapping data through SparkSQL, identify the data format to generate a data format tag, and perform adaptive conversion on different format data.

[0013] Further, in the above-mentioned big data-based geographic mapping data processing method, the big data processing platform is established based on Hadoop and Spark, real-time acquisition of geographic mapping data, intelligent format conversion, adaptive coordinate system unification and dynamic noise filtering preprocessing of the collected raw data based on the big data processing platform, to obtain preprocessed mapping data, including:

[0014] Extract the metadata information of the geographic mapping data, identify the current coordinate system type, and establish a conversion mapping table between common coordinate systems based on the mapping industry standard;

[0015] Use the Proj4 geographic projection library to perform batch conversion on data in different coordinate systems, and calculate the conversion error in real time during the conversion process. If the error exceeds the threshold, adjust the conversion parameters;

[0016] Perform statistical analysis on the preprocessed data through SparkCore, identify the noise type, and use the K-nearest neighbor filtering algorithm to denoise the noise data to obtain the preprocessed mapping data.

[0017] Further, in the above-mentioned big data-based geographic mapping data processing method, the preprocessed multi-source data is input into the shared backbone network based on the improved ConvNeXt architecture, the shared backbone network is combined with the private branch network, and the shared features and private features of the data are extracted, including:

[0018] Replace the original ConvNeXt convolution residual block with a deep convolution, replace the ReLU activation function with a GELU activation function, and add a LayerNorm normalization layer after each convolution layer to obtain the shared backbone network;

[0019] Input the preprocessed mapping data into the improved shared backbone network to minimize the feature reconstruction error, and use the Adam optimizer for training.

[0020] Further, in the above-mentioned geographic surveying and mapping data processing method based on big data, the pre-processed multi-source data is input into the shared backbone network based on the improved ConvNeXt architecture, the shared backbone network is combined with the private branch network, the shared features and private features of the data are extracted, and the method comprises the following steps:

[0021] For different types of surveying and mapping data, three types of private branch networks are designed, including image data private branch, point cloud data private branch and vector data private branch.

[0022] The remote sensing image is input into the shared backbone to the image private branch, and the LiDAR point cloud is input into the shared backbone to the point cloud private branch. During the forward propagation process of the network, the shared backbone outputs shared features, and the private branch outputs private features.

[0023] Further, in the above-mentioned geographic surveying and mapping data processing method based on big data, the shared features and private features are fused by using the spatial attention mechanism and the channel attention mechanism to generate an adaptively weighted fusion feature map, comprising:

[0024] The shared features and the private features are obtained, the spatial dimensions of the two types of features are unified through convolution operation, and the global average pool is performed on the feature map after the unified dimension to obtain spatial attention enhanced features;

[0025] Global maximum pooling and global average pooling are performed on the spatial attention enhanced features, and the two results are spliced and input into the layer full connection layer to generate a channel weight vector;

[0026] The channel weight vector is multiplied with the spatial attention enhanced features channel by channel to generate an adaptively weighted fusion feature map.

[0027] Further, in the above-mentioned geographic surveying and mapping data processing method based on big data, the fusion feature map is input into the Transformer encoder to identify the environmental state, and the geographic information recognition result is output, comprising:

[0028] The fusion feature map is input into the Transformer encoder, and the geographic information recognition result is obtained by performing forward propagation. The geographic information recognition result is output by threshold judgment.

[0029] Further, in the geographic surveying and mapping data processing system based on big data, the geographic surveying and mapping data processing system comprises the following modules:

[0030] The surveying and mapping data acquisition module is used for establishing a big data processing platform based on Hadoop and Spark, collecting geographic surveying and mapping data in real time, and intelligently converting the collected original data, adaptively unifying the coordinate system, and dynamically filtering and preprocessing the noise to obtain preprocessed surveying and mapping data.

[0031] a data feature extraction module configured to input the preprocessed multi-source data into a shared backbone network based on an improved ConvNeXt architecture, combine the shared backbone network with a private branch network, and extract shared features and private features of the data;

[0032] a data feature fusion module configured to fuse the shared features and the private features by using a spatial attention mechanism and a channel attention mechanism, and generate a self-adaptive weighted fusion feature map;

[0033] a survey data recognition module configured to input the fusion feature map into a Transformer encoder to recognize an environment state and output a geographic information recognition result.

[0034] Further, in the geographic survey data processing system based on big data, the data feature fusion module comprises the following sub-modules:

[0035] a convolution sub-module configured to obtain the shared features and the private features, unify the spatial dimensions of the two types of features by convolution operation, perform global average pooling on the feature maps after the unified dimensions, and obtain spatial attention enhanced features;

[0036] a generation sub-module configured to perform global maximum pooling and global average pooling on the spatial attention enhanced features, input the two results after splicing into a full connection layer of a layer, and generate a channel weight vector;

[0037] a multiplication sub-module configured to multiply the channel weight vector and the spatial attention enhanced features channel by channel to generate a self-adaptive weighted fusion feature map.

[0038] Further, in the geographic survey data processing system based on big data, the data feature fusion module comprises the following sub-modules:

[0039] an output sub-module configured to input the fusion feature map into a Transformer encoder, perform forward propagation to obtain a geographic information recognition result, filter the result by threshold judgment, and output the geographic information recognition result.

[0040] The beneficial effects are that, through the multi-modal unified representation and self-adaptive fusion mechanism, the complementary information between different geographic data sources is fully mined, the information island problem is solved, and the interpretation robustness and precision in complex scenes such as shadow areas and occluded areas are significantly improved. The end-to-end processing flow reduces manual intervention, realizes the overall automation of geographic information extraction, improves the processing efficiency compared with traditional methods, and improves the recognition efficiency and accuracy of geographic survey data. BRIEF DESCRIPTION OF DRAWINGS

[0041] Various other advantages and benefits will become apparent to those of ordinary skill in the art, upon reading the following detailed description of the preferred embodiment. The accompanying drawings are included to provide a description of preferred embodiments and are not intended to limit the scope of the application.

[0042] Figure 1 The first embodiment schematic diagram of the geographic surveying and mapping data processing method based on big data in the embodiment of the application;

[0043] Figure 2 The second embodiment schematic diagram of the geographic surveying and mapping data processing method based on big data in the embodiment of the application;

[0044] Figure 3 The first embodiment schematic diagram of the geographic surveying and mapping data processing system based on big data in the embodiment of the application. DETAILED DESCRIPTION

[0045] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0046] Those skilled in the art can understand that, unless specifically stated otherwise, the singular form "one", "an", and "the" used herein can also include the plural form. It should be further understood that the use of the phrase "comprising" in the specification of the present application means that there are features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0047] The present application will be described in detail below with reference to the drawings, Figure 1 As shown, the geographic surveying and mapping data processing method based on big data includes the following steps:

[0048] Step 101, a big data processing platform is established based on Hadoop and Spark, real-time geographic surveying and mapping data is collected, and the collected original data is preprocessed by intelligent format conversion, adaptive coordinate system and dynamic noise filtering based on the big data processing platform, to obtain preprocessed surveying and mapping data;

[0049] Specifically, in the embodiment, core components are deployed based on the Hadoop ecosystem, including the HDFS distributed file system, the YARN resource scheduling framework, the integrated Spark framework, the SparkCore basic computing engine, the SparkStreaming real-time data processing module, and the SparkSQL structured data query module, to establish a big data processing platform; geographic surveying and mapping data, at least including point cloud data, image stream data, and remote sensing satellite image data, are collected in real time; the file information of the geographic surveying and mapping data is read through the SparkSQL, data format tags are generated by identifying the data format and different format data is executed for adaptive conversion. Metadata information of the geographic surveying and mapping data is extracted, the current coordinate system type is identified, a conversion mapping table between common coordinate systems is established based on the surveying and mapping industry standard; the data of different coordinate systems is executed for batch conversion by using the Proj4 geographic projection library, conversion errors are calculated in real time during the conversion process, and if the error exceeds a threshold, the conversion parameters are adjusted; the pretreated data is subjected to statistical analysis through the SparkCore, noise types are identified, noise data is reduced by using a K nearest neighbor filtering algorithm, and pretreated surveying and mapping data is obtained.

[0050] In the process of building a big data processing platform, the core is to build a basic framework around the needs of massive storage of geographic surveying and mapping data, efficient computing, and multi-source access. In terms of component selection, the HDFS distributed file system based on the Hadoop ecological deployment can specifically carry TB / PB level raw data in the surveying and mapping scenario. Whether it is high-resolution images collected by unmanned aerial vehicles or high-density point clouds generated by ground laser scanners, they can be safely stored and flexibly expanded through distributed storage architecture. The YARN resource scheduling framework avoids resource contention when multiple tasks are running in parallel by dynamically allocating CPU and memory resources. For example, when processing image streams and point cloud data simultaneously, more memory can be allocated to the computationally intensive point cloud preprocessing task. The integrated Spark framework further enhances processing capabilities: SparkCore as the basic computing engine supports batch operations on offline data, SparkStreaming with low latency characteristics interfaces real-time data streams from GNSS receivers, unmanned aerial vehicles and other devices, and realizes second-level access and temporary storage of point clouds and images. SparkSQL facilitates subsequent fast queries on structured surveying and mapping data such as vector attribute tables, and the three work together to meet the mixed processing needs of real-time and offline in the surveying and mapping scenario. Platform configuration optimization is more in line with data characteristics, adjusting the default 128MB block size of HDFS to 256MB can reduce the number of block divisions for large files such as images and point clouds, reducing storage overhead; Reserving more than 50% of the cluster memory for Spark tasks can effectively prevent memory overflow during point cloud density analysis, image noise reduction and other computing processes; The strategy of 3 copies of key raw data and 2 copies of preprocessed data ensures the safety of non-renewable raw surveying and mapping data, and avoids the waste of cost caused by redundant storage. The data access interface realizes full-scenario coverage, and the real-time interface adapts to the low-latency data transmission needs of devices through Socket / HTTP protocol, the offline interface is compatible with mainstream GIS databases such as PostgreSQL / PostGIS, CAD.dwg files and remote sensing satellite.tiff / .img files, and batch uploading is completed through HDFS client tools, completely solving the problem of single interface and adaptation difficulty of traditional platforms.

[0051] In the preprocessing link of geographic mapping raw data, the data quality is ensured through intelligent conversion, accurate unification and dynamic filtering. Intelligent format conversion first relies on SparkSQL to read file header information, automatically identifies.las / .pcd, point cloud,.tiff / .jpg, image,.shp / .geojson, vector and other formats and generates labels, and then executes targeted conversion logic: converts CAD.dwg files to GeoJSON format, which can completely retain the coordinate and attribute information of vector graphics, facilitating subsequent spatial analysis; converts laser point cloud.las files to Parquet columnar storage format, which can improve subsequent query efficiency by more than 30%; converts remote sensing image.tiff files to SequenceFile format supported by HDFS, which can adapt to distributed reading scenarios. After conversion, the data integrity is verified through the GDAL library. If problems such as missing coordinates and missing attribute fields are found, the re-conversion process is triggered immediately to prevent invalid data from flowing into the downstream. Adaptive coordinate system solves the problem of coordinate chaos of multi-source data. First, extract image geographic header files, point cloud coordinate reference fields and other metadata, identify WGS84, CGCS2000, UTM and other coordinate system types, and then establish a conversion mapping table based on the mapping industry standard, such as parameter conversion of WGS84 and CGCS2000, coordinate conversion of UTM projection and Gauss-Kruger projection. When calling the Proj4 geographic projection library to perform batch conversion, real-time monitoring of plane coordinate error, ≤0.5m and elevation error, ≤0.1m is performed. If the error exceeds the standard, a regional correction factor is added to adjust the parameters to ensure that all data are unified to the target coordinate system. Dynamic noise filtering realizes on-demand denoising. Through SparkCore statistical analysis, noise types are identified: calculate the neighborhood density of point cloud, judge low-density points as isolated points; analyze the gray value distribution of image pixels, locate the salt and pepper noise outside the normal range; detect the node density of vector curves, and select redundant nodes. Then, according to the noise type, the algorithm is dynamically selected: K-neighborhood filtering for point cloud isolated points, K=20 for high density and K=10 for low density, median filtering for image salt and pepper noise, 3x3 window for 1080P image and 5x5 window for 4K image, and Douglas-Peucker algorithm for vector redundant nodes. Finally, the plane deviation of point cloud and the image clarity before and after processing are compared, the parameters are adjusted, and the preprocessed data meeting the accuracy requirements are generated and stored in the corresponding directory of HDFS.

[0052] In step 102, the preprocessed multi-source data is input into a shared backbone network based on an improved ConvNeXt architecture, the shared backbone network is combined with a private branch network, and shared features and private features of the data are extracted;

[0053] Specifically, in the embodiment, the convolution residual block of the original ConvNeXt is replaced with a deep convolution, the ReLU activation function is replaced with a GELU activation function, and a LayerNorm normalization layer is added after each convolution layer to obtain a shared backbone network; the improved shared backbone network is input with preprocessed surveying and mapping data, and is trained by using an Adam optimizer with the objective of minimizing feature reconstruction error. For different types of surveying and mapping data, three types of private branch networks are designed, including an image data private branch, a point cloud data private branch, and a vector data private branch; a remote sensing image is input into the shared backbone to the image private branch, and a LiDAR point cloud is input into the shared backbone to the point cloud private branch; during the forward propagation process of the network, the shared backbone outputs shared features, and the private branch outputs private features.

[0054] Three targeted optimizations are made to the original ConvNeXt architecture, and a high-efficiency shared backbone network is constructed through scientific training. In the residual block structure adjustment, although the 1x1 convolution→3x3 depth convolution→1x1 convolution process of the original ConvNeXt can extract basic features, the limited receptive field leads to incomplete feature capture when facing large-scale features such as mountains and lakes in the surveying and mapping scene. Therefore, the scheme introduces a dilated convolution with a dilated rate of 2 at the 3x3 depth convolution stage. By setting interval sampling points in the convolution kernel, the receptive field range is expanded by more than 2 times without increasing the computational complexity, which can completely cover the spatial distribution characteristics of large features, such as the continuous edge profile of a lake boundary spanning more than 100 meters, effectively capturing the continuous edge profile and avoiding feature fragmentation caused by insufficient receptive field of traditional convolution. The activation function optimization focuses on the pain point of weak feature extraction: the original ReLU activation function directly sets the gradient to zero when the input value is less than 0, which easily loses features such as road edges, small buildings, and other features with gentle grayscale changes or weak signals. After using the GELU activation function, the smooth nonlinear characteristics of the Gaussian error linear unit preserve and amplify the gradient of weak feature signals. For example, in urban suburban mapping, the grayscale gradient recognition sensitivity of road edges can be improved by more than 30%, reducing the detection of small features. The normalization layer position adjustment solves the problem of data distribution deviation in the training process: the original ConvNeXt only sets the normalization layer after the residual connection, which leads to the output data of the convolution layer prone to distribution drift due to scale differences, affecting feature consistency. The scheme adds a LayerNorm normalization layer after each convolution layer to adjust the mean and variance of the convolution output in real time, so that the data distribution of each layer is stable in the preset interval, not only improving the network training convergence speed by 40%, but also avoiding feature extraction bias caused by image light differences and point cloud density fluctuations. In the training stage of the shared backbone network, multi-source data, remote sensing images, LiDAR point clouds, and GIS vectors are aligned to a unified grid through spatial indexing, and then input into the improved ConvNeXt network regardless of data type. The training focuses on minimizing the feature reconstruction error as the core objective, and uses the MSE loss function to calculate the difference between the network output features and the original data features, constantly optimizing the parameters to reduce the error and ensure that the network learns the common features of multi-source data, such as spatial topological relationships and feature texture structure rules. When using the Adam optimizer, the initial learning rate is set to 1e-4 to avoid parameter oscillation in the early stage of training, and is reduced to 0.5 of the original every 10 rounds to ensure stable convergence of the parameters in the later stage. After training, the bottom 80% of the network parameters are fixed, which are responsible for extracting basic spatial features and have strong universality. Only the top 20% of the parameters can be fine-tuned, which not only ensures the stability of feature extraction, but also provides flexible adjustment space for subsequent adaptation to different data types.

[0055] For the unique attribute differences of multi-source surveying and mapping data, three types of customized private branch networks are designed to realize the complete extraction of general features and exclusive features in cooperation with the shared backbone network. The image data private branch focuses on capturing spectral and color information: a 2-layer 1x1 convolution + 1-layer max pooling structure is adopted, the first layer of 1x1 convolution, 256 convolution kernels are responsible for channel expansion and preliminary filtering of spectral information on the basis of the shared backbone output, and the red, near-infrared and other spectral channel features in the image that are strongly related to the feature type are retained; the second layer of 1x1 convolution, 128 convolution kernels further compress the redundant channels and strengthen the key spectral gradients, such as the spectral differences between vegetation and water; finally, through the 2x2 max pooling layer, the key information of the feature map is retained, while the data dimension is reduced to avoid redundant calculations. This structure can effectively extract the NDVI index related features of cultivated vegetation and the blue band reflection features of water bodies, providing accurate spectral basis for subsequent feature classification. The point cloud data private branch is designed for three-dimensional spatial features: a 3D convolution layer with a 3x3x3 kernel and a global average pooling structure is adopted, the 3D convolution kernel, 128, can directly perform convolution operations on the three-dimensional voxel data of LiDAR point cloud, capturing three-dimensional features such as elevation fluctuations, such as slope changes in mountainous areas, surface roughness, such as the surface differences between bare land and forest land, etc. Compared with traditional 2D convolution, the representation accuracy of three-dimensional spatial features can be improved by more than 50%; the global average pooling layer converts the feature voxels output by the 3D convolution into fixed-dimensional vectors, realizing efficient integration and subsequent processing of three-dimensional features. The vector data private branch focuses on the combination of attribute features and spatial correlation: a fully connected layer + attention layer structure is adopted, the output dimension of the fully connected layer is 64, which first maps the spatial features extracted by the shared backbone and the attribute information of the vector data, such as road width, building height, and feature category code, into low-dimensional attribute feature vectors; the attention layer highlights key attributes such as building category labels and road width parameters through weight distribution, and suppresses irrelevant attributes such as redundant ownership information, for example, in urban road mapping, the feature weight of the road width attribute can be increased to twice that of other attributes to ensure accurate identification of trunk roads and branch roads. In the feature extraction execution stage, multi-source data is respectively input into the corresponding shared backbone + private branch channel: remote sensing images are input into the shared backbone, and the spatial position features and spectral features are output by the image private branch; LiDAR point clouds are input into the shared backbone, and the spatial position features and three-dimensional structure features are output by the point cloud private branch; GIS vectors are input into the shared backbone, and the spatial position features and attribute features are output by the vector private branch; all features are finally stored in the feature cache area, laying a data foundation for subsequent double attention mechanism fusion.

[0056] Step 103, using spatial attention mechanism and channel attention mechanism to fuse shared features and private features to generate adaptively weighted fusion feature maps;

[0057] Specifically, in this embodiment, shared features and private features are obtained. The spatial dimensions of the two types of features are unified through convolution operations. Global average pooling is performed on the feature map after unifying the dimensions to obtain spatial attention-enhanced features. Global max pooling and global average pooling are performed on the spatial attention-enhanced features. The two results are concatenated and input into the fully connected layer to generate channel weight vectors. The channel weight vectors are multiplied with the spatial attention-enhanced features channel by channel to generate an adaptively weighted fusion feature map.

[0058] By employing spatial attention and channel attention mechanisms, feature selection and enhancement are performed from the spatial and channel dimensions, respectively, addressing the problems of blurred key spatial regions and interference from redundant channel information in traditional feature processing. In the spatial attention mechanism stage, feature map dimensions are first unified. Due to differences in data types (e.g., image feature maps are 224×224×C, while point cloud feature maps are 112×112×C), spatial dimension mismatches exist. Bilinear interpolation or convolution operations are used to unify the two types of features to the same size, such as 112×112×C, ensuring accurate correspondence between features at each spatial location. For example, the 3D feature map of LiDAR point cloud is interpolated to the feature map of remote sensing image. Figure 1 A unified spatial grid is used to avoid spatial misalignment during subsequent weighting. A spatial weight map is then generated: global average pooling is performed on the feature map of uniform dimension, compressing the H×W×C features into a 1×1×C channel mean. Then, a 1×1 convolution reduces the number of channels to 1, eliminating channel dimension interference. Finally, a sigmoid activation function maps the data to the [0,1] interval, resulting in an H×W×1 spatial weight map. This weight map can accurately locate spatial regions valuable for identification. For example, in urban mapping, densely built-up areas, due to their rich features, edges, and textures, can have weight values ​​above 0.8, while sparsely populated suburbs have weight values ​​below 0.3. This weight difference distinguishes the priority of spatial features. Finally, spatial attention feature enhancement is performed: the spatial weight map is multiplied element-wise with the original feature map, amplifying features in high-weight areas, such as building outlines and road networks, while suppressing noise in low-weight areas, such as random vegetation in suburbs and image noise. For example, in road recognition scenarios, the signal strength of road edge features can be increased by more than 50%, reducing edge breakage problems caused by background interference.

[0059] The channel attention mechanism focuses on the effectiveness screening of feature channels. First, the key channel features are captured by channel importance calculation: the spatial attention enhanced features are simultaneously subjected to global max pooling and global average pooling, the two pooling results are spliced and input into a 2-layer fully connected layer, the first fully connected layer compresses the channel number to C / 4, reduces the computational complexity and avoids overfitting, and the second fully connected layer restores to C channels, and then a 1x1xC channel weight vector is generated through a Sigmoid activation function. The vector can accurately quantify the value of each channel, for example, in the vegetation identification task, the weight values of the red spectrum channel and the near-infrared spectrum channel can reach 0.9, and the weight value of the irrelevant blue spectrum channel is only 0.1; in the building identification task, the weight values of the point cloud elevation channel and the surface roughness channel are more than 0.8, and the weight value of the redundant reflectivity channel is less than 0.2. Then, through channel attention feature screening, the channel weight vector and the spatial attention enhanced features are multiplied channel by channel, the effective features of the high-weight channels are retained, the redundant information of the low-weight channels is removed, the feature dimension is reduced by 30%-40% under the premise of keeping the key information, and an efficient feature basis is laid for subsequent fusion.

[0060] In the adaptive weighted fusion stage, the scheme discards the disadvantages of traditional fixed proportion fusion, and realizes efficient integration of two types of optimized features through dynamic weight distribution. First, the fusion weight dynamic distribution is performed: the feature entropy of the spatial attention enhanced features and the channel attention screened features is calculated. Then, the feature map fusion and optimization are performed: the two types of features are subjected to element-wise weighted summation according to the dynamically distributed weights to obtain the initial fusion feature map; the feature channel number is unified to 256 through 1x1 convolution to adapt to the input dimension of the subsequent Transformer encoder, and then the LayerNorm normalization layer is used to adjust the mean and variance of the feature map to avoid feature distribution deviation caused by data scale difference, and finally the adaptive weighted fusion feature map is generated and stored in the fusion feature directory. The feature map not only retains the details of the spatial key regions such as building boundaries and road directions, but also integrates the information of high-value channels such as vegetation spectrum and terrain elevation, providing high-quality feature input for subsequent geographic information recognition.

[0061] Step 104, input the fusion feature map into the Transformer encoder to identify the environmental state and output the geographic information recognition result.

[0062] Specifically, in the present embodiment, the fusion feature map is input into the Transformer encoder, forward propagation is performed to obtain the geographic information recognition result, threshold judgment is performed for screening, and the geographic information recognition result is output.

[0063] In the fusion feature map input processing stage, the feature format adaptation needs to be completed to meet the input requirements of the Transformer encoder: for the 112x112x256 fusion feature map, a fixed size of 16x16 pixels is used for feature block, Patch cutting. This size selection can not only retain the feature details, but also control the number of Patches. 112 / 16x112 / 16=49, which can avoid excessive Patches from causing a sharp increase in computational load; each Patch needs to complete dimension conversion through a linear embedding layer. Since the Transformer encoder does not have native spatial order perception capability, position encoding, PositionalEncoding, needs to be introduced to supplement spatial correlation information: according to the row number and column number of each Patch in the original feature map, a 1024-dimensional position encoding vector is generated using the sine-cosine position encoding logic. For example, the Patch in the 3rd row and 5th column has a coding vector that represents the row position through the sine function and the column position through the cosine function, ensuring that Patches at different spatial positions have unique encoding identifiers; after adding the position encoding vector and the Patch embedding vector element by element, the embedding vector with position information is obtained, enabling the Transformer to accurately capture the spatial topological relationship between Patches, such as the small difference in position encoding between adjacent Patches and the large difference in position encoding between distant Patches, avoiding confusion of the spatial order of features.

[0064] After completing the input processing, the Transformer encoder feature processing link is entered: a 6-layer encoder stacking structure is constructed, each layer containing a multi-head self-attention module and a feedforward neural network module, and the 6-layer design can balance between capturing global features and controlling computing cost, each sub-vector independently learns attention weights of different dimensions, for example, some attention heads focus on the edge correlation of ground objects, such as the boundary connection of roads and buildings, and some focus on the spectral feature correlation, such as the spectral difference between vegetation and water bodies, through multi-view attention learning, the complex correlation between patches can be fully captured; the initial attention score is obtained by calculating the Query, Key, and Value matrices, and then the weight is adjusted according to the similarity of the patch space and the feature, and after normalization by the Softmax function, the weighted sum is generated, which generates the attention-enhanced feature vector. The vector strengthens the key correlation features and weakens the irrelevant patch interference. The feedforward neural network module adopts the process of linear transformation -> GELU activation -> linear transformation, and the hidden layer dimension is set to 4096, which can provide sufficient nonlinear expression space. The attention-enhanced feature vector is mapped to a high-dimensional space for complex feature combination, and then the GELU activation function is used to realize smooth nonlinear conversion, and finally it is converted back to 1024 dimensions; residual connection and LayerNorm normalization are added after each encoder to effectively alleviate the gradient decay in the training process, which improves the convergence speed by more than 35%. After the 6-layer encoder processing is completed, the global average pooling is performed on the output 49 1024-dimensional feature vectors, and the mean value of all patch features is calculated to obtain a 1024-dimensional global feature vector containing multi-source mapping data fusion information, image spectrum, point cloud three-dimensional, vector attribute and global spatial correlation, ground object overall distribution, topological relationship, which provides comprehensive feature support for subsequent recognition tasks.

[0065] The beneficial effects are that through the multi-modal unified representation and adaptive fusion mechanism, the complementary information between different geographic data sources is fully tapped, the information silo problem is solved, and the interpretation robustness and accuracy in complex scenes such as shadow areas and occluded areas are significantly improved. The end-to-end processing process reduces manual intervention, realizes the overall automation of geographic information extraction, improves the processing efficiency compared with traditional methods, and improves the recognition efficiency and accuracy of geographic mapping data.

[0066] Please refer to Figure 2 In the geographic mapping data processing method based on big data, a big data processing platform is established based on Hadoop and Spark, geographic mapping data is collected in real time, and the collected original data is preprocessed by intelligent format conversion, adaptive coordinate system unification and dynamic noise filtering based on the big data processing platform. The preprocessed mapping data includes the following steps:

[0067] Step 201, extract the metadata information of the geographic mapping data, identify the current coordinate system type, and establish a conversion mapping table between common coordinate systems based on the mapping industry standard;

[0068] Step 201, use the Proj4 geographic projection library to perform batch conversion on data of different coordinate systems, and calculate conversion error in real time during the conversion process. If the error exceeds the threshold, adjust the conversion parameters;

[0069] Step 201, perform statistical analysis on the preprocessed data through SparkCore, identify noise types, and use K-neighbor filtering algorithm to denoise the noise data to obtain preprocessed mapping data.

[0070] The above describes the embodiments of the geographic mapping data processing method based on big data of the present application. Please refer to Figure 3 In the geographic mapping data processing system based on big data, the geographic mapping data processing system comprises the following modules:

[0071] The mapping data acquisition module is used to establish a big data processing platform based on Hadoop and Spark, to collect geographic mapping data in real time, and to perform intelligent format conversion, adaptive coordinate system integration and dynamic noise filtering preprocessing on the collected raw data based on the big data processing platform to obtain preprocessed mapping data.

[0072] The data feature extraction module is used to input the preprocessed multi-source data into a shared backbone network based on an improved ConvNeXt architecture, to combine the shared backbone network with a private branch network, and to extract shared features and private features of the data.

[0073] The data feature fusion module is used to fuse the shared features and the private features by using a spatial attention mechanism and a channel attention mechanism to generate a self-adaptive weighted fusion feature map.

[0074] The mapping data recognition module is used to input the fusion feature map into a Transformer encoder to recognize the environmental state and output a geographic information recognition result.

[0075] The basic principles, main features and advantages of the present application are shown and described above. It should be understood by those skilled in the art that the present application is not limited by the above embodiments, and the above embodiments and descriptions in the specification are only preferred examples of the present application and are not intended to limit the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and equivalents.

Claims

1. A geographic surveying and mapping data processing method based on big data, characterized in that, The geographic surveying and mapping data processing method includes the following steps: A big data processing platform based on Hadoop and Spark is built to collect geographic mapping data in real time. The platform performs intelligent format conversion, adaptive coordinate system 1, and dynamic noise filtering on the collected raw data to obtain preprocessed mapping data. The preprocessed multi-source data is input into a shared backbone network based on the improved ConvNeXt architecture. The shared backbone network is combined with a private branch network to extract the shared and private features of the data. The shared and private features are fused using spatial attention and channel attention mechanisms to generate an adaptively weighted fused feature map; The fused feature map is input into the Transformer encoder to identify the environmental state and output the geographic information identification result. The process involves inputting the preprocessed multi-source data into a shared backbone network based on an improved ConvNeXt architecture, combining the shared backbone network with private branch networks, and extracting the shared and private features of the data, including: The original ConvNeXt convolutional residual blocks are replaced with depthwise convolutions, the ReLU activation function is replaced with the GELU activation function, and a LayerNorm normalization layer is added after each convolutional layer to obtain a shared backbone network. Preprocessed mapping data is input into an improved shared backbone network, and the Adam optimizer is used for training with the goal of minimizing feature reconstruction error. The process involves inputting the preprocessed multi-source data into a shared backbone network based on an improved ConvNeXt architecture, combining the shared backbone network with private branch networks, and extracting the shared and private features of the data, including: For different types of surveying and mapping data, three types of private branch networks are designed, including private branches for image data, private branches for point cloud data, and private branches for vector data. Remote sensing images are input into the shared backbone to the image private branch, and LiDAR point clouds are input into the shared backbone to the point cloud private branch. During the network forward propagation, the shared backbone outputs shared features, and the private branch outputs private features. The process of fusing shared and private features using spatial attention and channel attention mechanisms to generate an adaptively weighted fused feature map includes: We acquire shared and private features, unify the spatial dimensions of the two types of features through convolution, and perform global average pooling on the feature map after unification to obtain spatial attention-enhanced features. Global max pooling and global average pooling are performed on the spatial attention enhancement features. The two results are concatenated and then input into the fully connected layer to generate the channel weight vector. The channel weight vector is multiplied with the spatial attention enhancement feature channel by channel to generate an adaptively weighted fusion feature map; The step of inputting the fused feature map into the Transformer encoder to identify the environmental state and outputting the geographic information identification result includes: The fused feature map is input into the Transformer encoder, and forward propagation is performed to obtain the geographic information recognition result. The result is then filtered by a threshold and output as the geographic information recognition result.

2. The geographic mapping data processing method based on big data as described in claim 1, characterized in that, The aforementioned big data processing platform, built upon Hadoop and Spark, collects geographic mapping data in real time. Based on this platform, it performs intelligent format conversion, adaptive coordinate system 1, and dynamic noise filtering preprocessing on the collected raw data to obtain preprocessed mapping data, including: Deploy core components based on the Hadoop ecosystem, including the HDFS distributed file system and the YARN resource scheduling framework, integrate the Spark framework to deploy the SparkCore basic computing engine, the SparkStreaming real-time data processing module, and the SparkSQL structured data query module, and establish a big data processing platform. Real-time acquisition of geographic mapping data, including at least point cloud data, image stream data, and remote sensing satellite image data; SparkSQL reads file information of geographic mapping data, identifies data format, generates data format labels, and performs adaptation conversions for data of different formats.

3. The geographic mapping data processing method based on big data as described in claim 1, characterized in that, The aforementioned big data processing platform, built upon Hadoop and Spark, collects geographic mapping data in real time. Based on this platform, it performs intelligent format conversion, adaptive coordinate system 1, and dynamic noise filtering preprocessing on the collected raw data to obtain preprocessed mapping data, including: Extract metadata information from geographic surveying and mapping data, identify the current coordinate system type, and establish a conversion mapping table between common coordinate systems based on surveying and mapping industry standards; The Proj4 geoprojection library is used to perform batch transformations of data in different coordinate systems. The transformation error is calculated in real time during the transformation process, and the transformation parameters are adjusted if the error exceeds the threshold. Statistical analysis of the preprocessed data was performed using SparkCore to identify noise types, and the K-nearest neighbor filtering algorithm was used to reduce noise in the data to obtain preprocessed mapping data.

4. A geographic surveying and mapping data processing system based on big data, used to execute the geographic surveying and mapping data processing method based on big data as described in claim 1, characterized in that, The geographic mapping data processing system includes the following modules: The mapping data acquisition module is used to build a big data processing platform based on Hadoop and Spark, collect geographic mapping data in real time, and perform intelligent format conversion, adaptive coordinate system 1 and dynamic noise filtering preprocessing on the collected raw data based on the big data processing platform to obtain preprocessed mapping data. The data feature extraction module is used to input preprocessed multi-source data into a shared backbone network based on the improved ConvNeXt architecture, and combine the shared backbone network with the private branch network to extract the shared and private features of the data. The data feature fusion module is used to fuse the shared features and private features using spatial attention mechanism and channel attention mechanism to generate an adaptive weighted fusion feature map; The mapping data recognition module is used to input the fused feature map into the Transformer encoder to recognize the environmental state and output the geographic information recognition result.

5. The geographic mapping data processing system based on big data as described in claim 4, characterized in that, The data feature fusion module Includes the following sub-modules: The convolution submodule is used to obtain shared and private features. It unifies the spatial dimensions of the two types of features through convolution operations, and performs global average pooling on the feature map after unification to obtain spatial attention-enhanced features. The generation submodule is used to perform global max pooling and global average pooling on the spatial attention enhancement features, and the two results are concatenated and input into the fully connected layer to generate the channel weight vector. The multiplication submodule is used to multiply the channel weight vector with the spatial attention enhancement feature channel by channel to generate an adaptively weighted fusion feature map.

6. The geographic mapping data processing system based on big data as described in claim 4, characterized in that, The data feature fusion module includes the following sub-modules: The output submodule is used to input the fused feature map into the Transformer encoder, perform forward propagation to obtain the geographic information recognition result, filter it through threshold judgment, and output the geographic information recognition result.

Images