Remote sensing data processing method and device, equipment and storage medium

Abstract

The invention discloses a remote sensing data processing method and device, equipment and a storage medium. The remote sensing data processing method comprises the steps of firstly reading the number of first Polyons of a vector file, the number of ring midpoints and resolution data of a tif file corresponding to the vector file; inputting the first Polygon number and the ring midpoint number into a preset ring complex coefficient processing module, processing the first Polygon number and the ring midpoint number by the ring complex coefficient processing module, and outputting a corresponding ring complex coefficient; processing the first Polygon number, the resolution data and the ring complex coefficient by a factor processing module to obtain a second Polygon number of a single vector file; and segmenting the vector file and the tif file according to the second Polygon number, and performing parallel processing to obtain a final segmentation data set, thereby solving the technical problems of low efficiency and overlong time consumption caused by sequential calculation of single tasks when the tif file segmented by the existing geographic space data processing library is relatively large.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing data processing, and in particular to a remote sensing data processing method, device, equipment and storage medium. Background Art

[0002] Remote sensing data processing is an important part of remote sensing technology application, which includes multiple steps to ensure the availability and accuracy of image data. The following are some key steps in remote sensing image preprocessing: Geometric correction: This is the process of ensuring that image data matches the real-world geographic location. It includes geolocation, geometric correction, image registration, orthorectification, etc. Geometric correction can correct systematic and non-systematic geometric deformations caused by sensors. This usually involves selecting ground control points (GCPs), establishing geometric correction models, and performing image resampling. Image fusion: This step combines image data of different resolutions to improve the spatial resolution and spectral information of the image. For example, the newly added GS fusion method can fuse multispectral data and panchromatic data to obtain higher-resolution multispectral data. Image mosaicking: Stitching multiple images together to form a continuous, seamless large image. Image cropping: Crop the image to a specific area as needed. Cloud and shadow processing: Remove clouds and shadows from the image for easy analysis. Atmospheric correction: This step is used to eliminate the impact of the atmosphere on the image, making the image closer to the true reflectance of the surface.

[0003] Among them, the Shapefile (.shp) file in remote sensing data is a commonly used geographic information system (GIS) vector data format, which contains geographic spatial information, such as points, lines, and polygons. For example, in remote sensing data processing, if a shpfile vector map of a certain crop (such as corn) is identified in a certain place, it is necessary to segment the remote sensing data (such as ndvi of corn) based on this shp file to specifically study the growth characteristics of a single crop.

[0004] Currently, the existing large-area Shapefile (.shp) vector data is cut into multiple small-area raster (.tif) data, usually for more detailed geospatial analysis. This process can be completed by geospatial systems or programming libraries. As well as the technology of cutting shp into tif, it is usually using command line tools, and geospatial data processing libraries can be used to cut raster data.

[0005] However, although the use of the geospatial data processing library is easy to implement and the results are feasible, it takes too long when the shp file is large and the tif file to be cut is also large. For example, when the shp file is 88M and the tif file is 54M, it takes at least 8430 seconds, about 2 hours and 20 minutes. Analyzing the internal algorithm implementation, the core algorithm is to determine whether each point in the tif file falls on a certain ring of the shpfile (the relationship between points and polygons). Since there are a large number of rings in the large shpfile (each ring may have multiple inner rings in addition to an outer ring), this leads to the situation that when the tif is too large (the amount of point data is too large) and the shpfile is too large (the number of rings is too large), the single task is calculated sequentially inefficiently and takes too long.

[0006] Therefore, when the tif files cut by the current geospatial data processing library are large, it leads to technical problems such as low efficiency and long time consumption in sequential calculation of single tasks. Summary of the invention

[0007] The present application provides a remote sensing data processing method, device, equipment and storage medium to solve the technical problem that when the tif file cut by the existing geospatial data processing library is large, the calculation efficiency of a single task is low and the time consumption is too long.

[0008] In view of this, the first aspect of the present application provides a remote sensing data processing method, the method comprising:

[0009] Read the number of the first Polygon and the number of points in the ring from the vector file;

[0010] Read the resolution data of the tif file corresponding to the vector file;

[0011] Input the number of the first Polygons and the number of the ring midpoints into a preset ring complexity coefficient processing module, and the ring complexity coefficient processing module processes and outputs the corresponding ring complexity coefficient;

[0012] The factor processing module processes the first number of Polygons, the resolution data and the ring complexity coefficient to obtain the second number of Polygons of the single vector file;

[0013] The vector file and the tif file are divided according to the number of the second Polygons, and the final cut data set is obtained by parallel processing.

[0014] A second aspect of the present application provides a remote sensing data processing device, the device comprising:

[0015] A reading module, used to read the number of the first Polygon of the vector file, the number of points in the ring, and the resolution data of the tif file corresponding to the vector file;

[0016] A ring complexity coefficient processing module, used for processing the number of the first Polygons and the number of the ring midpoints and outputting the corresponding ring complexity coefficient;

[0017] A factor processing module, used for processing the first number of Polygons, the resolution data and the ring complexity coefficient to obtain the second number of Polygons of a single vector file;

[0018] A parallel processing unit is used to divide the vector file and the tif file according to the number of the second Polygons, and to perform parallel processing to obtain a final cut data set.

[0019] A third aspect of the present application provides a remote sensing data processing device, the device comprising a processor and a memory:

[0020] The memory is used to store the program code and transmit the program code to the processor;

[0021] The processor is used to execute the steps of the remote sensing data processing method of the first aspect according to the instructions in the program code.

[0022] A fourth aspect of the present application provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the method of the first aspect above.

[0023] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0024] In the present application, a remote sensing data processing method, apparatus, device and storage medium are provided, which read the first number of Polygons and the number of ring midpoints of a vector file; read the resolution data of a tif file corresponding to the vector file; input the first number of Polygons and the number of ring midpoints into a preset ring complexity coefficient processing module, and the ring complexity coefficient processing module processes and outputs the corresponding ring complexity coefficient; the factor processing module processes the first number of Polygons, the resolution data and the ring complexity coefficient to obtain the second number of Polygons of a single vector file; the vector file and the tif file are divided according to the second number of Polygons, and the final cut data set is obtained by parallel processing, thereby solving the technical problem that when the tif file cut by the existing geospatial data processing library is large, the single task is calculated sequentially inefficiently and takes too long.

[0025] At the same time, in this application, by splitting the large shp file, the large shp data is split into multiple small shp data, the corresponding tif file is split, and then the small shp file is used to cut the corresponding small tif file, and finally the cut small tifs are combined to obtain the final result, which greatly improves the calculation speed when using it to cut the corresponding tif file.

[0026] It also avoids the computer crash caused by long-term calculations and the waste of time resources by single-process tasks. The large numbers calculated in the early stage are all recorded in the original shpfile, which can be directly read without traversal, achieving the beneficial effect of high efficiency. Compared with the traditional mode, which takes two and a half hours to cut, after the above algorithm optimization, the corresponding tif files are split, and then the corresponding small tif files are cut with small shp files. Finally, the final result of the combined calculation of the cut small tif files can be completed in more than 10 minutes, which greatly improves the calculation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A method flow chart of a remote sensing data processing method in an embodiment of the present application;

[0028] Figure 2 This is a schematic diagram of the structure of the remote sensing data processing device in the embodiment of the present application;

[0029] Figure 3 Schematic diagram of the structure of the remote sensing data processing device in the embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] The present application designs a remote sensing data processing method, which reads the first Polygon number, the number of ring midpoints, and the resolution data of the tif file corresponding to the vector file in a vector file; inputs the first Polygon number and the number of ring midpoints into a preset ring complexity coefficient processing module, which processes and outputs the corresponding ring complexity coefficient; processes the first Polygon number, the resolution data and the ring complexity coefficient by a factor processing module to obtain the second Polygon number of a single vector file; divides the vector file and the tif file according to the second Polygon number, and processes them in parallel to obtain the final cut data set, thereby solving the technical problem that when the tif file cut by the existing geospatial data processing library is large, the single task is calculated sequentially inefficiently and takes too long.

[0032] For easier understanding, see Figure 1 , Figure 1 is a flow chart of the remote sensing data processing method in the embodiment of the present application, such as Figure 1 As shown, specifically:

[0033] S101, reading the number of the first Polygon of the vector file, the number of the midpoints of the ring, and the resolution data of the tif file corresponding to the vector file;

[0034] When you need to cut tif for a large area, first read the number of the first Polygon and the number of midpoints in the ring of the vector file.

[0035] It should be noted that the aforementioned reading of the number of first Polygons and the number of midpoints in the ring of the vector file requires first obtaining the header file information of the vector file;

[0036] Read the header file information to determine the number of the first Polygons and the number of the midpoints of the ring;

[0037] Secondly, read the grid size of the raster data to be cut from the vector file.

[0038] After that, read the number of polygons and the number of all ring midpoints, as well as the resolution of the tif file from the shp file header information. In the .shp header file, read the number of Polygons NumParts in the shapefile, recorded as the component P_n, the number of all ring midpoints is recorded as the component p_k, and read the shape size (n*m) of the tif file to be cut, recorded as T_n.

[0039] 102. Input the number of the first Polygons and the number of the ring midpoints into a preset ring complexity coefficient processing module, and the ring complexity coefficient processing module processes and outputs a corresponding ring complexity coefficient;

[0040] In this embodiment, the number of the first Polygons and the number of the ring midpoints are input into a preset ring complexity coefficient processing module, and the ring complexity coefficient processing module processes and outputs the corresponding ring complexity coefficient, including:

[0041] Determine a ring point number density function of a single ring according to the number of the first Polygons and the number of points in the ring;

[0042] The ring point number density function is processed by a Gaussian distribution function to obtain a corresponding ring complexity coefficient.

[0043] The specific process can be as follows:

[0044] (1) Calculate the complexity coefficient of the shp ring. The complexity of a single ring in the shp file is determined by the number of points that make up the ring. The number of points will be a random variable, and the ring point density function μ, mean μ = p k / P n -1, where P k is the number of points in the ring, P n is the number of the first Polygon NumParts (the ring is connected end to end);

[0045] (2) The ring point density function conforms to the Gaussian distribution, so the shp ring complexity coefficient is Where μ is the ring point number density function;

[0046] In this embodiment, it should be noted that polygon is a geometric object used to represent a closed area defined by multiple points. In the geospatial system, the Polygon object usually has the following characteristics:

[0047] Exterior Ring: defines the boundary of a polygon. A Polygon needs at least one exterior ring.

[0048] Interior Rings: Also called holes, they define the empty space inside a polygon. Interior rings are optional, and a Polygon can have no interior rings or multiple interior rings.

[0049] Vertices: Points that make up the boundary of a polygon. Each vertex is usually represented by a pair of coordinates (x, y).

[0050] It can be further understood that before reading the first Polygon number and the number of points in the ring of the vector file, the following is also included:

[0051] Determine whether the vector file outline and the resolution data of the tif file exceed a threshold;

[0052] If so, a minimum intersection deletion operation is performed on the vector file outline and the resolution data.

[0053] The above operation is used to remove scraps. If the overall contour of the shpfile file is too different from the overall contour of the tif to be cut, the scraps can be removed by taking the minimum intersection of the two overall maximum contours to save computing resources for the subsequent process.

[0054] To remove the scraps between Shapefile (.shp) and raster data (.tif), that is, to ensure that the boundaries of the two match as much as possible, this can be achieved by calculating the minimum intersection of their boundaries. This can reduce unnecessary calculations in subsequent processing and improve efficiency.

[0055] Here are the steps to implement this process using the relevant libraries: Read Shapefile and Raster Data: Use the Geodata Processing library to read Shapefile and raster data using rasterio. Calculate Boundaries: Extract boundaries from Shapefile and raster data. Calculate Minimum Intersection: Use the Shapely library to calculate the minimum intersection of the two boundaries. Crop Shapefile: Crop the Shapefile based on the minimum intersection. Save the cropped Shapefile: Save the cropped Shapefile as a new file.

[0056] The aforementioned geospatial data processing library provides support for reading and writing a variety of raster and vector data formats. The geospatial data processing library was originally developed for processing raster data, and was later merged with OGR (for processing vector data) to form a unified library that can process both raster and vector data.

[0057] Its main functions include: supporting multiple raster data formats. Providing operations such as data conversion, reprojection, clipping, merging, segmentation and statistics. Supporting multiple coordinate systems and projection conversions. Providing interfaces to facilitate geospatial data processing and analysis. The data model of the geospatial data processing library includes concepts such as dataset, raster band, and color table. A dataset is a collection of raster bands, responsible for geographic reference definition and coordinate system definition. The coordinate system is represented by an OGC WKT string, including information such as the full name of the coordinate system, the name of the geographic coordinate system, the datum, the ellipsoid name, the prime meridian, and the projection method. The geospatial data processing library also supports affine geographic transformation and geographic control points (GCP) to describe the geographic reference of raster datasets. Affine transformation converts pixel / row coordinates to geographic reference space through six parameters. GCP associates raster positions with geographic reference systems through a set of control points. Geographic control points (GCP) are one of the methods used by the geospatial data processing library to describe the geographic reference of raster datasets. Using this approach, a dataset will have a set of control points that associate a raster position with one or more locations in a geographic reference system. All control points share a geographic reference coordinate system. Each control point (described by the Geospatial Data Processing Library - GCP class) consists of the following parts: a unique identifier (pszId), information (pszInfo), pixel position (dfGCPPixel and dfGCPLine), and a geographic reference position (dfGCPX, dfGCPY, and dfGCPZ). GCPs can be used to establish a conversion relationship from pixel coordinates to geographic coordinates, which is usually used for geometric correction of images. The Geospatial Data Processing Library provides the Geospatial Data Processing Library conversion tool, which can be used to convert coordinates based on GCPs. This tool can handle a variety of conversion methods including GCPs, and can read coordinate pairs from the command line for conversion. When using the Geospatial Data Processing Library conversion tool, GCPs can be provided through the -gcp parameter in the format of -gcp <pixel> <line> <easting> <northing>[elevation]. When performing geometric correction, you can use the geospatial data processing library .Warp function, which supports multiple resampling algorithms and can use GCPs for correction. The correction process involves establishing a mathematical relationship between pixel coordinates and geographic coordinates, then creating a new image and filling the pixel values ​​of the new image using resampling techniques.

[0058] In addition, the geospatial data processing library also provides command line tools for performing tasks such as data conversion and terrain analysis. These tools can be called directly from the command line or used through a graphical interface in the geospatial system.

[0059] It can be understood that the geospatial data processing library is used for reprojection, transformation and mosaicking of images. Reprojection: The source and target coordinate systems can be specified using the -s_srs and -t_srs parameters. Setting output resolution: The resolution of the output image can be set using the -tr parameter. Cropping: The geographic extent of the output image can be specified using the -te parameter. Setting output dimensions: The -ts parameter is used to set the pixel dimensions of the output image. Using overview levels: The -ovr parameter can specify which overview level of the source file to use. Setting memory limits: The -wm parameter is used to control the amount of memory used by the geospatial data processing library when processing. Multithreaded processing: The -multi parameter can enable multithreaded processing for improved efficiency. Output format: The -of parameter is used to specify the format of the output file. Creation options: The -co parameter is used to set format-specific creation options, such as compression type.

[0060] It should be noted that the tif file mentioned in this embodiment is also often called TIFF file (Tagged Image File Format), which is a flexible and widely used image file format, mainly used to store raster image data. The TIFF format supports a variety of colors and data types, including black and white, grayscale, RGB, CMYK and more. It also supports a variety of different compression algorithms, such as lossless compression (such as LZW, Deflate) and lossy compression (such as JPEG compression). The main features of the TIFF format include: Compatibility: TIFF is a cross-platform format that can be used in a variety of operating systems. Flexibility: Supports a variety of color modes, data types and compression algorithms. High quality: Commonly used in the image scanning and printing industry because it can store image data losslessly. Metadata support: Additional information about the image can be stored, such as camera settings, copyright information, etc. Multi-page: TIFF files can contain multiple images, similar to an image collection.

[0061] In geospatial data processing, TIFF files are often used to store raster data in geographic information systems. These data usually contain geographic reference information, such as coordinate system, pixel size, and the location of the image in the real world. Geospatial data processing libraries provide extensive support for TIFF files, including reading, writing, and processing these files.

[0062] If TIFF files need to be processed, you can use the Geospatial Data Processing Library command line tools or Geospatial Data Processing Library bindings to perform operations. For example, you can use Geospatial Data Processing Library_translate to convert the format of a TIFF file, and use Geospatial Data Processing Library to perform reprojection or rectification.

[0063] S103, a factor processing module processes the first number of Polygons, the resolution data and the ring complexity coefficient to obtain the second number of Polygons of a single vector file;

[0064] It should be noted that the segmentation factor of the vector file is obtained by taking the logarithm and square root algorithm according to the number of the first Polygons, the resolution data and the ring complexity coefficient;

[0065] The number of second Polygons corresponding to a single vector file is obtained through the segmentation factor.

[0066] In one embodiment, the segmentation is not only related to the complexity of the shpfile, but also has a positive correlation with the size of the tif itself. Therefore, the number of regions of the shpfile segmented by the algorithm is as follows:

[0067] Among them, P n is the number of the first Polygon NumParts, T n is the shape size of the tif file that needs to be cut (n*m), and ρ is the shp ring complexity coefficient.

[0068] In addition to being related to the complexity of the shpfile, cutting is also about finding a suitable l value, which can help decide how to split the Shapefile into smaller parts in order to reduce the consumption of computing resources when processing large-size raster data.

[0069] The process can use the math library to calculate logarithms and square roots. The math.log2() function is used to calculate the logarithm with base 2, which is a common logarithm base when dealing with calculations related to information theory. Make sure the values ​​of Pn and Tn are correct because they directly affect the calculation results of l. In addition, the value of ρ should also be adjusted according to actual conditions to reflect the relationship between resource consumption and performance requirements during processing.

[0070] It can be further explained that the number of the first Polygon in a single shpfile is approximately:

[0071] P single ≈P n / l

[0072] It is used to estimate how many Polygons should be included in each Shapefile area after segmentation when cutting raster data. single is the approximate number of Polygons in each segmented area. Pn is the total number of Polygons in the Shapefile, and l is the segmentation factor calculated by the previous formula, which is used to determine how many regions the Shapefile is segmented into. This estimate can help us decide how many Polygons each segment should contain, so that we can balance the computational load and improve efficiency when processing raster data.

[0073] In practice, adjusting the value of ρ (called the "adjustment factor") is an empirical process that depends on a variety of factors, including computing resources, data complexity, and desired accuracy. Here are some steps and suggestions to help adjust the value of ρ to improve the accuracy of the calculation based on actual geographic data: Understand the role of ρ: ρ is an adjustment factor that balances the impact of shapefile complexity and raster data size on the number of segmentations. Higher ρ values ​​increase the number of segmentations, which may improve the accuracy of the processing, but also increase the complexity of the calculation and resource consumption. Lower ρ values ​​reduce the number of segmentations, which may reduce resource consumption, but may reduce the accuracy of the processing. Benchmarking: Benchmark at different ρ values ​​to observe the impact on calculation time and result accuracy. Choose a baseline ρ value and then adjust it step by step to observe the changes. Consider data characteristics: Analyze the number, size, and complexity of polygons in the shapefile. Analyze the size and resolution of the raster data. Consider computing resources: Adjust the ρ value based on the available computing resources (such as CPU, memory). If resources are limited, you may need to choose a smaller ρ value. Iterative adjustment: Initially set a ρ value and perform a complete calculation. Analyze the results. If you find that the computing resource consumption is too high or the results are not accurate enough, increase or decrease the ρ value as needed. Automated adjustment: Consider developing an automated adjustment mechanism to dynamically adjust the ρ value based on real-time calculation feedback. Record and analyze: Record the calculation results and resource consumption after each adjustment of the ρ value. Analyze the data to find the optimal ρ value. Consider the actual application scenario: Adjust the ρ value according to the needs of the actual application. For example, if the application is very sensitive to computing time, you may need to choose a smaller ρ value. Software and tools: Use the analysis functions in the geospatial system or programming tools to help determine the optimal ρ value. Ultimately, adjusting the value of ρ is a trial and error process that requires fine-tuning based on actual conditions and needs. Through the above steps, adjustments can be made more systematically to find the ρ value that best suits the data and needs.

[0074] S104: Split the vector file and the tif file according to the number of the second Polygons, and process them in parallel to obtain a final cut data set.

[0075] Optionally, the step S104 specifically includes:

[0076] Read the vector file and the polygons corresponding to the number of the second Polygons;

[0077] Reorganize the polygons corresponding to the number of the second Polygons to generate at least two new vector files, and generate a new vector file set based on all the new vector files;

[0078] Segment the tif file according to the latest circumscribed rectangle in the new vector file to generate a new tif file set including at least 2 segmented tif files;

[0079] The new vector file and the segmented tif file are processed in parallel to obtain a final segmentation data set.

[0080] It can be understood that the number of polygons corresponding to the number of the second Polygon is read from the vector file;

[0081] It should be further explained that the new vector file and the segmented tif file are processed in parallel to obtain the final segmentation data set, specifically including:

[0082] Processing the new vector file and the segmented tif file in parallel to extract corresponding segmentation result data;

[0083] The segmentation result data are integrated into a final cutting data set.

[0084] Split shapefile and tif original file: According to the above calculation results P single , take the integer P, read the shpfile file, read P polygons from the shpfile each time, reorganize the new small shpfile, and continue until all the polygons are taken, denoted as SF1, SF2, ...SF i ...、SF m ,m≦l, at the same time, according to SF i The minimum bounding rectangle in the information splits the original tif file and obtains TF i ;

[0085] The process of splitting Shapefile (.shp) and the corresponding tif file (.tif) can be divided into the following steps:

[0086] 1. Calculate the number of Polygons contained in each segment: Based on the previous calculation, we get the approximate value Psingle of the number of Polygons that each segment should contain. Round this value up to get P.

[0087] 2. Read Shapefile: Use the geo data processing library to read the original Shapefile file.

[0088] 3. Split Shapefile: Each time P Polygons are read from the Shapefile, a new small Shapefile is created until all Polygons are read.

[0089] 4. Split the raster file: For each new small Shapefile, calculate its minimum bounding rectangle and use this rectangle to crop the original raster file (.tif).

[0090] 5. Save the segmented files: Save each segmented small Shapefile and the corresponding raster file as a new file.

[0091] Parallel processing: SF i Cut TF i , that is, the result set of small modules F = {f i , i∈{1, 2, ...m}};

[0092] It is understandable that to achieve parallel processing, each small Shapefile (SF i ) Cut the corresponding raster file (TF i ), you can use Python's multithreading or multiprocessing libraries, such as asynchronous execution callable objects, to parallelize processing, you need to use, for example, thread pools and process pools to reproject multiple files in parallel. Parallel processing can significantly improve the efficiency of processing large amounts of data.

[0093] Each small Shapefile (SF i ) Cut the corresponding raster file (TF i ) can be implemented as follows:

[0094] 1. Read Shapefile: Use the geographic data processing library to read the original Shapefile file.

[0095] 2. Calculate the number of Polygons contained in each segment: According to the previous calculation results, round up to get the number of Polygons in each segment.

[0096] 3. Define the cutting function: The clip_raster function is used to cut each small Shapefile and the corresponding raster file.

[0097] 4. Use thread pool and process pool for parallel processing:

[0098] οCreate a thread pool and process pool instance.

[0099] οFor each small Shapefile, submit a task to the executor for parallel cutting.

[0100] ο Use the result instruction to get the result of each task.

[0101] Finally, we merge the elements in set F to get the calculation result: Where F = {f i , i∈{1, 2, ...m}}, which is understood as merging the elements in the result set F to obtain the final calculation result A.

[0102] It is understood that after completing the parallel processing of all the small modules, in GIS, this usually means merging all the small shapefiles into a large shapefile, or merging all the small raster files (.tif) into a large raster file.

[0103] It should be noted that when merging raster files, you need to ensure that the coordinate reference system (CRS) and resolution of all files are consistent, otherwise the merged result may be wrong. In addition, when merging shapefiles, you also need to ensure that they have the same geometry type and coordinate reference system.

[0104] Furthermore, when merging large amounts of GIS data, it is very important to optimize the data processing process to improve efficiency. Use efficient libraries and tools: Select GIS libraries with excellent performance, such as the Geospatial Data Processing Library, which provide optimized data processing functions. Parallel processing: Use parallel processing to accelerate data processing by leveraging the power of multi-core processors. Adopting asynchronous execution callable object modules can be used for parallel processing. Batch processing: Dividing large data sets into smaller batches for processing can reduce memory consumption and increase processing speed. Memory management: Monitor and manage memory usage to avoid memory leaks. Data simplification: Simplify data structures without affecting analysis results, such as reducing the number of vertices of vector data or reducing the resolution of raster data. Use spatial indexes: Use spatial indexes (such as R-trees) to speed up spatial queries and data filtering, which can significantly increase data processing speed. Optimize read and write operations: Reduce disk I / O operations and optimize read and write performance through buffering, batch writing, and reasonable selection of data formats. Use cloud services: Use GIS services and elastic computing resources provided by cloud computing platforms to expand processing capabilities on demand. Data preprocessing: Before merging, preprocess the data, such as clipping, filtering, and reprojection, to reduce the amount of data that needs to be processed during the merging process. Use dedicated hardware: Using faster storage devices (such as SSD) and more powerful CPU / GPU can increase the speed of data processing. Avoid unnecessary data conversion: Try to minimize data format conversion, because the conversion process may consume a lot of time and resources. Code optimization: Optimize code logic, avoid unnecessary loops and complex query operations, and use vectorized operations to improve performance. Use appropriate data structures: Choose appropriate data structures according to data characteristics, such as using geographic data frames for vector data processing and using xarray for multidimensional raster data processing. Cache mechanism: For repeatedly used data, implement a cache mechanism to avoid repeated reading and processing. Staged processing: Decompose the entire processing flow into multiple stages, and each stage outputs intermediate results, so that problems can be quickly located and optimized when they occur. Monitoring and analysis: Use performance monitoring tools to analyze bottlenecks in the data processing process and optimize them in a targeted manner. Reasonable use of databases: For very large data sets, consider using spatial database object-relational database extensions to manage data, which provide optimized spatial data storage and query functions. Through these strategies, the efficiency of processing large amounts of GIS data can be significantly improved, and processing time and resource consumption can be reduced.

[0105] The aforementioned geographic data processing library is a library for geospatial data processing. Based on the extension of the data processing library, it inherits the data processing and analysis capabilities of the data processing library and can easily process tabular data. Geospatial data frames can be processed like ordinary data frames, making data filtering, aggregation, merging and other operations simple and efficient. Supports multiple geospatial data formats: can read and write common geospatial data formats such as Shapefile, GeoJSON, GPKG, etc. Convenient for data interaction with other geographic information system software. Powerful geospatial operation functions: provides a wealth of geospatial operation functions, such as spatial connection, buffer analysis, overlay analysis, etc. Spatial queries can be performed to quickly filter out geographic elements that meet specific spatial conditions. Its application scenarios: Geographic data analysis: statistical analysis of geospatial data, calculation of geometric attributes such as area, length, and perimeter. Perform spatial clustering analysis to discover clustering patterns in geographic space: map production combines the drawing library to produce various types of maps, including thematic maps, contour maps, etc. Customize the map annotation and color matching to make the map more intuitive and beautiful. Spatial planning and decision support: In the fields of urban planning, resource management, environmental protection, etc., spatial analysis is performed using geographic data processing libraries to support decision making. For example, assessing the spatial impact of different planning options, determining the scope of protected areas, etc.

[0106] It can be understood that the object-relational database mentioned in this embodiment is an object-relational database extension that adds the functionality of a geographic information system (GIS) to a SQL database. This enables the object-relational database extension to not only store spatial data, but also perform spatial queries and analysis. The following are some key points for managing spatial data using an object-relational database extension: Spatial data type: The object-relational database extension supports a variety of spatial data types, including geometry and geography types. Geometry types are used to represent points, lines, and polygons on a plane, while geography types are used to represent geographic locations on the earth. Spatial index: In order to improve the efficiency of spatial queries, the object-relational database extension uses an R-Tree index based on GiST. These indexes can help quickly locate spatial data, thereby speeding up the query process. Spatial function: The object-relational database extension provides a large number of spatial functions for performing various spatial analyses and operations, such as calculating the distance, area, and length between two geometric objects, and performing spatial relationship queries. Spatial reference system: The object-relational database extension allows users to define and use different spatial reference systems (SRS), which define how to associate geometric data with real-world locations. Data import and export: The object-relational database extension supports importing spatial data from a variety of formats, including Shapefile, GeoJSON, KML, etc., and also supports exporting data to these formats. Raster data processing: The object-relational database extension also supports the storage and processing of raster data, which is very useful when processing remote sensing imagery and terrain analysis. Metadata management: The object-relational database extension provides two tables, geometry_columns and spatial_ref_sys, for tracking and reporting geometry columns and spatial reference systems in the database. Data projection: The object-relational database extension allows users to project spatial data so that data can be represented and analyzed in different coordinate systems. Validity checking: The object-relational database extension provides tools to check the validity of geometric data and provides functions to repair invalid geometric objects. Triggers and historical edit tracking: The object-relational database extension supports the use of database triggers to track the edit history of data, which is very useful for data version control and change management. 3D and 4D data support: The object-relational database extension supports not only 2D geometric data, but also 3D and 4D data, allowing users to store and query spatial data with height and time dimensions. Raster data management: The object-relational database extension also provides support for the management, query, and analysis of raster data.

[0107] The remote sensing data processing method in this embodiment first reads the first Polygon number, the number of ring midpoints, and the resolution data of the tif file corresponding to the vector file; inputs the first Polygon number and the number of ring midpoints into a preset ring complexity coefficient processing module, and the ring complexity coefficient processing module processes and outputs the corresponding ring complexity coefficient; the factor processing module processes the first Polygon number, the resolution data and the ring complexity coefficient to obtain the second Polygon number of a single vector file; the vector file and the tif file are divided according to the second Polygon number, and the final cut data set is obtained by parallel processing, which solves the technical problem that when the tif file cut by the existing geospatial data processing library is large, the single task is calculated sequentially inefficiently and takes too long.

[0108] At the same time, in this embodiment, by splitting the large shp file, the large shp data is split into multiple small shp data, the corresponding tif file is split, and then the small shp file is used to cut the corresponding small tif file. Finally, the cut small tifs are combined and calculated to obtain the final result, which greatly improves the calculation speed when using it to cut the corresponding tif file.

[0109] In this embodiment, the computer crash caused by long-term calculation and the waste of time resources by single-process tasks are avoided. The large numbers calculated in the early stage are all recorded in the original shpfile, and can be directly read without traversal, achieving the beneficial effect of high efficiency. Compared with the two and a half hours of cutting required by the traditional mode, after the above algorithm optimization, the corresponding tif files are segmented, and then the corresponding small tif files are cut with small shp files. Finally, the final result can be obtained by the joint calculation of the cut small tif files, which can complete the task in more than 10 minutes, greatly improving the calculation speed.

[0110] See also Figure 2 , Figure 2 Schematic diagram of the structure of the remote sensing data processing device in the embodiment of the present application. Figure 2 As shown, specifically:

[0111] The reading module 201 is used to read the number of the first Polygon of the vector file, the number of the midpoints of the ring, and the resolution data of the tif file corresponding to the vector file;

[0112] In one embodiment, the first reading module 201 is specifically configured to:

[0113] Get the header file information of the vector file;

[0114] The number of the first Polygons, the number of points in the ring and the resolution of the tif file are determined according to the header file information.

[0115] A ring complexity coefficient processing module 202, used to process the number of the first Polygons and the number of the ring midpoints and output the corresponding ring complexity coefficient;

[0116] In one embodiment, the ring complexity coefficient processing module 202 is specifically used to:

[0117] Determine a ring point number density function of a single ring according to the number of the first Polygons and the number of points in the ring;

[0118] The ring point density function is processed by a Gaussian distribution function to obtain a corresponding ring complexity coefficient. A factor processing module 203 is used to process the first number of Polygons, the resolution data and the ring complexity coefficient by a factor processing module to obtain a second number of Polygons of a single vector file;

[0119] The factor processing module 203′ is specifically used for:

[0120] Obtaining the segmentation factor of the vector file by taking a logarithm and a square root algorithm according to the number of the first Polygons, the resolution data and the ring complexity coefficient;

[0121] The number of second Polygons corresponding to a single vector file is obtained through the segmentation factor.

[0122] The parallel processing module 204 is used to divide the vector file and the tif file according to the number of the second Polygons, and perform parallel processing to obtain a final cut data set.

[0123] The parallel processing module 204 includes:

[0124] A reading unit, used for reading polygons corresponding to the number of the second Polygons in the vector file;

[0125] a reorganization unit, configured to reorganize the polygons corresponding to the number of the second Polygons to generate at least two new vector files and generate a new vector file set;

[0126] A segmentation unit, used for segmenting the tif file according to the latest circumscribed rectangle in the new vector file, to generate a new tif file set including at least two segmented tif files;

[0127] The parallel processing unit is used to process the new vector file and the segmented tif file in parallel to obtain a final segmentation data set.

[0128] The parallel processing module is specifically used to: process the new vector file and the segmented tif file in parallel, extract the corresponding segmentation result data, and integrate the segmentation result data into a final cutting data set.

[0129] The remote sensing data processing device in this embodiment first reads the first Polygon number, the number of ring midpoints, and the resolution data of the tif file corresponding to the vector file; inputs the first Polygon number and the number of ring midpoints into a preset ring complexity coefficient processing module, which processes and outputs the corresponding ring complexity coefficient; processes the first Polygon number, the resolution data and the ring complexity coefficient by a factor processing module to obtain the second Polygon number of a single vector file; divides the vector file and the tif file according to the second Polygon number, and processes them in parallel to obtain the final cut data set, thereby solving the technical problem that when the tif file cut by the existing geospatial data processing library is large, the single task is calculated sequentially inefficiently and takes too long to calculate.

[0130] At the same time, in this embodiment, by splitting the large shp file, the large shp data is split into multiple small shp data, the corresponding tif file is split, and then the small shp file is used to cut the corresponding small tif file. Finally, the cut small tifs are combined and calculated to obtain the final result, which greatly improves the calculation speed when using it to cut the corresponding tif file.

[0131] In this embodiment, the computer crash caused by long-term calculation and the waste of time resources by single-process tasks are avoided. The large numbers calculated in the early stage are all recorded in the original shpfile, and can be directly read without traversal, achieving the beneficial effect of high efficiency. Compared with the two and a half hours of cutting required by the traditional mode, after the above algorithm optimization, the corresponding tif files are segmented, and then the corresponding small tif files are cut with small shp files. Finally, the final result can be obtained by the joint calculation of the cut small tif files, which can complete the task in more than 10 minutes, greatly improving the calculation speed.

[0132] The present application also provides another remote sensing data processing device, such as Figure 3 As shown, the device 10 includes:

[0133] One or more processors 110 and a memory 120 are described below by taking one processor 110 as an example. The processor 110 and the memory 120 may be connected via a bus or other methods, with connection via a bus being taken as an example.

[0134] The processor 110 is used to complete various control logics of the device 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISCMachine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0135] The memory 120 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the method for online processing of fixed asset data in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the method for online processing of fixed asset data in the above method embodiment.

[0136] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] One or more units are stored in the memory 120, and when executed by one or more processors 110, the following steps are implemented:

[0138] Read the number of the first Polygon and the number of points in the ring from the vector file;

[0139] Read the resolution data of the tif file corresponding to the vector file;

[0140] Input the number of the first Polygons and the number of the ring midpoints into a preset ring complexity coefficient processing module, and the ring complexity coefficient processing module processes and outputs the corresponding ring complexity coefficient;

[0141] The factor processing module processes the first number of Polygons, the resolution data and the ring complexity coefficient to obtain the second number of Polygons of the single vector file;

[0142] The vector file and the tif file are divided according to the number of the second Polygons, and the final cut data set is obtained by parallel processing.

[0143] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the above steps S101 to S104 are implemented.

[0144] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0145] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0146] In an embodiment of the present application, a remote sensing data processing method, apparatus, device and storage medium are provided, which first reads the first number of Polygons and the number of ring midpoints of a vector file; reads the resolution data of a tif file corresponding to the vector file; inputs the first number of Polygons and the number of ring midpoints into a preset ring complexity coefficient processing module, which processes and outputs the corresponding ring complexity coefficient; processes the first number of Polygons, the resolution data and the ring complexity coefficient by a factor processing module to obtain the second number of Polygons of a single vector file; divides the vector file and the tif file according to the second number of Polygons, and processes them in parallel to obtain a final cut data set, thereby solving the technical problem that when the tif file cut by the existing geospatial data processing library is large, the single task is calculated sequentially inefficiently and takes too long to calculate.

[0147] At the same time, in this embodiment, by splitting the large shp file, the large shp data is split into multiple small shp data, the corresponding tif file is split, and then the small shp file is used to cut the corresponding small tif file. Finally, the cut small tifs are combined and calculated to obtain the final result, which greatly improves the calculation speed when using it to cut the corresponding tif file.

[0148] In this embodiment, the computer crash caused by long-term calculation and the waste of time resources by single-process tasks are avoided. The large numbers calculated in the early stage are all recorded in the original shpfile, and can be directly read without traversal, achieving the beneficial effect of high efficiency. Compared with the two and a half hours of cutting required by the traditional mode, after the above algorithm optimization, the corresponding tif files are segmented, and then the corresponding small tif files are cut with small shp files. Finally, the final result can be obtained by the joint calculation of the cut small tif files, which can complete the task in more than 10 minutes, greatly improving the calculation speed.

[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0150] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.< / northing> < / easting> < / line> < / pixel>

Claims

1. A remote sensing data processing method, characterized in that: include: Read the number of the first Polygon of the vector file, the number of points in the ring, and the resolution data of the tif file corresponding to the vector file; Input the number of the first Polygons and the number of the ring midpoints into a preset ring complexity coefficient processing module, and the ring complexity coefficient processing module processes and outputs the corresponding ring complexity coefficient; The factor processing module processes the first number of Polygons, the resolution data and the ring complexity coefficient to obtain the second number of Polygons of a single vector file; The vector file and the tif file are divided according to the number of the second Polygons, and the final cut data set is obtained by parallel processing.

2. The remote sensing data processing method according to claim 1, characterized in that: The first Polygon number of the read vector file, the number of points in the ring, and the resolution data of the tif file corresponding to the vector file specifically include: Get the header file information of the vector file; The number of the first Polygons, the number of points in the ring and the resolution data of the tif file are determined according to the header file information.

3. The remote sensing data processing method according to claim 1, characterized in that: Inputting the first number of Polygons and the number of midpoints of the ring into a preset ring complexity coefficient processing module, and the ring complexity coefficient processing module processes and outputs the corresponding ring complexity coefficient, including: Determine a ring point number density function of a single ring according to the number of the first Polygons and the number of points in the ring; The ring point number density function is processed by a Gaussian distribution function to obtain a corresponding ring complexity coefficient.

4. The remote sensing data processing method according to claim 1, characterized in that: The factor processing module processes the first number of Polygons, the resolution data and the ring complexity coefficient to obtain the second number of Polygons of a single vector file, specifically including: Obtaining the segmentation factor of the vector file by taking a logarithm and a square root algorithm according to the number of the first Polygons, the resolution data and the ring complexity coefficient; The number of second Polygons corresponding to a single vector file is obtained through the segmentation factor.

5. The remote sensing data processing method according to claim 1, characterized in that: The vector file and the tif file are divided according to the number of the second Polygons, and the final cutting data set is obtained by parallel processing, including: Read the vector file and the polygons corresponding to the number of the second Polygons; Reorganizing the polygons to generate at least two new vector files; Segment the tif file according to the latest circumscribed rectangle in the new vector file to generate a new tif file set including at least 2 segmented tif files; The new vector file and the segmented tif file are processed in parallel to obtain a final segmentation data set.

6. The remote sensing data processing method according to claim 5, characterized in that: The new vector file and the segmented tif file are processed in parallel to obtain a final cutting data set, specifically including: Processing the new vector file and the segmented tif file in parallel to extract corresponding segmentation result data; The segmentation result data are integrated into a final cutting data set.

7. The remote sensing data processing method according to claim 1, characterized in that: The reading of the first Polygon number of the vector file, the number of points in the ring, and the resolution data of the tif file corresponding to the vector file also includes: Determine whether the vector file outline and the resolution data of the tif file exceed a threshold; If so, a minimum intersection deletion operation is performed on the vector file outline and the resolution data.

8. A remote sensing data processing device, characterized in that: include: A reading module, used to read the number of the first Polygon of the vector file, the number of points in the ring, and the resolution data of the tif file corresponding to the vector file; A ring complexity coefficient processing module, used for processing the number of the first Polygons and the number of the ring midpoints and outputting the corresponding ring complexity coefficient; A factor processing module, used for processing the first number of Polygons, the resolution data and the ring complexity coefficient to obtain the second number of Polygons of a single vector file; A parallel processing unit is used to divide the vector file and the tif file according to the number of the second Polygons, and to perform parallel processing to obtain a final cut data set.

9. A remote sensing data processing device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the remote sensing data processing method according to any one of claims 1 to 7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the remote sensing data processing method according to any one of claims 1 to 7.

Images