A remote sensing data processing method, apparatus, device, and storage medium

Abstract

This application discloses a remote sensing data processing method, apparatus, device, and storage medium. The method involves first reading the number of first polygons, the number of ring midpoints, and the resolution data of the corresponding TIFF file from a vector file; inputting the first polygon count and the number of ring midpoints into a preset ring complexity coefficient processing module, which processes and outputs the corresponding ring complexity coefficient; then, a factor processing module processes the first polygon count, the resolution data, and the ring complexity coefficient to obtain the second polygon count for a single vector file; finally, the vector file and the TIFF file are segmented based on the second polygon count, and parallel processing is performed to obtain the final segmented data set. This method solves the technical problem of low efficiency and excessive time consumption in sequential calculations for single tasks when the TIFF files segmented by existing geospatial data processing libraries are large.

Description

Technical Field

[0001] This application relates to the field of remote sensing data processing technology, and in particular to a remote sensing data processing method, apparatus, device and storage medium. Background Technology

[0002] Remote sensing data processing is a crucial aspect of remote sensing technology applications. It involves multiple steps to ensure the usability and accuracy of image data. Here are some key steps in remote sensing image preprocessing: Geometric Correction: This process ensures that image data matches real-world geographic locations. It includes geolocation, geometric fine correction, image registration, and orthorectification. Geometric fine correction corrects systematic and non-systematic geometric distortions caused by the sensor. This typically involves selecting ground control points (GCPs), establishing a geometric correction model, and image resampling. Image Fusion: This step combines image data of different resolutions to improve the spatial resolution and spectral information of the image. For example, the newly added GS fusion method can fuse multispectral and panchromatic data to obtain higher-resolution multispectral data. Image Mosaicing: This step stitches multiple images together to form a continuous, seamless large image. Image Cropping: Cropping images to specific areas as needed. Cloud and Shadow Removal: Removing clouds and shadows from the image to facilitate analysis. Atmospheric Correction: This step eliminates the influence of the atmosphere on the image, making the image closer to the true reflectance of the Earth's surface.

[0003] Among them, Shapefile (.shp) files in remote sensing data are a commonly used geographic information system (GIS) vector data format, which contains geospatial information such as points, lines, and polygons. For example, in remote sensing data processing, given a known shapefile vector map of a certain crop (such as corn) in a certain location, it is necessary to extract remote sensing data (such as the ndvi file for corn) from this shapefile for specific research on the growth characteristics of a single crop.

[0004] Currently, the existing practice of cutting large-area Shapefile (.shp) vector data into multiple smaller-area raster (.tif) data is typically for more refined geospatial analysis. This process can be accomplished using geospatial systems or programming libraries. The technique for cutting .shp into .tif data usually involves using command-line tools, and geospatial data processing libraries can be used to cut the raster data.

[0005] However, while geospatial data processing libraries are simple to implement and yield feasible results, they are time-consuming when the shapefile and the resulting tif file are both large. For example, when the shapefile is 88MB and the tif file is 54MB, the minimum processing time is 8430 seconds, approximately 2 hours and 20 minutes. Analyzing the internal algorithm, the core algorithm determines whether each point in the tif file falls on a ring in the shapefile (the relationship between points and polygons). Since large shapefiles contain a large number of rings (each ring may have multiple inner rings in addition to an outer ring), when the tif file is too large (too much point data) and the shapefile is too large (too many rings), the sequential calculation of a single task is inefficient and time-consuming.

[0006] Therefore, when the TIFF files segmented by the current geospatial data processing library are large, it leads to the technical problem of low efficiency and excessive time consumption when a single task is calculated sequentially. Summary of the Invention

[0007] This application provides a remote sensing data processing method, apparatus, device, and storage medium to solve the technical problem that when the existing geospatial data processing library has large TIFF files, the single task is inefficient and takes too long to calculate sequentially.

[0008] In view of the above, the first aspect of this application provides a remote sensing data processing method, the method comprising:

[0009] Read the number of the first polygon and the number of ring midpoints from the vector file;

[0010] Read the resolution data of the TIFF file corresponding to the vector file;

[0011] The number of the first Polygon and the number of the ring midpoints are input into the preset ring complexity coefficient processing module, which processes and outputs the corresponding ring complexity coefficient.

[0012] The factor processing module processes the first number of polygons, the resolution data, and the ring complexity coefficient to obtain the second number of polygons in a single vector file;

[0013] The vector file and the tif file are split according to the second number of Polygons, and the final split data set is obtained by parallel processing.

[0014] A second aspect of this application provides a remote sensing data processing apparatus, the apparatus comprising:

[0015] The reading module is used to read the number of the first polygons, the number of ring midpoints, and the resolution data of the corresponding TIFF file of the vector file;

[0016] The ring complexity coefficient processing module is used to process the number of the first Polygon and the number of the ring midpoints and output the corresponding ring complexity coefficient;

[0017] The factor processing module is used to process the first number of Polygons, the resolution data, and the ring complexity coefficient to obtain the second number of Polygons in a single vector file;

[0018] A parallel processing unit is used to segment the vector file and the tif file according to the number of the second Polygon, and to obtain the final segmented data set through parallel processing.

[0019] A third aspect of this application provides a remote sensing data processing device, the device including a processor and a memory:

[0020] The memory is used to store program code and transfer the program code to the processor;

[0021] The processor is used to execute the steps of the remote sensing data processing method described in the first aspect above, according to instructions in the program code.

[0022] A fourth aspect of this application provides a computer-readable storage medium for storing program code for performing the method described in the first aspect.

[0023] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0024] This application provides a remote sensing data processing method, apparatus, device, and storage medium. The method involves reading the first number of polygons and the number of ring midpoints from a vector file; reading the resolution data of the corresponding TIFF file; inputting the first number of polygons and the number of ring midpoints into a preset ring complexity coefficient processing module, which processes and outputs the corresponding ring complexity coefficient; a factor processing module processes the first number of polygons, the resolution data, and the ring complexity coefficient to obtain the second number of polygons in a single vector file; and segmenting the vector file and the TIFF file according to the second number of polygons, processing them in parallel to obtain the final segmented data set. This solves the technical problem of low efficiency and excessive time consumption in sequential calculations for single tasks when the TIFF files segmented by existing geospatial data processing libraries are large.

[0025] Meanwhile, this application divides large shapefiles into multiple smaller shapefiles, splits the corresponding tif files, and then uses the smaller shapefiles to cut the corresponding smaller tif files. Finally, the cut smaller tif files are combined to obtain the final result, which greatly improves the calculation speed when using it to cut the corresponding tif files.

[0026] This approach avoids system crashes caused by prolonged computation and the waste of time resources associated with single-process tasks. The large numbers calculated in the early stages are already recorded in the original shapefile, allowing for direct reading without traversal, thus achieving high efficiency. Compared to the traditional method of splitting data into smaller tif files, this algorithm optimization, which involves segmenting the corresponding tif files, further splitting the smaller shapefiles into smaller tif files, and finally combining the smaller tif files to obtain the final result, can complete the task in just over 10 minutes, significantly improving computation speed. Attached Figure Description

[0027] Figure 1 This is a flowchart of the remote sensing data processing method in the embodiments of this application;

[0028] Figure 2 This is a schematic diagram of the remote sensing data processing device in the embodiments of this application;

[0029] Figure 3 This is a schematic diagram of the structure of the remote sensing data processing device in the embodiments of this application. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0031] This application designs a remote sensing data processing method. It reads the first number of polygons and the number of ring midpoints from a vector file, along with the resolution data of the corresponding TIFF file. The first number of polygons and the number of ring midpoints are input into a preset ring complexity coefficient processing module, which processes and outputs the corresponding ring complexity coefficient. A factor processing module processes the first number of polygons, the resolution data, and the ring complexity coefficient to obtain the second number of polygons in a single vector file. The vector file and the TIFF file are then segmented based on the second number of polygons, and parallel processing is performed to obtain the final segmented data set. This method solves the technical problem of low efficiency and excessive time consumption in sequential calculations for single tasks when the TIFF files segmented by existing geospatial data processing libraries are large.

[0032] For easier understanding, please refer to Figure 1 , Figure 1 This is a flowchart of the remote sensing data processing method in the embodiments of this application, as follows: Figure 1 As shown, specifically:

[0033] S101. Read the number of the first polygons, the number of ring midpoints, and the resolution data of the corresponding tif file of the vector file;

[0034] When it is necessary to cut a large area shapefile into a tif file, first read the number of the first polygon and the number of ring midpoints of the vector file.

[0035] It should be noted that the aforementioned method of reading the first number of polygons and the number of ring midpoints of a vector file first requires obtaining the header information of the vector file;

[0036] The header file information is read to determine the number of the first Polygon and the number of the ring midpoints;

[0037] Next, read the raster dimensions of the raster data to be cut from the vector file.

[0038] Next, the number of polygons and the number of all loop midpoints, as well as the resolution of the tif file, are read from the header information of the shapefile. In the .shp header file, the number of polygons NumParts in the shapefile is read and denoted as P_n, the number of all loop midpoints is denoted as p_k, and the shape size (n*m) of the tif file to be cut is read and denoted as T_n.

[0039] 102. Input the number of the first Polygons and the number of the ring midpoints into the preset ring complexity coefficient processing module, which will process and output the corresponding ring complexity coefficient.

[0040] In this embodiment, the number of the first Polygon and the number of the ring midpoints are input into a preset ring complexity coefficient processing module, which processes and outputs the corresponding ring complexity coefficient, including:

[0041] The ring point density function of a single ring is determined based on the number of the first Polygon and the number of the ring midpoints;

[0042] The ring point number density function is processed by a Gaussian distribution function to obtain the corresponding ring complexity coefficient.

[0043] The specific process can be summarized as follows:

[0044] (1) Calculate the complexity coefficient of a shapefile ring. The complexity of a single ring in a shapefile is determined by the number of points constituting the ring. The number of points is a random variable. The ring point density function is μ, with a mean of μ = p. k / P n -1, where P k Let P be the number of points in the ring. n It is the number of the first Polygon, NumParts (the end of the loop is connected);

[0045] (2) The number density function of the ring points follows a Gaussian distribution, therefore the ring complexity coefficient of the shapefile is: Where μ is the number density function of the ring points;

[0046] In this embodiment, it should be noted that a polygon is a geometric object used to represent a closed region defined by multiple points. In geospatial systems, polygon objects typically have the following characteristics:

[0047] Exterior Ring: Defines the boundary of the polygon. A polygon requires at least one exterior ring.

[0048] Interior Rings: Also known as holes, these define the empty spaces inside a polygon. Interior rings are optional; a polygon can have no interior rings or multiple interior rings.

[0049] Vertices: Points that form the boundary of a polygon. Each vertex is typically represented by a pair of coordinates (x, y).

[0050] Furthermore, it can be understood that before reading the number of the first polygon and the number of ring midpoints in the vector file, the following steps are also taken:

[0051] Determine whether the resolution data of the vector file outline and the tif file exceed a threshold;

[0052] If so, the vector file outline and the resolution data are subjected to a minimum intersection deletion operation.

[0053] By performing the above operations to remove scrap, if the overall outline of the shapefile file differs too much from the overall outline of the tif file to be cut, the scrap can be removed by taking the minimum intersection of the two overall maximum outlines, thus saving computational resources for the subsequent process.

[0054] To remove the margins between Shapefile (.shp) and raster data (.tif), ensuring their boundaries match as closely as possible, this can be achieved by calculating the minimum intersection of their boundaries. This reduces unnecessary computation in subsequent processing, improving efficiency.

[0055] The following are the steps to implement this process using relevant libraries: Read Shapefile and raster data: Use the geodata processing library to read the Shapefile and rasterio to read the raster data. Calculate boundaries: Extract boundaries from the Shapefile and raster data. Calculate minimum intersection: Use the Shapely library to calculate the minimum intersection of their boundaries. Clip the Shapefile: Clip the Shapefile based on the minimum intersection. Save the clipped Shapefile: Save the clipped Shapefile as a new file.

[0056] The aforementioned geospatial data processing library provides read and write support for various raster and vector data formats. Originally developed for processing raster data, the geospatial data processing library was later merged with OGR (for vector data processing) to form a unified library capable of handling both raster and vector data.

[0057] Its main functions include: supporting multiple raster data formats; providing operations such as data transformation, reprojection, clipping, merging, splitting, and statistics; supporting multiple coordinate systems and projection transformations; and providing interfaces for convenient geospatial data processing and analysis. The geospatial data processing library's data model includes concepts such as datasets, raster bands, and color tables. A dataset is a collection of raster bands, responsible for defining the georeference and coordinate system. The coordinate system is represented by an OGC WKT string, including the full name of the coordinate system, the geographic coordinate system name, the datum, the ellipsoid name, the prime meridian, and the projection method. The geospatial data processing library also supports affine geographic transformations and geographic control points (GCPs) to describe the georeference of raster datasets. Affine transformations convert pixel / row coordinates to georeference space using six parameters. GCPs associate raster locations with the georeference system using a set of control points. Geographic control points (GCPs) are one method used by the geospatial data processing library to describe the georeference of raster datasets. Using this method, a dataset will have a set of control points, associated with raster locations and one or more locations in a georeferenced system. All control points share a single georeferenced coordinate system. Each control point (GCP class description in Geospatial Data Processing Library) includes the following parts: a unique identifier (pszId), information (pszInfo), pixel location (dfGCPPixel and dfGCPLine), and georeferenced location (dfGCPX, dfGCPY, and dfGCPZ). GCPs can be used to establish transformation relationships from pixel coordinates to geographic coordinates, typically for geometric correction of imagery. Geospatial Data Processing Library provides a Geospatial Data Processing Library Transformation Tool, which can be used for coordinate transformations based on GCPs. This tool can handle various transformation methods, including GCPs, and can read coordinate pairs from the command line for transformation. When using the Geospatial Data Processing Library Transformation Tool, GCPs can be provided via the -gcp parameter, in the format -gcp. <pixel> <line> <easting> <northing>[elevation]. When performing geometric correction, the `.Warp` function from the Geospatial Data Processing Library can be used. It supports various resampling algorithms and can utilize GCPs for correction. The correction process involves establishing a mathematical relationship between pixel coordinates and geographic coordinates, then creating a new image and using resampling techniques to populate the pixel values ​​of the new image.

[0058] In addition, the geospatial data processing library provides command-line tools for performing tasks such as data transformation and terrain analysis. These tools can be invoked directly via the command line or used through a graphical interface within the geospatial system.

[0059] Understandably, the geospatial data processing library is used for image reprojection, transformation, and mosaicking. Reprojection: The source and target coordinate systems can be specified using the `-s_srs` and `-t_srs` parameters. Setting output resolution: The output image resolution can be set using the `-tr` parameter. Cropping: The geographic extent of the output image can be specified using the `-te` parameter. Setting output size: The `-ts` parameter is used to set the pixel size of the output image. Using overview level: The `-ovr` parameter specifies which overview level of the source file to use. Setting memory limits: The `-wm` parameter controls the amount of memory used by the geospatial data processing library during processing. Multi-threaded processing: The `-multi` parameter enables multi-threaded processing for improved efficiency. Output format: The `-of` parameter specifies the output file format. Creation options: The `-co` parameter sets creation options for specific formats, such as compression type.

[0060] It should be noted that the TIFF file mentioned in this embodiment is also commonly referred to as a TIFF (Tagged Image File Format), a flexible and widely used image file format primarily used to store raster image data. The TIFF format supports various color and data types, including black and white, grayscale, RGB, CMYK, and more. It also supports various compression algorithms, such as lossless compression (e.g., LZW, Deflate) and lossy compression (e.g., JPEG compression). Key features of the TIFF format include: Compatibility: TIFF is a cross-platform format and can be used on various operating systems. Flexibility: Supports multiple color modes, data types, and compression algorithms. High Quality: Commonly used in the image scanning and printing industries because it can store image data losslessly. Metadata Support: Can store additional information about the image, such as camera settings, copyright information, etc. Multi-page: A TIFF file can contain multiple images, similar to an image collection.

[0061] In geospatial data processing, TIFF files are frequently used to store raster data from geographic information systems. This data typically contains georeferenced information such as coordinate system, pixel size, and the image's location in the real world. Geospatial data processing libraries provide extensive support for TIFF files, including reading, writing, and processing them.

[0062] If TIFF files need processing, you can use geospatial data processing library command-line tools or geospatial data processing library bindings. For example, you can use the geospatial data processing library `_translate` to convert the format of TIFF files, and use geospatial data processing libraries for reprojection or correction.

[0063] S103. The factor processing module processes the first number of Polygons, the resolution data, and the ring complexity coefficient to obtain the second number of Polygons in a single vector file.

[0064] It should be noted that the segmentation factor of the vector file is obtained by taking the logarithm and square root algorithm based on the number of the first Polygon, the resolution data, and the cyclic complexity coefficient.

[0065] The number of second polygons corresponding to a single vector file is obtained by using the segmentation factor.

[0066] In one embodiment, the segmentation is not only related to the complexity of the shapefile, but also positively correlated with the size of the tif file itself. Therefore, the number of shapefile regions segmented by the algorithm is as follows:

[0067] Among them, P n It is the number of the first Polygon, NumParts, T n ρ is the shape size (n*m) of the TIFF file to be cut, and ρ is the shape complexity coefficient of the shapefile.

[0068] Besides being related to the complexity of the shapefile, cutting is also about finding a suitable value for l, which can help determine how to divide the shapefile into smaller parts in order to reduce the consumption of computing resources when processing large-sized raster data.

[0069] This process can use the `math` library to calculate the logarithm and square root. The `math.log2()` function calculates the logarithm to base 2, a commonly used logarithmic base in information theory-related calculations. Ensure that the values ​​of `Pn` and `Tn` are correct, as they directly affect the calculation result of `l`. Furthermore, the value of `ρ` should be adjusted according to the actual situation to reflect the relationship between resource consumption and performance requirements during processing.

[0070] Furthermore, it can be concluded that the approximate number of the first Polygon in a single shapefile is:

[0071] P single ≈P n / l

[0072] Used to estimate approximately how many polygons should be contained in each segmented Shapefile region when cutting raster data. Here: P single This represents the approximate number of polygons in each segmented region. Pn is the total number of polygons in the Shapefile, and l is the segmentation factor calculated using the previous formula, used to determine how many regions to divide the Shapefile into. This estimation helps us decide how many polygons each segment should contain, so as to balance computational load and improve efficiency when processing raster data.

[0073] In practical applications, adjusting the value of ρ (called the "adjustment factor") is an empirical process that depends on various factors, including computational resources, data complexity, and required precision. Here are some steps and suggestions to help adjust the value of ρ to improve computational accuracy based on actual geographic data: Understanding the role of ρ: ρ is an adjustment factor used to balance the impact of Shapefile complexity and raster data size on the number of segments. A higher ρ value increases the number of segments, potentially improving processing precision, but also increases computational complexity and resource consumption. A lower ρ value reduces the number of segments, potentially reducing resource consumption, but may decrease processing precision. Benchmarking: Benchmark at different ρ values ​​and observe the impact on computation time and result precision. Choose a baseline ρ value and then gradually adjust it, observing the changes. Considering data characteristics: Analyze the number, size, and complexity of polygons in the Shapefile. Analyze the size and resolution of the raster data. Considering computational resources: Adjust the ρ value based on available computational resources (such as CPU and memory). If resources are limited, a smaller ρ value may be necessary. Iterative adjustment: Initially set a ρ value and perform a complete calculation. Analyze the results. If excessive computational resource consumption or inaccurate results are found, increase or decrease the ρ value as needed. Automated Adjustment: Consider developing an automated adjustment mechanism to dynamically adjust the ρ value based on real-time computational feedback. Recording and Analysis: Record the computational results and resource consumption after each ρ value adjustment. Analyze the data to find the optimal ρ value. Consider Real-World Application Scenarios: Adjust the ρ value according to the needs of the actual application. For example, if the application is highly sensitive to computation time, a smaller ρ value may be necessary. Software and Tools: Utilize the analytical functions in geospatial systems or programming tools to help determine the optimal ρ value. Ultimately, adjusting the ρ value is a process of trial and error, requiring fine-tuning based on actual conditions and needs. Through the above steps, adjustments can be made more systematically to find the ρ value that best suits the data and requirements.

[0074] S104. The vector file and the tif file are split according to the number of the second Polygon, and the final split data set is obtained by parallel processing.

[0075] Optionally, step S104 specifically includes:

[0076] Read the number of polygons in the vector file corresponding to the second number of polygons;

[0077] Reorganize the number of polygons corresponding to the number of the second polygons to generate at least two new vector files, and generate a new set of vector files based on all the new vector files;

[0078] The tif file is segmented according to the latest bounding rectangle in the new vector file to generate a new set of tif files including at least two segmented tif files;

[0079] The new vector file and the segmented TIFF file are processed in parallel to obtain the final segmented data set.

[0080] Understandably, this involves reading the vector file and identifying the number of polygons corresponding to the second number of polygons.

[0081] It should be further explained that the parallel processing of the new vector file and the segmented TIFF file to obtain the final segmented data set specifically includes:

[0082] The new vector file and the segmented TIFF file are processed in parallel to extract the corresponding segmentation result data;

[0083] The segmentation results are then integrated into a final cut data set.

[0084] Segmenting the shapefile and tif source file: Based on the above calculation results P single Take the integer P, read the shapefile file, and each time read P polygons from the shapefile, reconstruct a new small shapefile, until all polygons are read, denoted as SF1, SF2, ... SF3. i ...、SF m ,m≦l,At the same time, according to SF i The minimum bounding rectangle in the information is used to divide the original TIFF file and obtain the TF file. i ;

[0085] The process of splitting a Shapefile (.shp) and its corresponding tif file (.tif) can be divided into the following steps:

[0086] 1. Calculate the number of polygons in each segment: Based on the previous calculations, we obtained an approximate value, Psingle, for the number of polygons that each segment should contain. Round this value up to get P.

[0087] 2. Read Shapefile: Use the geo data processing library to read the raw Shapefile.

[0088] 3. Split Shapefile: Read P Polygons from Shapefile each time and create a new small Shapefile, until all Polygons have been read.

[0089] 4. Split raster files: For each new small Shapefile, calculate its minimum bounding rectangle and use this rectangle to crop the original raster file (.tif).

[0090] 5. Save the split files: Save each split small Shapefile and its corresponding raster file as a new file.

[0091] Parallel processing: SF i Cut TF i That is, to obtain the result set F = {f} of the small module. i , i∈{1,2,...m}};

[0092] Understandably, to achieve parallel processing, each small Shapefile (SF) needs to be processed. i Cut the corresponding raster file (TF) i To parallelize processing, you can use Python's multithreading or multiprocessing libraries, such as asynchronous callable objects. Using asynchronous callable objects allows for parallel processing, and may require thread pools and process pools to reproject multiple files in parallel. Parallel processing can significantly improve efficiency when handling large amounts of data.

[0093] Each small Shapefile (SF) i Cut the corresponding raster file (TF) i The implementation can be as follows:

[0094] 1. Read Shapefile: Use the geographic data processing library to read the raw Shapefile file.

[0095] 2. Calculate the number of Polygons in each segment: Based on the previous calculation results, round up to get the number of Polygons in each segment.

[0096] 3. Define the clip function: The clip_raster function is used to cut each small Shapefile and its corresponding raster file.

[0097] 4. Use thread pools and process pools for parallel processing:

[0098] Create a thread pool and a process pool instance.

[0099] For each small Shapefile, submit a task to the executor for parallel cutting.

[0100] Use the results command to get the results for each task.

[0101] Finally, perform a union operation on the elements in set F to obtain the calculated result: Where, F = {f i , i∈{1,2,...m}}, can be understood as merging the elements in the result set F to obtain the final calculation result A.

[0102] Understandably, after completing the parallel processing of all the small modules, in GIS, this usually means merging all the small Shapefiles into one large Shapefile, or merging all the small raster files (.tif) into one large raster file.

[0103] It's important to note that when merging raster files, you must ensure that all files have the same coordinate reference system (CRS) and resolution; otherwise, the merged result may contain errors. Furthermore, when merging shapefiles, you also need to ensure that they have the same geometry type and coordinate reference system.

[0104] Furthermore, when merging large amounts of GIS data, optimizing data processing workflows to improve efficiency is crucial. This includes: using efficient libraries and tools: selecting high-performance GIS libraries, such as geospatial data processing libraries, which offer optimized data processing capabilities; parallel processing: leveraging the capabilities of multi-core processors to accelerate data processing using parallel processing; employing asynchronous execution of callable object modules for parallel processing; batch processing: dividing large datasets into smaller batches for processing reduces memory consumption and increases processing speed; memory management: monitoring and managing memory usage to prevent memory leaks; data simplification: simplifying data structures without affecting analysis results, such as reducing the number of vertices in vector data or lowering the resolution of raster data; using spatial indexes: utilizing spatial indexes (such as R trees) to accelerate spatial queries and data filtering, which can significantly improve data processing speed; optimizing read / write operations: reducing disk I / O operations through buffering, batch writing, and choosing appropriate data formats to optimize read / write performance; and using cloud services: utilizing GIS services and elastic computing resources provided by cloud computing platforms to scale processing capacity on demand. Data Preprocessing: Preprocess the data before merging, such as pruning, filtering, and reprojection, to reduce the amount of data to be processed during the merging process. Use Dedicated Hardware: Using faster storage devices (such as SSDs) and more powerful CPUs / GPUs can improve data processing speed. Avoid Unnecessary Data Conversion: Minimize data format conversions, as these processes can consume significant time and resources. Code Optimization: Optimize code logic, avoid unnecessary loops and complex query operations, and use vectorized operations to improve performance. Use Appropriate Data Structures: Choose appropriate data structures based on data characteristics, such as using geographic data frames for vector data processing and xarrays for multidimensional raster data processing. Caching Mechanisms: Implement caching mechanisms for frequently used data to avoid repeated reading and processing. Phased Processing: Decompose the entire processing flow into multiple phases, with each phase outputting intermediate results. This allows for quick identification and optimization of problems when they arise. Monitoring and Analysis: Use performance monitoring tools to analyze bottlenecks in the data processing flow and optimize them accordingly. Reasonable Database Use: For very large datasets, consider using spatial database object relational database extensions to manage the data, as they provide optimized spatial data storage and query capabilities. These strategies can significantly improve the efficiency of processing large amounts of GIS data and reduce processing time and resource consumption.

[0105] The aforementioned geographic data processing library is a library for geospatial data processing. Based on the data processing library extension, it inherits the data processing and analysis capabilities of the original library, enabling convenient processing of tabular data. It can process geospatial data frames like ordinary data frames, making data filtering, aggregation, and merging operations simple and efficient. It supports multiple geospatial data formats: it can read and write common geospatial data formats such as Shapefile, GeoJSON, and GPKG. It facilitates data interaction with other geographic information system software. Its geospatial operation functions are powerful: it provides a rich set of geospatial operation functions, such as spatial joining, buffer analysis, and overlay analysis. It can perform spatial queries to quickly filter out geographic features that meet specific spatial conditions. Its application scenarios include: Geographic data analysis: performing statistical analysis on geospatial data, calculating geometric attributes such as area, length, and perimeter; performing spatial cluster analysis to discover clustering patterns in geospatial space; and map creation: combining with the drawing library to create various types of maps, including thematic maps and contour maps. It allows for customized map annotations and color schemes, making maps more intuitive and aesthetically pleasing. Spatial planning and decision support: In fields such as urban planning, resource management, and environmental protection, spatial analysis is conducted using geographic data processing databases to support decision-making. For example, assessing the spatial impact of different planning schemes and determining the boundaries of protected areas.

[0106] It is understood that the object-relational database mentioned in this embodiment is an object-relational database extension that adds Geographic Information System (GIS) functionality to an SQL database. This enables the object-relational database extension not only to store spatial data but also to perform spatial queries and analyses. The following are some key points for managing spatial data using the object-relational database extension: Spatial Data Types: The object-relational database extension supports various spatial data types, including Geometry and Geography types. Geometry types are used to represent points, lines, and polygons on a plane, while Geography types are used to represent geographical locations on Earth. Spatial Indexes: To improve the efficiency of spatial queries, the object-relational database extension uses GiST-based R-Tree indexes. These indexes help to quickly locate spatial data, thereby accelerating the query process. Spatial Functions: The object-relational database extension provides a large number of spatial functions for performing various spatial analyses and operations, such as calculating the distance, area, and length between two geometric objects, and performing spatial relationship queries. Spatial Reference Systems: The object-relational database extension allows users to define and use different Spatial Reference Systems (SRSs), which define how geometric data is associated with real-world locations. Data Import and Export: The Object-Relational Database Extension supports importing spatial data from various formats, including Shapefile, GeoJSON, and KML, and also supports exporting data to these formats. Raster Data Processing: The Object-Relational Database Extension also supports the storage and processing of raster data, which is very useful when processing remote sensing imagery and terrain analysis. Metadata Management: The Object-Relational Database Extension provides two tables, geometry_columns and spatial_ref_sys, for tracking and reporting geometric columns and spatial reference systems in the database. Data Projection: The Object-Relational Database Extension allows users to perform projection transformations on spatial data for representation and analysis in different coordinate systems. Validity Check: The Object-Relational Database Extension provides tools to check the validity of geometric data and functions to fix invalid geometric objects. Triggers and Edit History Tracking: The Object-Relational Database Extension supports the use of database triggers to track the data's edit history, which is very useful for data version control and change management. 3D and 4D Data Support: The Object-Relational Database Extension supports not only 2D geometric data but also 3D and 4D data, allowing users to store and query spatial data with height and time dimensions. Raster data management: The Object Relational Database extension also provides support for the management, querying, and analysis of raster data.

[0107] The remote sensing data processing method in this embodiment first reads the number of first polygons, the number of ring midpoints, and the resolution data of the corresponding TIFF file from the vector file; then inputs the number of first polygons and the number of ring midpoints into a preset ring complexity coefficient processing module, which processes and outputs the corresponding ring complexity coefficient; the factor processing module processes the number of first polygons, the resolution data, and the ring complexity coefficient to obtain the number of second polygons in a single vector file; and finally, the vector file and the TIFF file are segmented according to the number of second polygons, and processed in parallel to obtain the final segmented data set. This solves the technical problem of low efficiency and excessive time consumption caused by sequential calculation of large TIFF files in existing geospatial data processing libraries.

[0108] Meanwhile, in this embodiment, by segmenting the large shapefile, the large shapefile data is divided into multiple small shapefile data, and the corresponding tif file is segmented. Then, the small shapefiles are used to cut the corresponding small tif files. Finally, the segmented small tif files are combined to obtain the final result. This greatly improves the calculation speed when using it to cut the corresponding tif files.

[0109] In this embodiment, system crashes caused by prolonged computation and the waste of time resources by single-process tasks are avoided. The large numbers calculated in the early stages are all recorded in the original shapefile, allowing direct reading without traversal, achieving high efficiency. Compared to the traditional method of splitting data that takes two and a half hours, the above algorithm optimization, which segments the corresponding tif files, then uses the smaller shapefiles to further split the smaller tif files, and finally performs joint calculations on the split tif files to obtain the final result, can complete the task in just over 10 minutes, greatly improving computation speed.

[0110] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of the remote sensing data processing device in the embodiments of this application, as shown below. Figure 2 As shown, specifically:

[0111] The reading module 201 is used to read the number of the first polygons, the number of ring midpoints, and the resolution data of the corresponding tif file of the vector file;

[0112] In one embodiment, the first reading module 201 is specifically used for:

[0113] Obtain the header information of the vector file;

[0114] The number of the first Polygon, the number of the ring midpoints, and the resolution of the tif file are determined based on the header file information.

[0115] The ring complexity coefficient processing module 202 is used to process the number of the first Polygon and the number of the ring midpoints and output the corresponding ring complexity coefficient;

[0116] In one embodiment, the cyclic complexity coefficient processing module 202 is specifically used for:

[0117] The ring point density function of a single ring is determined based on the number of the first Polygon and the number of the ring midpoints;

[0118] The ring point density function is processed using a Gaussian distribution function to obtain the corresponding ring complexity coefficient. Factor processing module 203 is used to process the first number of polygons, the resolution data, and the ring complexity coefficient to obtain the second number of polygons in a single vector file.

[0119] Factor processing module 203 is specifically used for:

[0120] The segmentation factor of the vector file is obtained by taking the logarithm and square root algorithm based on the number of the first polygons, the resolution data, and the ring complexity coefficient.

[0121] The number of second polygons corresponding to a single vector file is obtained by using the segmentation factor.

[0122] The parallel processing module 204 is used to split the vector file and the tif file according to the number of the second Polygon, and to obtain the final split data set through parallel processing.

[0123] Parallel processing module 204 includes:

[0124] The reading unit is used to read the number of polygons in the vector file corresponding to the number of the second polygons;

[0125] A recombination unit is used to recombine the number of polygons corresponding to the number of the second polygons to generate at least two new vector files, and to generate a new set of vector files.

[0126] A segmentation unit is used to segment the tif file according to the latest bounding rectangle in the new vector file, and generate a new set of tif files including at least two segmented tif files;

[0127] A parallel processing unit is used to process the new vector file and the segmented TIFF file in parallel to obtain the final segmented data set.

[0128] The parallel processing module is specifically used to: process the new vector file and the segmented TIFF file in parallel, extract the corresponding segmentation result data, and integrate the segmentation result data into a final cut data set.

[0129] The remote sensing data processing device in this embodiment first reads the number of first polygons, the number of ring midpoints, and the resolution data of the corresponding TIFF file from the vector file; then inputs the number of first polygons and the number of ring midpoints into a preset ring complexity coefficient processing module, which processes and outputs the corresponding ring complexity coefficient; the factor processing module processes the number of first polygons, the resolution data, and the ring complexity coefficient to obtain the number of second polygons in a single vector file; and finally, the vector file and the TIFF file are segmented according to the number of second polygons, and processed in parallel to obtain the final segmented data set. This solves the technical problem of low efficiency and excessive time consumption caused by sequential calculation of large TIFF files in existing geospatial data processing libraries.

[0130] Meanwhile, in this embodiment, by segmenting the large shapefile, the large shapefile data is divided into multiple small shapefile data, and the corresponding tif file is segmented. Then, the small shapefiles are used to cut the corresponding small tif files. Finally, the segmented small tif files are combined to obtain the final result. This greatly improves the calculation speed when using it to cut the corresponding tif files.

[0131] In this embodiment, system crashes caused by prolonged computation and the waste of time resources by single-process tasks are avoided. The large numbers calculated in the early stages are all recorded in the original shapefile, allowing direct reading without traversal, achieving high efficiency. Compared to the traditional method of splitting data that takes two and a half hours, the above algorithm optimization, which segments the corresponding tif files, then uses the smaller shapefiles to further split the smaller tif files, and finally performs joint calculations on the split tif files to obtain the final result, can complete the task in just over 10 minutes, greatly improving computation speed.

[0132] This application also provides another remote sensing data processing device, such as... Figure 3 As shown, device 10 includes:

[0133] One or more processors 110 and memory 120 are described, taking one processor 110 as an example. The processor 110 and memory 120 can be connected by a bus or other means, taking connection by a bus as an example.

[0134] Processor 110 is used to perform various control logics of device 10, and can be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), microcontroller, ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. Furthermore, processor 110 can also be any conventional processor, microprocessor, or state machine. Processor 110 can also be implemented as a combination of computing devices, such as a combination of DSP and microprocessor, multiple microprocessors, one or more microprocessors combined with DSP and / or any other such configuration.

[0135] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions corresponding to the online fixed asset data processing method in the embodiments of the present invention. The processor 110 executes various functional applications and data processing of the device 10 by running the non-volatile software programs, instructions, and units stored in the memory 120, thereby implementing the online fixed asset data processing method in the above method embodiments.

[0136] The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created according to the use of the device 10. Furthermore, the memory 120 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include memory remotely located relative to the processor 110, and these remote memories may be connected to the device 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0137] One or more units are stored in memory 120, and when executed by one or more processors 110, perform the following steps:

[0138] Read the number of the first polygon and the number of ring midpoints from the vector file;

[0139] Read the resolution data of the TIFF file corresponding to the vector file;

[0140] The number of the first Polygon and the number of the ring midpoints are input into the preset ring complexity coefficient processing module, which processes and outputs the corresponding ring complexity coefficient.

[0141] The factor processing module processes the first number of polygons, the resolution data, and the ring complexity coefficient to obtain the second number of polygons in a single vector file;

[0142] The vector file and the tif file are split according to the second number of Polygons, and the final split data set is obtained by parallel processing.

[0143] This invention provides a non-volatile computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by one or more processors, they implement the steps S101 to S104 described above.

[0144] As examples, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The memory components or memories disclosed in the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0145] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The computer program can be stored in a non-volatile, computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The storage medium can be a memory, magnetic disk, floppy disk, flash memory, optical storage, etc.

[0146] This application provides a remote sensing data processing method, apparatus, device, and storage medium. The method involves first reading the first number of polygons and the number of ring midpoints from a vector file; then reading the resolution data of the corresponding TIFF file; inputting the first number of polygons and the number of ring midpoints into a preset ring complexity coefficient processing module, which processes and outputs the corresponding ring complexity coefficient; a factor processing module processes the first number of polygons, the resolution data, and the ring complexity coefficient to obtain the second number of polygons in a single vector file; and finally, the vector file and the TIFF file are segmented according to the second number of polygons, and processed in parallel to obtain the final segmented data set. This solves the technical problem of low efficiency and excessive time consumption in sequential calculations for single tasks when the TIFF files segmented by existing geospatial data processing libraries are large.

[0147] Meanwhile, in this embodiment, by segmenting the large shapefile, the large shapefile data is divided into multiple small shapefile data, and the corresponding tif file is segmented. Then, the small shapefiles are used to cut the corresponding small tif files. Finally, the segmented small tif files are combined to obtain the final result. This greatly improves the calculation speed when using it to cut the corresponding tif files.

[0148] In this embodiment, system crashes caused by prolonged computation and the waste of time resources by single-process tasks are avoided. The large numbers calculated in the early stages are all recorded in the original shapefile, allowing direct reading without traversal, achieving high efficiency. Compared to the traditional method of splitting data that takes two and a half hours, the above algorithm optimization, which segments the corresponding tif files, then uses the smaller shapefiles to further split the smaller tif files, and finally performs joint calculations on the split tif files to obtain the final result, can complete the task in just over 10 minutes, greatly improving computation speed.

[0149] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0150] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.< / northing> < / easting> < / line> < / pixel>

Claims

1. A remote sensing data processing method, characterized in that, include: Read the number of the first polygon in the vector file, the number of ring midpoints, and the resolution data of the corresponding TIFF file; The number of the first Polygon and the number of the ring midpoints are input into the preset ring complexity coefficient processing module, which processes and outputs the corresponding ring complexity coefficient. The factor processing module processes the first number of polygons, the resolution data, and the ring complexity coefficient to obtain the second number of polygons in a single vector file, specifically including: The segmentation factor of the vector file is obtained by taking the logarithm and square root algorithm based on the number of the first polygons, the resolution data, and the ring complexity coefficient. The number of second polygons corresponding to a single vector file is obtained through the segmentation factor; The vector file and the tif file are split according to the second number of polygons, and parallel processing is performed to obtain the final segmented data set, including: Read the number of polygons in the vector file corresponding to the second number of polygons; The polygons are recombined to generate at least two new vector files; The tif file is segmented according to the latest bounding rectangle in the new vector file to generate a new set of tif files including at least two segmented tif files; The new vector file and the segmented TIFF file are processed in parallel to obtain the final segmented data set.

2. The remote sensing data processing method according to claim 1, characterized in that, The reading of the first polygon count, the number of ring midpoints, and the resolution data of the corresponding TIFF file from the vector file specifically includes: Obtain the header information of the vector file; The number of the first Polygon, the number of the ring midpoints, and the resolution data of the TIFF file are determined based on the header file information.

3. The remote sensing data processing method according to claim 1, characterized in that, The number of the first polygons and the number of the ring midpoints are input into a preset ring complexity coefficient processing module, which processes and outputs the corresponding ring complexity coefficient, including: The ring point density function of a single ring is determined based on the number of the first Polygon and the number of the ring midpoints; The ring point number density function is processed by a Gaussian distribution function to obtain the corresponding ring complexity coefficient.

4. The remote sensing data processing method according to claim 1, characterized in that, The new vector file and the segmented TIFF file are processed in parallel to obtain the final segmented data set, specifically including: The new vector file and the segmented TIFF file are processed in parallel to extract the corresponding segmentation result data; The segmentation results are then integrated into a final cut data set.

5. The remote sensing data processing method according to claim 1, characterized in that, Before reading the number of the first polygons, the number of ring midpoints, and the resolution data of the corresponding TIFF file of the vector file, the process also includes: Determine whether the resolution data of the vector file outline and the tif file exceed a threshold; If so, the vector file outline and the resolution data are subjected to a minimum intersection deletion operation.

6. A remote sensing data processing device, characterized in that, include: The reading module is used to read the number of the first polygons, the number of ring midpoints, and the resolution data of the corresponding TIFF file of the vector file; The ring complexity coefficient processing module is used to process the number of the first Polygon and the number of the ring midpoints and output the corresponding ring complexity coefficient; The factor processing module is used to process the first number of polygons, the resolution data, and the ring complexity coefficient to obtain the second number of polygons in a single vector file. Specifically, it is used for: The segmentation factor of the vector file is obtained by taking the logarithm and square root algorithm based on the number of the first polygons, the resolution data, and the ring complexity coefficient. The parallel processing unit is used to segment the vector file and the tif file according to the number of the second polygons, and to obtain the final segmented data set through parallel processing. Specifically, it is used for: Read the number of polygons in the vector file corresponding to the second number of polygons; The polygons are recombined to generate at least two new vector files; The tif file is segmented according to the latest bounding rectangle in the new vector file to generate a new set of tif files including at least two segmented tif files; The new vector file and the segmented TIFF file are processed in parallel to obtain the final segmented data set.

7. A remote sensing data processing device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the remote sensing data processing method according to any one of claims 1-5 according to the instructions in the program code.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the remote sensing data processing method according to any one of claims 1-5.

Images