A batch sample manufacturing method and system based on multi-source data

Abstract

The application provides a batch sample production method and system based on multi-source data, automatically identifies and constructs data pairs through an intelligent file matching algorithm, combines a coordinate system processing module, fundamentally solves the problem of multi-source data coordinate mismatch, ensures the accuracy of cutting operation, adopts a sliding window algorithm and a spatial index optimization technology, realizes synchronous and accurate batch cutting of vector and image data according to a specified size, guarantees the consistency of sample data, introduces a multi-thread parallel processing mechanism, decomposes and executes large-scale tasks in parallel, optimizes data structure and algorithm, greatly improves the processing speed, reduces resource occupation, effectively breaks through the performance bottleneck, adopts metadata-driven attribute assignment technology, automatically extracts and inherits key attribute information in images and vectors, realizes full-factor description of sample data, and provides more abundant and more accurate sample data for subsequent intelligent interpretation model training.

Description

Technical Field

[0001] This invention relates to the field of geographic information system data processing, and in particular to a method and system for batch sample production based on multi-source data. Background Technology

[0002] Intelligent remote sensing interpretation is primarily based on machine learning and deep learning algorithms, and its performance is highly dependent on the quality and quantity of training data. The sample set, as the core of the training data, provides the model with learning objectives and knowledge sources. In the field of remote sensing data processing, a large amount of high-precision vector data already exists, such as land use maps, digital agriculture maps, and vegetation cover maps. These vector data accurately describe the location and shape of surface features. Simultaneously, massive amounts of corresponding remote sensing imagery record the spectral and spatial information of the land surface. Theoretically, by using precise vector data as a mask to crop registered remote sensing images, sample data for model training can be directly generated. This method effectively utilizes existing data assets, greatly reduces the workload of manual annotation, and significantly improves the efficiency of sample creation. Therefore, using existing vector data to batch crop corresponding images has become an important technical approach for constructing large-scale intelligent remote sensing interpretation sample sets.

[0003] However, existing geographic information data processing software and technical solutions have many shortcomings in achieving the aforementioned batch cropping to generate samples, severely restricting the efficiency and quality of data processing. First, there is a serious coordinate system mismatch problem. Vector data and image data from different sources often use different projections or geographic coordinate systems, resulting in a lack of direct correspondence between image pixel positions and vector graphic geographic locations. This makes precise cropping difficult and necessitates tedious manual coordinate registration. Second, there is a significant performance bottleneck. Existing technologies are extremely slow when processing large-scale, high-resolution images and dense vector data. Completing a single batch cropping task can take hours or even days, failing to meet the demands of real-time or efficient processing. Furthermore, the processing consumes excessive system resources, easily leading to processing interruptions or system instability. Third, software support has significant limitations. Existing tools have limited functionality. For example, ArcGIS is inefficient at cropping high-density polygons, and ENVI's batch cropping function is mostly limited to cropping multiple images within the same vector range, lacking the ability to perform multi-vector-to-multi-image, gridded batch cropping at specified sizes. Integration between different software programs is poor, making seamless workflow integration difficult. Finally, there is the issue of missing attribute inheritance. Metadata such as shooting time and resolution contained in the image header file, as well as attribute information such as the name and category of the vector data itself, are often lost during the cropping process of existing technologies. These attributes cannot be fully inherited into the final sample data, resulting in the generated samples lacking key descriptive information, which affects subsequent sample management and model training.

[0004] Therefore, how to efficiently and accurately achieve batch cropping of vector data and image data, simultaneously solve the problems of coordinate system differences, performance bottlenecks, software function limitations, and attribute information loss in existing technologies, and automatically generate standardized sample data containing complete vector and image content and attribute information, is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for batch sample production based on multi-source data, so as to solve the problems existing in the prior art.

[0006] To achieve the above objectives, the present invention provides the following solution: This invention provides a method for batch sample production based on multi-source data, comprising: Collect image data files and vector data files in the root directory specified by the user, and use an intelligent file matching algorithm to identify and construct data pairs to be processed consisting of the image data files and vector data files from the same geographical area; Spatially align the data pairs to be processed. Based on the cropping parameters input by the user, use a sliding window algorithm to generate a regular grid cropping region within the image space. Simultaneously perform the same spatial cropping operation on the vector data to generate multiple image-vector sample blocks. While performing the cropping operation, a metadata-driven approach is used to extract metadata information from the image data file and add standardized attribute fields to the vector data, and assign the metadata information to the corresponding standardized attribute fields; The multi-threaded parallel processing mechanism is used to process multiple pairs of data to be processed or multiple cropping windows within a single pair of data to be processed in parallel, and the processed image-vector sample blocks are saved to the output directory according to a preset naming rule.

[0007] Preferably, the intelligent file matching algorithm includes: Recursively scan the root directory and all its subfolders at all levels; Based on a preset filename pattern, regular expressions are used to match filenames within a folder and extract filename prefixes; By combining file type verification, raster image files and vector data files are filtered out. Image files and vector files located in the same folder with the same filename prefix are identified as a valid pair of data to be processed.

[0008] Preferably, the sliding window algorithm determines the window size and sliding step size based on the target size and overlap rate in the cropping parameters, moves the window from the upper left corner of the image according to the sliding step size, and generates a regular grid cropping area covering the entire image range; and by constructing a spatial index, the vector features in the current window are quickly located during each cropping to avoid traversing the entire vector data.

[0009] Preferably, the metadata information includes the shooting time, number of bands, and resolution extracted from the image header file; the standardized attribute fields include sample type, classification code, and cropping region ID.

[0010] Preferably, before the spatial alignment step, a coordinate system processing step is included: automatically identifying the coordinate systems of the image data file and the vector data file; if the two are inconsistent, they are uniformly converted to a preset target coordinate system to ensure that the image pixel position and the vector graphic geographical position correspond accurately.

[0011] Preferably, the multi-threaded parallel processing mechanism includes: using a thread pool to manage the creation and destruction of threads, and implementing data interaction between threads through shared memory or message queues; wherein, the queue of tasks to be processed is stored in shared memory for threads to retrieve, and the processing results are transmitted to the output module through a message queue; and mutexes and semaphores are used to ensure safe access to shared resources.

[0012] This invention also provides a batch sample production system based on multi-source data, comprising: The data traversal module is used for deep traversal of the root directory specified by the user, and identifies and constructs a queue of tasks to be processed consisting of image data files and vector data files through an intelligent file matching algorithm. The cropping module is used to spatially align the data pairs to be processed. Based on the cropping parameters input by the user, it uses a sliding window algorithm to generate a regular grid cropping area and simultaneously performs spatial cropping on the image and vector data. The attribute assignment module is used to extract metadata information from the image header file, add standardized attribute fields to the vector data, assign the metadata information to the corresponding attribute fields, and generate sample data for full feature description. The output module is used to persistently save the processed image-vector data pairs according to preset naming rules and directory structure. The parallel processing module is used to schedule the clipping module and the attribute assignment module through a multi-threaded parallel processing mechanism to perform parallel operations on multiple pairs of data to be processed or multiple clipping windows.

[0013] Preferably, the intelligent file matching algorithm combines filename pattern matching and file type dual verification to identify image data files and vector data files located in the same folder with the same filename prefix.

[0014] Preferably, the cropping module employs spatial index optimization technology to quickly locate vector elements intersecting with the current cropping window during the sliding window cropping process, thereby avoiding traversing the entire vector data and improving cropping efficiency.

[0015] Preferably, the system further includes a log recording and exception handling module, which is used to record processing logs during large-scale batch operations and to capture and process exceptions when they occur, so as to ensure data integrity and traceability of the processing flow.

[0016] The present invention achieves the following beneficial technical effects compared to the prior art: This invention provides a method and system for batch sample creation based on multi-source data, offering significant technical advantages. The invention automatically identifies and constructs data pairs through an intelligent file matching algorithm, combined with a coordinate system processing module, fundamentally solving the problem of coordinate mismatch in multi-source data and ensuring the accuracy of the cropping operation. By employing a sliding window algorithm and spatial index optimization technology, it achieves synchronous and precise batch cropping of vector and image data according to specified sizes, guaranteeing the consistency of sample data. In particular, the invention introduces a multi-threaded parallel processing mechanism, decomposing large-scale tasks into parallel executions and optimizing data structures and algorithms, greatly improving processing speed, reducing resource consumption, and effectively overcoming performance bottlenecks. Simultaneously, the invention uses metadata-driven attribute assignment technology to automatically extract and inherit key attribute information from images and vectors, achieving a full-element description of the sample data and providing richer and more accurate sample data for subsequent intelligent interpretation model training. In summary, this invention provides an efficient, accurate, and automated integrated tool that comprehensively solves many shortcomings of existing technologies and can fully meet the needs of large-scale geographic information data batch processing and sample library construction. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 The system architecture diagram for batch sample production based on multi-source data provided by this invention is shown. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The purpose of this invention is to provide a method and system for batch sample production based on multi-source data, so as to solve the problems existing in the prior art.

[0021] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] Example 1: like Figure 1 As shown, the batch sample production system provided in this embodiment adopts a highly cohesive and loosely coupled modular architecture design. Its core processing flow is completed collaboratively by a data traversal module, a coordinate unification module, a pruning module, an attribute assignment module, an output module, and a parallel processing module. The system is also equipped with a log recording and exception handling module, which is used to record processing logs during large-scale batch operations and to capture and handle exceptions when they occur, ensuring data integrity and traceability of the processing flow. The following is a combination of... Figure 1 The specific embodiments of the present invention will be described in detail.

[0023] The method for making batch samples based on multi-source data of the present invention first performs a data collection and traversal step. The user specifies a root directory through an interactive interface, and this root directory may contain a multi-layer nested folder structure. The data traversal module uses a depth-first search algorithm to recursively scan the user-specified root directory and all levels of its sub-folders. During the scanning process, the module uses a dual verification mechanism of file name pattern matching and file type to accurately identify valid folders that contain both raster image files and vector data files. Specifically, the system uses regular expressions to match each file name in the folder according to the file name pattern preset by the user. For each file, the system checks whether it conforms to the preset pattern and extracts the file name prefix. At the same time, the system verifies the file type through the file extension or file header information, screening out image files that support common raster formats such as GeoTIFF and IMG, and vector data files that support the ESRI Shapefile format. Then, the system compares whether the prefixes of the image file and the vector file in the same folder are the same. If the prefixes are the same, it is considered that they form a valid data pair, that is, the image and the vector describe the same geographical area. For example, if there are files "farmland_1.tif" and "farmland_1.shp" in the folder, through regular expression matching and prefix comparison, the system can determine that they are valid data pairs. All identified valid data pairs are sent to the pending task queue for subsequent processing.

[0024] Before starting the cropping process, the present invention also preferably includes a coordinate system unification processing step. Since vector data and image data from different sources may use different projection coordinate systems or geographic coordinate systems, resulting in the inability to directly correspond the pixel positions of the image with the geographical positions of the vector graphics. To solve this problem, the coordinate unification module automatically identifies the coordinate systems of the image data file and the vector data file. If it detects that the coordinate systems of the two are inconsistent, the system will uniformly convert them to a preset target coordinate system, for example, converting the GCJ02 coordinate system to the WGS84 coordinate system. This conversion ensures the precise matching of the image and the vector in terms of spatial position, laying a reliable foundation for subsequent precise cropping operations.

[0025] Next, it enters the core cropping and attribute assignment stage. The cropping module generates regular grid cropping areas within the image space according to the cropping parameters input by the user through the interactive interface, such as a target size of 512×512 pixels or 1024×1024 pixels, and optional overlap rate and other indicators. Specifically, the sliding window algorithm determines the size of the window and the sliding step according to the target size and overlap rate in the cropping parameters input by the user. Starting from the upper left corner of the image, the window is gradually moved to the right and downwards according to the calculated step, thereby generating regular grid cropping areas covering the entire image range. The position and size of each window are recorded as an independent cropping unit.

[0026] During the sliding window cropping process, this invention simultaneously performs the same spatial cropping operation on the vector data. To improve cropping efficiency, the cropping module employs spatial index optimization technology. The system pre-constructs spatial indexes for the vector data, such as R-tree indexes or quadtree indexes. When performing vector cropping on a specific cropping window, the system quickly locates the vector features intersecting with the current cropping window using this spatial index, without traversing the entire vector dataset. This optimization significantly reduces computation and dramatically improves cropping speed. In this way, image pixels within each cropping window achieve strict matching with corresponding vector features, ensuring that the images and vector data in the generated sample blocks correspond completely spatially.

[0027] While the cropping operation is underway, the attribute assignment module employs a metadata-driven architecture to automatically add rich attribute information to the sample data. Specifically, the attribute assignment module first extracts key metadata information from the image header file, including shooting time, number of bands, and resolution. For example, the system reads the TIFFTAG_DATETIME tag from the GeoTIFF file to obtain the shooting time, reads IMAGEWIDTH and IMAGELENGTH to obtain the image size, and reads BANDCOUNT to obtain the number of bands. Simultaneously, the system automatically adds standardized attribute fields to the vector data according to preset rules. These fields cover business attributes such as sample type, classification code, and cropping region ID. After completing metadata extraction and attribute field addition, the system converts the extracted metadata information into a suitable format and assigns it to the corresponding attribute fields. For example, the shooting time is converted from a specific format in the image header file to the standard "YYYY-MM-DD" date format and then assigned to the "Shooting Time" attribute field. Through this process, the system achieves a full-feature description of the sample data, ensuring that the metadata and attribute fields in the original data are completely inherited into the sample data.

[0028] To handle large-scale data processing tasks, this invention introduces a multi-threaded parallel processing mechanism. The system uses a thread pool to manage thread creation and destruction, allocating the number of threads rationally based on the number of CPU cores and memory resources of the current hardware system. The queue of tasks to be processed is stored in shared memory, and each thread retrieves a task from shared memory for independent processing. Image-vector sample blocks and their attribute information generated during task processing are transmitted to the output module via a message queue. Simultaneously, the system employs synchronization mechanisms such as mutexes and semaphores to ensure safe access to shared resources by multiple threads, avoiding data races and deadlocks. In this way, the system can simultaneously process multiple pairs of data in parallel, and can also perform parallel operations on multiple cropping windows within a single large data pair, fully utilizing system resources and significantly improving overall processing efficiency.

[0029] During the output phase, the output module automatically generates standardized filenames according to preset sample numbering and naming rules. For example, filenames can include key information such as the original data prefix, the row and column numbers of the cropping window, and the sample type. Simultaneously, the system categorizes and stores output files using a multi-level directory structure, such as creating first-level subfolders by sample type and second-level subfolders by time phase, persistently saving the processed image-vector data pairs to the user-specified storage location. Throughout the processing flow, the logging and exception handling module records the status information of each step in real time. When a cropping window or a data pair fails to process, the system captures the exception and records a detailed error log, while continuing to process the next task. This ensures that the entire batch operation is not interrupted due to individual data anomalies, thereby guaranteeing the reliability and traceability of large-scale batch operations.

[0030] In another preferred embodiment, the present invention also provides a batch adjustment function. When a user discovers that data of the same type needs to be corrected uniformly, such as coordinate system deviation data from the same data source, the user can select the batch adjustment option through the interactive interface. The system automatically performs the same coordinate translation or format conversion operation on these data and displays the data correction effect on the GIS map in real time. This function further improves the ease of use and processing efficiency of the present invention.

[0031] In summary, this invention forms a complete, efficient, and automated batch sample production technology solution through intelligent file matching in the data traversal module, automatic coordinate system transformation in the coordinate unification module, sliding window and spatial index optimization in the cropping module, metadata-driven assignment in the attribute assignment module, multi-threaded scheduling in the parallel processing module, and standardized storage in the output module. Those skilled in the art will understand that the specific implementation of the above modules and steps can employ any suitable programming language and geographic information processing library. For example, using Python combined with the GDAL library for raster and vector data reading, writing, and processing; using OpenCV for image cropping; and using the thread pool module in the standard library for parallel processing—all of these fall within the scope of this invention.

[0032] This invention has illustrated its principles and implementation methods using specific examples. The descriptions of these embodiments are merely illustrative of the method and its core ideas; furthermore, those skilled in the art will recognize that modifications may be made to the specific implementation methods and application scope based on the principles of this invention. Therefore, the content of this specification should not be construed as limiting the invention.

Claims

1. A method for batch sample production based on multi-source data, characterized in that, include: Collect image data files and vector data files in the root directory specified by the user, and use an intelligent file matching algorithm to identify and construct data pairs to be processed consisting of the image data files and vector data files from the same geographical area; Spatially align the data pairs to be processed. Based on the cropping parameters input by the user, use a sliding window algorithm to generate a regular grid cropping region within the image space. Simultaneously perform the same spatial cropping operation on the vector data to generate multiple image-vector sample blocks. While performing the cropping operation, a metadata-driven approach is used to extract metadata information from the image data file and add standardized attribute fields to the vector data, and assign the metadata information to the corresponding standardized attribute fields; The multi-threaded parallel processing mechanism is used to process multiple pairs of data to be processed or multiple cropping windows within a single pair of data to be processed in parallel, and the processed image-vector sample blocks are saved to the output directory according to a preset naming rule.

2. The method for batch sample production based on multi-source data according to claim 1, characterized in that, The intelligent file matching algorithm includes: Recursively scan the root directory and all its subfolders at all levels; Based on a preset filename pattern, regular expressions are used to match filenames within a folder and extract filename prefixes; By combining file type verification, raster image files and vector data files are filtered out. Image files and vector files located in the same folder with the same filename prefix are identified as a valid pair of data to be processed.

3. The method for batch sample production based on multi-source data according to claim 1, characterized in that, The sliding window algorithm determines the window size and sliding step size based on the target size and overlap rate in the cropping parameters, and moves the window from the upper left corner of the image according to the sliding step size to generate a regular grid cropping area covering the entire image range. Furthermore, by constructing a spatial index, vector features within the current window can be quickly located during each clipping, thus avoiding traversing the entire vector data.

4. The method for batch sample production based on multi-source data according to claim 1, characterized in that, The metadata information includes the shooting time, number of bands, and resolution extracted from the image header file; the standardized attribute fields include sample type, classification code, and cropping region ID.

5. The method for batch sample production based on multi-source data according to claim 1, characterized in that, Before the spatial alignment step, a coordinate system processing step is also included: automatically identifying the coordinate systems of the image data file and the vector data file; if the two are inconsistent, they are uniformly converted to a preset target coordinate system to ensure that the image pixel position and the vector graphic geographical position correspond accurately.

6. The method for batch sample production based on multi-source data according to claim 1, characterized in that, The multi-threaded parallel processing mechanism includes: using a thread pool to manage the creation and destruction of threads, and implementing data interaction between threads through shared memory or message queues; wherein, the queue of tasks to be processed is stored in shared memory for threads to retrieve, and the processing results are passed to the output module through a message queue; and mutexes and semaphores are used to ensure safe access to shared resources.

7. A batch sample production system based on multi-source data, characterized in that, include: The data traversal module is used for deep traversal of the root directory specified by the user, and identifies and constructs a queue of tasks to be processed consisting of image data files and vector data files through an intelligent file matching algorithm. The cropping module is used to spatially align the data pairs to be processed. Based on the cropping parameters input by the user, it uses a sliding window algorithm to generate a regular grid cropping area and simultaneously performs spatial cropping on the image and vector data. The attribute assignment module is used to extract metadata information from the image header file, add standardized attribute fields to the vector data, assign the metadata information to the corresponding attribute fields, and generate sample data for full feature description. The output module is used to persistently save the processed image-vector data pairs according to preset naming rules and directory structure. The parallel processing module is used to schedule the clipping module and the attribute assignment module through a multi-threaded parallel processing mechanism to perform parallel operations on multiple pairs of data to be processed or multiple clipping windows.

8. The batch sample production system based on multi-source data according to claim 7, characterized in that, The intelligent file matching algorithm combines filename pattern matching and file type dual verification to identify image data files and vector data files located in the same folder with the same filename prefix.

9. The batch sample production system based on multi-source data according to claim 7, characterized in that, The cropping module employs spatial index optimization technology to quickly locate vector elements intersecting with the current cropping window during the sliding window cropping process, thereby avoiding traversing the entire vector data and improving cropping efficiency.

10. The batch sample production system based on multi-source data according to claim 7, characterized in that, The system also includes a log recording and exception handling module, which is used to record processing logs during large-scale batch operations and to capture and handle exceptions when they occur, ensuring data integrity and traceability of the processing flow.

Images