Storage management method and system for multi-source space-based data images

Abstract

The invention provides a storage management method and system for a multi-source space-based data image, and relates to the technical field of data processing, and the method comprises the steps: obtaining original space-based image data, carrying out the feature analysis, and generating a standard structured data item; constructing a space-time grid division rule, carrying out classification according to data attributes, and generating a global logic storage path identifier; performing association to generate an association mapping relationship, and persisting the association mapping relationship to the distributed metadatabase; responding to the data query request, analyzing the data query request through the distributed metadatabase, and positioning target data; and constructing a distributed object storage cluster, and performing parallel reading to obtain a target image file. The technical problems that in the prior art, the parallel processing capacity of large-scale data is limited, parallel reading and efficient query are difficult to support effectively in the environment of a large-scale data set, the data access performance is poor, and then query delay and load increase are caused are solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a storage management method and system for multi-source space-based data images. Background Technology

[0002] With the rapid development of remote sensing and satellite technologies, the quantity and complexity of space-based data, especially multi-source space-based imagery data, are gradually increasing. These data come from a wide range of sources, including different satellite platforms, sensors, observation targets, and time periods, covering multiple dimensions such as space, time, and attributes. This makes the storage, management, and querying of space-based imagery data particularly complex. Therefore, it is necessary to develop efficient storage management methods to ensure the efficient access, analysis, and application of this large-scale data. Traditional storage systems, such as single database systems or file systems, have limited parallel processing capabilities for large-scale data. Although modern database systems provide distributed storage capabilities, parallel reading and efficient querying still face bottlenecks. In the context of large-scale datasets, traditional storage systems struggle to effectively support parallel reading and efficient querying, leading to poor data access performance, especially when queries involve multiple data nodes, resulting in bandwidth bottlenecks or node access latency issues. Summary of the Invention

[0003] This application provides a storage management method and system for multi-source space-based data images, aiming to solve the technical problems of the limited parallel processing capability of existing technologies for large-scale data, the difficulty in effectively supporting parallel reading and efficient querying in the environment of large-scale datasets, resulting in poor data access performance, and consequently, query latency and increased load.

[0004] The first aspect disclosed in this application provides a storage management method for multi-source space-based imagery. The method includes: acquiring raw space-based imagery data, performing feature analysis to generate standard structured data items; constructing spatiotemporal grid partitioning rules to classify the standard structured data items according to data attributes, generating global logical storage path identifiers; associating the standard structured data items with the global logical storage path identifiers to generate an association mapping relationship, and persisting the association mapping relationship to a distributed metadata database; responding to a data query request, parsing the data query request through the distributed metadata database, locating target data, wherein the target data contains a target logical storage path identifier; constructing a distributed object storage cluster, traversing the distributed object storage cluster based on the target logical storage path identifier to perform parallel reading, and obtaining the target image file.

[0005] The second aspect of this application discloses a storage management system for multi-source space-based data imagery. This system is used in the aforementioned storage management method for multi-source space-based data imagery. The system includes: a feature parsing module for acquiring raw space-based imagery data, performing feature parsing, and generating standard structured data items; a data item classification module for constructing spatiotemporal grid partitioning rules to classify the standard structured data items according to data attributes, generating global logical storage path identifiers; an association mapping relationship generation module for associating the standard structured data items with the global logical storage path identifiers, generating association mapping relationships, and persisting the association mapping relationships to a distributed metadata database; a target data location module for responding to data query requests, parsing the data query requests through the distributed metadata database, and locating target data, wherein the target data includes a target logical storage path identifier; and a target image file acquisition module for constructing a distributed object storage cluster, traversing the distributed object storage cluster based on the target logical storage path identifier, and performing parallel reading to obtain the target image file.

[0006] One or more technical solutions provided in this application have at least the following beneficial effects: 1. This invention generates standard structured data items by performing feature analysis on raw space-based image data, which can ensure that image data from different sources and in different formats are processed and stored uniformly, making different types of data compatible in subsequent processing.

[0007] 2. This invention generates a global logical storage path identifier by constructing spatiotemporal grid partitioning rules and classifying data according to data attributes. This can generate a unique identifier for each data item, ensuring that each image data has a clear and unique storage location in large-scale data storage, which helps to locate and manage image data more efficiently.

[0008] 3. This invention associates standard structured data items with global logical storage path identifiers and persists the association mapping relationship to a distributed meta-database, realizing efficient mapping and tracking of data and storage locations. This persistence of the mapping relationship not only helps with long-term data management but also provides a basis for data access, ensuring the traceability and reliability of data access.

[0009] 4. When responding to a data query request, this invention parses the query request and locates the target data through a distributed metadata database, achieving rapid data retrieval. The target logical storage path identifier in the data query process provides accurate positioning, reducing data retrieval time and storage costs.

[0010] 5. This invention constructs a distributed object storage cluster and performs parallel reading based on the target logical storage path identifier, realizing efficient access to image data. Through parallel reading, it can make full use of multi-core processing and distributed computing capabilities, significantly improving the data reading speed. In addition, the distributed storage cluster enhances the elasticity and fault tolerance of data storage, improving the stability and scalability of the system.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0012] Figure 1 This is a schematic flowchart of a storage management method for multi-source space-based data images provided in an embodiment of this application.

[0013] Figure 2 This is a schematic diagram of the storage management system structure for multi-source space-based data images provided in an embodiment of this application.

[0014] Figure labeling: Feature parsing module 10, data item classification module 20, association mapping relationship generation module 30, target data location module 40, target image file acquisition module 50. Detailed Implementation

[0015] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0016] Example 1, as Figure 1 As shown in the embodiments of this application, a storage management method for multi-source space-based data imagery is provided, the method comprising: Raw space-based imagery data is acquired, its features are analyzed, and standard structured data items are generated.

[0017] Raw space-based imagery data is acquired through multi-source space-based platforms, such as satellites and drones. This data includes not only the pixel information of the images themselves but also auxiliary data, such as raw metadata, sensor configuration, and imaging environment. By analyzing the raw imagery data and its auxiliary data, key metadata is extracted, specifically spatiotemporal metadata, payload attributes, and quality identifiers. The parsed metadata is then organized and encapsulated according to a standard format to generate standard structured data items, which are used for subsequent storage, management, and retrieval.

[0018] The spatiotemporal grid partitioning rules are constructed to classify the standard structured data items according to data attributes, and a global logical storage path identifier is generated.

[0019] Based on the acquired standard structured data items, a spatiotemporal grid is partitioned. This partitioning organizes and divides the data along both spatial and temporal dimensions, ensuring reasonable spatial coverage and slicing the data temporally. Building upon this spatiotemporal partitioning, the data is further categorized according to the data attributes within the standard structured data items. These categorizations include classification based on sensor type, resolution, and image quality labels. The spatiotemporal partitioning, data attribute classifications, and other metadata are encoded. By concatenating spatial grid encoding, temporal slice encoding, and classification attribute encoding, a globally unique logical storage path identifier is generated. This unique identifier ensures the location of each image data item within the storage system.

[0020] The standard structured data items are associated with the global logical storage path identifier to generate an association mapping relationship, and the association mapping relationship is persisted to the distributed metadata database.

[0021] Standard structured data items are matched and associated with generated global logical storage path identifiers. Each standard structured data item corresponds to a unique global logical storage path identifier. Through association operations, an association mapping relationship is generated, which represents the relationship between data items and storage paths. For example, image data of a specific spatiotemporal region and time period is linked to a certain storage path identifier. This mapping relationship is stored in the form of key-value pairs. The above association mapping relationship is stored in a distributed metadata database. A distributed metadata database is a system that can efficiently store and manage large amounts of metadata, ensuring that the target data can be quickly located during queries. The persistent mapping relationship supports efficient data retrieval and access.

[0022] In response to a data query request, the system parses the data query request using a distributed metadata database to locate the target data, which includes a target logical storage path identifier.

[0023] The system receives query requests from users or other systems via a data query interface. These requests include specific requirements for image data, such as time range, spatial range, and sensor type. First, the query request undergoes security verification to ensure the requester has permission to access the requested data; if permission is lacking, the query is rejected. The system checks the completeness and validity of all parameters included in the query request, such as whether it contains the required time, spatial, and image quality conditions. Based on the parameter completeness check results, the query conditions are translated into a database query statement. An index combination is built in the distributed metadata database based on the database query statement, and query analysis is performed to quickly locate the target data, including the logical storage path identifier of the target data. This path identifier directly points to the storage location of the image data, ensuring effective data location.

[0024] A distributed object storage cluster is constructed, and the target image file is obtained by traversing the distributed object storage cluster in parallel based on the target logical storage path identifier.

[0025] A distributed object storage cluster is established, comprising multiple storage nodes that store different portions of the space-based imagery data. This distributed storage design improves data access speed and fault tolerance. Based on the target logical storage path identifier, storage location information, such as storage node and object key, is extracted through parsing. This information indicates the specific location of the target imagery data. Following the target logical storage path identifier, multiple storage nodes are traversed within the distributed storage cluster for parallel read operations. Parallel reading significantly improves data access speed, especially for large-scale imagery data. Data blocks from multiple storage nodes are traversed and read in parallel, then merged to form a complete target imagery file.

[0026] Furthermore, methods for acquiring raw space-based imagery data, performing feature analysis, and generating standard structured data items include: Raw space-based imagery data is collected from multiple space-based platforms, including auxiliary data. Based on this auxiliary data, a data source retrieval is performed, and a target data parsing adapter is invoked. The auxiliary data is analyzed using the target data parsing adapter to extract raw metadata, which includes raw spatiotemporal information, payload configuration parameters, and imaging environment parameters. Based on this raw spatiotemporal information, spatiotemporal transformation is performed to construct standard metadata items, which include a spatiotemporal reference. The raw space-based imagery data is then evaluated for quality according to this spatiotemporal reference to construct image quality labels. Finally, the payload configuration parameters and imaging environment parameters are structurally encapsulated according to the image quality labels to construct the standard structured data items.

[0027] Raw space-based imagery data is acquired through multiple space-based platforms, including various types of satellites, remote sensing equipment, and drones. Each platform acquires imagery data from different perspectives or regions. The raw space-based imagery data collected by each platform includes pixel data and other auxiliary data, such as: sensor configuration (e.g., sensor type, resolution, band information); imaging environment parameters (e.g., lighting conditions, atmospheric effects); and system metadata (e.g., capture time, image location).

[0028] Based on the relevant information contained in the auxiliary data, the data sources are retrieved. These data sources include different sensors, imaging platforms, and the image data they acquire. The retrieved auxiliary data is processed and parsed using a target data parsing adapter, a data processing module responsible for extracting raw metadata from different data source formats. Raw metadata includes: raw spatiotemporal information, such as image capture time, geographic coordinates, and sensor orientation; payload configuration parameters, such as sensor type, resolution, and band information; and imaging environment parameters, such as lighting conditions, atmospheric parameters, and sensor operating status, which affect image quality.

[0029] Using raw spatiotemporal information, such as capture time and geographic coordinates, the data undergoes spatiotemporal transformation. This transformation maps the raw data to a unified spatiotemporal reference system, ensuring consistent comparison and processing of data collected from different platforms and at different times. After the spatiotemporal transformation, standard metadata items are constructed, including the spatiotemporal reference, such as the positioning system coordinate system and a unified time reference. Based on the spatiotemporal reference, the raw space-based imagery data undergoes quality assessment. This assessment is based on several indicators: image sharpness (assessing image resolution and sharpness); geometric accuracy (checking for geometric distortion and assessing positioning accuracy); and imaging environment impact (assessing the influence of lighting, atmospheric conditions, etc., on image quality). Based on the quality assessment results, image quality labels are constructed. These labels help distinguish data of different qualities, determining how to further process and store the data.

[0030] Based on the generated image quality labels, the image data is classified and labeled, and the image payload configuration parameters are structured and encapsulated. These parameters are usually in a fixed format to facilitate subsequent querying and processing. The image imaging environment parameters are also structured and encapsulated. These parameters usually affect the final image quality. Structured storage makes it easier to analyze and compare data under different conditions.

[0031] By combining image quality labels, payload configuration parameters, and imaging environment parameters, a standard structured data item is finally constructed. This data item is a detailed description of the original image data, including spatiotemporal information, quality labels, sensor configuration, and imaging environment.

[0032] Furthermore, the method for constructing spatiotemporal grid partitioning rules to classify the standard structured data items according to data attributes and generate global logical storage path identifiers includes: Spatial coverage analysis is performed based on the standard structured data items to define the spatial coverage area; spatial grid mapping is performed according to the spatial coverage area to construct a spatial grid code; a data attribute classification tree is constructed to classify and match the standard structured data items to generate classification attribute codes; imaging time data of the standard structured data items is extracted, and time slice granularity is set according to the imaging time data; the standard structured data items are matched based on the time slice granularity to generate time slice codes; file identifiers are concatenated according to the spatial grid code, the time slice code, and the classification attribute code to construct the global logical storage path identifier.

[0033] Based on the raw spatiotemporal information in standard structured data items, spatial coverage analysis is performed. This analysis calculates the distribution range of the image in geospatial space, i.e., the image's coverage area. Using the four-corner coordinates provided in the standard structured data items, the image's boundaries are determined. These coordinates identify the image's extent, determining its geographical location and coverage area. Spatial coverage analysis generates the image's spatial coverage area, which can be a rectangular region calculated geometrically from the four-corner coordinates, or other geometric shapes, depending on the image acquisition and projection methods.

[0034] The defined spatial coverage area is mapped onto a grid system according to predetermined grid division rules. The grid is divided based on the image's resolution, geographic extent, and granularity, with each grid cell representing a small area of ​​the image. The grid can be divided into multiple levels, such as multi-level grid division, from coarse to fine, ensuring that the image data is reasonably refined and divided spatially. The granularity of the grid, i.e., the size of each grid cell, is adjusted according to the actual needs of the image data. Each grid cell has a unique code to identify the area. These codes are generated based on the grid's location, size, and other spatial characteristics to ensure that the spatial location of all image data can be uniquely identified throughout the entire data storage and retrieval process. The construction of spatial grid coding ensures the spatial consistency of all image data, allowing the geographic location of images to be uniformly identified through the coding system, facilitating subsequent querying, storage, and processing.

[0035] A data attribute classification tree is a hierarchical structure used to represent various attributes of image data, such as sensor type, resolution, and image quality. In the classification tree, different attribute types are arranged hierarchically, forming a tree-like structure. Based on the attributes in standard structured data items, such as the spatiotemporal information and payload configuration parameters of the image, they are matched with the corresponding categories in the data attribute classification tree. Each data item is assigned to an appropriate category, thus forming an ordered data structure. According to the category of the data item, a classification attribute code is generated for each data item. This code is a structured identifier based on the classification tree, used to represent the type, attribute, and classification information of the data item. Classification attribute coding can map complex image data attributes into concise codes, making subsequent storage, retrieval, and processing more efficient.

[0036] Standard structured data items contain the imaging time of image data, such as the specific timestamp of the image capture. First, this time information is extracted from the data items. Then, according to the data requirements, the time slice granularity is set. The time slice granularity refers to dividing the data into multiple time periods, such as hours, days, and months.

[0037] The imaging time of each standard structured data item is matched according to the set time slice granularity. The imaging time of each image data is checked to determine which time period it belongs to, i.e., which time slice. After determining the time slice to which each data item belongs, a unique code is generated for each time slice. This time slice code can identify the time period to which the data item belongs, which helps with the time sequence management of the data.

[0038] The generated spatial grid code, time slice code, and classification attribute code are concatenated. Each code represents a different dimension of the image data. Combining them can create a globally unique identifier for the data and generate a global logical storage path identifier. This identifier can uniquely determine the storage location of each image data item and can be used for data storage, retrieval, and management.

[0039] Furthermore, spatial coverage analysis is performed based on the aforementioned standard structured data items to delineate the spatial coverage area. The methods include: A two-dimensional plane coordinate system is constructed based on the original space-based image data; the standard structured data items are mapped to the two-dimensional plane coordinate system for image corner positioning to determine the corner coordinates; the center point is calculated based on the corner coordinates to determine the center point coordinates; geographic coverage is calculated proportionally based on the corner coordinates and the center point coordinates to delineate the geographic coverage area; geometric distortion analysis is performed on the standard structured data items based on the geographic coverage area to calculate the positioning error value; the geographic coverage area is corrected based on the positioning error value to set the minimum bounding polygon; the minimum bounding polygon is output as the spatial coverage range.

[0040] A two-dimensional plane coordinate system is constructed based on the original spatiotemporal information of the imagery. This can be a geographic coordinate system, such as WGS84 (Global Positioning System coordinate system), or a specific projection coordinate system of the imagery. If the original imagery data is not in a standard geographic coordinate system, necessary coordinate transformations are performed to convert the coordinate data of the original imagery into coordinates in a two-dimensional plane coordinate system for further analysis.

[0041] Based on the geographic location information in the standard structured data items, the image data is mapped to a two-dimensional plane coordinate system. This process transforms the raw spatiotemporal information of the image into specific coordinate values ​​in the plane coordinate system. Using the mapped coordinate system, the positions of the four corners of the image are determined. These corner coordinates help determine the image's coverage area and its specific location in geospatial space. Through geometric calculations, the system obtains the accurate coordinates of the four corners of the image; these coordinates are used for subsequent geometric correction, spatial analysis, and positioning error correction.

[0042] Using four-corner coordinates, the center point coordinates of the image are calculated through a geometric algorithm. This process is accomplished by calculating the midpoint or geometric center of the four corners. The calculated center point coordinates represent the geometric center of the image and are used to correct positioning errors and perform subsequent geometric adjustments.

[0043] Based on the coordinates of the four corners and the center point, a proportional calculation of geographic coverage is performed. This proportional calculation extrapolates the coverage area of ​​the image based on its geometric characteristics, such as image size, scale, and geographic projection. It calculates the precise geographic coverage area of ​​the image based on factors such as the image scale, geographic coordinate system, and image resolution. The geographic coverage area of ​​the image is then determined based on the results of the proportional calculation. This process generates a geographic region, which can be rectangular, square, or other shapes, depending on the image's geometry and the calculation results.

[0044] Using geographic coverage areas, we analyze the geometric distortion of image data in geospatial space. Geometric distortion is caused by multiple factors, such as image projection error, sensor error, and platform motion. Based on the geometric distortion analysis, we calculate the positioning error value of the image data. The positioning error value refers to the degree of deviation between the image data and the actual geographical location, and it is calculated by comparing it with known reference data. The positioning error value is a key evaluation indicator for image quality and positioning accuracy, and it is recorded to provide a reference for subsequent correction steps.

[0045] Based on the calculated positioning error value, the geographic coverage area is corrected. The correction process includes adjusting the geometry of the image to correct deviations caused by the error, making the geographic coverage of the image more accurate. The corrected geographic coverage area is a polygon. The minimum bounding polygon is then calculated. The minimum bounding polygon is the smallest polygon that encloses the entire geographic coverage area. It has the smallest perimeter or area and can accurately describe the actual coverage of the image.

[0046] The generated minimum bounding polygon is used as the spatial coverage of the image data. This polygon accurately represents the coverage area of ​​the image in geospace and is used for subsequent spatial querying, storage and processing.

[0047] Furthermore, spatial grid mapping is performed according to the aforementioned spatial coverage area to construct spatial grid coding, the method including: A multi-level grid partitioning model is constructed to divide the target area into multiple hierarchical grid units; the multiple hierarchical grid units are traversed and matched according to the spatial coverage range to determine multiple target hierarchical network units; the four corner coordinates and the center point coordinates are mapped to the multiple target hierarchical network units to divide the area into main grid units and auxiliary grid units; the main grid units are grid-coded to construct a first initial grid code; the auxiliary grid units are analyzed across image grids, and encoded based on multiple cross-grid image units to construct a second initial grid code; the first initial grid code and the second initial grid code are traversed and verified, and when the verification is successful, the first initial grid code and the second initial grid code are stored and integrated to construct the spatial grid code.

[0048] Based on the size of the target area and the requirements of the image data, the granularity of the grid division is determined. The granularity can be selected based on the image resolution, the coverage area, and the system's performance requirements. The granularity setting determines the size of each grid unit. The target area is divided into multiple grid units, which are divided according to a preset granularity. Regular square grids can be used, or irregular grids can be used depending on specific needs. A multi-level grid structure means that the grid units of the target area can be further subdivided. For example, a coarse grid unit can be further subdivided into multiple smaller grid units. Different levels of grid units can provide different resolutions, allowing the system to flexibly perform fine-grained spatial analysis as needed. This multi-level grid division mode enables image data to be managed at different levels. Lower-level grids have larger data granularity, while higher-level grids have finer data granularity, thereby improving data storage efficiency and query accuracy.

[0049] The process iterates through all grid cells at all levels, checking each cell individually to see if it intersects or overlaps with the spatial coverage of the target area. Specifically, for each level of grid cell, matching is performed based on spatial coverage. This means that each grid cell is checked to determine if it contains a portion of the target area. Through this traversal and matching process, multiple target-level grid cells are ultimately identified. Each target-level grid cell represents a storage or query area for one or more image data, ensuring effective access and retrieval of relevant data in subsequent operations. By matching across multiple levels, flexible and refined data management and access are possible. Lower-level grid cells are suitable for fast queries and large-scale data retrieval, while higher-level grid cells are used for refined data analysis and localization.

[0050] By mapping the four corner coordinates and center point coordinates of an image to target-level grid cells, the spatial location of the image is determined. The four corner coordinates represent the image boundaries, and the center point coordinates represent the geometric center of the image. By mapping these two coordinates to grid cells, the precise location of the image within the grid can be determined. Based on the four corner coordinates and center point coordinates, the target area is divided into main grid cells and auxiliary grid cells. The main grid cell is the primary storage unit for image data, representing the core area of ​​the image. The main grid cell is used to efficiently store and manage image data. The auxiliary grid cells represent the edge areas of the image data or areas adjacent to the main grid cells. The auxiliary grid cells help manage the extended areas of the image, ensuring data integrity and area coverage.

[0051] Based on the grid's hierarchy, location, and size, an appropriate encoding rule is selected. This rule maps the spatial location of grid cells to a set of numbers or characters, generating a unique identifier for each grid cell. Based on the geometric location of the master grid cell, a unique code is generated for each master grid cell. This code is generated hierarchically according to the grid's structure, ensuring uniqueness and sequentiality. Using the encoding rule and the grid's location, an initial grid code is generated. This code uniquely identifies the master grid cell and supports subsequent data storage and retrieval.

[0052] In the edge regions of image data, the image may span multiple grid cells. By analyzing the overlapping areas of the image with multiple grid cells, cross-grid image cells are identified. Cross-grid image cells refer to the parts where data overlaps or intersects between different grid cells. For cross-grid image cells, encoding is performed based on their distribution across multiple grid cells. Each cross-grid image cell's encoding represents its position and size within multiple grid cells. Through the analysis of cross-grid image cells, a second initial grid code is generated for each auxiliary grid cell. This code differs from the primary grid cell's code and is used to identify the distribution of image data within the auxiliary grid region.

[0053] The first and second initial grid codes are traversed, and their validity is checked. The verification process includes checking whether the codes conform to the rules, whether there are duplicate or erroneous codes, and whether they accurately reflect the location of the grid cells. If both the first and second initial grid codes pass the verification, it means that they can effectively identify the grid cells and can be correctly used for data storage and retrieval. After successful verification, the first and second initial grid codes are stored and integrated. The integration process combines the two codes into a complete spatial grid code, which can simultaneously represent the location information of the main grid cells and auxiliary grid cells.

[0054] Furthermore, constructing a multi-level grid partitioning model divides the target area into multiple levels of grid units. Methods include: A hierarchical recursive approach is used to perform grid analysis and identification on the target area, defining level 0, level 1, ..., level N grids, where N is an integer greater than 1. Based on the level 0, level 1, ..., level N grids, rules are applied to divide the area and construct a multi-level grid system. The multi-level grid system is traversed and spliced ​​according to hierarchical order to construct a hierarchical code. The hierarchical code is then subjected to bidirectional geographic coordinate transformation and mapping to construct the multiple hierarchical grid units.

[0055] A hierarchical recursive algorithm is used to divide the target area into multiple levels of grid cells. The hierarchical recursive process progressively refines the size of the grid cells. The level 0 grid represents the coarsest grid, covering the entire target area. The level 1 grid further subdivides the level 0 grid, and so on. Each subsequent level of grid subdivides the previous grid to increase spatial resolution, up to the Nth level grid, where N is an integer greater than 1. This means that the grid division has at least two levels, and the specific number of levels depends on the size of the target area, the requirements of the image data, and the system performance. Through recursion, the detail and accuracy gradually increase, ensuring that each level of grid can adapt to different spatial analysis needs.

[0056] Based on grid levels 0, 1, ..., N, each grid is divided according to predefined rules, forming a hierarchical grid system. Each level of the grid system represents spatial units with different resolutions and precisions. As the level increases, the size of the grid units gradually decreases, while the level of detail gradually increases. This multi-level grid system can flexibly handle spatial query needs at different levels. Lower-level grids are suitable for large-scale spatial queries, while higher-level grids are suitable for small-scale data processing requiring high-precision analysis and positioning.

[0057] The multi-level grid system is traversed hierarchically, starting with level 0, then level 1, level 2, and so on up to level N. Each grid cell is visited layer by layer in space. Based on the level and location of each grid, the grid codes from different levels are concatenated. For example, the code for level 0 might be "0001", level 1 might be "0001-01", and level 2 might be "0001-01-001". Each level inherits the coding structure of the previous level, ensuring the hierarchy and uniqueness of the codes. This concatenation method generates a hierarchical code for each grid cell, uniquely identifying its location and level, supporting subsequent storage and retrieval.

[0058] A bidirectional geographic coordinate transformation method is employed to map hierarchical codes onto a geographic coordinate system. This bidirectional transformation means that geographic coordinates can be converted to grid codes, and grid codes can be converted back to their corresponding geographic coordinates. Through this bidirectional mapping, hierarchical codes can be accurately matched with geographic coordinates in the target area. Each grid cell's geographic coordinates correspond one-to-one with its corresponding hierarchical code, ensuring data accuracy. The mapping process generates multiple hierarchical grid cells, each representing a specific spatial region and possessing a unique hierarchical code, which can be used for data storage, querying, and retrieval.

[0059] Furthermore, in response to a data query request, the data query request is parsed through a distributed metadata database to locate the target data, which includes a target logical storage path identifier. The method includes: The system receives data query requests through a data query interface and verifies that the data query requests contain access permissions. When the data query requests contain access permissions, it checks the integrity of the parameters of the data query requests and generates parameter check results. Based on the parameter check results, the data query requests are formatted to generate multiple query statements, which correspond to the data query requests. Query analysis is performed based on the multiple query statements to determine query conditions, and index combinations are constructed according to the query conditions. The index combinations are distributed to a distributed metadata database for index querying to obtain multiple query nodes to locate target data. When the target data is a single target data record, the logical storage path identifier field value of the target data is extracted as the target logical storage path identifier. When the target data is multiple target data records, multiple logical storage path identifier field values ​​of the target data are extracted in batches for deduplication processing to determine the target logical storage path identifier.

[0060] The system receives data query requests through a data query interface. These requests include the data type the user wishes to access, query conditions, and request parameters, and can originate from the user interface, API calls, or internal system mechanisms. The system first verifies the validity of the request, including checking whether the requester has sufficient permissions to access the requested data. This involves access control systems, such as role-based access control or other permission verification mechanisms. If the query request contains valid access permissions, the system continues with subsequent steps; otherwise, an error message is returned, the request is rejected, and the user or system is notified.

[0061] After verifying access permissions, a completeness check is performed on all parameters in the query request. This includes: checking for necessary query conditions (e.g., some data queries require specific time ranges, spatial locations, etc., and the query cannot be executed if missing); checking the correctness of parameter formats (e.g., time formats should be valid date formats, spatial coordinates should conform to the expected coordinate system); and checking whether parameter values ​​are within reasonable ranges (e.g., whether the query time is within the allowed time range, and whether the spatial coordinates are within a valid region). Parameter check results are generated: if all parameters meet the requirements, the check is passed, and subsequent query operations continue; if missing or invalid parameters are found, an error message is generated and the user or requester is prompted to correct them.

[0062] Based on the parameter check results, the conditions in the query request are converted into a query statement format that the database or storage system can understand. For example, based on conditions such as time range and spatial range in the request, corresponding SQL or NoSQL query statements are generated. For complex query requests, a single query request is converted into multiple query statements for execution on a distributed database or multiple data sources. For example, query conditions are broken down into: querying data by spatial grid, querying data by time slice, and further subdividing query conditions based on classification attributes. During the generation of multiple query statements, it is ensured that these query statements are consistent with the conditions of the original query request, establishing a correspondence between each query statement and the original query request, ensuring that the query results can be merged or summarized into a complete query response.

[0063] Based on multiple query statements, the query conditions in each statement are analyzed. These conditions include multi-dimensional information such as time range, spatial location, and data attributes. For example, if the query request is filtered based on time and space, the query conditions will include filtering criteria for both dimensions. Based on the query conditions, index combinations are constructed. Index combinations refer to combining multiple query conditions to ensure more efficient query operations.

[0064] Query requests are distributed to different query nodes in the distributed metadata database based on index combinations. Each query node is responsible for storing and processing a portion of the data. Each query node in the distributed metadata database executes query operations based on the index combinations. Because indexes are already used, query nodes can directly locate relevant data records without scanning the entire database. Each query node quickly locates the target data that meets the index query conditions. The target data includes information such as the storage path and metadata of the image data.

[0065] After the query operation is completed, check the query results. If the query result is a single target data record, meaning only one data record matches the query conditions, then continue with the subsequent steps. For a single target data record, extract the logical storage path identifier field from that data record. The logical storage path identifier field is a unique identifier for that data record in the storage system, indicating the storage location of the data.

[0066] If the query returns multiple target data records, batch processing is performed. Multiple logical storage path identifier fields are extracted from these records, with each target data record having its own associated logical storage path identifier. Since multiple target data records may share the same storage path, the extracted logical storage path identifiers need to be deduplicated to ensure that each identifier appears only once, avoiding duplicate queries and data accesses. Through deduplication, a unique target logical storage path identifier is ultimately determined, ensuring that subsequent operations can efficiently locate and access the target data.

[0067] Furthermore, a distributed object storage cluster is constructed, and the target image file is obtained by traversing the distributed object storage cluster in parallel based on the target logical storage path identifier to obtain the target image file. The method includes: Parse the target logical storage path identifier to extract target storage location information; map the target storage location information to the distributed object storage cluster to locate the target storage node and object key; extract multiple read tasks based on the target logical storage path identifier and split them into multiple parallel read subtasks; read multiple image file data blocks from the distributed object storage cluster based on the object key; integrate the multiple parallel read subtasks with the multiple image file data blocks to construct the target image file.

[0068] The target logical storage path identifier is an identifier that contains storage path information. It can uniquely identify the location of data in the storage system. This identifier will be parsed to obtain the target storage location information contained therein, such as the storage bucket name, file path, data partition, etc.

[0069] Based on the extracted target storage location information, this information is mapped to a distributed object storage cluster. The distributed storage cluster consists of multiple storage nodes. The specific target storage node is located within the cluster based on the target storage location information. Each storage node contains a portion or the complete object of a stored file, and the target storage node indicates the physical location of the data. Based on the target storage location information, the object key of the corresponding data in the storage is determined. The object key is a unique identifier for a file or data, helping to quickly locate the target data in the storage.

[0070] Based on the target logical storage path identifier, the system analyzes whether the target data needs to be read in blocks, especially when the data volume is large. The read task is then split into multiple subtasks, each responsible for reading a portion of the data, ensuring efficient processing of large amounts of data. Depending on the physical distribution of storage and network bandwidth, each parallel read subtask is assigned to an appropriate storage node for processing. In this way, multiple data blocks can be read in parallel across multiple nodes, and parallelization helps reduce the overall data read time.

[0071] Based on the determined object key, a read request is sent to the distributed object storage cluster. Each request reads a data block of the target data, which is a portion of the content of a large file. Multiple data blocks are read in parallel across multiple storage nodes, with each node responsible for reading the data block assigned to it, ensuring that the data reading process is efficient.

[0072] Data blocks obtained from multiple parallel reading subtasks are merged, each representing a part of the image file. Specifically, the data blocks are stitched together in the correct order according to the order in which they are read, ensuring the integrity and accuracy of the image. The stitching process includes arranging each part of the image in spatial and temporal order. After merging, a complete target image file is generated, which contains all parts of the original image data and is ready for subsequent storage, analysis, or display.

[0073] Furthermore, the method for parsing the target logical storage path identifier to extract target storage location information includes: The target logical storage path identifier is parsed to identify multiple path components; storage bucket name information is extracted based on the multiple path components, and a data storage logical container is determined based on the storage bucket name information; data storage availability analysis is performed based on the target logical storage path identifier to determine the data storage location; the target storage location information is extracted by locating the data storage logical container in conjunction with the data storage location.

[0074] The target logical storage path identifier undergoes syntax parsing. This parsing process includes: extracting the bucket name (the container or group name for data storage); extracting the file path (the path of the data file in the storage system); and extracting the file name (the name of the specific data object). Through syntax parsing, the path identifier is broken down into multiple path components, which help the system locate and position the data storage location.

[0075] From the parsed path components, the bucket name information is extracted. This is the first part of the path, identifying the container storing the data. For example, the bucket name might be "bucket1" or "data-storage". The bucket name is a logical container in the storage system. All data objects are stored in specific buckets. The extracted bucket name determines the logical container for data storage, which is then mapped to a specific physical storage resource, such as a container in cloud storage, a database, or a distributed file system.

[0076] Using the target logical storage path identifier, data storage availability analysis is performed. This includes: checking if the storage path is valid and points to an existing bucket or file; confirming the availability of the bucket or file path and whether access is permitted; and analyzing whether the data storage location supports read operations to ensure data validity within the storage system. Through availability analysis, the data storage location is ultimately determined. This location can be a physical disk, a storage node, or a location within a distributed storage cluster. Determining the data storage location facilitates subsequent storage operations and data access.

[0077] The system combines the data storage logical container and the data storage location for precise location tracking. The logical container specifies the scope of data storage, while the location points to a specific physical storage resource. By combining these two elements, the storage location of the target data can be accurately pinpointed. At this point, the storage path is fully determined, and the target data can be located within the physical storage system or distributed storage cluster based on this location information. Finally, the target storage location information, including the storage node, storage location, and object key, is extracted. This information is used for subsequent data access operations to ensure accurate access to the target data.

[0078] Example 2, based on the same inventive concept as the storage management method for multi-source space-based data imagery in the foregoing examples, such as... Figure 2 As shown in the embodiment of this application, a storage management system for multi-source space-based data imagery is provided, the system comprising: The feature parsing module 10 is used to acquire raw space-based image data, perform feature parsing, and generate standard structured data items; the data item classification module 20 is used to construct spatiotemporal grid partitioning rules to classify the standard structured data items according to data attributes and generate global logical storage path identifiers; the association mapping relationship generation module 30 is used to associate the standard structured data items with the global logical storage path identifiers to generate association mapping relationships and persist the association mapping relationships to the distributed metadata database; the target data location module 40 is used to respond to data query requests, parse the data query requests through the distributed metadata database, locate the target data, and the target data includes the target logical storage path identifier; the target image file acquisition module 50 is used to construct a distributed object storage cluster, traverse the distributed object storage cluster based on the target logical storage path identifier, and perform parallel reading to obtain the target image file.

[0079] Furthermore, the feature parsing module 10 is used to perform the following operation steps: Raw space-based imagery data is collected from multiple space-based platforms, including auxiliary data. Based on this auxiliary data, a data source retrieval is performed, and a target data parsing adapter is invoked. The auxiliary data is analyzed using the target data parsing adapter to extract raw metadata, which includes raw spatiotemporal information, payload configuration parameters, and imaging environment parameters. Based on this raw spatiotemporal information, spatiotemporal transformation is performed to construct standard metadata items, which include a spatiotemporal reference. The raw space-based imagery data is then evaluated for quality according to this spatiotemporal reference to construct image quality labels. Finally, the payload configuration parameters and imaging environment parameters are structurally encapsulated according to the image quality labels to construct the standard structured data items.

[0080] Furthermore, the data item classification module 20 is used to perform the following operation steps: Spatial coverage analysis is performed based on the standard structured data items to define the spatial coverage area; spatial grid mapping is performed according to the spatial coverage area to construct a spatial grid code; a data attribute classification tree is constructed to classify and match the standard structured data items to generate classification attribute codes; imaging time data of the standard structured data items is extracted, and time slice granularity is set according to the imaging time data; the standard structured data items are matched based on the time slice granularity to generate time slice codes; file identifiers are concatenated according to the spatial grid code, the time slice code, and the classification attribute code to construct the global logical storage path identifier.

[0081] Furthermore, the data item classification module 20 is used to perform the following operation steps: A two-dimensional plane coordinate system is constructed based on the original space-based image data; the standard structured data items are mapped to the two-dimensional plane coordinate system for image corner positioning to determine the corner coordinates; the center point is calculated based on the corner coordinates to determine the center point coordinates; geographic coverage is calculated proportionally based on the corner coordinates and the center point coordinates to delineate the geographic coverage area; geometric distortion analysis is performed on the standard structured data items based on the geographic coverage area to calculate the positioning error value; the geographic coverage area is corrected based on the positioning error value to set the minimum bounding polygon; the minimum bounding polygon is output as the spatial coverage range.

[0082] Furthermore, the data item classification module 20 is used to perform the following operation steps: A multi-level grid partitioning model is constructed to divide the target area into multiple hierarchical grid units; the multiple hierarchical grid units are traversed and matched according to the spatial coverage range to determine multiple target hierarchical network units; the four corner coordinates and the center point coordinates are mapped to the multiple target hierarchical network units to divide the area into main grid units and auxiliary grid units; the main grid units are grid-coded to construct a first initial grid code; the auxiliary grid units are analyzed across image grids, and encoded based on multiple cross-grid image units to construct a second initial grid code; the first initial grid code and the second initial grid code are traversed and verified, and when the verification is successful, the first initial grid code and the second initial grid code are stored and integrated to construct the spatial grid code.

[0083] Furthermore, the data item classification module 20 is used to perform the following operation steps: A hierarchical recursive approach is used to perform grid analysis and identification on the target area, defining level 0, level 1, ..., level N grids, where N is an integer greater than 1. Based on the level 0, level 1, ..., level N grids, rules are applied to divide the area and construct a multi-level grid system. The multi-level grid system is traversed and spliced ​​according to hierarchical order to construct a hierarchical code. The hierarchical code is then subjected to bidirectional geographic coordinate transformation and mapping to construct the multiple hierarchical grid units.

[0084] Furthermore, the target data positioning module 40 is used to perform the following operation steps: The system receives data query requests through a data query interface and verifies that the data query requests contain access permissions. When the data query requests contain access permissions, it checks the integrity of the parameters of the data query requests and generates parameter check results. Based on the parameter check results, the data query requests are formatted to generate multiple query statements, which correspond to the data query requests. Query analysis is performed based on the multiple query statements to determine query conditions, and index combinations are constructed according to the query conditions. The index combinations are distributed to a distributed metadata database for index querying to obtain multiple query nodes to locate target data. When the target data is a single target data record, the logical storage path identifier field value of the target data is extracted as the target logical storage path identifier. When the target data is multiple target data records, multiple logical storage path identifier field values ​​of the target data are extracted in batches for deduplication processing to determine the target logical storage path identifier.

[0085] Furthermore, the target image file acquisition module 50 is used to perform the following operation steps: Parse the target logical storage path identifier to extract target storage location information; map the target storage location information to the distributed object storage cluster to locate the target storage node and object key; extract multiple read tasks based on the target logical storage path identifier and split them into multiple parallel read subtasks; read multiple image file data blocks from the distributed object storage cluster based on the object key; integrate the multiple parallel read subtasks with the multiple image file data blocks to construct the target image file.

[0086] Furthermore, the target image file acquisition module 50 is used to perform the following operation steps: The target logical storage path identifier is parsed to identify multiple path components; storage bucket name information is extracted based on the multiple path components, and a data storage logical container is determined based on the storage bucket name information; data storage availability analysis is performed based on the target logical storage path identifier to determine the data storage location; the target storage location information is extracted by locating the data storage logical container in conjunction with the data storage location.

[0087] Through the foregoing detailed description of the storage management method for multi-source space-based data images, those skilled in the art can clearly understand the storage management system for multi-source space-based data images in this embodiment. Since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and relevant parts can be referred to the method section.

[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for storing and managing multi-source space-based data imagery, characterized in that, The method includes: The raw space-based imagery data is acquired, its features are analyzed, and standard structured data items are generated. The spatiotemporal grid partitioning rules are constructed to classify the standard structured data items according to data attributes, and a global logical storage path identifier is generated; Associate the standard structured data items with the global logical storage path identifier to generate an association mapping relationship, and persist the association mapping relationship to the distributed metadata database; In response to a data query request, the data query request is parsed through a distributed metadata database to locate the target data, which includes a target logical storage path identifier; A distributed object storage cluster is constructed, and the target image file is obtained by traversing the distributed object storage cluster in parallel based on the target logical storage path identifier.

2. The storage management method for multi-source space-based data imagery as described in claim 1, characterized in that, The methods for acquiring raw space-based imagery data, performing feature analysis, and generating standard structured data items include: Raw space-based image data is collected by traversing multiple space-based platforms, and the raw space-based image data includes auxiliary data; Based on the auxiliary data, a data source retrieval is performed, and the target data parsing adapter is invoked. The auxiliary data is analyzed through the target data parsing adapter to extract the original metadata, which includes original spatiotemporal information, payload configuration parameters, and imaging environment parameters. Based on the original spatiotemporal information, spatiotemporal transformation is performed to construct standard metadata items. The standard metadata items include spatiotemporal benchmarks. The original space-based image data is then evaluated for quality according to the spatiotemporal benchmarks to construct image quality labels. The payload configuration parameters and imaging environment parameters are structurally encapsulated according to the image quality labels to construct the standard structured data items.

3. The storage management method for multi-source space-based data imagery as described in claim 1, characterized in that, The method for constructing spatiotemporal grid partitioning rules, classifying the standard structured data items according to data attributes, and generating global logical storage path identifiers includes: Spatial coverage analysis is performed based on the aforementioned standard structured data items to define the spatial coverage area; Spatial grid mapping is performed according to the spatial coverage area to construct spatial grid coding; A data attribute classification tree is constructed to classify and match the standard structured data items, generating classification attribute codes; Extract the imaging time data of the standard structured data items, and set the time slice granularity according to the imaging time data; The standard structured data items are matched based on the time slice granularity to generate time slice codes; The global logical storage path identifier is constructed by concatenating the file identifiers according to the spatial grid encoding, the time slice encoding, and the classification attribute encoding.

4. The storage management method for multi-source space-based data imagery as described in claim 3, characterized in that, Spatial coverage analysis is performed based on the aforementioned standard structured data items to delineate the spatial coverage area. The method includes: A two-dimensional plane coordinate system is constructed based on the original space-based image data; The standard structured data items are mapped to the two-dimensional plane coordinate system to perform image four-corner positioning and determine the four-corner coordinates; The center point coordinates are calculated based on the four corner coordinates to determine the center point coordinates. Geographic coverage area is delineated by performing a proportional calculation based on the coordinates of the four corners and the coordinates of the center point. Geometric distortion analysis is performed on the standard structured data items based on the geographical coverage area to calculate the positioning error value; The geographic coverage area is corrected based on the positioning error value, and a minimum bounding polygon is set. The smallest bounding polygon is output as the spatial coverage area.

5. The storage management method for multi-source space-based data imagery as described in claim 4, characterized in that, The method for mapping spatial grids according to the spatial coverage area and constructing spatial grid codes includes: A multi-level grid partitioning model is constructed to divide the target area into multiple levels of grid units; The multiple hierarchical grid units are traversed according to the spatial coverage area to match and determine multiple target hierarchical network units; The four corner coordinates and the center point coordinates are mapped to the multiple target-level network units to divide the main grid unit and the auxiliary grid unit. The main mesh cells are mesh-coded to construct a first initial mesh code; The auxiliary grid unit is subjected to cross-grid image analysis, and a second initial grid code is constructed based on multiple cross-grid image units. The first initial grid code and the second initial grid code are traversed and verified. When the verification is successful, the first initial grid code and the second initial grid code are stored and integrated to construct the spatial grid code.

6. The storage management method for multi-source space-based data imagery as described in claim 5, characterized in that, Constructing a multi-level mesh generation model involves dividing the target area into multiple levels of mesh units. Methods include: A hierarchical recursive approach is used to perform grid analysis and identification on the target area, setting the level 0 grid, level 1 grid, ..., level N grid, where N is an integer greater than 1; Based on the level 0 grid, the level 1 grid, ... the level N grid, a multi-level grid system is constructed by dividing the grid according to rules. The multi-level grid system is traversed and spliced ​​together in hierarchical order to construct a hierarchical code; The hierarchical encoding is transformed and mapped bidirectionally to geographic coordinates to construct the multiple hierarchical grid units.

7. The storage management method for multi-source space-based data imagery as described in claim 1, characterized in that, In response to a data query request, the method parses the data query request using a distributed metadata database to locate the target data, wherein the target data includes a target logical storage path identifier. The data query request is received through the data query interface, and the data query request is verified to include access permissions. When the data query request includes access permissions, check the integrity of the parameters of the data query request and generate parameter check results; Based on the parameter check results, the data query request is formatted and multiple query statements are generated, and the multiple query statements correspond to the data query request. Based on the multiple query statements, perform query analysis to determine the query conditions, and construct an index combination according to the query conditions; The indexes are distributed to a distributed metadata database for index querying, thereby obtaining multiple query nodes to locate the target data. When the target data is a single target data record, the logical storage path identifier field value of the target data is extracted as the target logical storage path identifier; When the target data consists of multiple target data records, the values ​​of multiple logical storage path identifier fields of the target data are extracted in batches for deduplication to determine the target logical storage path identifier.

8. The storage management method for multi-source space-based data imagery as described in claim 1, characterized in that, Constructing a distributed object storage cluster, and traversing the distributed object storage cluster in parallel based on the target logical storage path identifier to obtain the target image file, the method includes: Parse the target logical storage path identifier to extract the target storage location information; Based on the target storage location information, the target storage node and object key are located by mapping to the distributed object storage cluster. Based on the target logical storage path identifier, multiple read tasks are extracted and split into multiple parallel read subtasks. Multiple image file data blocks are read from the distributed object storage cluster based on the object key; The multiple parallel reading subtasks are combined with the multiple image file data blocks to construct the target image file.

9. The storage management method for multi-source space-based data imagery as described in claim 8, characterized in that, The method for parsing the target logical storage path identifier to extract target storage location information includes: The target logical storage path identifier is parsed to identify multiple path components; Based on the components of the multiple paths, extract the bucket name information, and determine the data storage logical container based on the bucket name information; Based on the target logical storage path identifier, perform data storage availability analysis to determine the data storage location; The target storage location information is extracted by locating the data storage location using the data storage logical container and the data storage location.

10. A storage and management system for multi-source space-based data imagery, characterized in that, The system is used to implement the storage management method for multi-source space-based data imagery as described in any one of claims 1-9, the system comprising: The feature parsing module is used to acquire raw space-based image data, perform feature parsing, and generate standard structured data items; The data item classification module is used to construct spatiotemporal grid partitioning rules, classify the standard structured data items according to data attributes, and generate global logical storage path identifiers; The association mapping relationship generation module is used to associate the standard structured data items with the global logical storage path identifier, generate an association mapping relationship, and persist the association mapping relationship to the distributed metadata database; The target data location module is used to respond to data query requests, parse the data query requests through a distributed metadata database, and locate the target data, which includes a target logical storage path identifier. The target image file acquisition module is used to construct a distributed object storage cluster, and traverse the distributed object storage cluster in parallel based on the target logical storage path identifier to obtain the target image file.

Images