Mass data organization and storage method, device, electronic equipment and storage medium

Abstract

The application provides a mass data organization and storage method, device, electronic equipment and storage medium, in the method, a target data segment in a target physical storage partition can be independently executed data maintenance from a whole data set composed of data segments in other physical storage partitions by using a partition unloading method, that is, when data maintenance is performed due to problems of data in the target physical storage partition, data in other physical storage partitions can still be used, high-availability data requirements are met, when data maintenance is required, only the target physical storage partition corresponding to the data needs to be maintained, easy-to-manage data requirements are met, the data segment in the physical storage partition is taken as an atomic unit, consistency of data content is ensured, and data in an incomplete physical storage partition during data processing is invisible to a business application (that is, an application system), and requirements of an atomic application scene of a super transaction are met.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and in particular to a method, apparatus, electronic device, and storage medium for organizing and storing massive amounts of data. Background Technology

[0002] The natural resources sector has accumulated tens of millions of data points through various initiatives. This data plays a crucial role in natural resources operations. However, due to its massive volume, diverse sources, heterogeneous structure, and dynamic nature, it requires continuous maintenance and dynamic updates. Currently, database partitioning technology has solved the problems of massive data storage and high-performance, high-concurrency applications. However, from the perspective of the inherent characteristics of natural resources data (spatial and temporal features) and application needs, the requirements for high availability and manageability of the data have not been fully met, and the need for super-transactional atomicity in handling massive amounts of data under concurrent access has been overlooked.

[0003] Natural resource operations, which generate massive amounts of data, often involve multi-level collaboration at the national, provincial, municipal, and county levels. The organization and management of these results should simultaneously meet the needs of all four levels, requiring multi-granularity organization and management. Currently, the database construction for these natural resource data at the municipal level and above either involves simply piling up data at the most basic unit of data production (e.g., county-level administrative regions) or haphazardly arranging data from the entire region within a single storage space. The technical logic of data construction does not fully reflect the business characteristics of multi-level collaborative data management, and lacks sufficient data control capabilities independent of application logic and multi-granularity atomicity support. For example, databases built with a single storage space struggle to ensure the integrity of data entry and updates at the county-level administrative region management unit, and fault isolation cannot be minimized. Provincial databases, simply piling up results at the county-level administrative region granularity, increase the complexity of backup and recovery operations at the municipal and provincial levels. Both of these database construction methods also suffer from the problem that failures in local data and its derived data often affect the overall database accessibility, significantly compromising the high availability requirements of ubiquitous sharing.

[0004] Currently, the natural resources system uses database partitioning technology to address the issues of high performance and high concurrency in applications with massive amounts of data. However, it largely ignores the needs of high availability, ease of management, and atomicity in application scenarios involving massive amounts of data. It often relies on high-cost, high-tech-complexity, and difficult-to-maintain cluster hardware and software deployments to meet the high availability requirements of massive databases, and on writing complex business code to achieve atomicity in business scenarios involving massive transactions.

[0005] In summary, traditional methods of organizing and storing massive amounts of data are insufficient to fully meet the needs of application scenarios requiring high availability, ease of management, and atomicity beyond transactions. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a method, apparatus, electronic device and storage medium for organizing and storing massive amounts of data, so as to alleviate the difficulty of traditional massive data organization and storage methods in fully meeting the needs of application scenarios with high availability, ease of management and atomicity beyond transactions.

[0007] In a first aspect, embodiments of the present invention provide a method for organizing and storing massive amounts of data, applied to a data management business interface at the partition granularity, the method comprising:

[0008] According to the partition unloading method called by the application system, the target dataset segment in the target physical storage partition is unloaded from the overall dataset, so that the target dataset segment in the target physical storage partition can perform data addition, deletion and modification operations independently of the overall dataset composed of dataset segments in other physical storage partitions. Except for the application system that calls the partition unloading method, other application systems cannot access the target dataset segment in the target physical storage partition. When performing data addition, deletion and modification operations on the target dataset segment in the target physical storage partition, the corresponding target segmentation rule is followed. The target segmentation rule is the segmentation rule corresponding to the target dataset segment when the overall dataset is segmented.

[0009] The target data set segment in the target physical storage partition is managed according to the target partition management function called by the application system to obtain the managed target physical storage partition. The target partition management function includes at least one of the following: clearing partition, deleting partition, migrating partition, cloning partition, splitting partition, merging partition, and maintaining partition index.

[0010] The target segmentation rules corresponding to the target physical storage partition that has been managed are verified against the segmentation rule set of the overall dataset to verify whether there is a conflict. The data structure of the target dataset segment in the target physical storage partition that has been managed is verified against the data structure of the overall dataset to verify whether the data structure is consistent.

[0011] After successful verification, the target dataset segment in the target physical storage partition that has been managed is incorporated into its overall dataset according to the partition mounting method called by the application system, making it accessible to all application systems.

[0012] Furthermore, before unloading the target dataset segment from the overall dataset from the target physical storage partition according to the partition unloading method called by the application system, the method further includes:

[0013] When the application system calls the data management business interface, it determines a set of segmentation rules for segmenting the overall dataset, and segments the overall dataset according to the set of segmentation rules to obtain multiple dataset segments;

[0014] Multiple physical storage partitions are created according to the segmentation rule set, and a mapping relationship is established between each dataset segment and the corresponding physical storage partition, so that each dataset segment is stored in the corresponding physical storage partition.

[0015] Furthermore, a set of segmentation rules for segmenting the overall dataset is determined, including:

[0016] A set of segmentation rules for segmenting the overall dataset is determined based on the inherent characteristics of the manually selected overall dataset, wherein the inherent characteristics include: temporal features and spatial features.

[0017] Furthermore, multiple physical storage partitions are created according to the segmentation rule set, and a mapping relationship is established between each dataset segment and its corresponding physical storage partition, including:

[0018] Based on the user's customization, a physical storage partition is created corresponding to each segmentation rule, and then a mapping relationship between each dataset segment and its corresponding physical storage partition is established according to each segmentation rule.

[0019] Furthermore, the partition unloading method carries data isolation boundary information, and unloads the target dataset segment in the target physical storage partition from the overall dataset according to the partition unloading method called by the application system, including:

[0020] Determine the target physical storage partition corresponding to the data isolation boundary information;

[0021] Unload the target dataset segment from the overall dataset in the target physical storage partition.

[0022] Furthermore, the clearing of the partition includes: logical clearing and physical clearing. The logical clearing involves deleting the target dataset segment and the corresponding index partition in the target physical storage partition, but without releasing the physical storage space occupied by the target physical storage partition. The physical clearing involves deleting the target dataset segment and the corresponding index partition in the target physical storage partition, and releasing the other physical storage space occupied by the target physical storage partition, except for retaining the necessary physical storage space.

[0023] While deleting the target physical storage partition, the segmentation rules of the deleted target physical storage partition are merged into the data segmentation rule set of the overall dataset.

[0024] The migration partition involves copying the target dataset segment from the target physical storage partition to the first new physical storage partition, deleting the target physical storage partition, updating the mapping relationship between the target dataset segment and the corresponding first new physical storage partition, and rebuilding the partition index corresponding to the target dataset segment.

[0025] The cloning partition involves copying the target dataset segment from the target physical storage partition to the second new physical storage partition, and creating a corresponding partition index for the target dataset segment in the second new physical storage partition according to the index corresponding to the cloned target dataset segment and its index type and parameters.

[0026] The partitioning process involves generating multiple sub-segmentation rules based on the segmentation rules corresponding to the split dataset segments, allocating a physical storage partition for each sub-segmentation rule to obtain multiple target physical storage sub-partitions, migrating dataset segments from the split target physical storage partitions to the corresponding target physical storage sub-partitions according to the sub-segmentation rules, and creating corresponding partition indexes for the multiple target dataset sub-segments formed by the migration based on the indexes corresponding to the split target dataset segments and their index types and parameters.

[0027] The merging partition is generated by generating merging rules based on the target segmentation rules corresponding to the multiple target dataset segments to be merged, allocating corresponding physical storage partitions to the merging rules, obtaining the merged target physical storage partition, migrating the multiple target dataset segments to be merged to the merged target physical storage partition to form a merged dataset segment, releasing the physical storage of the merged target physical storage partition, and creating a corresponding partition index on the merged dataset segment according to the index of the overall dataset.

[0028] The maintenance of the partition index involves rebuilding and maintaining the partition index on the target dataset segment in the target physical storage partition that has been unloaded.

[0029] Furthermore, the partition mounting method carries data boundary information, and according to the partition mounting method called by the application system, incorporates the target dataset segment in the managed target physical storage partition into its overall dataset for access by all application systems, including:

[0030] Determine the target physical storage partition for which management has been completed, corresponding to the data boundary information;

[0031] The target dataset segment in the target physical storage partition that has been managed is incorporated into its overall dataset for access by all application systems.

[0032] Secondly, embodiments of the present invention also provide an apparatus for organizing and storing massive amounts of data, applied to a data management business interface at the partition granularity, the apparatus comprising:

[0033] The unloading unit is used to unload the target dataset segment in the target physical storage partition from the overall dataset according to the partition unloading method called by the application system, so that the target dataset segment in the target physical storage partition can perform data addition, deletion and modification operations independently of the overall dataset composed of dataset segments in other physical storage partitions. Except for the application system that calls the partition unloading method, other application systems cannot access the target dataset segment in the target physical storage partition. When performing data addition, deletion and modification operations on the target dataset segment in the target physical storage partition, the corresponding target segmentation rule is followed. The target segmentation rule is the segmentation rule corresponding to the target dataset segment when the overall dataset is segmented.

[0034] The management unit is used to manage the target data segment in the target physical storage partition according to the target partition management function called by the application system, and obtain the managed target physical storage partition. The target partition management function includes at least one of the following: clearing partition, deleting partition, migrating partition, cloning partition, splitting partition, merging partition, and maintaining partition index.

[0035] The verification unit is used to verify the target segmentation rules corresponding to the target physical storage partition that has been managed by comparing them with the segmentation rule set of the overall dataset to verify whether there is a conflict, and to verify the data structure of the target dataset segment in the target physical storage partition that has been managed by comparing it with the data structure of the overall dataset to verify whether the data structure is consistent.

[0036] The mounting unit, after successful verification, incorporates the target dataset segment in the target physical storage partition into its overall dataset for access by all application systems, according to the partition mounting method called by the application system.

[0037] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0038] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the method described in any of the first aspects above.

[0039] In this embodiment of the invention, a method for organizing and storing massive amounts of data is provided, applied to a data management business interface at the partition granularity. The method includes: unloading a target dataset segment from the overall dataset in a target physical storage partition according to a partition unloading method called by an application system, so that the target dataset segment in the target physical storage partition can perform data addition, deletion, and modification operations independently of the overall dataset composed of dataset segments in other physical storage partitions. Except for the application system that called the partition unloading method, other application systems cannot access the target dataset segment in the target physical storage partition. When performing data addition, deletion, and modification operations on the target dataset segment in the target physical storage partition, the corresponding target segmentation rule is followed. The target segmentation rule is the segmentation rule corresponding to the target dataset segment when segmenting the overall dataset. According to the application... The system calls the target partition management function to manage the target dataset segment in the target physical storage partition, resulting in a managed target physical storage partition. The target partition management function includes at least one of the following: clearing a partition, deleting a partition, migrating a partition, cloning a partition, splitting a partition, merging a partition, and maintaining a partition index. The system then verifies the target segmentation rules corresponding to the managed target physical storage partition against the segmentation rule set of the overall dataset to check for conflicts. It also verifies the data structure of the target dataset segment in the managed target physical storage partition against the data structure of the overall dataset to ensure consistency. After successful verification, the system incorporates the target dataset segment in the managed target physical storage partition into its overall dataset according to the partition mounting method called by the application system, making it accessible to all application systems.As described above, the method for organizing and storing massive amounts of data in this invention employs a partitioning and offloading approach. This allows the target dataset segment in the target physical storage partition to be maintained independently from the overall dataset comprised of dataset segments in other physical storage partitions. In other words, data management and processing operations occurring on the target physical storage partition do not affect the availability of data in other physical storage partitions within the same overall dataset to the application system. Specifically, when data in the target physical storage partition is experiencing problems and requires maintenance, data in other physical storage partitions remains usable, meeting the high availability data requirements. Furthermore, when data maintenance is needed, only the target physical storage partition corresponding to that data needs to be maintained; there is no need for overall maintenance of the entire dataset. Only local data updates and maintenance are required, eliminating the need for overall dataset maintenance. This significantly improves data maintenance efficiency and meets the demand for easily manageable data. Furthermore, by using dataset segments within physical storage partitions as atomic units, data consistency is guaranteed. Incomplete data within physical storage partitions during data processing is invisible to business applications (i.e., application systems). This means that for massive datasets, the visibility of local data to the business can be dynamically controlled, meeting the requirements of atomic application scenarios beyond transactions. In other words, the massive data organization and storage method of this invention can meet the needs of high availability, easy management, and atomic application scenarios beyond transactions, alleviating the difficulty of traditional massive data organization and storage methods in fully meeting these requirements. Attached Figure Description

[0040] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 A flowchart illustrating a method for organizing and storing massive amounts of data, as provided in an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of a device for organizing and storing massive amounts of data, provided as an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0044] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Traditional methods of organizing and storing massive amounts of data are insufficient to meet the needs of high availability, ease of management, and atomicity beyond transactions in application scenarios.

[0046] Based on this, the method for organizing and storing massive amounts of data in this invention employs a partitioning offloading method. This allows the target dataset segment in the target physical storage partition to be maintained independently from the overall dataset composed of dataset segments in other physical storage partitions. In other words, data management and processing operations occurring on the target physical storage partition do not affect the availability of data in other physical storage partitions within the same overall dataset to the application system. Specifically, when data maintenance is required on the target physical storage partition, data in other physical storage partitions remains usable, meeting the high availability data requirements. Furthermore, when data maintenance is needed, only the target physical storage partition corresponding to the data needs to be maintained. This eliminates the need for overall maintenance of the entire dataset; only local data updates and maintenance are required, significantly improving data maintenance efficiency and meeting the demand for easily manageable data. Furthermore, by using dataset segments within physical storage partitions as atomic units, data consistency is guaranteed. Incomplete data in physical storage partitions during data processing is invisible to business applications (i.e., application systems). This means that for massive datasets, the visibility of local data to the business can be dynamically controlled, meeting the requirements of atomic application scenarios beyond transactions. In other words, the massive data organization and storage method of this invention can meet the requirements of high availability, easy management, and atomic application scenarios beyond transactions.

[0047] To facilitate understanding of this embodiment, a method for organizing and storing massive amounts of data disclosed in this embodiment of the invention will first be described in detail.

[0048] Example 1:

[0049] According to an embodiment of the present invention, an embodiment of a method for organizing and storing massive amounts of data is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0050] Figure 1This is a flowchart of a method for organizing and storing massive amounts of data according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0051] Step S102: According to the partition unloading method called by the application system, the target dataset segment in the target physical storage partition is unloaded from the overall dataset, so that the target dataset segment in the target physical storage partition can perform data addition, deletion and modification operations independently of the overall dataset composed of dataset segments in other physical storage partitions. Except for the application system that calls the partition unloading method, other application systems cannot access the target dataset segment in the target physical storage partition. When performing data addition, deletion and modification operations on the target dataset segment in the target physical storage partition, the corresponding target segmentation rule is followed. The target segmentation rule is the segmentation rule corresponding to the target dataset segment when the overall dataset is segmented.

[0052] In this embodiment of the invention, the above-described method for organizing and storing massive amounts of data can be applied to data management business interfaces at the partition granularity. When an application system expects to operate directly on a target dataset segment without going through the overall dataset, such as isolating flawed data or optimizing and rebuilding local indexes, the application system calls the partition unloading method. When the partition unloading method is called, data isolation boundary information (e.g., logical isolation boundaries and / or start time boundaries) of a target dataset segment is independently managed and maintained within the overall dataset, thereby unloading the target dataset segment from the overall dataset in the target physical storage partition.

[0053] The target physical storage partition (i.e., the unmounted physical storage partition) can be a newly created empty physical storage partition or a physical storage partition that already has the target dataset segment. Although the unmounted physical storage partition becomes another dataset independent of the overall dataset, the interface's program logic implements the data segmentation rules that the original physical storage partition (i.e., the physical storage partition before it was unmounted) should follow when performing data add, delete, or modify operations independently on the target physical storage partition, so that the target physical storage partition can still follow the segmentation rules of the overall dataset when it is mounted.

[0054] Taking the land use map dataset of Province X as an example, the subset of land use map data of County Y in Province X (i.e., the target dataset segment) that is unloaded is not allowed to have land use map data of other county-level administrative regions in Province X inserted into it; it can only have land use map data of County Y in Province X inserted into it. That is, when data addition, deletion, and modification operations are performed on the target dataset segment in the target physical storage partition, the corresponding target segment rules are followed. Since the partition unloading operation only involves updating the metadata describing the overall dataset and the target dataset segment of its partition, the impact on business applications (i.e., application systems) accessing the overall dataset (i.e., the overall logical partition dataset) during this operation is negligible. Except for the application that is maintaining the unloaded target dataset segment, the data in the target physical storage partition is not visible to business applications on the overall dataset after it is unloaded (i.e., except for the application system that calls the partition unloading method, other application systems cannot access the target dataset segment in the target physical storage partition).

[0055] Step S104: Manage the target data set segment in the target physical storage partition according to the target partition management function called by the application system to obtain the managed target physical storage partition. The target partition management function includes at least one of the following: clear partition, delete partition, migrate partition, clone partition, split partition, merge partition, maintain partition index.

[0056] Step S106: Verify the target segmentation rules corresponding to the managed target physical storage partition against the segmentation rule set of the overall dataset to check for conflicts, and verify the data structure of the target dataset segment in the managed target physical storage partition against the data structure of the overall dataset to check for consistency.

[0057] Specifically, before mounting the partition, the interface (i.e., the data management business interface at the partition granularity) compares the target segmentation rules corresponding to the managed target physical storage partition with the segmentation rule set of the overall dataset (each segmentation rule corresponding to the overall dataset), and compares the data structure of the target dataset segment in the managed target physical storage partition with the data structure of the overall dataset. This verifies whether the target physical storage partition in the unmounted state can be mounted onto the overall dataset. If the target segmentation rules corresponding to the managed target physical storage partition conflict with each segmentation rule corresponding to the overall dataset, or if the data structure of the target dataset segment in the managed target physical storage partition is inconsistent with the data structure of the overall dataset, then the subsequent mounting will not be performed, and the process will return to step S104 above until the verification is successful.

[0058] Step S108: After successful verification, the target dataset segment in the target physical storage partition is incorporated into its overall dataset for access by all application systems according to the partition mounting method called by the application system.

[0059] Specifically, contrary to partition unloading, when an application system deems that a partitioning operation independent of the overall dataset has been completed and needs to incorporate the target dataset segment contained in the managed target physical storage partition into the overall dataset for runtime, partition mounting can be used to disable the independent management and maintenance of data boundary information (i.e., data region boundaries and / or time windows) of a target dataset segment within the overall dataset. Through partition mounting, the target dataset segment contained in the target physical storage partition is incorporated into the overall dataset (i.e., the overall logical partition dataset), and accessed by the business application system along with the original data in the overall logical partition dataset. It should be noted that the segmentation rules used to describe the target physical storage partition should not conflict with the segmentation rules of the overall dataset (i.e., the overall logical partition dataset) to which it is mounted. Taking the land use map dataset of Province X as an example, after the unloaded subset of land use map data from County Y in Province X has completed data update maintenance, index optimization, or storage migration, it can be mounted and incorporated into the land use map dataset of Province X for access by business applications. Since the partition mounting operation only involves updating the metadata describing the overall logical partition dataset and its corresponding subset of partition data (i.e., the target dataset segment), the impact on business applications accessing the overall dataset during this operation is negligible.

[0060] In this embodiment of the invention, a method for organizing and storing massive amounts of data is provided, applied to a data management business interface at the partition granularity. The method includes: unloading a target dataset segment from the overall dataset in a target physical storage partition according to a partition unloading method called by an application system, so that the target dataset segment in the target physical storage partition can perform data addition, deletion, and modification operations independently of the overall dataset composed of dataset segments in other physical storage partitions. Except for the application system that called the partition unloading method, other application systems cannot access the target dataset segment in the target physical storage partition. When performing data addition, deletion, and modification operations on the target dataset segment in the target physical storage partition, the corresponding target segmentation rule is followed. The target segmentation rule is the segmentation rule corresponding to the target dataset segment when segmenting the overall dataset. According to the application... The system calls the target partition management function to manage the target dataset segment in the target physical storage partition, resulting in a managed target physical storage partition. The target partition management function includes at least one of the following: clearing a partition, deleting a partition, migrating a partition, cloning a partition, splitting a partition, merging a partition, and maintaining a partition index. The system then verifies the target segmentation rules corresponding to the managed target physical storage partition against the segmentation rule set of the overall dataset to check for conflicts. It also verifies the data structure of the target dataset segment in the managed target physical storage partition against the data structure of the overall dataset to ensure consistency. After successful verification, the system incorporates the target dataset segment in the managed target physical storage partition into its overall dataset according to the partition mounting method called by the application system, making it accessible to all application systems.As described above, the method for organizing and storing massive amounts of data in this invention employs a partitioning and offloading approach. This allows the target dataset segment in the target physical storage partition to be maintained independently from the overall dataset comprised of dataset segments in other physical storage partitions. In other words, data management and processing operations occurring on the target physical storage partition do not affect the availability of data in other physical storage partitions within the same overall dataset to the application system. Specifically, when data in the target physical storage partition is experiencing problems and requires maintenance, data in other physical storage partitions remains usable, meeting the high availability data requirements. Furthermore, when data maintenance is needed, only the target physical storage partition corresponding to that data needs to be maintained; there is no need for overall maintenance of the entire dataset. Only local data updates and maintenance are required, eliminating the need for overall dataset maintenance. This significantly improves data maintenance efficiency and meets the demand for easily manageable data. Furthermore, by using dataset segments within physical storage partitions as atomic units, data consistency is guaranteed. Incomplete data within physical storage partitions during data processing is invisible to business applications (i.e., application systems). This means that for massive datasets, the visibility of local data to the business can be dynamically controlled, meeting the requirements of atomic application scenarios beyond transactions. In other words, the massive data organization and storage method of this invention can meet the needs of high availability, easy management, and atomic application scenarios beyond transactions, alleviating the difficulty of traditional massive data organization and storage methods in fully meeting these requirements.

[0061] The above provides a brief overview of the method for organizing and storing massive amounts of data according to the present invention. The specific details involved are described in detail below.

[0062] In an optional embodiment of the present invention, the premise for unloading the target dataset segment in the target physical storage partition from the overall dataset according to the partition unloading method called by the application system is that the overall dataset has been segmented and allocated with physical storage partitions based on the segmented configuration. The method for segmenting and allocating physical storage partitions is as follows:

[0063] (1) When the application system calls the data management business interface, it determines the segmentation rule set for segmenting the overall dataset, and segments the overall dataset according to the segmentation rule set to obtain multiple dataset segments;

[0064] Specifically, the overall dataset is segmented based on the inherent characteristics of the manually selected dataset, which include temporal and spatial features.

[0065] In implementation, for a dataset that may contain tens of millions of records, manual selection of the dataset's inherent features is performed based on requirements (e.g., based on continuous accumulation over time or geographical coverage, selection by time period, administrative region, business category, etc.; if by time period, features by day, month, quarter, year; if by administrative region, features by county, city, province, etc.). The interface of this invention (the data management business interface with partitioning granularity) then determines the segmentation rule set for dividing the dataset. If by month, the segmentation rule set could be January 2024, February 2024, and March 2024. For example, in April 2024, the entire dataset is segmented by month according to the segmentation rule set. As shown above, the data in January 2024 is taken as one dataset segment, the data in February 2024 is taken as another dataset segment, and the data in March 2024 is taken as yet another dataset segment. In this way, multiple dataset segments are obtained, and each dataset segment corresponds to a segmentation rule. That is, based on the segmentation rule set, the data in this massive dataset (i.e., the overall dataset) can be aggregated into multiple dataset segments. Records (i.e., data) in the same dataset segment have good proximity in terms of time, spatial distribution, or other classification features. Like ordinary datasets, logically partitioned datasets (i.e., dataset segments) are known to relevant business applications, but business applications may not know the underlying partitioning logic (i.e., the specific segmentation rules). Through partition unloading (as described in step S102 above), business applications can avoid accessing partial data that has failed, requires offline maintenance (e.g., migration to storage media with different performance), cannot meet the needs of ultra-transactional atomicity scenarios (e.g., statistical analysis during database ingestion), or requires complete content replacement, thus maintaining the sustainability of business application operation. For business applications that need to avoid partial data to operate, since the volume of that partial data is much smaller, the time required for fault recovery and offline maintenance of that partial data is much shorter. This means that the business application interruption time caused by fault recovery or maintenance of partial data is much shorter, and the recovery time is generally predictable. Taking the land use map dataset of Province X as an example, this land use map dataset contains approximately 17 million land use maps. The land use map data can be segmented based on the county-level administrative region where the land use map is located, forming a subset of land use map data (i.e., multiple dataset segments) that correspond one-to-one with the 103 county-level administrative regions of Province X. Each data subset contains only the land use map data of the administrative region to which it belongs.This data segmentation and aggregation strategy avoids random data storage layout for map features. In map application scenarios, the visualization of spatially adjacent map features means that these map feature data need to be read simultaneously, greatly improving the efficiency of physical data reading and avoiding a large number of data filtering operations. The user's direct experience is an improvement in the performance of map feature visualization. This data segmentation strategy also caters to the multi-level authorized business application scenarios of provinces, cities, and counties. That is, county-level applications only need to be authorized to access their own dataset segments, city-level applications only need to be authorized to access the dataset segments of a few counties under their jurisdiction, and provincial-level applications only need to be authorized to access all county-level dataset segments. In addition, dataset segments can be maintained independently of the overall dataset after being partitioned and unloaded. During the maintenance of one or more county-level land use map feature data (such as data entry, spatial index creation, and migration from traditional disks to high-performance SSD storage), the operation of various natural resource business applications is generally not interrupted, ensuring good availability.

[0066] (2) Create multiple physical storage partitions according to the segmentation rule set, and establish a mapping relationship between each dataset segment and the corresponding physical storage partition, so that each dataset segment is stored in the corresponding physical storage partition;

[0067] Specifically, physical storage partitions are created for each segmentation rule based on user-defined criteria, and then a mapping relationship is established between each dataset segment and its corresponding physical storage partition based on each segmentation rule.

[0068] In implementation, users can customize the storage location for each dataset segment based on factors such as usage frequency and fault tolerance. Then, physical storage partitions corresponding to each dataset segment are created in the corresponding storage location according to the user's customization. The created physical storage partitions also correspond to the corresponding segmentation rules. Based on the segmentation rules, a mapping relationship between each dataset segment and the corresponding physical storage partition can be established. After the mapping relationship is established, each dataset segment is stored in the corresponding physical storage partition according to the database's own data storage location function.

[0069] When creating each physical storage partition, a minimum capacity of physical storage (e.g., a database segment) matching the data storage requirements of that physical storage partition is allocated (e.g., different technologies, different disk redundancy levels). The newly created physical storage partition becomes part of the entire logical partition dataset as an empty subset of data. Since only the minimum capacity physical storage partition is allocated, the impact of creating a new physical storage partition on the synchronous access of the overall dataset by business applications is negligible.

[0070] In an optional embodiment of the present invention, the partition unloading method carries data isolation boundary information, and unloads the target data set segment in the target physical storage partition from the overall dataset according to the partition unloading method called by the application system, specifically including the following steps:

[0071] 1) Determine the target physical storage partition corresponding to the data isolation boundary information;

[0072] 2) Unload the target dataset segment from the overall dataset in the target physical storage partition.

[0073] In an optional embodiment of the present invention, clearing the partition includes: logical clearing and physical clearing. Logical clearing involves deleting the target dataset segment and the corresponding index partition in the target physical storage partition, but without releasing the physical storage space occupied by the target physical storage partition. Physical clearing involves deleting the target dataset segment and its corresponding index partition in the target physical storage partition, and releasing the other physical storage space occupied by the target physical storage partition, except for retaining the necessary physical storage space (such as a database block or page). The aforementioned index partition is a physical storage area that stores only the data index information of the target dataset segment. It is a sub-region of the overall dataset index storage area. The index of this dataset segment is referred to as the partition index.

[0074] Deleting a partition involves deleting the target physical storage partition and simultaneously integrating the segmentation rules of the deleted target physical storage partition into the data segmentation rule set of the overall dataset.

[0075] The migration partition involves copying the target dataset segment from the target physical storage partition to the first new physical storage partition, deleting the target physical storage partition, updating the mapping relationship between the target dataset segment and the corresponding first new physical storage partition, and rebuilding the partition index corresponding to the target dataset segment.

[0076] The clone partition is to copy the target dataset segment from the target physical storage partition to the second new physical storage partition, and create a corresponding partition index for the target dataset segment in the second new physical storage partition according to the index corresponding to the cloned target dataset segment and its index type and parameters.

[0077] The partitioning process involves generating multiple sub-segmentation rules based on the segmentation rules corresponding to the split dataset segments, allocating a physical storage partition for each sub-segmentation rule, resulting in multiple target physical storage sub-partitions, migrating dataset segments from the split target physical storage partitions to the corresponding target physical storage sub-partitions according to the sub-segmentation rules, and creating corresponding partition indexes for the multiple target dataset sub-partitions formed by the migration based on the indexes corresponding to the split target dataset segments and their index types and parameters.

[0078] The process of merging partitions involves generating merge rules based on the target segmentation rules corresponding to the multiple target dataset segments to be merged, allocating corresponding physical storage partitions to the merge rules, obtaining the target physical storage partitions for merging, migrating the multiple target dataset segments to be merged to the target physical storage partitions for merging, forming merged dataset segments, releasing the physical storage of the target physical storage partitions for merging, and creating corresponding partition indexes on the merged dataset segments according to the indexing of the overall dataset.

[0079] Maintaining the partition index involves rebuilding and maintaining the partition index on the target dataset segment within the target physical storage partition that has been unloaded.

[0080] The following is a detailed description of each of the above management functions:

[0081] Clearing a partition includes logical clearing and physical clearing. Logical clearing deletes the data content and its derived data such as indexes of the target physical storage partition without releasing the physical storage space occupied by the target physical storage partition. Logical clearing is achieved by simply marking the data and indexes of the target physical storage partition as invalid by the interface, without calling the operating system to reclaim the storage space already occupied by the target physical storage partition. Physical clearing deletes the data content and its derived data such as indexes of the target physical storage partition, and calls the operating system to release the other physical storage space occupied by the target physical storage partition, except for the necessary physical storage space (such as a database block or page).

[0082] When deleting a partition, after physically clearing the partition, the interface program, based on the parameters provided by the application system when calling the partition deletion operation, determines whether to merge the segmentation rules of the target dataset segment with the relevant segmentation rules in the overall dataset by adjusting the segmentation rules of the overall dataset. For example, when deleting the January 2024 partition, it can be specified whether to modify the existing segmentation rules of the February 2024 partition in the segmentation rule set of the overall dataset to accommodate data records belonging to January 2024; alternatively, the segmentation rules of the determined January 2024 dataset segment can be directly deleted from the segmentation rule set of the overall dataset.

[0083] The partition migration interface creates a new physical storage partition in the target storage area, updates the mapping relationship between the target dataset segment and the corresponding new physical storage partition, copies the data content of the target physical storage partition to be migrated to the new physical storage partition, and deletes the target physical storage partition to be migrated. The storage areas before and after migration can be different areas of the same physical storage or belong to different physical storage. When migrating a partition, the partition index can be rebuilt synchronously or asynchronously.

[0084] The clone partition function retains the data content of the target physical storage partition while creating a new physical storage partition in the target storage area. The data content of the cloned physical storage partition is copied from the target physical storage partition. You can choose whether to create a corresponding index for the new physical storage partition according to the index type and index columns on the overall dataset.

[0085] The partitioning process involves creating multiple sub-segmentation rules based on the target segmentation rules of the target physical storage partition. These sub-segmentation rules define multiple target dataset segments that cannot overlap, and the union of these sub-segmentation rules is equivalent to the target dataset segment of the split target physical storage partition. For each sub-segmentation rule, an unloaded target physical storage sub-partition is created. Data from the split target physical storage partition is migrated to the corresponding target physical storage sub-partition according to the sub-segmentation rules. Optionally, partition indexes can be created for the migrated target dataset segments based on the index types and columns present in the overall dataset. For massive datasets partitioned by administrative regions, this partitioning operation can handle scenarios involving newly established administrative regions. This interface maintains that the split partitions belong to the overall dataset by updating the segmentation rules of the overall dataset.

[0086] Merging partitions, the opposite of splitting partitions, involves creating a merge rule for multiple target physical storage partitions that are already unloaded. This merge rule uses the target segmentation rules of these partitions to form a merge rule, which defines the dataset as the union of the datasets from the multiple target physical storage partitions. A new target physical storage partition is then created, unloaded, and data is migrated from each of the target partitions to it. Optionally, partition indexes can be created for the merged dataset segments based on the index types and columns present in the overall dataset. This interface updates the segmentation rules of the overall dataset to ensure that the merged target dataset segments belong to the overall dataset. For massive datasets partitioned by administrative regions, the merge partitioning operation can handle scenarios involving administrative region merging.

[0087] Maintaining partition indexes: For unloaded target physical storage partitions, partition indexes can be rebuilt based on the data content of the target physical storage partition for the purpose of optimizing the index, referring to the index types and index column specifications on the overall dataset; alternatively, partition indexes can be asynchronously created for new partitions formed by partition management and maintenance operations, referring to the indexes on the overall dataset.

[0088] In an optional embodiment of the present invention, the partition mounting method carries data boundary information, and according to the partition mounting method called by the application system, the target dataset segment in the target physical storage partition that has been managed is incorporated into its overall dataset for access by all application systems, specifically including the following steps:

[0089] (1) Determine the target physical storage partition for management completion corresponding to the data boundary information;

[0090] (2) Incorporate the target dataset segment in the target physical storage partition that has been managed into its overall dataset for access by all application systems.

[0091] This invention, based on existing database partitioning management technology and large table partitioning operations, develops a partition-granular data management business interface, providing a method for organizing and storing massive datasets in a refined manner based on their inherent characteristics, ensuring high availability and ease of management. This includes: dividing the massive dataset into multiple dataset segments based on data classification characteristics such as time period and spatial distribution, and the main access and utilization patterns; implementing a storage layout that matches the storage medium and reliability requirements based on the differences in access performance, security strength, and management needs caused by the differences in time period and spatial distribution among the dataset segments; and separating the overall massive data business application from the local data management load. This maintains the high performance and high concurrency capabilities of massive data while achieving high availability, ease of management, and support for hyper-transactional atomicity applications based on the inherent characteristics of the massive data.

[0092] It should be noted that, in addition to the interface methods mentioned above, this invention also provides auxiliary interfaces such as GetDatasetPartitionInfo, SetDatasetPartitionInfo, GetPartitionInfo, SetPartitionInfo, and ValidatePartitionToDataset. Among them, GetDatasetPartitionInfo and SetDatasetPartitionInfo are used to query or configure the segmentation rules of the overall logical partition dataset, the included partitions, and information about partitions in the unloaded state; GetPartitionInfo and SetPartitionInfo are used to query or configure the data segmentation rules, partition status, and related index information of a specific partition.

[0093] The method for organizing and storing massive amounts of data in this invention has the following advantages:

[0094] (1) Data management and processing operations occurring on the target physical storage partition can be performed without affecting the availability of data in other physical storage partitions on the same overall logical partition dataset (i.e., the overall dataset) to the business.

[0095] (2) Using physical storage partitions as atomic units, the consistency of data content is guaranteed. Data incomplete in physical storage partitions during the data processing period is not visible to business applications.

[0096] (3) For massive datasets, the visibility of local data to the business can be dynamically controlled;

[0097] (4) Indexes are maintained at the partition level, and index maintenance on a specific physical storage partition will not affect the validity of data indexes on other partitions;

[0098] (5) While quickly isolating the physical storage partition of the problematic data, the availability of the data content in other physical storage partitions to the business is not affected;

[0099] (6) Implement differentiated storage allocation and on-demand adjustment for massive amounts of data, such as allocating data from physical storage partitions that require high-performance access to high-performance storage.

[0100] (7) Break down the long-term, uninterrupted, continuous data management and maintenance work that affects the whole system into multiple local, short-duration, independently concurrent data management and maintenance work units;

[0101] (8) Implement data maintenance with predictable duration and narrow time windows;

[0102] (9) Provide adjustable and configurable data segmentation rules that reflect the inherent characteristics of natural resource industry data;

[0103] (10) It provides a mechanism for users to customize data organization, layout and storage according to the inherent characteristics of data objects. Users can implement differentiated management and maintenance of local data of massive scale sets at different life cycles.

[0104] (11) It improves the availability of massive datasets in non-cluster configurations and has certain advantages in terms of low cost and easy maintenance compared to cluster deployment.

[0105] Taking the land survey data entry and land statistics application scenarios of Province X as an example, in accordance with national regulations on conducting land surveys at the county-level administrative region level, Province X has constructed a provincial land survey database based on the land survey data of its 103 counties and cities. The core content of this database consists of approximately 17 million land parcels across the province, representing a large data volume. Relying on the province's existing and operational land spatial information platform, the land survey data supports land statistics, farmland protection, land approval, and other business operations. To ensure that the construction of the land survey data database in Province X is as uninterrupted as possible and that the entered data can serve the aforementioned applications as soon as possible, this invention can be used to carry out its construction:

[0106] 1. The land parcel data for the entire province is segmented according to the county-level administrative region code. At this point, the application functions can run, but there is no output content because the overall dataset of land parcels is empty.

[0107] 2. Now we need to import the land parcel data of County Y in Province X into the database. To do this, we will create a physical storage partition. The land parcel data with the administrative code of County Y will be placed in this physical storage partition.

[0108] 3. In order to ensure that land statistics operations can only be carried out when the land category map data of County Y is complete, the land category map data segment of County Y is unloaded from the overall land category map dataset of the province, so that empty or incomplete land category map data of County Y cannot be accessed by land statistics operations.

[0109] 4. Load data into the target physical storage partition of the Y County land parcel in the unloading state using standard SQL. This data loading may be controlled by multiple database transactions, the specific number of which often depends on the technical personnel calling this interface.

[0110] 5. After the data is entered into the database, the target physical storage partition of the land parcel data segment of County Y will be mounted onto the overall dataset of land parcels in the province. Land statistics operations can then calculate land area by multi-level land parcels on the complete land parcel data of County Y.

[0111] Without a partitioned unloading and mounting strategy, land statistics operations can access incomplete land parcel data for County Y during the data entry process. The resulting output is often meaningless and misleading to statisticians. Using a traditional database entry transaction to address this issue is limited by the large data volume and database log configuration. By employing a partitioned unloading and mounting strategy, the atomicity of data entry beyond a traditional database transaction can be achieved, as illustrated in this example.

[0112] Example 2:

[0113] This invention also provides an apparatus for organizing and storing massive amounts of data. This apparatus is mainly used to execute the method for organizing and storing massive amounts of data provided in Embodiment 1 of this invention. The apparatus for organizing and storing massive amounts of data provided in this invention will be described in detail below.

[0114] Figure 2 This is a schematic diagram of a device for organizing and storing massive amounts of data according to an embodiment of the present invention, as shown below. Figure 2 As shown, the data management business interface applied to partition granularity mainly includes: an unloading unit 10, a management unit 20, a verification unit 30, and a mounting unit 40, wherein:

[0115] The unloading unit is used to unload the target dataset segment in the target physical storage partition from the overall dataset according to the partition unloading method called by the application system. This allows the target dataset segment in the target physical storage partition to perform data addition, deletion, and modification operations independently of the overall dataset composed of dataset segments in other physical storage partitions. Except for the application system that calls the partition unloading method, other application systems cannot access the target dataset segment in the target physical storage partition. When performing data addition, deletion, and modification operations on the target dataset segment in the target physical storage partition, the corresponding target segmentation rule is followed. The target segmentation rule is the segmentation rule corresponding to the target dataset segment when the overall dataset is segmented.

[0116] The management unit is used to manage the target data segment in the target physical storage partition according to the target partition management function called by the application system, and obtain the managed target physical storage partition. The target partition management function includes at least one of the following: clearing partition, deleting partition, migrating partition, cloning partition, splitting partition, merging partition, and maintaining partition index.

[0117] The verification unit is used to verify the target segmentation rules corresponding to the managed target physical storage partition against the segmentation rule set of the overall dataset to verify whether there are any conflicts, and to verify the data structure of the target dataset segment in the managed target physical storage partition against the data structure of the overall dataset to verify whether the data structure is consistent.

[0118] The mounting unit, after successful verification, incorporates the target dataset segment from the target physical storage partition into its overall dataset for access by all application systems, based on the partition mounting method called by the application system.

[0119] In this embodiment of the invention, a device for organizing and storing massive amounts of data is provided, applied to a data management business interface at the partition granularity. The device includes: unloading a target dataset segment from the overall dataset in a target physical storage partition according to a partition unloading method called by an application system, so that the target dataset segment in the target physical storage partition can perform data addition, deletion, and modification operations independently of the overall dataset composed of dataset segments in other physical storage partitions. Except for the application system that called the partition unloading method, other application systems cannot access the target dataset segment in the target physical storage partition. When performing data addition, deletion, and modification operations on the target dataset segment in the target physical storage partition, the corresponding target segmentation rule is followed. The target segmentation rule is the segmentation rule corresponding to the target dataset segment when segmenting the overall dataset. According to the application... The system calls the target partition management function to manage the target dataset segment in the target physical storage partition, resulting in a managed target physical storage partition. The target partition management function includes at least one of the following: clearing a partition, deleting a partition, migrating a partition, cloning a partition, splitting a partition, merging a partition, and maintaining a partition index. The system then verifies the target segmentation rules corresponding to the managed target physical storage partition against the segmentation rule set of the overall dataset to check for conflicts. It also verifies the data structure of the target dataset segment in the managed target physical storage partition against the data structure of the overall dataset to ensure consistency. After successful verification, the system incorporates the target dataset segment in the managed target physical storage partition into its overall dataset according to the partition mounting method called by the application system, making it accessible to all application systems.As described above, the massive data organization and storage apparatus of the present invention employs a partitioning offloading method to allow the target dataset segment in the target physical storage partition to be maintained independently from the overall dataset composed of dataset segments in other physical storage partitions. In other words, data management and processing operations occurring on the target physical storage partition do not affect the availability of data in other physical storage partitions within the same overall dataset to the application system. Specifically, when data in the target physical storage partition is experiencing problems and requires maintenance, data in other physical storage partitions remains usable, meeting the high availability data requirements. Furthermore, when data maintenance is needed, only the target physical storage partition corresponding to the data needs to be maintained; there is no need for overall maintenance of the entire dataset. Only local data updates and maintenance are required, eliminating the need for overall dataset maintenance. This significantly improves data maintenance efficiency and meets the demand for easily manageable data. Furthermore, by using dataset segments within physical storage partitions as atomic units, data consistency is guaranteed. Incomplete data within physical storage partitions during data processing is invisible to business applications (i.e., application systems). This means that for massive datasets, the visibility of local data to the business can be dynamically controlled, meeting the requirements of atomic application scenarios beyond transactions. In other words, the massive data organization and storage method of this invention can meet the needs of high availability, easy management, and atomic application scenarios beyond transactions, alleviating the difficulty of traditional massive data organization and storage methods in fully meeting these requirements.

[0120] Optionally, the device is also used to: when the application system calls the data management business interface, determine the segmentation rule set for segmenting the overall dataset, and segment the overall dataset according to the segmentation rule set to obtain multiple dataset segments; create multiple physical storage partitions according to the segmentation rule set, and establish a mapping relationship between each dataset segment and the corresponding physical storage partition, so that each dataset segment is stored in the corresponding physical storage partition.

[0121] Optionally, the device is also used to: determine a set of segmentation rules for segmenting the whole dataset based on the intrinsic characteristics of the manually selected whole dataset, wherein the intrinsic characteristics include: temporal features and spatial features.

[0122] Optionally, the device is also used to: create physical storage partitions corresponding to each segmentation rule according to the user's customization, and then establish a mapping relationship between each dataset segment and the corresponding physical storage partition according to each segmentation rule.

[0123] Optionally, the partition unloading method carries data isolation boundary information, and the unloading unit is also used to: determine the target physical storage partition corresponding to the data isolation boundary information; and unload the target data set segment in the target physical storage partition from the overall dataset.

[0124] Optionally, clearing a partition includes: logical clearing and physical clearing. Logical clearing involves deleting the target dataset segment and its corresponding index partition in the target physical storage partition, but without releasing the physical storage space occupied by the target physical storage partition. Physical clearing involves deleting the target dataset segment and its corresponding index partition in the target physical storage partition, releasing the other physical storage space occupied by the target physical storage partition except for the necessary physical storage space. Deleting a partition involves merging the segmentation rules of the deleted target physical storage partition into the data segmentation rule set of the overall dataset while deleting the target physical storage partition. Migrating a partition involves copying the target dataset segment in the target physical storage partition to be migrated to a first new physical storage partition, deleting the migrated target physical storage partition, updating the mapping relationship between the target dataset segment and the corresponding first new physical storage partition, and rebuilding the partition index corresponding to the target dataset segment. Cloning a partition involves copying the target dataset segment in the target physical storage partition to be cloned to a second new physical storage partition, and according to the index corresponding to the cloned target dataset segment and its index type and parameters, modifying the second new physical storage partition. The process involves: 1) Creating corresponding partition indexes for target dataset segments within the physical storage partition; 2) Splitting partitions by generating multiple sub-segmentation rules based on the segmentation rules corresponding to the split dataset segments, allocating physical storage partitions for each sub-segmentation rule to obtain multiple target physical storage sub-partitions, migrating dataset segments from the split target physical storage partitions to their corresponding target physical storage sub-partitions according to the sub-segmentation rules, and creating corresponding partition indexes for the migrated target dataset sub-partitions based on the indexes, index types, and parameters of the split target dataset segments; 3) Merging partitions by generating merge rules based on the target segmentation rules corresponding to the merged target dataset segments, allocating corresponding physical storage partitions for the merge rules to obtain merged target physical storage partitions, migrating the merged target dataset segments to the merged target physical storage partitions to form merged dataset segments, releasing the physical storage of the merged target physical storage partitions, and creating corresponding partition indexes on the merged dataset segments based on the overall dataset index situation; and 4) Maintaining partition indexes by rebuilding and maintaining the partition indexes on the target dataset segments in the unloaded target physical storage partitions.

[0125] Optionally, the partition mounting method carries data boundary information, and the mounting unit is also used to: determine the target physical storage partition that has been managed and completed corresponding to the data boundary information; and incorporate the target dataset segment in the target physical storage partition into its overall dataset for access by all application systems.

[0126] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0127] like Figure 3 As shown in the embodiment of this application, an electronic device 600 includes a processor 601, a memory 602, and a bus. The memory 602 stores machine-readable instructions executable by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 via the bus, and the processor 601 executes the machine-readable instructions to perform the steps of the method for organizing and storing massive amounts of data as described above.

[0128] Specifically, the memory 602 and processor 601 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 601 runs the computer program stored in the memory 602, it can execute the above-mentioned method for organizing and storing massive amounts of data.

[0129] The processor 601 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 601 or by instructions in software form. The processor 601 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 602, and processor 601 reads the information from memory 602 and, in conjunction with its hardware, completes the steps of the above method.

[0130] Corresponding to the above-described method for organizing and storing massive amounts of data, this application also provides a computer-readable storage medium storing machine-executable instructions. When these machine-executable instructions are invoked and executed by a processor, they cause the processor to perform the steps of the above-described method for organizing and storing massive amounts of data.

[0131] The apparatus for organizing and storing massive amounts of data provided in this application embodiment can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the apparatus provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the apparatus embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, apparatuses, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0132] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0133] For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0134] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0136] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the massive data organization and storage method described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0138] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for organizing and storing massive amounts of data, characterized in that, The method, applied to a data management business interface at the partition granularity, includes: According to the partition unloading method called by the application system, the target dataset segment in the target physical storage partition is unloaded from the overall dataset, so that the target dataset segment in the target physical storage partition can perform data addition, deletion, and modification operations independently of the overall dataset composed of dataset segments in other physical storage partitions. Except for the application system that calls the partition unloading method, other application systems cannot access the target dataset segment in the target physical storage partition. When performing data addition, deletion, and modification operations on the target dataset segment in the target physical storage partition, the corresponding target segmentation rule is followed. The target segmentation rule is the segmentation rule corresponding to the target dataset segment when the overall dataset is segmented. When the partition unloading method is called, the data isolation boundary information of a target dataset segment is independently managed and maintained in the overall dataset, thereby unloading the target dataset segment in the target physical storage partition from the overall dataset. The target data set segment in the target physical storage partition is managed according to the target partition management function called by the application system to obtain the managed target physical storage partition. The target partition management function includes at least one of the following: clearing partition, deleting partition, migrating partition, cloning partition, splitting partition, merging partition, and maintaining partition index. The target segmentation rules corresponding to the target physical storage partition that has been managed are verified against the segmentation rule set of the overall dataset to verify whether there is a conflict. The data structure of the target dataset segment in the target physical storage partition that has been managed is verified against the data structure of the overall dataset to verify whether the data structure is consistent. After successful verification, the target dataset segment in the target physical storage partition that has been managed is incorporated into its overall dataset according to the partition mounting method called by the application system, making it accessible to all application systems.

2. The method according to claim 1, characterized in that, Before unloading the target dataset segment from the overall dataset according to the partition unloading method called by the application system, the method further includes: When the application system calls the data management business interface, it determines a set of segmentation rules for segmenting the overall dataset, and segments the overall dataset according to the set of segmentation rules to obtain multiple dataset segments; Multiple physical storage partitions are created according to the segmentation rule set, and a mapping relationship is established between each dataset segment and the corresponding physical storage partition, so that each dataset segment is stored in the corresponding physical storage partition.

3. The method according to claim 2, characterized in that, Determine the set of segmentation rules for segmenting the overall dataset, including: A set of segmentation rules for segmenting the overall dataset is determined based on the inherent characteristics of the manually selected overall dataset, wherein the inherent characteristics include: temporal features and spatial features.

4. The method according to claim 2, characterized in that, Multiple physical storage partitions are created according to the segmentation rule set, and a mapping relationship is established between each dataset segment and its corresponding physical storage partition, including: Based on the user's customization, a physical storage partition is created corresponding to each segmentation rule, and then a mapping relationship between each dataset segment and its corresponding physical storage partition is established according to each segmentation rule.

5. The method according to claim 1, characterized in that, The partition unloading method carries data isolation boundary information and unloads the target dataset segment from the overall dataset from the target physical storage partition according to the partition unloading method called by the application system, including: Determine the target physical storage partition corresponding to the data isolation boundary information; Unload the target dataset segment from the overall dataset in the target physical storage partition.

6. The method according to claim 1, characterized in that: The clearing of the partition includes: logical clearing and physical clearing. Logical clearing involves deleting the target dataset segment and the corresponding index partition in the target physical storage partition, but without releasing the physical storage space occupied by the target physical storage partition. Physical clearing involves deleting the target dataset segment and the corresponding index partition in the target physical storage partition, and releasing the other physical storage space occupied by the target physical storage partition, except for retaining the necessary physical storage space. While deleting the target physical storage partition, the segmentation rules of the deleted target physical storage partition are merged into the data segmentation rule set of the overall dataset. The migration partition involves copying the target dataset segment from the target physical storage partition to the first new physical storage partition, deleting the target physical storage partition, updating the mapping relationship between the target dataset segment and the corresponding first new physical storage partition, and rebuilding the partition index corresponding to the target dataset segment. The cloning partition involves copying the target dataset segment from the target physical storage partition to the second new physical storage partition, and creating a corresponding partition index for the target dataset segment in the second new physical storage partition according to the index corresponding to the cloned target dataset segment and its index type and parameters. The partitioning process involves generating multiple sub-segmentation rules based on the segmentation rules corresponding to the split dataset segments, allocating a physical storage partition for each sub-segmentation rule to obtain multiple target physical storage sub-partitions, migrating dataset segments from the split target physical storage partitions to the corresponding target physical storage sub-partitions according to the sub-segmentation rules, and creating corresponding partition indexes for the multiple target dataset sub-segments formed by the migration based on the indexes corresponding to the split target dataset segments and their index types and parameters. The merging partition is generated by generating merging rules based on the target segmentation rules corresponding to the multiple target dataset segments to be merged, allocating corresponding physical storage partitions to the merging rules, obtaining the merged target physical storage partition, migrating the multiple target dataset segments to be merged to the merged target physical storage partition to form a merged dataset segment, releasing the physical storage of the merged target physical storage partition, and creating a corresponding partition index on the merged dataset segment according to the index of the overall dataset. The maintenance of the partition index involves rebuilding and maintaining the partition index on the target dataset segment in the target physical storage partition that has been unloaded.

7. The method according to claim 1, characterized in that, The partition mounting method carries data boundary information. Based on the partition mounting method invoked by the application system, the target dataset segment in the managed target physical storage partition is incorporated into its overall dataset for access by all application systems, including: Determine the target physical storage partition for which management has been completed, corresponding to the data boundary information; The target dataset segment in the target physical storage partition that has been managed is incorporated into its overall dataset for access by all application systems.

8. A device for organizing and storing massive amounts of data, characterized in that, A data management business interface applied to partition granularity, the device comprising: The unloading unit is used to unload the target dataset segment in the target physical storage partition from the overall dataset according to the partition unloading method called by the application system. This allows the target dataset segment in the target physical storage partition to perform data addition, deletion, and modification operations independently of the overall dataset composed of dataset segments in other physical storage partitions. Except for the application system that calls the partition unloading method, other application systems cannot access the target dataset segment in the target physical storage partition. When performing data addition, deletion, and modification operations on the target dataset segment in the target physical storage partition, the corresponding target segmentation rule is followed. The target segmentation rule is the segmentation rule corresponding to the target dataset segment when the overall dataset is segmented. When the partition unloading method is called, data isolation boundary information for a target dataset segment is established and maintained independently in the overall dataset, thereby unloading the target dataset segment in the target physical storage partition from the overall dataset. The management unit is used to manage the target data segment in the target physical storage partition according to the target partition management function called by the application system, and obtain the managed target physical storage partition. The target partition management function includes at least one of the following: clearing partition, deleting partition, migrating partition, cloning partition, splitting partition, merging partition, and maintaining partition index. The verification unit is used to verify the target segmentation rules corresponding to the target physical storage partition that has been managed by comparing them with the segmentation rule set of the overall dataset to verify whether there is a conflict, and to verify the data structure of the target dataset segment in the target physical storage partition that has been managed by comparing it with the data structure of the overall dataset to verify whether the data structure is consistent. The mounting unit, after successful verification, incorporates the target dataset segment in the target physical storage partition into its overall dataset for access by all application systems, according to the partition mounting method called by the application system.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images