A distributed file management query method, system, terminal and storage medium

Abstract

The application discloses a kind of distributed file management query method, system, terminal and storage medium, the method includes: based on the source data on all target computing power hubs, using data mixing strategy, generate local GID table on each target computing power hub;According to the local GID table on each target computing power hub, generate global GID table;According to target interface, real-time detection data file change on all target computing power hubs, and according to the data file change, real-time update corresponding local GID table;Whenever target query operation is acquired, according to the current local GID table update global GID table, according to the updated global GID table, obtain index information, according to the index information, complete current query.The application can be realized in computing power network fast data synchronization by the local GID table and global GID table built.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and in particular to a distributed file management query method, system, terminal and storage medium. Background Art

[0002] The Computing Power Network (CPN) is a new type of network that brings together powerful computing, storage, and network resources to provide computing power support for large-scale intelligent applications with massive amounts of data, such as training large language models or autonomous driving models. It is particularly critical for ensuring data consistency, availability, disaster recovery, and efficient access when facing ultra-large-scale data processing and storage requirements.

[0003] However, the current computing power network is relatively inconvenient in the data synchronization process, resulting in high complexity in data management and inconvenience for users.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a distributed file management query method, system, terminal and computer-readable storage medium, aiming to solve the problem in the prior art that the current computing power network is relatively inconvenient in the data synchronization process, resulting in high complexity of data management and inconvenience for users.

[0006] To achieve the above object, the present invention provides a distributed file management query method, which includes the following steps:

[0007] Based on the source data on all target computing hubs, a data shuffling strategy is adopted to generate a local GID table on each target computing hub;

[0008] Generate a global GID table based on the local GID table on each target computing hub;

[0009] Detect data file changes on all target computing hubs in real time according to the target interface, and update the corresponding local GID table in real time according to the data file changes;

[0010] Whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information.

[0011] Optionally, the data shuffling strategy includes a local stage and a global stage;

[0012] The local stage performs random processing on the local full amount of data stored on multiple computing power hubs to obtain local random sample division data blocks;

[0013] In the global stage, a data block divided by a local random sample is selected on each computing power hub, and all the selected data blocks divided by the local random sample are merged to obtain a data block divided by a cross-geographic random sample.

[0014] Optionally, based on the source data on all target computing power hubs, a data shuffling strategy is adopted to generate a local GID table on each target computing power hub, specifically including:

[0015] Based on the source data on all target computing hubs, a data shuffling strategy is adopted to generate local random sample partitioning data blocks and cross-geographic random sample partitioning data blocks in each target computing hub;

[0016] Generate a local source data GID based on the source data, divide the data blocks according to the local random sample in each target computing power hub, generate a local local random sample divided data GID in each target computing power hub, divide the data blocks according to the cross-geographic random sample in each target computing power hub, and generate a local cross-geographic random sample divided data GID in each computing power hub;

[0017] According to the local source data GID, local random sample partition data GID and local cross-geographic random sample partition data GID of each computing power hub, and after adding the target tag field, the local GID table of each computing power hub is generated.

[0018] Optionally, generating a global GID table based on the local GID table on each target computing power hub specifically includes:

[0019] Generate global source data GID based on all local source data GIDs in the local GID table on all target computing hubs, generate global local random sample partition data GID based on the local local random sample partition data GIDs on all target computing hubs, and generate global cross-geographic random sample partition data GID based on the local cross-geographic random sample partition data GIDs on all target computing hubs;

[0020] The global GID table is generated according to the global source data GID, the global local random sample partition data GID and the global cross-geographic random sample partition data GID.

[0021] Optionally, the real-time detection of data file changes on all target computing hubs according to the target interface, and the real-time updating of the corresponding local GID table according to the data file changes, specifically includes:

[0022] Detect data file changes on all target computing power hubs in real time according to the target interface;

[0023] When a data file change is detected, the state value of the target flag field is adjusted according to the data file change, and the corresponding local GID table is updated according to the state value of the target flag field.

[0024] Optionally, whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information, specifically including:

[0025] Whenever a target query operation is obtained, the global GID table is updated accordingly based on the local source data GID, local random sample partition data GID, and local cross-geographic random sample partition data GID of each computing power hub in all current local GID tables;

[0026] According to the target query operation, obtaining index information in the updated global GID table;

[0027] Generate a query result corresponding to the target query operation based on the index information and output it.

[0028] Optionally, the distributed file management query method further includes:

[0029] Get the preset heartbeat time and detect in real time whether the current time reaches the heartbeat time based on the timer;

[0030] When the time is reached, the timer is set to re-time, and the global GID table is updated according to the current local GID table.

[0031] In addition, to achieve the above-mentioned purpose, the present invention further provides a distributed file management and query system, wherein the distributed file management and query system includes:

[0032] The local GID table generation module is used to generate a local GID table on each target computing power hub based on the source data on all target computing power hubs and adopt a data shuffling strategy;

[0033] The global GID table generation module is used to generate a global GID table based on the local GID table on each target computing power hub;

[0034] An update module is used to detect data file changes on all target computing hubs in real time according to the target interface, and update the corresponding local GID table in real time according to the data file changes;

[0035] The query module is used to update the global GID table according to the current local GID table whenever a target query operation is obtained, obtain index information according to the updated global GID table, and complete the current query according to the index information.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a distributed file management query program stored on the memory and runnable on the processor, and when the distributed file management query program is executed by the processor, the steps of the distributed file management query method as described above are implemented.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a distributed file management query program, and when the distributed file management query program is executed by the processor, the steps of the distributed file management query method as described above are implemented.

[0038] In the present invention, based on the source data on all target computing power hubs, a data shuffling strategy is adopted to generate a local GID table on each target computing power hub; based on the local GID table on each target computing power hub, a global GID table is generated; data file changes on all target computing power hubs are detected in real time according to the target interface, and the corresponding local GID table is updated in real time according to the data file changes; whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information. The present invention converts the large data files scattered in different computing power nodes in the computing power network environment into global random samples that conform to the data model characteristics of random sample partitioning through the designed data shuffling strategy, thereby deploying the local GID table on the corresponding target computing power hub and the global GID table on the middleware, so that the entire computing power network can update the local GID table in time when an update occurs, realize fast data synchronization, and when a query operation is obtained, the global GID table is updated according to the updated local GID table, thereby making the query result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of a preferred embodiment of the distributed file management query method of the present invention;

[0040] Figure 2 It is a flow chart of the data shuffling strategy in the distributed file management query method of the present invention;

[0041] Figure 3 This is a schematic diagram of the logical hierarchy of data files in the distributed file management query method of the present invention;

[0042] Figure 4 This is a schematic diagram of the global GID table synchronization mechanism corresponding to the middleware in the distributed file management query method of the present invention;

[0043] Figure 5 This is a schematic diagram of data synchronization in the distributed file management query method of the present invention;

[0044] Figure 6 It is a structural diagram of a preferred embodiment of the distributed file management and query system of the present invention;

[0045] Figure 7 FIG. 4 is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0047] The Computing Power Network (CPN) is a new type of network that brings together powerful computing, storage, and network resources to provide computing power support for large-scale intelligent applications that require massive amounts of data, such as training large language models or autonomous driving models. It is particularly critical for ensuring data consistency, availability, disaster recovery, and efficient access when facing ultra-large-scale data processing and storage requirements. However, current computing power networks are relatively inconvenient during data synchronization, resulting in complex data management and inconvenience for users.

[0048] In response to one or more of the above problems, the present invention adopts a data shuffling strategy based on the source data on all target computing power hubs to generate a local GID table on each target computing power hub; generates a global GID table based on the local GID table on each target computing power hub; detects data file changes on all target computing power hubs in real time according to the target interface, and updates the corresponding local GID table in real time according to the data file changes; whenever a target query operation is obtained, updates the global GID table according to the current local GID table, obtains index information according to the updated global GID table, and completes the current query according to the index information.

[0049] The distributed file management query method described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the distributed file management query method includes the following steps:

[0050] Step S10: Based on the source data on all target computing power hubs, a data shuffling strategy is adopted to generate a local GID table on each target computing power hub.

[0051] It should be noted that in the present invention, the data shuffling strategy is a shuffling strategy designed by the present invention, specifically an RSP (Random Sample Partition) data shuffling strategy in a computing power network environment. In the computing power network environment, the big data files scattered in different computing power nodes are converted into global random samples that conform to the characteristics of the RSP data model, that is, the Geo-RSP (Geo-Distributed Random Sample Partition) data model.

[0052] Furthermore, the data shuffling strategy includes a local stage and a global stage;

[0053] The local stage performs random processing on the local full amount of data stored on multiple computing power hubs to obtain local random sample division data blocks;

[0054] In the global stage, a data block divided by a local random sample is selected on each computing power hub, and all the selected data blocks divided by the local random sample are merged to obtain a data block divided by a cross-geographic random sample.

[0055] Specifically, the data shuffling strategy includes a local stage and a global stage; Figure 2 As shown, local RSP is performed during the local phase. For a large set of data distributed across different computing hubs, the feature distribution of local data may differ from that of the full data. Within each computing hub, a distributed file system such as HDFS (Hadoop Distributed File System), GFS (Google Distributed File System), or Gfarm (Gfarm File System) is typically used to manage these local data blocks. The feature distributions of these local data blocks may vary. Local RSP refers to randomizing the local full data stored on each computing hub or data center. This step can be accomplished using the toRSP() operator. Local RSP makes the feature distribution of data uniform across all computing hubs. Each data block that has undergone local RSP, i.e., the data block partitioned by the local random sample, can be considered a random sample of the current computing hub and has similar feature distributions.

[0056] The global phase is known as RSP shuffling. After local RSP is complete, the global phase begins. The core operation of this phase is to randomly select a randomized local RSP data block from each computing hub and merge these blocks into a new data block, called a Geo-RSP data block. This is a cross-geographic random sample partitioning of the data block. The characteristic distribution of this Geo-RSP data block is similar to that of the full data set. This means that by analyzing the Geo-RSP data block, we can approximate the statistical characteristics of the full data set across the entire computing network. In this way, each computing hub can generate and acquire a large number of Geo-RSP blocks. This complete process is called RSP shuffling.

[0057] The primary advantage of the Geo-RSP data model is that it introduces a new management strategy for geographically distributed file systems. This strategy avoids comprehensive data synchronization across all computing hubs, requiring only limited data backup within the computing hubs for fault tolerance. Furthermore, only minimal shuffling and synchronization of Geo-RSP data blocks between different computing hubs ensures that each computing hub has a global view without having to bear the significant WAN load associated with global data synchronization. This not only ensures data security and fault tolerance, but also significantly reduces the WAN resource requirements of the data synchronization process. Furthermore, both local RSPization and Geo-RSP shuffling can be performed offline, further reducing the negative impact of WAN constraints on computation, thereby enabling more efficient and cost-effective data analysis in distributed data processing.

[0058] Therefore, it can be seen that the global RSP data block (i.e., Geo-RSP data block) in the Geo-RSP data model can be used to approximately estimate the statistical results of the full amount of data. That is to say, during actual data analysis, there is no need to use data backup and synchronization mechanisms on the computing power hub to ensure strong consistency of data in different locations. Only a small number of Geo-RSP data blocks need to be synchronized to ensure the normal progress of the calculation and analysis. Even if problems such as computing interruption or hardware failure occur, no serious consequences will occur whether the computing task is reallocated or the current calculation is continued. At the same time, protecting the integrity of the source data is also very critical, but for the present invention, sudden abnormal situations can be dealt with by saving copies of the data on a few computing power hubs or local computing nodes.

[0059] In response to the data shuffling strategy, the present invention also designs a new file management system for computing power networks - Geo-Distributed Indexed Datasets File System, referred to as GIDFS file system, that is, a local GID table is generated on each target computing power hub, and a global GID table is generated on the middleware to synchronize the information of all target computing power hubs. GIDFS manages physical data files through the mechanism of indexing metadata information and geographic location information. This method not only improves data retrieval efficiency, but also simplifies data synchronization and backup processes, thereby effectively reducing the complexity of data management. Through the GIDFS file system, the computing system of the computing power network can quickly and accurately locate physical data, and can more efficiently process cross-geographic distributed computing tasks, while ensuring data security and high availability.

[0060] The GIDFS file system does not directly store physical data files; it only stores and manages index information for data files. For example, it can store global file names, local file names, data storage paths, RSP block counts, and computing hub numbers. This information can be directly read from the metadata source data in traditional file systems (such as HDFS) or automatically scanned using simple program code. Combining this index information creates a GID (Geo-Distributed Indexed Dataset) record, with each GID representing a record in a data table. In a computing network environment, each physical data file stored on a computing hub generates a corresponding GID record. Using these GID records, the computing network's computing systems can quickly and accurately locate specific data files and determine the execution parameters for computing tasks based on the index information. Accurately obtaining index information is crucial for the reliable operation of the GIDFS file system. The GIDFS file system can directly obtain the required index information from the metadata table (i.e., metadata table) of existing distributed file systems (such as HDFS). Alternatively, it can construct the initial GID table through disk scanning and continuously maintain and iterate the GID table.

[0061] Furthermore, the index information of the GIDFS file system can be abstracted into a general API interface, using SQL statements to complete information indexing operations, thereby promoting the execution of computing tasks. For example, by executing an SQL query statement such as "Select filepath, GeoRSPBlockNums from Global GeoRSPTable where GlobalFileName=FileName", key information such as the specific storage path of a specific dataset and the number of global Geo-RSP blocks can be obtained.

[0062] In a computing network environment, due to differences in data types and processing stages, the generated GID information will also be different. For example, there are significant differences in the recording requirements of GID information between local source data distributed and stored on a computing hub and local RSP data processed by a data shuffling strategy. For the latter, in addition to recording the global file name and the file name of the local source data, it is also necessary to record the name of the RSP file and the number of RSP data blocks. Therefore, if Figure 3 As shown, to effectively manage data index information with different core characteristics, the GIDFS file system divides cross-geographic GID data indexes into four different levels based on the Geo-RSP data shuffling process. Level I, global source data, represents the logical whole of the data in the computing network and is a logical data representation, consisting of three global components corresponding to levels II, III, and IV. Level II: local source data refers to the original data files stored on each computing hub. Level III: local RSP data represents data files that have undergone preliminary randomization within the computing hub. Level IV: local Geo-RSP data refers to Geo-RSP data files that have undergone RSP shuffling between computing hubs. Furthermore, local source data refers to the original files; local RSP data corresponds to the local stage of the data shuffling strategy; and local Geo-RSP data corresponds to the global stage of the data shuffling strategy, namely, the Geo-RSP data blocks obtained after local RSP data is transmitted and merged across data centers.

[0063] Among them, Levels II, II, and IV are physical data, typically stored on computing hubs. While global source data (Level I) logically represents the entire data in the computing network, it is essentially achieved by synchronizing and integrating local GIDs (Levels II, II, and IV) and is the collection of all local GID information. That is, based on the data logic within the computing network environment, six types of GID information tables, totaling three groups, are generated, each corresponding to the other and requiring synchronous updates. This hierarchical and segmented data index management method not only improves the efficiency of data retrieval and management, but also provides a clear logical structure for data processing, ensuring data consistency and integrity during cross-geographic distributed computing.

[0064] Usually, the GID information of global data is maintained in the form of a GID table on a middleware (the central control node of the computing power network environment), covering information of three types of logical data: global source data, global RSP data, and global Geo-RSP data; however, it is unrealistic to use the global GID table on the middleware to directly monitor the GID information in the entire computing power network, which will put a lot of pressure on the network load of the middleware and is easily affected by the wide area network constraints. Therefore, the present invention designs and deploys a local GID table responsible for synchronizing local data information on each computing power hub. This table contains most of the same information as the global GID table, but adds three important target tag fields: "is_modified", "is_deleted" and "is_sync", the purpose of which is to record the status changes of the file represented by the current GID.

[0065] Among them, is_modified: This status flag indicates whether the current data file has been modified since the last synchronization. If the data file has been modified, this flag will be set to true, indicating that the change needs to be synchronized to the global GID table. is_deleted: This status flag indicates whether the current data file has been deleted. If so, this flag will be set to true, and the corresponding global GID table will be updated to reflect this change during the next synchronization. is_sync: This status flag indicates whether the current data file has been synchronized. If the file's GID information has been successfully synchronized to the global GID table and no additions, modifications, or deletions have been performed since synchronization, this flag will be set to true, indicating that the current GID record is up to date. By introducing these three flag fields, the computing system of the computing power network can effectively track and manage the status changes of data files on each computing power hub. This synchronization mechanism between the local GID table and the global GID table ensures the consistency, freshness, and reliability of data in the computing power network.

[0066] Furthermore, based on the source data on all target computing hubs, a data shuffling strategy is adopted to generate a local GID table on each target computing hub, specifically including:

[0067] Based on the source data on all target computing hubs, a data shuffling strategy is adopted to generate local random sample partitioning data blocks and cross-geographic random sample partitioning data blocks in each target computing hub;

[0068] Generate a local source data GID based on the source data, divide the data blocks according to the local random sample in each target computing power hub, generate a local local random sample divided data GID in each target computing power hub, divide the data blocks according to the cross-geographic random sample in each target computing power hub, and generate a local cross-geographic random sample divided data GID in each computing power hub;

[0069] According to the local source data GID, local random sample partition data GID and local cross-geographic random sample partition data GID of each computing power hub, and after adding the target tag field, the local GID table of each computing power hub is generated.

[0070] Specifically, in the present invention, three local GID information tables are maintained on all computing power hubs, which are used to record and update the index information of the source data, RSP data and Geo-RSP data on the computing power hub, namely the local source data GID, the local local random sample partitioning data GID and the local cross-geographic random sample partitioning data GID, and a target tag field is added to each of them to generate the local GID table of each computing power hub.

[0071] Step S20: Generate a global GID table based on the local GID table on each target computing power hub.

[0072] Specifically, if Figure 4 As shown, in the present invention, the GIDFS file system maintains three global GID master tables on the middleware, namely the global GID tables. Among them, the global source data GID is responsible for synchronizing the local source data GID on all computing power hubs, the global RSP data GID is responsible for synchronizing the GID information of the local RSP data on all computing power hubs, and the global Geo-RSP data GID synchronizes the GID information of the local Geo-RSP data on all computing power hubs. The synchronization between the three groups of GIDs does not interfere with each other, but the information content verifies each other. The main function of the three global GID tables is to provide indexing services for computing tasks to determine the computing power hub and resource allocation for executing tasks. The main function of the three local GID tables is to maintain and update the GID information of local data and update the global GID table to ensure the accuracy of the global GID information table. Since the GID table is an index information table, only very little communication overhead is required to ensure almost real-time information synchronization.

[0073] Furthermore, generating a global GID table based on the local GID table on each target computing power hub specifically includes:

[0074] Generate global source data GID based on all local source data GIDs in the local GID table on all target computing hubs, generate global local random sample partition data GID based on the local local random sample partition data GIDs on all target computing hubs, and generate global cross-geographic random sample partition data GID based on the local cross-geographic random sample partition data GIDs on all target computing hubs;

[0075] The global GID table is generated according to the global source data GID, the global local random sample partition data GID and the global cross-geographic random sample partition data GID.

[0076] Specifically, all local source data GIDs in the local GID tables on all target computing power hubs are summarized to generate a global source data GID, the local local random sample partition data GIDs on all target computing power hubs are summarized to generate a global local random sample partition data GID, and the local cross-geographic random sample partition data GIDs on all target computing power hubs are summarized to generate a global cross-geographic random sample partition data GID. Then, the global source data GID, the global local random sample partition data GID and the global cross-geographic random sample partition data GID are summarized to generate the global GID table.

[0077] Step S30: Detect data file changes on all target computing hubs in real time according to the target interface, and update the corresponding local GID table in real time according to the data file changes.

[0078] Specifically, the synchronization and construction of the global GID table is selective, rather than simply retrieving the contents of all local GID tables and merging them. Therefore, each synchronization only updates the GID information for data whose status has changed. This mechanism relies primarily on three additional information tags in the local table compared to the global table, namely the status value of the target tag field. When the status of physical data stored on the computing power hub changes, the change in the information tags can be used to determine which data files' local GID information needs to be updated in the global GID table.

[0079] Furthermore, the real-time detection of data file changes on all target computing hubs according to the target interface and the real-time updating of the corresponding local GID table according to the data file changes specifically include:

[0080] Detect data file changes on all target computing power hubs in real time according to the target interface;

[0081] When a data file change is detected, the state value of the target flag field is adjusted according to the data file change, and the corresponding local GID table is updated according to the state value of the target flag field.

[0082] Specifically, in one embodiment of the present invention, before a GID synchronization update, the three information flags for local data A in the local GID table are false, false, and true, respectively. This means that the local GID table has already been synchronized with the global GID table and no operations have been performed to change the data index information. Clearly, this local GID table does not need to be synchronized with the global GID table. For dataset B, its status flags are true, false, and false, indicating that it is a newly created dataset and has not yet been synchronized. Similarly, for dataset C, its status flags are true, true, and true, indicating that it has been previously synchronized with the global GID table, but an operation to delete this data has been performed on the computing hub. Therefore, the corresponding GID information in the local GID table needs to be synchronized and deleted from the global GID table. Therefore, when the next global GID update occurs, only three GID information in the local GID table need to be synchronized with the global table, rather than re-acquiring and merging all local GID tables. This approach makes the pressure on the wide area network bandwidth caused by synchronization updates almost negligible. At the same time, in actual applications, the computing resources of the computing network are not exclusively used by a single computing task; each computing hub typically also performs local computing tasks. Therefore, the state changes of local data files are not controllable from a global perspective. The differentiated design of the local GID table and the global GID table avoids these situations and improves the flexibility of the GIDFS file system.

[0083] Furthermore, in the present invention, when the local GID table is updated, Figure 5 As shown, the corresponding modules include two modules, namely the monitoring notification module (inotify module) and the update module (inotify module). The inotify module is logically located on the upper layer of the distributed file management system (usually HDFS) of the computing power hub, and can monitor event information at a specific location. It provides an event tracking API - InotifyEvent based on the DF SInotifyEventInputStream interface. The Stream can monitor various types of events, including Create, Rename, Unlink, Append, and other types. For example, a new folder is created on the computing power hub to store a data set, which will be monitored by InotifyEvent and trigger a Create event; operations such as renaming, moving folders, and adding data will trigger a Rename event; deleting files will trigger an Unlink event. While monitoring various events, the Stream will also provide corresponding processing solutions and provide the file storage path where the event occurred.

[0084] That is, when the computing system is not performing computing tasks, the imotify module monitors the changes in data files on all target computing power hubs in real time through the ImotifyEventAPI, that is, the target interface. When the local data file changes, the ImotifyEventAPI will generate a corresponding event based on the state change, and the DFSInotifyEventInputStream will return the path information (URL) where the event occurred. The local GID table will re-index the data index information under the path. At the same time, the status values ​​of the three tag fields, is modified, is delete, and is sync, are determined according to the specific type of the event, and then new local GID information is generated, the local GID table is updated accordingly, and the global GID table is updated if set.

[0085] Step S40: Whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information.

[0086] It should be noted that the Update module primarily updates the local GID table and the global GID table. The local GID table update is a post-event triggering solution. Event triggering is real-time, so local GID table updates are also real-time. Synchronous updates of the global and local GID tables are triggered in two main ways: the first is by setting a fixed heartbeat interval, which means regular updates are performed at intervals; the second is when a computing task requires computation or management operations and needs to retrieve index information from the global GID table. The former ensures the real-time nature of GID information, while the latter ensures its accuracy.

[0087] Furthermore, whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information, specifically including:

[0088] Whenever a target query operation is obtained, the global GID table is updated accordingly based on the local source data GID, local random sample partition data GID, and local cross-geographic random sample partition data GID of each computing power hub in all current local GID tables;

[0089] According to the target query operation, obtaining index information in the updated global GID table;

[0090] Generate a query result corresponding to the target query operation based on the index information and output it.

[0091] Specifically, when the target query operation is obtained, that is, calculation or management operations need to be performed at this time, in order to ensure the accuracy of the global GID table, the local source data GID, local local random sample partition data GID and local cross-geographic random sample partition data GID of each computing power hub in all current local GID tables are updated accordingly, so as to obtain the query results according to the corresponding global GID table.

[0092] Furthermore, the distributed file management query method further includes:

[0093] Get the preset heartbeat time and detect in real time whether the current time reaches the heartbeat time based on the timer;

[0094] When the time is reached, the timer is set to re-time, and the global GID table is updated according to the current local GID table.

[0095] Specifically, after the time set by the user starts counting, when the heartbeat time is reached, the global GID table is updated accordingly, thereby ensuring the real-time nature of the GID information.

[0096] The present invention adopts a data shuffling strategy based on the source data on all target computing power hubs to generate a local GID table on each target computing power hub; a global GID table is generated based on the local GID table on each target computing power hub; data file changes on all target computing power hubs are detected in real time according to the target interface, and the corresponding local GID table is updated in real time according to the data file changes; whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information. The present invention converts the large data files scattered in different computing power nodes in the computing power network environment into global random samples that conform to the data model characteristics of random sample partitioning through the designed data shuffling strategy, thereby deploying the local GID table on the corresponding target computing power hub and the global GID table on the middleware, so that the entire computing power network can update the local GID table in time when an update occurs, realize fast data synchronization, and update the global GID table according to the updated local GID table when a query operation is obtained, thereby making the query result more accurate.

[0097] Furthermore, if Figure 6 As shown, based on the above-mentioned distributed file management query method, the present invention also provides a distributed file management query system, wherein the distributed file management query system includes:

[0098] A local GID table generation module 61 is configured to generate a local GID table on each target computing power hub based on the source data on all target computing power hubs using a data shuffling strategy;

[0099] A global GID table generation module 62 is used to generate a global GID table based on the local GID table of each target computing power hub;

[0100] Update module 63, used to detect data file changes on all target computing hubs in real time according to the target interface, and update the corresponding local GID table in real time according to the data file changes;

[0101] The query module 64 is configured to update the global GID table according to the current local GID table whenever a target query operation is obtained, obtain index information according to the updated global GID table, and complete the current query according to the index information.

[0102] Furthermore, if Figure 7 As shown, based on the above-mentioned distributed file management query method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 7 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0103] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a distributed file management query program 40 is stored on the memory 20, and the distributed file management query program 40 can be executed by the processor 10, thereby realizing the distributed file management query method of the present invention.

[0104] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 20, such as executing the distributed file management query method.

[0105] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0106] In one embodiment, when the processor 10 executes the distributed file management query program 40 in the memory 20 , the steps of the above distributed file management query method are implemented.

[0107] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a distributed file management query program, and when the distributed file management query program is executed by a processor, the following steps are implemented:

[0108] Based on the source data on all target computing hubs, a data shuffling strategy is adopted to generate a local GID table on each target computing hub;

[0109] Generate a global GID table based on the local GID table on each target computing hub;

[0110] Detect data file changes on all target computing hubs in real time according to the target interface, and update the corresponding local GID table in real time according to the data file changes;

[0111] Whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information.

[0112] Wherein, the data shuffling strategy includes a local stage and a global stage;

[0113] The local stage performs random processing on the local full amount of data stored on multiple computing power hubs to obtain local random sample division data blocks;

[0114] In the global stage, a data block divided by a local random sample is selected on each computing power hub, and all the selected data blocks divided by the local random sample are merged to obtain a data block divided by a cross-geographic random sample.

[0115] The data shuffling strategy is used based on the source data on all target computing hubs to generate a local GID table on each target computing hub, specifically including:

[0116] Based on the source data on all target computing hubs, a data shuffling strategy is adopted to generate local random sample partitioning data blocks and cross-geographic random sample partitioning data blocks in each target computing hub;

[0117] Generate a local source data GID based on the source data, divide the data blocks according to the local random sample in each target computing power hub, generate a local local random sample divided data GID in each target computing power hub, divide the data blocks according to the cross-geographic random sample in each target computing power hub, and generate a local cross-geographic random sample divided data GID in each computing power hub;

[0118] According to the local source data GID, local random sample partition data GID and local cross-geographic random sample partition data GID of each computing power hub, and after adding the target tag field, the local GID table of each computing power hub is generated.

[0119] The process of generating a global GID table based on the local GID table on each target computing power hub specifically includes:

[0120] Generate global source data GID based on all local source data GIDs in the local GID table on all target computing hubs, generate global local random sample partition data GID based on the local local random sample partition data GIDs on all target computing hubs, and generate global cross-geographic random sample partition data GID based on the local cross-geographic random sample partition data GIDs on all target computing hubs;

[0121] The global GID table is generated according to the global source data GID, the global local random sample partition data GID and the global cross-geographic random sample partition data GID.

[0122] The method of detecting data file changes on all target computing hubs in real time according to the target interface and updating the corresponding local GID table in real time according to the data file changes specifically includes:

[0123] Detect data file changes on all target computing power hubs in real time according to the target interface;

[0124] When a data file change is detected, the state value of the target flag field is adjusted according to the data file change, and the corresponding local GID table is updated according to the state value of the target flag field.

[0125] Wherein, whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information, specifically including:

[0126] Whenever a target query operation is obtained, the global GID table is updated accordingly based on the local source data GID, local random sample partition data GID, and local cross-geographic random sample partition data GID of each computing power hub in all current local GID tables;

[0127] According to the target query operation, obtaining index information in the updated global GID table;

[0128] Generate a query result corresponding to the target query operation based on the index information and output it.

[0129] The distributed file management query method further includes:

[0130] Get the preset heartbeat time and detect in real time whether the current time reaches the heartbeat time based on the timer;

[0131] When the time is reached, the timer is set to re-time, and the global GID table is updated according to the current local GID table.

[0132] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0133] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0134] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A distributed file management query method, characterized in that: The distributed file management query method includes: Based on the source data on all target computing hubs, a data shuffling strategy is adopted to generate a local cross-geographic data distribution index table on each target computing hub; Generate a global cross-geographic data distribution index table based on the local cross-geographic data distribution index table on each target computing power hub; Detect data file changes on all target computing hubs in real time according to the target interface, and update the corresponding local cross-geographic data distribution index table in real time according to the data file changes; Whenever a target query operation is obtained, the global cross-geographic data distribution index table is updated according to the current local cross-geographic data distribution index table, index information is obtained according to the updated global cross-geographic data distribution index table, and the current query is completed according to the index information; The data shuffling strategy includes a local stage and a global stage; The local stage performs random processing on the local full amount of data stored on multiple computing power hubs to obtain local random sample division data blocks; In the global stage, a data block divided by a local random sample is selected on each computing power hub, and all the selected data blocks divided by the local random sample are merged to obtain a data block divided by a cross-geographic random sample.

2. The distributed file management query method according to claim 1, characterized in that: Based on the source data on all target computing power hubs, a data shuffling strategy is adopted to generate a local cross-geographic data distribution index table on each target computing power hub, specifically including: Based on the source data on all target computing hubs, a data shuffling strategy is adopted to generate local random sample partitioning data blocks and cross-geographic random sample partitioning data blocks in each target computing hub; Generate a cross-geographic data distribution index table for local source data based on the source data, divide the data blocks according to the local random samples in each target computing power hub, generate a cross-geographic data distribution index table for the data divided by the local random samples in each target computing power hub, divide the data blocks according to the cross-geographic random samples in each target computing power hub, and generate a cross-geographic data distribution index table for the data divided by the local cross-geographic random samples in each computing power hub; According to the cross-geographic data distribution index table of the local source data of each computing power hub, the cross-geographic data distribution index table of the local random sample partition data and the cross-geographic data distribution index table of the local cross-geographic random sample partition data, and after adding the target tag field, the local cross-geographic data distribution index table of each computing power hub is generated.

3. The distributed file management query method according to claim 2, characterized in that: Generating a global cross-geographic data distribution index table based on the local cross-geographic data distribution index table on each target computing power hub specifically includes: Generate a cross-geographic data distribution index table for global source data based on the cross-geographic data distribution index table of all local source data in the local cross-geographic data distribution index table on all target computing power hubs; generate a cross-geographic data distribution index table for global local random sample partition data based on the cross-geographic data distribution index table of local random sample partition data on all target computing power hubs; generate a cross-geographic data distribution index table for global cross-geographic random sample partition data based on the cross-geographic data distribution index table of local cross-geographic random sample partition data on all target computing power hubs; The global cross-geographic data distribution index table is generated according to the cross-geographic data distribution index table of the global source data, the cross-geographic data distribution index table of the global local random sample partitioning data and the cross-geographic data distribution index table of the global cross-geographic random sample partitioning data.

4. The distributed file management query method according to claim 2, characterized in that: The method of detecting data file changes on all target computing power hubs in real time according to the target interface and updating the corresponding local cross-geographic data distribution index table in real time according to the data file changes specifically includes: Detect data file changes on all target computing power hubs in real time according to the target interface; When a data file change is detected, the state value of the target tag field is adjusted according to the data file change, and the corresponding local cross-geographic data distribution index table is updated according to the state value of the target tag field.

5. The distributed file management query method according to claim 2, characterized in that: Whenever a target query operation is obtained, the global cross-geographic data distribution index table is updated according to the current local cross-geographic data distribution index table, index information is obtained according to the updated global cross-geographic data distribution index table, and the current query is completed according to the index information, specifically including: Whenever a target query operation is obtained, the global cross-geographic data distribution index table is updated accordingly based on the cross-geographic data distribution index table of local source data of each computing power hub in all current local cross-geographic data distribution index tables, the cross-geographic data distribution index table of local random sample partition data, and the cross-geographic data distribution index table of local cross-geographic random sample partition data; According to the target query operation, obtaining index information in the updated global cross-geographic data distribution index table; Generate a query result corresponding to the target query operation based on the index information and output it.

6. The distributed file management query method according to claim 1, characterized in that: The distributed file management query method further includes: Get the preset heartbeat time and detect in real time whether the current time reaches the heartbeat time based on the timer; When the time is reached, the timer is set to re-time, and the global cross-geographic data distribution index table is updated according to the current local cross-geographic data distribution index table.

7. A distributed file management and query system, characterized in that: The distributed file management query system includes: A local cross-geographic data distribution index table generation module is used to generate a local cross-geographic data distribution index table on each target computing power hub based on the source data on all target computing power hubs and using a data shuffling strategy; A global cross-geographic data distribution index table generation module is used to generate a global cross-geographic data distribution index table based on the local cross-geographic data distribution index table on each target computing power hub; An update module is used to detect data file changes on all target computing power hubs in real time according to the target interface, and to update the corresponding local cross-geographic data distribution index table in real time according to the data file changes; A query module, configured to update the global cross-geographic data distribution index table according to the current local cross-geographic data distribution index table whenever a target query operation is obtained, obtain index information according to the updated global cross-geographic data distribution index table, and complete the current query according to the index information; The data shuffling strategy includes a local stage and a global stage; The local stage performs random processing on the local full amount of data stored on multiple computing power hubs to obtain local random sample division data blocks; In the global stage, a data block divided by a local random sample is selected on each computing power hub, and all the selected data blocks divided by the local random sample are merged to obtain a data block divided by a cross-geographic random sample.

8. A terminal, characterized in that: The terminal includes: a memory, a processor, and a distributed file management query program stored in the memory and executable on the processor. When the distributed file management query program is executed by the processor, the steps of the distributed file management query method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a distributed file management query program, and when the distributed file management query program is executed by a processor, the steps of the distributed file management query method according to any one of claims 1 to 6 are implemented.

Images