Distributed file management query method and system, terminal and storage medium

Abstract

The invention discloses a distributed file management query method and system, a terminal and a storage medium, and the method comprises the steps: generating a local GID table on each target computing power hub by adopting a data shuffling strategy based on source data on all target computing power hubs; generating a global GID table according to the local GID table on each target computing power hub; detecting data file changes on all target computing power hubs in real time according to the target interface, and updating a corresponding local GID table in real time according to the data file changes; and when a target query operation is obtained, updating the global GID table according to the current local GID table, obtaining index information according to the updated global GID table, and completing the current query according to the index information. According to the method, rapid data synchronization on the computing power network can be realized through the constructed local GID table and the global GID table.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a distributed file management query method, system, terminal and storage medium. Background Art

[0002] Computing Power Network (CPN) is a new type of network that brings together powerful computing power, storage and network resources to provide computing power support for large-scale intelligent applications with intensive massive data, such as training large language models or autonomous driving models. Especially when facing ultra-large-scale data processing and storage needs, it is the key to ensuring data consistency, availability, disaster recovery and efficient access.

[0003] However, the current computing power network is inconvenient in the data synchronization process, which makes data management more complex and inconvenient for users.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a distributed file management query method, system, terminal and computer-readable storage medium, aiming to solve the problem in the prior art that the current computing power network is relatively inconvenient in the data synchronization process, resulting in high complexity of data management and inconvenience for users.

[0006] To achieve the above object, the present invention provides a distributed file management query method, which comprises the following steps:

[0007] Based on the source data on all target computing hubs, a data shuffling strategy is adopted to generate a local GID table on each target computing hub;

[0008] Generate a global GID table based on the local GID table on each target computing hub;

[0009] Detect data file changes on all target computing hubs in real time according to the target interface, and update the corresponding local GID table in real time according to the data file changes;

[0010] Whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information.

[0011] Optionally, the data shuffling strategy includes a local stage and a global stage;

[0012] The local stage performs random processing on the local full amount of data stored on multiple computing power hubs to obtain local random sample division data blocks;

[0013] The global stage selects a data block divided by a local random sample on each computing power hub, merges all the selected data blocks divided by local random samples, and obtains a data block divided by a cross-geographic random sample.

[0014] Optionally, based on the source data on all target computing power hubs, a data shuffling strategy is adopted to generate a local GID table on each target computing power hub, specifically including:

[0015] Based on the source data on all target computing power hubs, a data shuffling strategy is adopted to generate local random sample partitioning data blocks and cross-geographic random sample partitioning data blocks in each target computing power hub;

[0016] Generate a local source data GID based on the source data, divide the data blocks according to the local random samples in each target computing power hub, generate the local local random sample divided data GID in each target computing power hub, divide the data blocks according to the cross-geographic random samples in each target computing power hub, and generate the local cross-geographic random sample divided data GID in each computing power hub;

[0017] According to the local source data GID, local random sample partition data GID and local cross-geographic random sample partition data GID of each computing power hub, and after adding the target tag field, the local GID table of each computing power hub is generated.

[0018] Optionally, generating a global GID table according to a local GID table on each target computing power hub specifically includes:

[0019] Generate global source data GID based on all local source data GIDs in the local GID table on all target computing hubs, generate global local random sample partition data GID based on local local random sample partition data GIDs on all target computing hubs, and generate global cross-geographic random sample partition data GID based on local cross-geographic random sample partition data GIDs on all target computing hubs;

[0020] The global GID table is generated according to the global source data GID, the global local random sample partition data GID and the global cross-geographic random sample partition data GID.

[0021] Optionally, the real-time detection of data file changes on all target computing power hubs according to the target interface, and real-time updating of the corresponding local GID table according to the data file changes, specifically includes:

[0022] Detect data file changes on all target computing power hubs in real time according to the target interface;

[0023] When a data file change is detected, the state value of the target mark field is adjusted according to the data file change, and the corresponding local GID table is updated according to the state value of the target mark field.

[0024] Optionally, whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information, specifically including:

[0025] Whenever a target query operation is obtained, the global GID table is updated accordingly according to the local source data GID, local local random sample partition data GID and local cross-geographic random sample partition data GID of each computing power hub in all current local GID tables;

[0026] According to the target query operation, obtaining index information in the updated global GID table;

[0027] Generate a query result corresponding to the target query operation according to the index information and output it.

[0028] Optionally, the distributed file management query method further includes:

[0029] Get the preset heartbeat time, and detect in real time whether the current time reaches the heartbeat time based on the timer;

[0030] When the time limit is reached, the timer is set to restart, and the global GID table is updated according to the current local GID table.

[0031] In addition, to achieve the above-mentioned purpose, the present invention also provides a distributed file management query system, wherein the distributed file management query system comprises:

[0032] A local GID table generation module is used to generate a local GID table on each target computing power hub based on the source data on all target computing power hubs and using a data shuffling strategy;

[0033] A global GID table generation module is used to generate a global GID table based on the local GID table on each target computing power hub;

[0034] An update module is used to detect data file changes on all target computing power hubs in real time according to the target interface, and update the corresponding local GID table in real time according to the data file changes;

[0035] The query module is used to update the global GID table according to the current local GID table whenever a target query operation is obtained, obtain index information according to the updated global GID table, and complete the current query according to the index information.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a distributed file management query program stored on the memory and executable on the processor, and when the distributed file management query program is executed by the processor, the steps of the distributed file management query method as described above are implemented.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a distributed file management query program, and when the distributed file management query program is executed by the processor, the steps of the distributed file management query method as described above are implemented.

[0038] In the present invention, based on the source data on all target computing power hubs, a data shuffling strategy is adopted to generate a local GID table on each target computing power hub; a global GID table is generated according to the local GID table on each target computing power hub; data file changes on all target computing power hubs are detected in real time according to the target interface, and the corresponding local GID table is updated in real time according to the data file changes; whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, and index information is obtained according to the updated global GID table, and the current query is completed according to the index information. The present invention converts the large data files scattered in different computing power nodes in the computing power network environment into global random samples that conform to the data model characteristics of random sample division through the designed data shuffling strategy, thereby deploying the local GID table on the corresponding target computing power hub and the global GID table on the middleware, so that the entire computing power network can update the local GID table in time when an update occurs, realize fast data synchronization, and when a query operation is obtained, the global GID table is updated according to the updated local GID table, so that the query result obtained is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of a preferred embodiment of the distributed file management query method of the present invention;

[0040] Figure 2 It is a flow chart of the data shuffling strategy in the distributed file management query method of the present invention;

[0041] Figure 3 It is a schematic diagram of the logical hierarchy of data files in the distributed file management query method of the present invention;

[0042] Figure 4 It is a schematic diagram of the global GID table synchronization mechanism corresponding to the middleware in the distributed file management query method of the present invention;

[0043] Figure 5 It is a schematic diagram of data synchronization in the distributed file management query method of the present invention;

[0044] Figure 6 It is a structural diagram of a preferred embodiment of the distributed file management query system of the present invention;

[0045] Figure 7 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0047] Computing Power Network (CPN) is a new type of network that brings together powerful computing power, storage and network resources to provide computing power support for large-scale intelligent applications with intensive massive data, such as training large language models or autonomous driving models. Especially when facing ultra-large-scale data processing and storage needs, it is the key to ensuring data consistency, availability, disaster recovery and efficient access. However, the current computing power network is inconvenient in the data synchronization process, resulting in high complexity of data management and inconvenience for users.

[0048] In response to one or more of the above problems, the present invention adopts a data shuffling strategy based on the source data on all target computing power hubs to generate a local GID table on each target computing power hub; generates a global GID table based on the local GID table on each target computing power hub; detects data file changes on all target computing power hubs in real time according to the target interface, and updates the corresponding local GID table in real time according to the data file changes; whenever a target query operation is obtained, updates the global GID table according to the current local GID table, obtains index information according to the updated global GID table, and completes the current query according to the index information.

[0049] The distributed file management query method described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the distributed file management query method includes the following steps:

[0050] Step S10: Based on the source data on all target computing power hubs, a data shuffling strategy is adopted to generate a local GID table on each target computing power hub.

[0051] It should be noted that in the present invention, the data shuffling strategy is a shuffling strategy designed by the present invention, specifically an RSP (Random Sample Partition) data shuffling strategy in a computing power network environment. In the computing power network environment, the big data files scattered in different computing power nodes are converted into global random samples that conform to the characteristics of the RSP data model, that is, the Geo-RSP (Geo-Distributed Random Sample Partition) data model.

[0052] Further, the data shuffling strategy includes a local stage and a global stage;

[0053] The local stage performs random processing on the local full amount of data stored on multiple computing power hubs to obtain local random sample division data blocks;

[0054] The global stage selects a data block divided by a local random sample on each computing power hub, merges all the selected data blocks divided by local random samples, and obtains a data block divided by a cross-geographic random sample.

[0055] Specifically, the data shuffling strategy includes a local stage and a global stage; Figure 2 As shown, local RSP is performed in the local stage. For the big data as a whole that is distributed and stored on different computing power hubs, the characteristic distribution of its local data may be different from the characteristic distribution of the full data. Inside each computing power hub, a distributed file system such as HDFS (Hadoop Distributed File System), GFS (Google Distributed File System) or Gfarm (Gfarm File System) is usually used to manage these local data in units of data blocks. The characteristic distribution between these local data blocks may be different. Local RSP refers to the randomization of the local full data stored in each computing power hub or data center. Specifically, this step can be completed by the toRSP() operator. Through local RSP, the characteristic distribution of data on each computing power hub becomes uniform; the data blocks that have been locally RSPed, that is, the local random sample partition data blocks, each of which can be regarded as a random sample of the current computing power hub, and they have similar characteristic distributions.

[0056] The global stage is RSP shuffle. After the local RSP is completed, the global stage begins. The core operation of this stage is to randomly select a randomized local RSP data block from each computing power hub, and merge these data blocks into a new data block, called Geo-RSP data block, that is, cross-geographic random sample partitioning data block. The characteristic distribution of the Geo-RSP data block is similar to the characteristic distribution of the full data, that is, by analyzing the Geo-RSP data block, the statistical characteristics of the full data in the entire computing power network can be approximately estimated. In this way, each computing power hub can generate and obtain a large number of Geo-RSP blocks, and this complete process is called RSP shuffle.

[0057] The main advantage of the Geo-RSP data model is that it introduces a new management strategy for distributed file systems across geographies. This strategy avoids full data synchronization on all computing hubs, and only requires limited data backup within the computing hubs for fault tolerance. At the same time, only a small number of Geo-RSP data blocks need to be shuffled and synchronized between different computing hubs, ensuring that each computing hub can obtain a global perspective without having to bear the huge WAN load brought by global data synchronization; this not only ensures data security and fault tolerance, but also significantly reduces the demand for WAN resources during the data synchronization process. In addition, both local RSPization and Geo-RSP shuffling can be performed offline, which further reduces the negative impact of WAN constraints on computing, thereby achieving more efficient and economical data analysis in distributed data processing.

[0058] Therefore, it can be seen that the global RSP data block (i.e., Geo-RSP data block) in the Geo-RSP data model can be used to approximately estimate the statistical results of the full amount of data. That is to say, during actual data analysis, there is no need to use data backup and synchronization mechanisms on the computing power hub to ensure strong consistency of data in different locations. Only a small number of Geo-RSP data blocks need to be synchronized to ensure the normal progress of the calculation and analysis. Even if problems such as computing interruptions or hardware failures occur, no serious consequences will occur whether the computing tasks are reallocated or the current calculations are continued. At the same time, protecting the integrity of the source data is also very critical, but for the present invention, sudden abnormal situations can be dealt with by saving copies of the data on a few computing power hubs or local computing nodes.

[0059] In response to the data shuffling strategy, the present invention also designs a new file management system for the computing power network - Geo-Distributed Indexed Datasets File System, referred to as GIDFS file system, that is, a local GID table is generated on each target computing power hub, and a global GID table is generated on the middleware to synchronize the information of all target computing power hubs. GIDFS manages physical data files through the mechanism of indexing metadata information and geographic location information. This method not only improves data retrieval efficiency, but also simplifies data synchronization and backup processes, thereby effectively reducing the complexity of data management. Through the GIDFS file system, the computing system of the computing power network can quickly and accurately locate physical data, and can more efficiently process cross-geographic distributed computing tasks, while ensuring data security and high availability.

[0060] The GIDFS file system does not directly store physical data files, but only stores and manages the index information of data files. For example, it can store global file names, local file names, data storage paths, RSP block numbers, and computing hub numbers. This information can be directly read through the metadata source data information of traditional file systems (such as HDFS), or can be automatically scanned and recorded through simple program code. Combining these index information can form a GID (Geo-Distributed Indexed Datasets) record, and each GID is a record in a data table. In the computing network environment, each physical data file stored on the computing hub will generate a corresponding GID record. Through these GID records, the computing system of the computing network can quickly and accurately locate specific data files and determine the relevant execution parameters of the computing task based on the index information. Accurate acquisition of index information is crucial for the reliable operation of the GIDFS file system. The GIDFS file system can use the metadata information table (i.e., metadata table) of the existing distributed file system (such as HDFS) to directly obtain the required index information; it can also build the initial GID table through disk scanning, and then continuously maintain the iterative update of the GID table.

[0061] Furthermore, the index information of the GIDFS file system can be abstracted into a general API interface, and SQL statements can be used to complete information indexing operations, thereby promoting the execution of computing tasks. For example, by executing SQL query statements such as "Selectfilepath,GeoRSPBlockNums from Global GeoRSPTable where GlobalFileName=FileName", key information such as the specific storage path of a specific data set and the number of global Geo-RSP blocks can be obtained.

[0062] In a computing network environment, due to differences in data types and processing stages, the generated GID information will also be different. For example, there are significant differences in the recording requirements of GID information between local source data distributed and stored on a computing hub and local RSP data processed by a data shuffling strategy. For the latter, in addition to recording the global file name and the file name of the local source data, it is also necessary to record the name of the RSP file and the number of RSP data blocks. Therefore, if Figure 3 As shown in the figure, in order to effectively manage the data index information with different core features, the GIDFS file system divides the cross-geographic GID data index into four different levels according to the shuffling process of Geo-RSP data, among which, Level I, global source data, represents the logical whole of the data in the computing network, and is a data representation at a logical level, including three global parts corresponding to Level II, III, and IV; Level II: local source data, refers to the original data files stored on each computing hub; Level III, local RSP data, represents the data files that have undergone preliminary randomization processing inside the computing hub; Level IV, local Geo-RSP data, refers to the Geo-RSP data files that have been shuffled between computing hubs. Furthermore, local source data refers to the most original files; local RSP data corresponds to the local stage of the data shuffling strategy; local Geo-RSP data corresponds to the global stage of the data shuffling strategy, that is, the local RSP data is transmitted and merged across data centers to obtain Geo-RSP data blocks.

[0063] Among them, levels II, II, and IV are physical data, usually stored on computing power hubs. Although the global source data (level I) logically represents the entire data in the computing power network, it is actually achieved by synchronizing and integrating local GIDs (levels II, II, and IV), and is a collection of all local GID information; that is, according to the data logic in the computing power network environment, 6 types of GID information tables with 3 groups of one-to-one correspondence and need to be updated synchronously are generated. This hierarchical and segmented data index management method not only improves the efficiency of data retrieval and data management, but also provides a clear logical structure for data processing, ensuring the consistency and integrity of data in the cross-geographic distributed computing process.

[0064] Usually, the GID information of global data is maintained in the form of a GID table on a middleware (the central control node of the computing power network environment), covering the information of three logical data: global source data, global RSP data, and global Geo-RSP data; however, it is unrealistic to use the global GID table on the middleware to directly monitor the GID information in the entire computing power network, which will put a lot of pressure on the network load of the middleware and is easily affected by wide area network constraints. Therefore, the present invention designs and deploys a local GID table responsible for synchronizing local data information on each computing power hub. This table contains most of the same information as the global GID table, but adds three important target tag fields: "is_modified", "is_deleted" and "is_sync", the purpose of which is to record the status changes of the file represented by the current GID.

[0065] Among them, is_modified: This status mark indicates whether the current data file has undergone modification operations since the last synchronization. If the data file is modified, the mark will be set to true, indicating that the change needs to be synchronized to the global GID table; is_deleted: This status mark indicates whether the current data file has been deleted. If so, the mark will be set to true, and the corresponding global GID table will also be updated to reflect this change during the next synchronization; is_sync: This status mark indicates whether the current data file has been synchronized. If the GID information of the file has been successfully synchronized to the global GID table, and no addition, modification or deletion operations have been performed after synchronization, the mark will be set to true, indicating that the current GID record is already the latest. By introducing these three mark fields, the computing system of the computing power network can effectively track and manage the status changes of data files on each computing power hub. This synchronization mechanism between the local GID table and the global GID table ensures the consistency, latest and reliability of data in the computing power network.

[0066] Furthermore, based on the source data on all target computing power hubs, a data shuffling strategy is adopted to generate a local GID table on each target computing power hub, specifically including:

[0067] Based on the source data on all target computing power hubs, a data shuffling strategy is adopted to generate local random sample partitioning data blocks and cross-geographic random sample partitioning data blocks in each target computing power hub;

[0068] Generate a local source data GID based on the source data, divide the data blocks according to the local random samples in each target computing power hub, generate the local local random sample divided data GID in each target computing power hub, divide the data blocks according to the cross-geographic random samples in each target computing power hub, and generate the local cross-geographic random sample divided data GID in each computing power hub;

[0069] According to the local source data GID, local random sample partition data GID and local cross-geographic random sample partition data GID of each computing power hub, and after adding the target tag field, the local GID table of each computing power hub is generated.

[0070] Specifically, in the present invention, three local GID information tables are maintained on all computing power hubs, which are used to record and update the index information of source data, RSP data and Geo-RSP data on the computing power hub, namely, local source data GID, local local random sample partition data GID and local cross-geographic random sample partition data GID, and a target tag field is added to each of them to generate a local GID table for each computing power hub.

[0071] Step S20: Generate a global GID table based on the local GID table on each target computing power hub.

[0072] Specifically, Figure 4 As shown, in the present invention, on the middleware, the GIDFS file system maintains three global GID master tables, namely the global GID tables. Among them, the global source data GID is responsible for synchronizing the local source data GID on all computing power hubs, the global RSP data GID is responsible for synchronizing the GID information of the local RSP data on all computing power hubs, and the global Geo-RSP data GID synchronizes the GID information of the local Geo-RSP data on all computing power hubs. The synchronization between the three groups of GIDs does not interfere with each other, but the information content verifies each other. The main function of the three global GID tables is to provide indexing services for computing tasks to determine the computing power hub and resource allocation for executing tasks. The main function of the three local GID tables is to maintain and update the GID information of local data and update the global GID table to ensure the accuracy of the global GID information table. Since the GID table is an index information table, only very little communication overhead is required to ensure almost real-time information synchronization.

[0073] Furthermore, generating a global GID table according to the local GID table on each target computing power hub specifically includes:

[0074] Generate global source data GID based on all local source data GIDs in the local GID table on all target computing hubs, generate global local random sample partition data GID based on local local random sample partition data GIDs on all target computing hubs, and generate global cross-geographic random sample partition data GID based on local cross-geographic random sample partition data GIDs on all target computing hubs;

[0075] The global GID table is generated according to the global source data GID, the global local random sample partition data GID and the global cross-geographic random sample partition data GID.

[0076] Specifically, all local source data GIDs in the local GID tables on all target computing power hubs are aggregated to generate a global source data GID, the local local random sample partition data GIDs on all target computing power hubs are aggregated to generate a global local random sample partition data GID, and the local cross-geographic random sample partition data GIDs on all target computing power hubs are aggregated to generate a global cross-geographic random sample partition data GID, and then the global source data GID, the global local random sample partition data GID and the global cross-geographic random sample partition data GID are aggregated to generate the global GID table.

[0077] Step S30: Detect data file changes on all target computing power hubs in real time according to the target interface, and update the corresponding local GID table in real time according to the data file changes.

[0078] Specifically, the synchronization and construction process of the global GID table is not a simple re-acquisition of the contents of all local GID tables and then merging them, but is selective. Therefore, for each synchronization, only the GID information whose data status has changed needs to be updated. This mechanism is mainly implemented by relying on the three additional information tags that the local table has compared to the global table, namely the status value of the target tag field. When the state of the physical data stored on the computing power hub changes, the change in the information tag can be used to determine which data files' local GID information needs to be updated to the global GID table.

[0079] Furthermore, the real-time detection of data file changes on all target computing power hubs according to the target interface and the real-time updating of the corresponding local GID table according to the data file changes specifically include:

[0080] Detect data file changes on all target computing power hubs in real time according to the target interface;

[0081] When a data file change is detected, the state value of the target mark field is adjusted according to the data file change, and the corresponding local GID table is updated according to the state value of the target mark field.

[0082] Specifically, in one embodiment of the present invention, before the GID synchronization update, the three information marks of the local data A in the local GID table are false, false, and true, that is, the local GID table has been synchronized with the global GID table, and no operation of changing the data index information has been performed so far. Obviously, the local GID table does not need to perform synchronization operations with the global GID table. For data set B, its status is marked as true, false, false, which means that the data set is a newly created data set and has not been synchronized. Similarly, for data set C, its status is marked as true, true, true, which means that the data set has been synchronized in the global GID table before, but the operation of deleting the data has been performed on the computing power hub, so the corresponding GID information in the local GID table needs to be synchronized and deleted in the global GID table. Therefore, when the next global GID update occurs, only three GID information in the local GID table needs to be synchronized in the global table, instead of re-acquiring and merging all local GID tables. This method makes the pressure of synchronization update on the wide area network bandwidth almost negligible. At the same time, in actual applications, the computing resources of the computing network are not monopolized by a certain computing task, and each computing hub usually performs local computing tasks. Therefore, the status changes of local data files are not controllable from a global perspective, and the differentiated design of the local GID table and the global GID table avoids these situations and improves the flexibility of the GIDFS file system. The differentiated design of the global GID table avoids these situations and improves the flexibility of the GIDFS file system.

[0083] Further, in the present invention, when the local GID table is updated, Figure 5 As shown, the corresponding modules include two modules, namely the monitoring notification module (inotify module) and the update module (inotify module). The inotify module is logically located at the upper layer of the distributed file management system (usually HDFS) of the computing power hub, and can monitor event information at a specific location. It provides an event tracking API—InotifyEvent, which is implemented based on the DF SInotifyEventInputStream interface. The Stream can monitor various types of events, including Create, Rename, Unlink, Append, and other types. For example, a new folder is created on the computing power hub to store a data set, which will be monitored by In otifyEvent and trigger a Create event; operations such as renaming, moving folders, and adding new data will trigger a Rename event; deleting files will trigger an Unlink event. While monitoring various events, the Stream will also give corresponding processing solutions and provide the file storage path where the event occurred.

[0084] That is, when the computing system is not executing computing tasks, the imotify module monitors the changes of data files on all target computing power hubs in real time through the ImotifyEventAPI, that is, the target interface. When the local data file changes, the ImotifyEventAPI will generate corresponding events according to the state changes, and the DFSInotifyEventInputStream will return the path information (URL) where the event occurred. The local GID table will re-index the data index information under the path. At the same time, the status values ​​of the three mark fields, is modified, is deleted, and is sync, are determined according to the specific type of the event, and then new local GID information is generated, the local GID table is updated accordingly, and the global GID table is updated if set.

[0085] Step S40: Whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information.

[0086] It should be noted that the Update module is mainly divided into the update of the local GID table and the update of the global GID table. Among them, the update of the local GID table is a processing solution after the event is triggered. The event trigger is real-time, so the update of the local GID table is also real-time. The synchronous update of the global GID table and the local GID table is mainly divided into two triggering situations: the first is to set a fixed heartbeat time, that is, to perform regular updates at regular intervals; the second is when the computing task needs to perform computing or management operations and needs to obtain index information from the global GID table. The former ensures the real-time nature of the GID information, while the latter ensures the accuracy of the GID information.

[0087] Furthermore, whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information, specifically including:

[0088] Whenever a target query operation is obtained, the global GID table is updated accordingly according to the local source data GID, local local random sample partition data GID and local cross-geographic random sample partition data GID of each computing power hub in all current local GID tables;

[0089] According to the target query operation, obtaining index information in the updated global GID table;

[0090] Generate a query result corresponding to the target query operation according to the index information and output it.

[0091] Specifically, when the target query operation is obtained, that is, calculation or management operations need to be performed at this time, in order to ensure the accuracy of the global GID table, the local source data GID, local random sample partition data GID and local cross-geographic random sample partition data GID of each computing power hub in all current local GID tables are used to update the global GID table accordingly, so as to obtain the query results according to the corresponding global GID table.

[0092] Furthermore, the distributed file management query method further includes:

[0093] Get the preset heartbeat time, and detect in real time whether the current time reaches the heartbeat time based on the timer;

[0094] When the time limit is reached, the timer is set to restart, and the global GID table is updated according to the current local GID table.

[0095] Specifically, after the time set by the user starts counting, when the heartbeat time is reached, the global GID table is updated once, thereby ensuring the real-time nature of the GID information.

[0096] The present invention adopts a data shuffling strategy based on the source data on all target computing power hubs, generates a local GID table on each target computing power hub; generates a global GID table according to the local GID table on each target computing power hub; detects data file changes on all target computing power hubs in real time according to the target interface, and updates the corresponding local GID table in real time according to the data file changes; whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, and index information is obtained according to the updated global GID table, and the current query is completed according to the index information. The present invention converts the large data files scattered in different computing power nodes in the computing power network environment into global random samples that conform to the data model characteristics of random sample division through the designed data shuffling strategy, thereby deploying the local GID table on the corresponding target computing power hub and the global GID table on the middleware, so that the entire computing power network can update the local GID table in time when an update occurs, realize fast data synchronization, and when a query operation is obtained, the global GID table is correspondingly updated according to the updated local GID table, thereby making the query result obtained more accurate.

[0097] Furthermore, if Figure 6 As shown, based on the above-mentioned distributed file management query method, the present invention also provides a distributed file management query system, wherein the distributed file management query system includes:

[0098] A local GID table generation module 61 is used to generate a local GID table on each target computing power hub based on the source data on all target computing power hubs and using a data shuffling strategy;

[0099] A global GID table generation module 62, used to generate a global GID table according to the local GID table on each target computing power hub;

[0100] An update module 63 is used to detect data file changes on all target computing power hubs in real time according to the target interface, and update the corresponding local GID table in real time according to the data file changes;

[0101] The query module 64 is used to update the global GID table according to the current local GID table whenever a target query operation is obtained, obtain index information according to the updated global GID table, and complete the current query according to the index information.

[0102] Furthermore, if Figure 7 As shown, based on the above-mentioned distributed file management query method and system, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 7 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0103] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a distributed file management query program 40 is stored on the memory 20, and the distributed file management query program 40 can be executed by the processor 10, thereby realizing the distributed file management query method of the present invention.

[0104] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the distributed file management query method.

[0105] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0106] In one embodiment, when the processor 10 executes the distributed file management query program 40 in the memory 20, the steps of the above distributed file management query method are implemented.

[0107] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a distributed file management query program, and when the distributed file management query program is executed by a processor, the following steps are implemented:

[0108] Based on the source data on all target computing hubs, a data shuffling strategy is adopted to generate a local GID table on each target computing hub;

[0109] Generate a global GID table based on the local GID table on each target computing hub;

[0110] Detect data file changes on all target computing hubs in real time according to the target interface, and update the corresponding local GID table in real time according to the data file changes;

[0111] Whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information.

[0112] Wherein, the data shuffling strategy includes a local stage and a global stage;

[0113] The local stage performs random processing on the local full amount of data stored on multiple computing power hubs to obtain local random sample division data blocks;

[0114] The global stage selects a data block divided by a local random sample on each computing power hub, merges all the selected data blocks divided by local random samples, and obtains a data block divided by a cross-geographic random sample.

[0115] The method of generating a local GID table on each target computing power hub by adopting a data shuffling strategy based on the source data on all target computing power hubs specifically includes:

[0116] Based on the source data on all target computing power hubs, a data shuffling strategy is adopted to generate local random sample partitioning data blocks and cross-geographic random sample partitioning data blocks in each target computing power hub;

[0117] Generate a local source data GID based on the source data, divide the data blocks according to the local random samples in each target computing power hub, generate the local local random sample divided data GID in each target computing power hub, divide the data blocks according to the cross-geographic random samples in each target computing power hub, and generate the local cross-geographic random sample divided data GID in each computing power hub;

[0118] According to the local source data GID, local random sample partition data GID and local cross-geographic random sample partition data GID of each computing power hub, and after adding the target tag field, the local GID table of each computing power hub is generated.

[0119] The generating of the global GID table according to the local GID table on each target computing power hub specifically includes:

[0120] Generate global source data GID based on all local source data GIDs in the local GID table on all target computing hubs, generate global local random sample partition data GID based on local local random sample partition data GIDs on all target computing hubs, and generate global cross-geographic random sample partition data GID based on local cross-geographic random sample partition data GIDs on all target computing hubs;

[0121] The global GID table is generated according to the global source data GID, the global local random sample partition data GID and the global cross-geographic random sample partition data GID.

[0122] The method of detecting data file changes on all target computing power hubs in real time according to the target interface and updating the corresponding local GID table in real time according to the data file changes specifically includes:

[0123] Detect data file changes on all target computing power hubs in real time according to the target interface;

[0124] When a data file change is detected, the state value of the target mark field is adjusted according to the data file change, and the corresponding local GID table is updated according to the state value of the target mark field.

[0125] Wherein, whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information, specifically including:

[0126] Whenever a target query operation is obtained, the global GID table is updated accordingly according to the local source data GID, local local random sample partition data GID and local cross-geographic random sample partition data GID of each computing power hub in all current local GID tables;

[0127] According to the target query operation, obtaining index information in the updated global GID table;

[0128] Generate a query result corresponding to the target query operation according to the index information and output it.

[0129] Wherein, the distributed file management query method further includes:

[0130] Get the preset heartbeat time, and detect in real time whether the current time reaches the heartbeat time based on the timer;

[0131] When the time limit is reached, the timer is set to restart, and the global GID table is updated according to the current local GID table.

[0132] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or terminal including the element.

[0133] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0134] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A distributed file management query method, characterized in that: The distributed file management query method comprises: Based on the source data on all target computing hubs, a data shuffling strategy is adopted to generate a local GID table on each target computing hub; Generate a global GID table based on the local GID table on each target computing hub; Detect data file changes on all target computing hubs in real time according to the target interface, and update the corresponding local GID table in real time according to the data file changes; Whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information.

2. The distributed file management query method according to claim 1, characterized in that: The data shuffling strategy includes a local stage and a global stage; The local stage performs random processing on the local full amount of data stored on multiple computing power hubs to obtain local random sample division data blocks; The global stage selects a data block divided by a local random sample on each computing power hub, merges all the selected data blocks divided by local random samples, and obtains a data block divided by a cross-geographic random sample.

3. The distributed file management query method according to claim 1, characterized in that: Based on the source data on all target computing power hubs, a data shuffling strategy is adopted to generate a local GID table on each target computing power hub, specifically including: Based on the source data on all target computing power hubs, a data shuffling strategy is adopted to generate local random sample partitioning data blocks and cross-geographic random sample partitioning data blocks in each target computing power hub; Generate a local source data GID based on the source data, divide the data blocks according to the local random samples in each target computing power hub, generate the local local random sample divided data GID in each target computing power hub, divide the data blocks according to the cross-geographic random samples in each target computing power hub, and generate the local cross-geographic random sample divided data GID in each computing power hub; According to the local source data GID, local random sample partition data GID and local cross-geographic random sample partition data GID of each computing power hub, and after adding the target tag field, the local GID table of each computing power hub is generated.

4. The distributed file management query method according to claim 3, characterized in that: The generating of the global GID table according to the local GID table on each target computing power hub specifically includes: Generate global source data GID based on all local source data GIDs in the local GID table on all target computing hubs, generate global local random sample partition data GID based on local local random sample partition data GIDs on all target computing hubs, and generate global cross-geographic random sample partition data GID based on local cross-geographic random sample partition data GIDs on all target computing hubs; The global GID table is generated according to the global source data GID, the global local random sample partition data GID and the global cross-geographic random sample partition data GID.

5. The distributed file management query method according to claim 3, characterized in that: The method of detecting data file changes on all target computing power hubs in real time according to the target interface and updating the corresponding local GID table in real time according to the data file changes specifically includes: Detect data file changes on all target computing power hubs in real time according to the target interface; When a data file change is detected, the state value of the target mark field is adjusted according to the data file change, and the corresponding local GID table is updated according to the state value of the target mark field.

6. The distributed file management query method according to claim 3, characterized in that: Whenever a target query operation is obtained, the global GID table is updated according to the current local GID table, index information is obtained according to the updated global GID table, and the current query is completed according to the index information, specifically including: Whenever a target query operation is obtained, the global GID table is updated accordingly according to the local source data GID, local local random sample partition data GID and local cross-geographic random sample partition data GID of each computing power hub in all current local GID tables; According to the target query operation, obtaining index information in the updated global GID table; Generate a query result corresponding to the target query operation according to the index information and output it.

7. The distributed file management query method according to claim 1, characterized in that: The distributed file management query method further includes: Get the preset heartbeat time, and detect in real time whether the current time reaches the heartbeat time based on the timer; When the time limit is reached, the timer is set to restart, and the global GID table is updated according to the current local GID table.

8. A distributed file management query system, characterized in that: The distributed file management query system includes: A local GID table generation module is used to generate a local GID table on each target computing power hub based on the source data on all target computing power hubs and using a data shuffling strategy; A global GID table generation module is used to generate a global GID table based on the local GID table on each target computing power hub; An update module is used to detect data file changes on all target computing power hubs in real time according to the target interface, and update the corresponding local GID table in real time according to the data file changes; The query module is used to update the global GID table according to the current local GID table whenever a target query operation is obtained, obtain index information according to the updated global GID table, and complete the current query according to the index information.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a distributed file management query program stored in the memory and executable on the processor. When the distributed file management query program is executed by the processor, the steps of the distributed file management query method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a distributed file management query program, and when the distributed file management query program is executed by a processor, the steps of the distributed file management query method according to any one of claims 1 to 7 are implemented.

Images