Data access method and device between heterogeneous file systems, equipment and medium

Abstract

The invention relates to a data access method and device between heterogeneous file systems, equipment and a medium. The method comprises the steps of determining a bucket identifier of a target data bucket in an accessed database and a query timestamp of the bucket identifier based on a logic plan of a query statement; the metadata information is queried, database configuration information of the file systems matched with the storage time periods is determined according to the query time stamps, the protocol types of the file systems are different from one another, and the file systems store data files in different storage time periods; a physical plan is constructed, a corresponding query task is included in the physical plan corresponding to each target data bucket, and each query task comprises a storage path, which is determined based on database configuration information, bucket identifiers of the target data buckets and query timestamps of the target data buckets and carries protocol type identifiers, of the target data files; and according to the protocol type identifier of each query task, executing the query task through the corresponding client to obtain a query result. The efficiency and reliability of data migration among different file systems can be improved.

Description

Technical Field

[0001] The present application relates to big data processing technology, and in particular to a method for accessing data between heterogeneous file systems and its apparatus, equipment, and medium. Background Art

[0002] Data migration is a common operation in big data clusters, such as migrating data from HDFS (Hadoop Distributed File System) to OSS (Object Storage Service). Alternatively, when resources in certain data buckets reach a bottleneck, data needs to be moved from one bucket to another. HDFS upgrades also involve data migration. However, traditional data migration methods, such as using the Distcp tool, have numerous issues.

[0003] Distcp is the official distributed copy tool provided by Hadoop. It implements data migration by executing a distributed copy command. However, this approach has significant limitations in practical applications. First, during the data migration process, since data is generated on a daily or hourly basis, if the execution time of the migration task crosses the daily or hourly boundary, the task may fail, or the copied data may become inconsistent with the original data. This not only increases the complexity of data migration, but may also lead to data loss or corruption. Second, when migrating data across data centers or regions, data migration requires extremely high bandwidth, which undoubtedly increases the cost and difficulty of migration. Furthermore, after the migration is completed, the data needs to be verified to ensure its integrity and consistency, which is a tedious and time-consuming process.

[0004] More critically, traditional data migration methods cannot effectively support incremental migrations at the database, table, and partition levels. In practice, users may only need to migrate a portion of data, rather than an entire bucket. However, the distcp tool cannot meet this granular migration requirement, forcing users to perform a full migration, which not only wastes resources but also increases migration time and costs. Furthermore, traditional migration methods require high bandwidth capacity during the migration process, which can hinder the smooth progress of the migration process when resources are limited, further affecting data migration efficiency.

[0005] In summary, traditional data migration methods, when used in complex scenarios within big data clusters, suffer from technical issues such as prone task failures, difficulty ensuring data consistency, high cross-region migration costs, inability to support partition-level migration, and high bandwidth requirements. These issues severely impact the efficiency and reliability of data migration, causing significant distress to users. Summary of the Invention

[0006] The purpose of this application is to solve the above problems and provide a method for accessing data between heterogeneous file systems and its corresponding device, equipment, non-volatile readable storage medium, and computer program product.

[0007] According to one aspect of the present application, a method for accessing data between heterogeneous file systems is provided, comprising:

[0008] Receive a query statement, parse and generate a logical plan corresponding to the query statement, and determine from the logical plan the bucket identifiers of target data buckets in the database accessed by the query statement and the query timestamp of each target data bucket;

[0009] Query metadata information, determine a target distributed file system with a storage period that matches the target data bucket based on the query timestamp of the target data bucket, and obtain database configuration information corresponding to the target distributed file system for storing the database, wherein each distributed file system has a different protocol type and stores data files of the data bucket generated by the database within the corresponding storage period according to different storage periods;

[0010] Constructing a physical plan corresponding to the logical plan, wherein each target data bucket in the physical plan includes a corresponding query task, each query task includes a storage path of a target data file determined based on the database configuration information and a bucket identifier of the corresponding target data bucket and its query timestamp, and the storage path carries a protocol type identifier of the corresponding target distributed file system;

[0011] According to the protocol type identifier in the storage path of each query task in the physical plan, the corresponding query task is executed for the target data file pointed to by the corresponding storage path through the client of the corresponding target distributed file system, and the query results of each query task are constructed as query response information.

[0012] According to another aspect of the present application, a device for accessing data between heterogeneous file systems is provided, comprising:

[0013] a logic parsing module configured to receive a query statement, parse and generate a logic plan corresponding to the query statement, and determine from the logic plan the bucket identifiers of target data buckets in the database accessed by the query statement and a query timestamp for each target data bucket;

[0014] A configuration query module is configured to query metadata information, determine a target distributed file system with a storage period that matches the target data bucket based on a query timestamp of the target data bucket, and obtain database configuration information corresponding to the target distributed file system storing the database, wherein the protocol types of the distributed file systems are different from each other and the data files of the data buckets generated by the database within the corresponding storage period are stored respectively according to different storage periods;

[0015] a task construction module configured to construct a physical plan corresponding to the logical plan, wherein each target data bucket in the physical plan includes a corresponding query task, each query task includes a storage path of a target data file determined based on the database configuration information and a bucket identifier of the corresponding target data bucket and its query timestamp, and the storage path carries a protocol type identifier of the corresponding target distributed file system;

[0016] The task execution module is configured to execute the corresponding query task for the target data file pointed to by the corresponding storage path through the client of the corresponding target distributed file system according to the protocol type identifier in the storage path of each query task in the physical plan, and construct the query results of each query task into query response information.

[0017] According to another aspect of the present application, a device for accessing data between heterogeneous file systems is provided, comprising a central processing unit and a memory, wherein the central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method described in the present application.

[0018] According to another aspect of the present application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the data access method between heterogeneous file systems in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the method are executed.

[0019] According to another aspect of the present application, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method when executed by a processor.

[0020] The present application effectively solves the technical problems faced by traditional data migration methods and achieves significant technical advantages. First, the present application implements the diversion access of incremental query statements in the database based on the correspondence between query timestamps and storage periods between multiple heterogeneous distributed file systems, which not only ensures the smooth access of cross-system data, but also realizes the dynamic migration of data. Secondly, the incremental migration method of the present application only migrates incremental data, avoiding various anomalies that may occur in full migration, thereby ensuring the security and reliability of the data. Furthermore, the present application achieves load balancing when storing data in the same database by reasonably allocating data to different distributed file systems, thereby improving the operating efficiency and stability of the system. In addition, the present application reduces the bandwidth consumption and time cost of data migration by only migrating data files generated or modified since the last migration timestamp, reduces the implementation cost of data migration, and improves the economy and feasibility of data migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This application is an exemplary network architecture that can be used to deploy and implement product search services;

[0022] Figure 2 A flowchart of an embodiment of a method for accessing data between heterogeneous file systems of the present application is provided;

[0023] Figure 3 This is a principle block diagram of the device for accessing data between heterogeneous file systems of the present application;

[0024] Figure 4 This is a structural diagram of a data access device between heterogeneous file systems used in this application. DETAILED DESCRIPTION

[0025] like Figure 1 As shown, the exemplary network architecture of the present application includes a front-end server 81 and a terminal device 80. The front-end server can be a data query server in the service cluster of the e-commerce platform, and undertakes the core function of providing data query services to the terminal devices. The front-end server 81 runs a computer program product implemented according to the data access method between heterogeneous file systems of the present application. The program product implements a distributed data engine based on the technical solution of the present application, provides data query services, can process data query requests carrying query statements from terminal devices, execute corresponding query tasks to obtain corresponding query results, and construct query response information with the query results to respond to data query requests. It should be pointed out that the terminal device 80 can be either a user terminal or a server with relatively independent functions, such as a middle-end server in an e-commerce platform.

[0026] Furthermore, multiple distributed file systems can be connected to the front-end server 81. These distributed file systems can use the same protocol type or different protocol types. When the distributed file systems use different protocol types, a larger heterogeneous file system network architecture is provided as a whole, which can store data generated by the same database at different stages.

[0027] Distributed file systems and distributed data engines are key technologies for processing large-scale data. Distributed file systems, such as the Hadoop Distributed File System (HDFS) and the Google File System (GFS), achieve distributed data storage by partitioning data into shards and storing them across multiple nodes. This sharding technology not only improves data storage efficiency but also enhances the system's fault tolerance through data replication.

[0028] Distributed data engines, such as Apache Spark, further enhance the efficiency and flexibility of data processing. Spark, through its core abstraction of Resilient Distributed Datasets (RDDs), allows for efficient in-memory processing of data, significantly increasing data processing speed. Furthermore, Spark's DAG scheduler divides tasks into multiple stages based on the dependencies between RDDs and efficiently distributes them to various nodes in the cluster for execution.

[0029] In this network architecture built by a distributed file system and a distributed data engine, data warehouse tools can be used to build metadata information that stores database configuration information. By providing a metadata information management system and corresponding management interfaces or interfaces, various accesses to the database configuration information in the metadata information can be achieved.

[0030] In practical applications, such as e-commerce platforms or live streaming platforms, front-end servers are typically responsible for providing data query services to terminal devices. These front-end servers can be deployed in a service cluster and run a computer program product implemented according to the data access method between heterogeneous file systems disclosed herein to provide data query services. This program product can process data query requests carrying data query statements from terminal devices and return accurate query response information.

[0031] For example, on an e-commerce platform, the front-end server can quickly respond to user queries, such as searching for products or checking order status, by working in conjunction with the distributed file system and distributed data engine to provide efficient data query services. In this network architecture, the distributed file system is responsible for storing large amounts of product information and user data, while the distributed data engine is responsible for processing and analyzing this data to support fast query responses.

[0032] This architecture allows e-commerce platforms to quickly process user queries and provide real-time product information and order status, enhancing the user experience. Live streaming platforms can also leverage this architecture to quickly process audience interaction requests, such as comments or likes, ensuring smooth and interactive live streaming through the efficient processing capabilities of the distributed data engine.

[0033] The technical solution of this application combines the powerful functions of distributed file systems and distributed data engines to achieve balanced dynamic distributed storage of incremental data files in the database in multiple distributed file systems, realize dynamic, safe and efficient migration of incremental data files between heterogeneous file systems, and provide efficient and reliable data query services for application scenarios such as e-commerce platforms and online live broadcast platforms.

[0034] See also Figure 2According to a data access method between heterogeneous file systems provided by this application, a computer program product can be installed and run on a server to centrally provide data query services. In some embodiments, the method includes the following steps:

[0035] Step S3100: Receive a query statement, parse and generate a logical plan corresponding to the query statement, and determine from the logical plan the bucket identifiers of target data buckets in the database accessed by the query statement and the query timestamp of each target data bucket;

[0036] The query statements of this application support data operations using Structured Query Language (SQL) or SQL-like syntax. Query statements can include any one or any combination of data query operations (SELECT), data write operations (INSERT / UPDATE), data delete operations (DELETE), and metadata operations (SHOW / CREATE). These operations are processed by a unified syntax parser. Its syntax structure includes an operation type identifier, a target database identifier, a data bucket identifier, a field selection list, and a conditional expression. The conditional expression can include a query timestamp for specifying the target data file.

[0037] Taking the e-commerce order management system as an example, in the query statement "SELECT order_id FROM orders WHEREcreate_time>'2023-01-01'", "orders" is the bucket identifier of the data bucket, and "create_time>'2023-01-01'" constitutes the time filter condition. This conditional expression is parsed as the basis for generating the query timestamp.

[0038] The distributed file system of the present application uses the database as a logical container for data organization. Each database can contain several data buckets, which are logical units for storing similar data files. Data buckets are identified by globally unique bucket identifiers, and each data bucket contains data files named and sorted by creation timestamps (such as "orders / 20230101_000000.parquet"). In a heterogeneous file system architecture, different data buckets of the same database can be stored in multiple distributed file systems with different protocol types (such as HDFS and OSS), but logical consistency is maintained through the database configuration information in the metadata information.

[0039] When the distributed data engine of the present application receives a data query request, it extracts a corresponding query statement from the data query request and then converts the query statement into a logical plan.

[0040] To this end, the query statement is first converted into an abstract syntax tree (AST) by a parser. The node structure of the AST consists of operation type nodes, data source nodes, and predicate nodes. For example, in the AST generated by the statement "INSERT INTO orders VALUES(...) WHERE create_date = '2023-02-01'," the root node is the INSERT operation type identifier, the left child node contains the database identifier "default" and the data bucket identifier "orders," and the predicate node in the right child tree parses the time filter condition "create_date = '2023-02-01'." The logical plan generation module converts this AST into a structured logical plan. This logical plan contains at least the operation type field, the bucket identifier list of the target data bucket, the field mapping relationship, and the query timestamp range generated by the time predicate conversion.

[0041] When extracting the target bucket's bucket ID from the logical plan, different extraction strategies are used depending on the operation type: for direct operations (such as SELECT * FROM orders), the explicitly declared bucket ID is obtained directly from the data source node; for indirect operations (such as JOIN queries), the bucket ID of the associated bucket is resolved using metadata information. Query timestamps are generated by parsing time comparison expressions in predicate nodes, including exact timestamps (such as "timestamp = '2023-01-01 00:00:00'"), time ranges (such as "create_time BETWEEN '2023-01-01' AND '2023-01-31'"), and time functions (such as "date_trunc('month',create_time)"). In the order management scenario, the condition "create_time > '2023-01-01'" is converted to the query timestamp lower limit of 1654041600 (Unix timestamp format) for subsequent matching against the storage period in the distributed file system.

[0042] It can be seen from this that a query statement may involve one or more data buckets to execute a query, depending on the computational complexity of the query statement itself. By converting the query statement into a logical plan, the bucket identifier of the target data bucket required by the query statement and the query timestamp applicable to each target data bucket can be directly determined based on the structured information of the logical plan.

[0043] Step S3200: query metadata information, determine a target distributed file system that matches the storage period of each target data bucket based on the query timestamp of each target data bucket, and obtain database configuration information corresponding to the target distributed file system storing the database. The protocol types of the distributed file systems are different, and the data files of the data buckets generated by the database within the corresponding storage period are stored according to different storage periods.

[0044] Metadata information serves as a coordination hub between heterogeneous file systems. It can organize database configuration information in a hierarchical structure and can include two core parts: storage topology description and time period allocation strategy. The storage topology description records the distributed file system instances involved in the storage and their protocol type identifiers. The time period allocation strategy defines the range of storage time periods that each file system instance is responsible for. In one embodiment, the metadata information is stored in a key-value database, where the key is a combination of a database identifier and a data bucket identifier, and the value is a JSON structure containing a storage path mapping table and a time period distribution table. In another embodiment, the metadata information can be stored in a relational database table, and the relationship between the database, data bucket, and file system instance is maintained through foreign key associations.

[0045] When it is necessary to prepare to migrate the incremental data files of the database in the distributed file system, that is, the source file system, to the new distributed file system, that is, the target file system, this can be achieved by pre-configuring the metadata information, which can be implemented in a variety of ways. In one embodiment, first register the new distributed file system instance in the metadata information, and record its protocol type identifier and connection parameters. Then create a time period allocation strategy, and use the migration timestamp as the dividing point between the storage periods of the old and new distributed file systems. Finally, generate storage path mapping rules to define the naming conventions of data files in the new file system. In another embodiment, the transition can be further combined with a dual-write mechanism. Before the arrival of the predetermined migration timestamp, the newly written data is saved in both the source file system and the target file system to ensure the continuity of query operations during the data migration process.

[0046] The heterogeneous file system of this application, that is, it contains multiple distributed file systems using different protocols. Therefore, the protocol types of each distributed file system are different from each other, which means that each file system instance recorded in the metadata information uses an exclusive communication protocol and data access interface. For example, HDFS uses the hdfs: / / protocol and accesses using the Hadoop FileSystem API, OSS uses the oss: / / protocol and accesses using the Alibaba Cloud OSS SDK, and S3 uses the s3: / / protocol and accesses using the AWS S3 client; these file system instances store database data according to the pre-divided time period range, and each file system is only responsible for storing the data files generated within a specific time period. For example, the first file system stores data from January to February, and the second file system stores data after February. By strictly dividing the time period, it is ensured that the data stored in different file system instances does not overlap in the time dimension. This design enables the system to quickly locate the file system instance containing the target data according to the query timestamp, and at the same time supports the horizontal expansion of the storage system and data migration by time period.

[0047] Accordingly, determining the matching distributed file system according to the query timestamp of the target data bucket can be implemented by applying a time interval matching algorithm. In one embodiment, the algorithm first locates the time period allocation policy table corresponding to the target data bucket in the metadata information, and then uses binary search to determine the storage period to which the query timestamp belongs. The time interval comparison adopts the left-closed and right-open principle, and it is determined that the match is successful when the query timestamp t satisfies start ≤ t < end. In another embodiment, to improve the query efficiency of a large-scale time period table, a skip list data structure is used to optimize the search process, and the matching speed is improved by trading space for time. For queries without explicitly specifying time conditions, the current system timestamp is defaultly used as the query timestamp to ensure that the file system instance where the latest data is located can always be located. Thus, according to the query timestamp of each target data bucket, the target distributed file system where this target data bucket is located can be determined.

[0048] After determining the target distributed file system corresponding to each target data bucket, the database configuration information of this target distributed file system can be obtained from the metadata information, specifically from the storage topology description in the metadata information. In one embodiment, the version number of the database configuration information of each distributed file system is maintained in the metadata information in advance, and a configuration snapshot of the latest version of the data configuration information is obtained before the query execution. In another embodiment, a multi-level cache can be used to accelerate the access to the configuration information. The local cache stores the recently used database configuration information, and when the cache fails, the latest configuration is obtained from the central metadata storage.

[0049] Step S3300: Construct a physical plan corresponding to the logical plan. In the physical plan, each target data bucket includes a corresponding query task. Each query task includes a storage path of a target data file determined based on the database configuration information, the bucket identifier of the corresponding target data bucket, and its query timestamp. The storage path carries a protocol type identifier of the corresponding target distributed file system.

[0050] After determining the bucket identifiers of the target data buckets in the database accessed by the query statement, the query timestamp for each target data bucket, and the database configuration information for the target distributed file system where each target data bucket resides, we can proceed to construct a physical plan corresponding to the logical plan. This physical plan construction process takes into account multiple factors, including the protocol type of the target distributed file system, the storage path of the data files, and the specific execution details of the query task.

[0051] For each target data bucket, the storage path of the target data file is determined based on its bucket ID and query timestamp, combined with the database configuration information obtained from the metadata. This storage path not only includes the specific location of the data file but also carries the protocol type identifier of the corresponding target distributed file system. For example, if the target distributed file system is OSS, the storage path might be similar to "oss: / / bucket_name / path / to / datafile.parquet", where "oss: / / " is the protocol type identifier, indicating that the data file is stored in the OSS system.

[0052] Next, the physical plan includes a corresponding query task for each target data bucket. Each query task contains specific operation instructions based on the storage path and protocol type identified above. These operation instructions instruct the distributed file system client on how to access and process the target data files. For example, for a data query operation, the query task will instruct the client to read the data file from the specified storage path and return the query results; for a data write operation, the query task will instruct the client to write the data to the specified storage path.

[0053] When constructing a physical plan, you also need to consider the query execution order and resource allocation. In a distributed environment, multiple query tasks may execute simultaneously, so the execution order of the tasks must be properly arranged to avoid resource contention and conflicts. Furthermore, appropriate computing and storage resources must be allocated based on the query complexity and data volume to ensure efficient query execution.

[0054] When constructing a physical plan, you should also consider data consistency and integrity. In particular, when writing data, you need to ensure that the data is written and stored correctly. This may involve technical measures such as data validation, error handling, and retry mechanisms to ensure the security and reliability of data in the distributed file system.

[0055] Step S3400: According to the protocol type identifier in the storage path of each query task in the physical plan, the corresponding query task is executed for the target data file pointed to by the corresponding storage path through the client of the corresponding target distributed file system, and the query results of each query task are constructed as query response information.

[0056] After constructing the physical plan, the specific operation instructions and target data file storage paths for each query task are defined. These storage paths carry the protocol type identifier of the corresponding target distributed file system, such as "hdfs: / / ," "oss: / / ," or "s3: / / ." These protocol type identifiers are used to select the appropriate client to execute the query task.

[0057] For each query task, the corresponding target distributed file system client is called based on the protocol type identified in the storage path. For example, if the storage path begins with "oss: / / ," the system calls the OSS client; if it begins with "hdfs: / / ," the system calls the HDFS client. These clients are preconfigured to handle data access requests of specific protocol types.

[0058] After receiving a query task, each client accesses the corresponding target data file based on the storage path specified in the query task. For data query operations, the client reads the data file from the specified storage path and returns the data to the query execution module. For data write operations, the client writes the data to the specified storage path. During query execution, the client handles various possible exceptions, such as network errors, permission issues, or data format errors, and performs retries or error reporting as needed.

[0059] After the query task is completed, the client returns the query results. The query results of all query tasks are collected and constructed into a query response message. The query response message typically contains the query result data content, execution status information, and possible error information. This information is sent back to the query requester, such as the user terminal or the middleware server.

[0060] As can be seen from the above embodiments, compared with the existing technology, this application effectively solves the technical problems faced by traditional data migration methods and achieves significant technical advantages, including but not limited to:

[0061] First of all, this application realizes the diversion of access to query statements acting on the database between multiple heterogeneous distributed file systems, based on the correspondence between the query timestamp and the storage period corresponding to each distributed file system. This not only ensures smooth cross-system data access, but also enables dynamic data migration. Specifically, by receiving the query statement and parsing to generate a logical plan, the bucket identifier of the target data bucket in the database accessed by the query statement and the query timestamp of each target data bucket are determined, and then the metadata information is queried. According to the query timestamp, the target distributed file system with a storage period that matches it is determined, so that the flow direction of the data targeted by the query statement can be accurately identified, and data storage and access can be realized in the corresponding distributed file system. In the case where the query statement contains data write operations, the dynamic migration of incremental data across file systems is actually realized, effectively avoiding the various problems caused by the full migration in the traditional migration method.

[0062] Secondly, the incremental migration method implemented in this application only needs to migrate the incrementally generated data, and will not generate various anomalies corresponding to the full migration, which can ensure data security. In the traditional migration method, since the data is generated on a daily or hourly basis, if the execution time of the migration task crosses the boundary of the day or hour, the task may fail, or the copied data is no longer consistent with the original data. However, this application constructs a physical plan, and the physical plan contains corresponding query tasks for each target data bucket. Each query task determines the storage path of the target data file based on the database configuration information and the bucket identifier of the target data bucket and its query timestamp, and executes the query task through the client of the corresponding distributed file system. This migration method based on incremental data avoids the risk of data loss or damage that may occur in the full migration, and ensures the security and reliability of data migration.

[0063] Furthermore, the present application realizes load balancing when storing the same database data among multiple distributed file systems. Since the protocol types of each distributed file system are different from each other, and the data files of the data buckets generated by the database in the corresponding storage period are stored according to different storage time periods, the present application can reasonably distribute the data to different distributed file systems according to the query timestamp and storage time period of each target data bucket. In this way, not only can the storage resources of each distributed file system be fully utilized, but also the performance bottleneck caused by the excessive load of a single file system can be effectively avoided, load balancing between multiple distributed file systems is achieved, and the operating efficiency and stability of the entire system are improved.

[0064] In addition, this application can significantly reduce the implementation cost of data migration. Traditional data migration methods have extremely high bandwidth requirements when facing cross-computer room and cross-regional migration scenarios, which undoubtedly increases the cost and difficulty of migration. However, this application uses incremental migration to only migrate data files generated or modified since the last migration timestamp, reducing bandwidth consumption and time costs for data migration. At the same time, since various abnormal situations that may occur in full migration are avoided, the cost of repeated migration caused by migration failure is also reduced, thereby significantly reducing the implementation cost of data migration as a whole and improving the economy and feasibility of data migration.

[0065] Based on any embodiment of the method of the present application, constructing a physical plan corresponding to the logical plan includes:

[0066] Step S3410: Determine the bucket identifier and query timestamp of the target data bucket in the query statement based on the logical plan, compare the query timestamp with the storage time periods of each distributed file system, and determine whether the storage time period matches the target distributed file system and its protocol type identifier corresponding to the query timestamp.

[0067] As previously mentioned, after a query statement is parsed and a logical plan is generated, the logical plan contains key query information, such as the operation type, the bucket identifier of the target data bucket, the field selection list, and the conditional expression. The conditional expression can include a time filter, such as create_time > '2023-01-01', which is parsed as the basis for generating the query timestamp.

[0068] Taking an e-commerce order management system as an example, assume the query is SELECT order_id FROM orders WHERE create_time > '2023-01-01'. In this example, "orders" is the bucket ID of the target data bucket, and "create_time > '2023-01-01'" is the time filter. By parsing this conditional expression, the query timestamp can be determined.

[0069] After determining the bucket identifier and query timestamp of the target data bucket, the query timestamp of each target data bucket is compared with the storage time period of each distributed file system to determine the target distributed file system whose storage time period matches the query timestamp. In this application, the metadata information records the storage time period range responsible for each distributed file system. For example, the time period distribution table may record that HDFS stores data from December 2022, while the OSS system stores data from January 2023. For the query statement in the previous example, the OSS system is its corresponding target distributed file system.

[0070] After determining the target distributed file system, you can obtain the protocol type identifier for that file system. In this application, the protocol types of each distributed file system are different. For example, HDFS uses the "hdfs: / / " protocol, OSS uses the "oss: / / " protocol, and S3 uses the "s3: / / " protocol. These protocol type identifiers are used to access the target distributed file system through the corresponding client.

[0071] Step S3420: Determine, based on the logical plan, whether the data operation on the target data bucket includes a data write operation; if the data write operation is included, determine whether the target data bucket includes a target data file named with the query timestamp as a creation timestamp;

[0072] A data write operation is an operation that points to a new data record in the target data bucket, such as the INSERT INTO operation in an SQL statement. It usually points to the target data file more clearly and involves checking whether a new data file needs to be added. Therefore, it is also an operation type that needs to be considered to implement the dynamic migration of incremental data files in this application. In the logical plan, these operation types will be clearly identified. By parsing the operation type field in the logical plan, you can determine whether the current query statement contains a data write operation.

[0073] If the logical plan indicates that the query statement includes a data write operation, the system further checks whether a target data file exists in the target data bucket, with the query timestamp as its creation timestamp. This step ensures that the target data file exists when the data write operation is performed. If the target data file does not exist, the system proceeds to step S3430 to create a new target data file, named with the query timestamp as its creation timestamp.

[0074] If the target data file already exists or has been created, the process proceeds to step S3440 to directly construct a storage path and update the path to the database configuration information in the metadata information. In this way, subsequent data write operations will be able to correctly write data to the specified storage path.

[0075] Step S3430: When the target data file is not included, create a target data file in the target data bucket, and name the target data file with the query timestamp as the creation timestamp;

[0076] When the logical plan indicates that the query statement includes a data write operation, and the target data file named with the query timestamp as the creation timestamp is not found in the target data bucket, in order to ensure that the corresponding target data file already exists before the data write operation, a clear storage path is provided for subsequent data write operations.

[0077] Specifically, the target data file's file name is first generated based on the query timestamp. For example, if the query timestamp is 2023-01-01 00:00:00, the target data file's file name might be "20230101_000000.parquet." This naming method not only clearly identifies the file's creation time but also facilitates subsequent file management and query operations.

[0078] Next, the target data file is created in the target data bucket. This creation process involves allocating storage space in the target distributed file system and initializing the file's metadata. For example, in an OSS system, creating a file might involve calling the OSS client API, specifying parameters such as the bucket name, file path, and file name.

[0079] After creating the target data file, update the file's storage path to the database configuration information in the metadata to ensure the accuracy and consistency of the metadata, so that subsequent queries and management operations can correctly locate the target data file. For example, the updated storage path will be similar to "oss: / / bucket_name / path / to / 20230101_000000.parquet".

[0080] In practice, the specific implementation for creating target data files can vary depending on the distributed file system. For example, in HDFS, you might need to use the Hadoop FileSystem API to create a file; in OSS, you might need to use the Alibaba Cloud OSS SDK; and in S3, you might need to use the AWS S3 client. Although the specific API calls and parameter settings may vary, the core logic remains the same: generating a file name based on the query timestamp and creating the corresponding data file in the target data bucket.

[0081] Step S3440: When a target data file is included, the protocol type identifier of the target distributed file system, the bucket identifier of the target data bucket and the creation timestamp are combined to form the storage path of the target data file, and the database configuration information in the metadata information is updated.

[0082] If the logical plan indicates that a query statement includes a data write operation, and the target data file, named with the query timestamp as its creation timestamp, already exists in the target data bucket, you must ensure that subsequent data write operations correctly write the data to the specified storage path. To do this, first obtain the protocol type identifier of the target distributed file system, such as "oss: / / " or "hdfs: / / ." These protocol type identifiers are key distinguishing features between distributed file systems and are used to access the target distributed file system through the corresponding client.

[0083] Next, the target data bucket's bucket ID and creation timestamp are combined to generate the complete storage path for the target data file. For example, if the target distributed file system is OSS, the target data bucket's bucket ID is "orders," and the query timestamp is 2023-01-01 00:00:00, the generated storage path might be "oss: / / bucket_name / path / to / orders / 20230101_000000.parquet." This path not only includes the file's specific location but also carries the protocol type identifier, clearly indicating which distributed file system the file is stored in.

[0084] Finally, this storage path is updated in the database configuration information within the metadata. Metadata is key to coordination between heterogeneous file systems. It records database configuration information, including the storage location and access methods of data files. Updating metadata ensures that subsequent queries and management operations can correctly locate the target data files. For example, in subsequent queries, the target data files can be quickly found and read based on the storage path in the metadata.

[0085] In actual applications, different distributed file systems may have different storage path formats and naming conventions. For example, in HDFS, the path may be similar to:

[0086] "hdfs: / / namenode_address / user / hive / warehouse / orders / 20230101_000000.parquet";

[0087] In S3, the path might look like:

[0088] "s3: / / bucket_name / path / to / orders / 20230101_000000.parquet".

[0089] Although the specific path formats may be different, the core logic is the same, that is, by combining the protocol type identifier, bucket identifier and creation timestamp, the storage path of the target data file is generated and updated to the metadata information.

[0090] Through the above embodiments, the present application has achieved significant technical advantages in data access and management between heterogeneous file systems. First, by accurately matching the query timestamp with the storage period of the distributed file system, the accuracy and efficiency of data access are ensured, unnecessary data scanning and processing are avoided, and query performance is significantly improved. Secondly, by dynamically identifying and creating target data files, the present application can flexibly respond to data write operations, ensure the correctness and integrity of data writes, and support dynamic migration of incremental data, avoiding resource waste and performance bottlenecks caused by full migration. In addition, by updating the storage path in the metadata information, the present application realizes real-time tracking and management of the storage location of data files, enhances scalability and flexibility, and enables the distributed file system to dynamically adjust according to changes in data growth and access patterns. These technical effects work together to not only improve the efficiency of data access and management, but also enhance the stability and reliability of the entire network architecture, providing a more efficient and flexible solution for data processing in a big data environment.

[0091] Based on any embodiment of the method of the present application, after determining whether the data operation on the target data bucket includes a data write operation according to the logical plan, the method includes:

[0092] Step S3450: When no data write operation is included, screen and determine the target data files in the target data bucket whose creation timestamp matches the query timestamp corresponding to the target data bucket;

[0093] When step S3430 determines that the logical plan indicates that the query statement does not include a data write operation, it may be an operation type such as a data read operation or a data update operation that does not involve creating a new data file. In this case, the steps of this embodiment will be executed.

[0094] Specifically, this step is responsible for filtering and identifying target data files in the target data bucket whose creation timestamp matches the query timestamp corresponding to that target data bucket. This process is implemented based on the time filter conditions that may be included in the query statement, such as "create_time>'2023-01-01'". This conditional expression is parsed as the basis for generating the query timestamp. By comparing the creation timestamp of each data file in the target data bucket with the query timestamp, data files that meet the conditions are filtered out. The creation timestamps of these data files match the query timestamp, indicating that they are the target data files of the query operation.

[0095] Step S3460: Combine the protocol type identifier of the target distributed file system, the bucket identifier of the target data bucket, and the creation timestamp of each target data file to form a corresponding storage path, and update the database configuration information in the metadata information.

[0096] After determining the target data files that meet the conditions, continue to combine the protocol type identifier of the target distributed file system, the bucket identifier of the target data bucket, and the creation timestamp of each target data file through this step to form the corresponding storage path. The implementation method is the same as step S3440 and will not be repeated here.

[0097] The storage path of each target data file is then updated to the database configuration information in the metadata. Metadata serves as the coordination hub between heterogeneous file systems, recording database configuration information, including the data file's storage location and access methods. By updating metadata, subsequent queries and management operations can accurately locate the target data file.

[0098] Through the above embodiments, the present application realizes accurate query of query statements between heterogeneous file systems, significantly improving the efficiency and accuracy of data access. Specifically, when the query statement does not contain a data write operation, by screening and determining the data files whose creation timestamps in the target data bucket match the query timestamp, it ensures that the query operation can accurately locate the required target data files, avoiding unnecessary data scanning and processing, thereby significantly improving query performance. Furthermore, by combining the protocol type identifier of the target distributed file system, the bucket identifier of the target data bucket, and the creation timestamp of the data file to form a storage path, and updating these paths to the metadata information, it is not only possible to accurately locate the data file, but also to flexibly support distributed file systems of different protocol types. This mechanism enables the system to efficiently access and manage data between heterogeneous file systems, enhances the scalability and flexibility of the system, and provides strong support for cross-system data queries.

[0099] Based on any embodiment of the method of the present application, before querying metadata information, the method includes:

[0100] Step S2100: Determine a migration timestamp and database configuration information of the database in the first type distributed file system in the metadata information, wherein the migration timestamp is used to divide a storage period between the first type distributed file system and the second type distributed file system;

[0101] This embodiment pre-deploys a heterogeneous file system consisting of a first type of distributed file system and a second type of distributed file system. The first type of distributed file system (e.g., HDFS) and the second type of distributed file system (e.g., OSS) have different protocol types and storage characteristics. To enable data migration between these heterogeneous systems, a migration timestamp can be manually or automatically determined in advance. This timestamp is used to demarcate the storage period between the first type of distributed file system and the second type of distributed file system.

[0102] The migration timestamp is a key time identifier for implementing heterogeneous file systems, used to distinguish the storage period of data in different distributed file systems. For example, the migration timestamp can be a specific point in time, such as January 1, 2023, 00:00:00, indicating that from this point in time, data will be migrated from the first type of distributed file system to the second type of distributed file system.

[0103] The relevant configuration of the first type of distributed file system is already present in the metadata information. Therefore, the database configuration information of the database in the first type of distributed file system in the metadata information is obtained. The database configuration information includes key information such as the database storage structure, data bucket organization, and data file storage path. This information is the basis for data migration and access, ensuring accurate data location and management.

[0104] Step S2200: Deploy the database in the second type distributed file system according to the storage structure information corresponding to the first type distributed file system in the database configuration information, and create the first data file of each data bucket of the database. The first data file is named with the migration timestamp as the creation timestamp.

[0105] The database is deployed in the second type of distributed file system based on the storage structure information in the first type of distributed file system. Specifically, the database structure and data buckets corresponding to the second type of distributed file system can be created based on the database configuration information in the first type of distributed file system, including the database storage structure, data bucket organization, and data file storage paths.

[0106] After creating the database and data buckets in the second-type distributed file system, you can create the first data file for each data bucket. The creation timestamp of these first data files is set to the migration timestamp. For example, if the migration timestamp is January 1, 2023, 00:00:00, the file name of the first data file can be "20230101_000000.parquet". This naming method not only clearly identifies the file creation time, but also facilitates subsequent file management and query operations.

[0107] In practice, different distributed file systems may have different creation and deployment mechanisms. For example, in HDFS, you may need to use the Hadoop FileSystem API to create files and directories; in OSS, you may need to use the Alibaba Cloud OSS SDK to create buckets and files; and in S3, you may need to use the AWS S3 client to create buckets and files. Although the specific API calls and parameter settings may vary, the core logic is consistent: based on the storage structure information of the first type of distributed file system, the corresponding database and data buckets are created in the second type of distributed file system. The first data file is created for each data bucket, and the creation timestamp of these files is set to the migration timestamp.

[0108] Step S2300: Update and store the storage path of the database, its data bucket, and its first data file in the second type distributed file system into the database configuration information of the metadata information.

[0109] To update the metadata, combine the storage path of each data file, including the protocol type identifier, bucket identifier, and file name, into a complete storage path and store it in the metadata. For example, if the target distributed file system is OSS, the bucket identifier of the target data bucket is "orders," and the migration timestamp is 2023-01-01 00:00:00, the generated storage path might be "oss: / / bucket_name / path / to / orders / 20230101_000000.parquet." These storage paths are updated in the database configuration information for the second type of distributed file system in the metadata to facilitate subsequent access.

[0110] Through the above embodiments, the present application realizes efficient data migration and management between heterogeneous file systems, significantly improving the flexibility and scalability of the system. First, by determining the migration timestamp, the storage period between the first type distributed file system and the second type distributed file system can be clearly divided, ensuring that the migration of data between different systems has a clear time limit, thereby avoiding confusion and duplication of data storage. Secondly, based on the storage structure information of the first type distributed file system, the database is deployed in the second type distributed file system and a data bucket and its first data file are created, which not only ensures the consistency of the data structure, but also enhances the traceability and management efficiency of the data file by using the migration timestamp as the basis for file naming. Finally, the storage path of the database, its data bucket and the first data file in the second type distributed file system is updated to the metadata information, ensuring the accuracy and real-time nature of the metadata information, enabling rapid positioning and access to the target data file, and improving the efficiency and accuracy of data access. These technical effects work together to provide strong support for data migration and management across heterogeneous file systems, significantly improving the performance and reliability of the entire data processing system.

[0111] Based on any embodiment of the method of the present application, determining the database configuration information of the database in the first type distributed file system in the migration timestamp and metadata information includes:

[0112] Step S2110: Acquire performance monitoring data of the first type of distributed file system, and decide whether to trigger a migration preparation instruction based on the performance monitoring data;

[0113] A performance monitoring process is pre-run on the servers of a first-type distributed file system (e.g., HDFS). This process continuously monitors system performance and generates performance monitoring data. This data includes, but is not limited to, key performance indicators such as CPU usage, memory usage, disk I / O, and network bandwidth. Distributed data engine servers, such as front-end servers, can continuously access this performance monitoring data, providing real-time insights into the operational status of the first-type distributed file system.

[0114] In one embodiment, a comprehensive performance score can be derived by weighting specific performance indicators. For example, CPU utilization, memory utilization, disk I / O, and network bandwidth can be assigned different weights to calculate a single performance score. This score is then compared with a preset threshold. When the score exceeds the threshold, it indicates that system performance has reached a critical state, triggering migration preparation instructions.

[0115] In another embodiment, a pre-trained deep learning model can be used to determine whether the system performance of the first type of distributed file system is reaching a critical state. The deep learning model can be trained based on historical performance data to learn patterns and trends in system performance changes. During runtime, the model can analyze current performance monitoring data in real time to predict whether system performance is about to reach a critical state. If the model determines that system performance is reaching a critical state, migration preparation instructions are triggered.

[0116] Step S2120: In response to the migration preparation instruction, determine storage structure information corresponding to storing the database in the first type distributed file system from the database configuration information in the metadata information, convert the information into storage structure information adapted for the second type distributed file system, and extend the current timestamp corresponding to the triggering of the migration preparation instruction by a preset time to serve as the migration timestamp;

[0117] After receiving the migration preparation instruction, the distributed data engine first extracts the storage structure information of the database in the first type of distributed file system from the metadata. This information includes key configurations such as the database's table structure, indexes, partitioning strategy, and data file storage paths. This configuration information is the foundation for data migration and ensures that the data maintains its structural and semantic consistency during the migration process.

[0118] Next, this storage structure information is converted into storage structure information that is compatible with the second type of distributed file system. This conversion process can involve multiple aspects, such as adjusting the storage format of data files to adapt to the characteristics of the target file system, or redesigning indexing and partitioning strategies to optimize data storage and access performance in the target system. For example, if the first type of distributed file system is HDFS and the second type of distributed file system is OSS, then the block storage format in HDFS needs to be converted to the object storage format in OSS, and the data organization method needs to be adjusted to adapt to the OSS access mode.

[0119] At the same time, the current timestamp corresponding to the triggering migration preparation instruction is extended by a preset time to serve as the migration timestamp. This migration timestamp is a key time marker for data migration and is used to demarcate the storage period between the first type distributed file system and the second type distributed file system. The extension of the preset time can be based on various factors, such as the estimated migration preparation time and system load forecast, to ensure that the file system is ready for data migration when the migration timestamp arrives.

[0120] Step S2130: When the migration timestamp arrives, determine the comprehensive performance score of the first type of distributed file system based on the continuously acquired performance monitoring data; when the comprehensive performance score meets the preset conditions, determine the database configuration information corresponding to the migration timestamp and the first type of distributed file system.

[0121] After the migration preparation instruction is triggered, the performance monitoring data of the first type of distributed file system can be continuously obtained to perform a more detailed performance evaluation. By collecting these performance data at multiple time points, a comprehensive performance score reflecting long-term characteristics can be generated. This scoring method can be implemented in a variety of ways. For example, according to the embodiment given in step S2110, a single performance score is first determined for the performance monitoring data generated at each time point, and then the average or weighted average of all single performance scores since the migration preparation instruction was triggered is calculated to determine a comprehensive performance score.

[0122] In one embodiment, a weighted average calculation can be performed on each individual performance score according to the time sequence, with the weight values increasing from the smallest to the largest, to calculate a comprehensive performance score. This comprehensive performance score can not only reflect long-term characteristics but also highlight the recent status.

[0123] When the comprehensive performance score meets a preset condition, the migration is confirmed to be effective. This preset condition can be a specific score threshold. For example, when the comprehensive performance score exceeds a certain threshold, it indicates that system performance has reached a critical state and data migration is required. At this point, the data migration process can be initiated based on the migration timestamp and the database configuration information corresponding to the first type of distributed file system, executing steps S2200 and S2300.

[0124] Through the above embodiments, the present application introduces a dynamic evaluation mechanism based on performance monitoring data in the data migration decision-making process between heterogeneous file systems, which significantly improves the intelligence and automation level of data migration. First, after triggering the migration preparation instruction, not only does it make a preliminary decision based on the initial performance monitoring data, but it also continuously obtains performance monitoring data and calculates a comprehensive performance score to ensure that the migration decision can reflect the long-term performance characteristics of the system and avoid misjudgments caused by instantaneous performance fluctuations. Secondly, the comprehensive performance score is calculated using methods such as weighted averaging, which can highlight the recent performance status of the system and make the migration decision more in line with current actual needs. Only when the comprehensive performance score meets the preset conditions is the migration determined to be effective, and the data migration process is started accordingly, ensuring the timeliness and necessity of the migration operation, avoiding waste of resources and unnecessary migration operations. Finally, in this way, the present application can automatically start data migration according to the actual operating status of the file system, improve the system's adaptability and resource utilization efficiency, and provide strong support for efficient data migration between distributed file systems.

[0125] Based on any embodiment of the method of the present application, the method further includes:

[0126] Step S4100: In response to the metadata information update event, record the update timestamp of the corresponding data file;

[0127] Metadata, the core of data management across heterogeneous file systems, records database configuration information, including key information such as the data file's storage location, access methods, and creation timestamp. When metadata is updated, it typically signifies a change in the data file's status, such as a modification, update, or migration. To ensure system data consistency and traceability, these update events can be handled in a responsive manner.

[0128] Specifically, when metadata is updated, an update event is triggered. In response to this event, the corresponding data file's update timestamp is recorded. The update timestamp records the time when the data file was last modified or updated. This timestamp helps the system determine the latest state of the data file and perform rollback or recovery operations when necessary.

[0129] Step S4200: respond to the abnormal rollback instruction, obtain the rollback timestamp specified by the instruction, and locate the target data file with a matching update timestamp in the metadata information according to the rollback timestamp as the file to be revoked;

[0130] An exception rollback instruction can be triggered by an administrative user. Upon receiving an exception rollback instruction, the rollback timestamp specified in the instruction is first retrieved. The rollback timestamp identifies the data state to which the rollback should be applied. Based on this rollback timestamp, a matching data file is searched for in the metadata. The metadata records the update timestamp corresponding to the last modification of each data file, which identifies the last update time of the data file. By comparing the rollback timestamp with the update timestamp, the target data file to be revoked can be precisely located.

[0131] When locating the target data file, the storage path and timestamp information in the metadata information will be checked. If the update timestamp of a data file matches the rollback timestamp or is after the rollback timestamp, then the data file will be marked as a pending undo file in preparation for the undo operation.

[0132] Step S4300: In response to the revocation confirmation instruction, a rollback operation is performed on the file to be revoked, and database configuration information corresponding to the file to be revoked in the metadata information is updated.

[0133] The undo confirmation instruction can be triggered by the management user. When the undo confirmation instruction is received, the rollback operation is first performed on the file that has been marked to be revoked. The purpose of the rollback operation is to restore the data file to the state corresponding to the specified rollback timestamp. If the data file is managed through version control, the rollback operation can simply switch the file version to the state before the rollback timestamp; if the data file is managed through incremental backup, the rollback operation can apply the corresponding incremental backup file to restore the data. In actual applications, the specific implementation of the rollback operation will vary depending on the storage format and management mechanism of the data file. After completing the rollback operation, it is necessary to update the database configuration information corresponding to the file to be revoked in the metadata information. Specifically, this can include updating the corresponding update timestamp.

[0134] Through the above embodiments, the present application implements an efficient and reliable data rollback mechanism in data management between heterogeneous file systems, significantly improving the stability and data consistency of the system. First, by responding to the update event of the metadata information and recording the update timestamp of the data file, the latest status of the data file is tracked in real time, providing an accurate time reference for subsequent rollback or recovery operations. Then, when an abnormal rollback instruction is received, the target data file that needs to be revoked is accurately located according to the rollback timestamp, ensuring the accuracy and pertinence of the rollback operation. Finally, by executing the rollback operation and updating the database configuration information in the metadata information, not only can the data file be restored to the specified state, but the accuracy and consistency of the metadata information can also be maintained, and it can quickly recover to the state before the abnormality occurs, reducing data loss and system unavailability time, thereby enhancing the reliability and recoverability of the system.

[0135] Based on any embodiment of the method of the present application, the method further includes:

[0136] Step S5100: Periodically scan the metadata information to determine the data bucket in the distributed file system after the storage period in which the data file is last created in the database, and obtain the creation timestamp of the last created data file;

[0137] Metadata records database configuration information, including key information such as data file storage location, access methods, and creation timestamps. Regularly scanning this metadata allows us to identify the last data file created in the database and its corresponding bucket in the distributed file system, which was stored later in the storage period. This process can be automatically triggered at pre-set intervals (such as daily, weekly, or monthly) to ensure continuous monitoring of data file creation timestamps.

[0138] In real-world applications, distributed file systems may be located in different geographic locations or on different storage media, each with its own specific storage period. For example, one distributed file system might store data from the last month, while another might store data from earlier periods. By scanning metadata, it's possible to identify the file system corresponding to the last storage period and determine the bucket where the last data file was created. After identifying the last created data file and its bucket, the creation timestamp of that data file can be retrieved.

[0139] Step S5200: Determine whether the creation timestamp exceeds the preset retention period of the corresponding data bucket. If the retention period is exceeded, move all data files in the data bucket before the retention period to the recycle bin.

[0140] By periodically scanning metadata information to identify the last data file created in the database and the data bucket in which it is located in the distributed file system with a later storage period, and after obtaining the creation timestamp of the data file, the creation timestamp is compared with the preset retention period of the data bucket. The retention period can be a preset fixed time length, such as one week, a natural month, 30 days, 60 days, etc., or it can be dynamically set according to the length of time between the migration timestamps of two data migrations. If the creation timestamp of the data file plus the length of the retention period is less than the current timestamp, that is, the retention period of the data file has ended, all data files in the data bucket before the retention period can be marked as expired, indicating that these expired data files will be moved to the recycle bin.

[0141] In practice, the Recycle Bin can also be a dedicated storage area where expired data files in the data bucket are directly migrated to, temporarily storing data files that are about to be deleted. This way, while clearing expired data, it also provides an opportunity for possible data recovery operations, enhancing the flexibility and reliability of the system.

[0142] Step S5300: regularly clearing data files stored in the recycle bin that have exceeded a preset time period.

[0143] In order to ensure that data files in the Recycle Bin do not occupy storage space indefinitely, cleanup operations can be performed regularly. The frequency of the cleanup operation can be set according to actual needs, such as daily, weekly, or monthly. During the cleanup process, the storage time of each data file in the Recycle Bin will be checked. If the storage time of a data file exceeds the preset duration, the data file will be permanently deleted. The preset duration can be determined based on the availability of storage resources, the importance of the data, and business needs. For example, for some less important data files, the preset duration can be set to a shorter time, such as one week; and for some important data files, the preset duration can be set to a longer time, such as one month or longer.

[0144] In practice, cleanup can be implemented in a variety of ways. For example, you can use a scheduled task to periodically trigger a cleanup script that checks the storage age of each data file in the recycle bin and deletes those that have exceeded the preset age.

[0145] Through the above embodiments, the present application realizes efficient lifecycle management of data files in a distributed file system, significantly improving the resource utilization efficiency of the system and the reliability of data management. First, by regularly scanning metadata information and determining the last created data file and its data bucket, the creation timestamp of the data file can be continuously monitored to ensure real-time tracking of the latest status of the data file. Secondly, by judging whether the creation timestamp of the data file exceeds the preset retention period of its corresponding data bucket, the expired data file can be automatically identified and marked and moved to the recycle bin, which not only optimizes the use of storage resources, but also provides a buffer opportunity for data recovery. Finally, by regularly cleaning up data files stored in the recycle bin for more than a preset time, storage space can be effectively freed up, and expired data files can be prevented from occupying valuable storage resources for a long time, while ensuring the automation and efficiency of the data cleaning process. These technical effects work together to enable the system to maintain efficient operation while also having good flexibility and scalability, and can adapt to diverse scenarios with different business needs and data importance levels.

[0146] Based on any embodiment of the method of the present application, after constructing the query results of each query task into query response information, the following steps are included:

[0147] Step S6100: Push the query response information to the computer device that submitted the query statement. The computer device encapsulates the query response information in a preset format and encapsulates each data record in the query response information into an interface component. The data record includes order information from the e-commerce platform.

[0148] Query response information is a collection of query execution results in a distributed file system. These results are processed into structured data records. For example, in an e-commerce platform application scenario, the query response information contains key data such as order number, order status, order amount, and customer information. To effectively display these data records to users, the query response information needs to be pushed to the computer device that submitted the query statement, typically the user's terminal device or server.

[0149] After receiving the query response, the computer device encapsulates it according to a pre-set format. This encapsulation process involves converting the data record into a format suitable for display and embedding it into an interface component. An interface component is a part of the user interface that displays data records, such as tables, lists, or charts. These interface components can be customized based on different application scenarios and user needs to provide an optimal user experience.

[0150] Step S6200: The computer device embeds the interface component into the order management page of the e-commerce platform for display.

[0151] In e-commerce platform applications, interface components typically display key order data, such as order number, order status, order amount, and customer information. After encapsulating these data records into interface components, these components need to be embedded into the e-commerce platform's order management page. The order management page is part of the e-commerce platform's user interface and is used to display and manage all order information. By embedding interface components into the order management page, users can easily view and manipulate order information, improving work efficiency and user experience.

[0152] By combining this embodiment with other embodiments of this application that implement dynamic migration of incremental data files in heterogeneous file systems, this application can significantly improve the efficiency and flexibility of data processing when faced with massive dynamically generated order data in e-commerce scenarios, while ensuring the real-time and accuracy of the data. Specifically, this application ensures that the e-commerce platform can process and display the latest order information in a timely manner by dynamically migrating incremental data files without having to wait for the full data migration to be completed. This incremental migration method not only reduces the time and resource consumption of data migration, but also improves the system's response speed and user experience.

[0153] During the data migration process, this application ensures the accuracy and consistency of data migration through precise timestamp management and metadata update mechanisms. When new order data is generated, these incremental data files can be identified and migrated in a timely manner, avoiding the problems of data loss or duplicate migration that may occur in traditional full migration methods. In addition, by encapsulating query response information into interface components and embedding them into the order management page of the e-commerce platform, users can view the latest order information in real time, improving data accessibility and operational convenience.

[0154] Combined with the technical features of other embodiments, such as the dynamic evaluation of performance monitoring data and the abnormal fallback mechanism, this application can intelligently adjust the migration strategy based on system performance and data status, ensuring that data migration tasks can be completed efficiently and stably even when the system load is high or data is abnormal. This intelligent data management method not only improves the system's adaptability, but also enhances the system's reliability and stability, allowing e-commerce platforms to maintain efficient and stable operation when faced with massive amounts of dynamic data.

[0155] See also Figure 3 According to one aspect of the present application, a data access device between heterogeneous file systems is provided, including a logic parsing module 3100, a configuration query module 3200, a task construction module 3300, and a task execution module 3400, wherein the logic parsing module 3100 is configured to receive a query statement, parse and generate a logical plan corresponding to the query statement, and determine the bucket identifier of the target data bucket in the database accessed by the query statement and the query timestamp of each target data bucket from the logical plan; the configuration query module 3200 is configured to query metadata information, determine the target distributed file system with a storage period matching the query timestamp of each target data bucket, and obtain database configuration information corresponding to the target distributed file system storing the database, and the protocol types of the distributed file systems are different from each other, and are respectively based on Different storage time periods store data files of the data buckets generated by the database within the corresponding storage time periods; the task construction module 3300 is configured to construct a physical plan corresponding to the logical plan, and each target data bucket in the physical plan includes a corresponding query task, and each query task includes a storage path of the target data file determined based on the database configuration information and the bucket identifier of the corresponding target data bucket and its query timestamp, and the storage path carries the protocol type identifier of the corresponding target distributed file system; the task execution module 3400 is configured to execute the corresponding query task for the target data file pointed to by the corresponding storage path through the client of the corresponding target distributed file system according to the protocol type identifier in the storage path of each query task in the physical plan, and construct the query results of each query task into query response information.

[0156] Based on any embodiment of the device of the present application, the task execution module 3400 includes: a system determination module, which is configured to determine the bucket identifier and query timestamp of the target data bucket in the query statement based on the logical plan, compare the query timestamp with the storage period of each distributed file system, and determine that the storage period matches the target distributed file system and its protocol type identifier corresponding to the query timestamp; a write judgment module, which is configured to judge whether the data operation of the target data bucket includes a data write operation according to the logical plan, and when it includes a data write operation, judge whether the target data bucket includes a target data file named with the query timestamp as the creation timestamp; a new creation module, which is configured to create a target data file in the target data bucket when the target data file is not included, and the target data file is named with the query timestamp as the creation timestamp; a file determination module, which is configured to combine the protocol type identifier of the target distributed file system, the bucket identifier of the target data bucket and the creation timestamp to form the storage path of the target data file when the target data file is included, and update the database configuration information in the metadata information.

[0157] Based on any embodiment of the device of the present application, the task execution module 3400 also includes: a non-write matching module, which is configured to screen and determine the target data files in the target data bucket whose creation timestamps match the query timestamps corresponding to the target data bucket when no data write operation is included; a path construction module, which is configured to combine the protocol type identifier of the target distributed file system, the bucket identifier of the target data bucket and the creation timestamps of each target data file to form a corresponding storage path, and update the database configuration information in the metadata information.

[0158] On the basis of any embodiment of the device of the present application, prior to the configuration query module 3200, it includes: an information determination module, configured to determine the migration timestamp and database configuration information of the database in the first type distributed file system in the metadata information, the migration timestamp is used to divide the storage period between the first type distributed file system and the second type distributed file system; a deployment processing module, configured to deploy the database in the second type distributed file system according to the storage structure information corresponding to the database in the first type distributed file system in the database configuration information, and create the first data file of each data bucket of the database, the first data file is named with the migration timestamp as the creation timestamp; an information update module, configured to update the storage path of the database, its data bucket and its first data file in the second type distributed file system to the database configuration information of the metadata information.

[0159] Based on any embodiment of the device of the present application, the information determination module includes: a trigger prediction module, configured to obtain performance monitoring data of a first type of distributed file system, and decide whether to trigger a migration preparation instruction based on the performance monitoring data; a migration preparation module, configured to respond to the migration preparation instruction, determine the storage structure information corresponding to the database stored in the first type of distributed file system from the database configuration information in the metadata information, convert it into storage structure information adapted to the second type of distributed file system, and extend the current timestamp corresponding to the triggering of the migration preparation instruction by a preset time as a migration timestamp; a decision start module, configured to determine a comprehensive performance score of the first type of distributed file system based on the performance monitoring data continuously obtained during the period when the migration timestamp arrives, and when the comprehensive performance score meets a preset condition, determine the database configuration information corresponding to the migration timestamp and the first type of distributed file system.

[0160] Based on any embodiment of the device of the present application, the device also includes: an update response module, configured to respond to the update event of the metadata information and record the update timestamp of the corresponding data file; an exception response module, configured to respond to the exception rollback instruction, obtain the rollback timestamp specified by the instruction, and locate the target data file whose update timestamp matches the rollback timestamp in the metadata information according to the rollback timestamp as the file to be revoked; a revocation response module, configured to respond to the revocation confirmation instruction, perform a rollback operation on the file to be revoked, and update the database configuration information corresponding to the file to be revoked in the metadata information.

[0161] Based on any embodiment of the device of the present application, the device also includes: a latest determination module, which is configured to periodically scan the metadata information, determine the data bucket in which the data file was last created in the distributed file system after the storage period, and obtain the creation timestamp of the last created data file; an overdue cleaning module, which is configured to determine whether the creation timestamp exceeds the preset retention period of its corresponding data bucket, and when the retention period is exceeded, move all data files in the data bucket before the retention period to a recycle bin; a permanent deletion module, which is configured to periodically clean up data files stored in the recycle bin that have exceeded a preset time length.

[0162] On the basis of any embodiment of the device of the present application, after the task execution module 3400, it includes: a push processing module, configured to push the query response information to the computer device that submits the query statement, and the computer device encapsulates it according to a preset format, and encapsulates each data record in the query response information into an interface component, wherein the data record includes the order information of the e-commerce platform; an output display module, configured to embed the interface component into the order management page of the e-commerce platform for display by the computer device.

[0163] Another embodiment of the present application also provides a data access device between heterogeneous file systems. Figure 4 FIG2 shows a schematic diagram of the internal structure of a device for accessing data between heterogeneous file systems. The device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The non-volatile computer-readable storage medium of the device stores an operating system, a database, and computer-readable instructions. The database may store information sequences. When executed by the processor, the computer-readable instructions enable the processor to implement a method for accessing data between heterogeneous file systems.

[0164] The processor of the device for accessing data between heterogeneous file systems is used to provide computing and control capabilities, supporting the operation of the entire device. The memory of the device for accessing data between heterogeneous file systems may store computer-readable instructions that, when executed by the processor, cause the processor to execute the method for accessing data between heterogeneous file systems disclosed herein. The network interface of the device for accessing data between heterogeneous file systems is used to connect and communicate with a terminal.

[0165] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the data access device between heterogeneous file systems to which the solution of the present application is applied. The specific data access device between heterogeneous file systems may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0166] In this embodiment, the processor is used to execute Figure 3 The memory stores the program code and various data required to execute the modules or submodules. The network interface is used to implement data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the heterogeneous file system data access device of this application. The server can call the server's program code and data to execute the functions of all modules.

[0167] The present application also provides a non-volatile readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for accessing data between heterogeneous file systems in any embodiment of the present application.

[0168] The present application also provides a computer program product, comprising a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0169] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0170] To sum up, this application effectively solves the technical problems faced by traditional data migration methods through a series of innovative technical features, realizes dynamic migration of data, ensures data security, achieves load balancing, and significantly reduces the implementation cost of data migration, providing an efficient, reliable and economical solution for data migration in big data clusters.

Claims

1. A method for accessing data between heterogeneous file systems, characterized in that: include: Receive a query statement, parse and generate a logical plan corresponding to the query statement, and determine from the logical plan the bucket identifiers of target data buckets in the database accessed by the query statement and the query timestamp of each target data bucket; Query metadata information, determine a target distributed file system with a storage period that matches the target data bucket based on the query timestamp of the target data bucket, and obtain database configuration information corresponding to the target distributed file system for storing the database, wherein each distributed file system has a different protocol type and stores data files of the data bucket generated by the database within the corresponding storage period according to different storage periods; Constructing a physical plan corresponding to the logical plan, wherein each target data bucket in the physical plan includes a corresponding query task, each query task includes a storage path of a target data file determined based on the database configuration information and a bucket identifier of the corresponding target data bucket and its query timestamp, and the storage path carries a protocol type identifier of the corresponding target distributed file system; According to the protocol type identifier in the storage path of each query task in the physical plan, the corresponding query task is executed for the target data file pointed to by the corresponding storage path through the client of the corresponding target distributed file system, and the query results of each query task are constructed as query response information.

2. The method for accessing data between heterogeneous file systems according to claim 1, wherein: Constructing a physical plan corresponding to the logical plan includes: Determine a bucket identifier and a query timestamp of a target data bucket in the query statement based on the logical plan, compare the query timestamp with the storage time periods of each distributed file system, and determine a target distributed file system and its protocol type identifier whose storage time period matches the query timestamp; Determine, according to the logical plan, whether the data operation of the target data bucket includes a data write operation, and if the data write operation is included, determine whether the target data bucket includes a target data file named with the query timestamp as a creation timestamp; When the target data file is not included, creating a target data file in the target data bucket, and naming the target data file with the query timestamp as the creation timestamp; When a target data file is included, the protocol type identifier of the target distributed file system, the bucket identifier of the target data bucket and the creation timestamp are combined to form the storage path of the target data file, and the database configuration information in the metadata information is updated.

3. The method for accessing data between heterogeneous file systems according to claim 2, wherein: After determining whether the data operation on the target data bucket includes a data write operation according to the logical plan, the method includes: When no data write operation is included, screening to determine the target data file whose creation timestamp in the target data bucket matches the query timestamp corresponding to the target data bucket; The protocol type identifier of the target distributed file system, the bucket identifier of the target data bucket, and the creation timestamp of each target data file are combined to form a corresponding storage path, which is updated to the database configuration information in the metadata information.

4. The method for accessing data between heterogeneous file systems according to claim 1, wherein: Before querying metadata information, include: Determining a migration timestamp and database configuration information of the database in the first type distributed file system in the metadata information, wherein the migration timestamp is used to divide a storage period between the first type distributed file system and the second type distributed file system; deploying the database in the second type of distributed file system according to the storage structure information corresponding to the first type of distributed file system in the database configuration information, and creating a first data file for each data bucket of the database, wherein each first data file is named using the migration timestamp as a creation timestamp; The storage path of the database, its data bucket, and its first data file in the second type distributed file system is updated and stored in the database configuration information of the metadata information.

5. The method for accessing data between heterogeneous file systems according to claim 4, wherein: Determining the migration timestamp and database configuration information of the database in the first type distributed file system in the metadata information includes: Obtaining performance monitoring data of the first type of distributed file system, and deciding whether to trigger a migration preparation instruction based on the performance monitoring data; In response to the migration preparation instruction, determining storage structure information corresponding to storing the database in the first type distributed file system from the database configuration information in the metadata information, converting the information into storage structure information adapted for the second type distributed file system, and extending a current timestamp corresponding to the triggering of the migration preparation instruction by a preset time to serve as a migration timestamp; During the period when the migration timestamp is reached, the comprehensive performance score of the first type of distributed file system is determined based on the continuously acquired performance monitoring data. When the comprehensive performance score meets the preset conditions, the database configuration information corresponding to the migration timestamp and the first type of distributed file system is determined.

6. The method for accessing data between heterogeneous file systems according to any one of claims 1 to 5, wherein: The method further comprises: In response to an update event of the metadata information, recording an update timestamp of the corresponding data file; In response to an abnormal rollback instruction, obtaining a rollback timestamp specified by the instruction, and locating a target data file whose update timestamp matches the rollback timestamp in the metadata information as a file to be revoked; In response to the revocation confirmation instruction, a rollback operation is performed on the file to be revoked, and database configuration information corresponding to the file to be revoked in the metadata information is updated.

7. The method for accessing data between heterogeneous file systems according to any one of claims 1 to 5, wherein: The method further comprises: Periodically scanning the metadata information, determining the data bucket of the last data file created in the database in the distributed file system after the storage period, and obtaining the creation timestamp of the last data file created; Determine whether the creation timestamp exceeds the preset retention period of the corresponding data bucket. If the retention period is exceeded, move all data files in the data bucket before the retention period to the recycle bin. Regularly clear the data files stored in the recycle bin that exceed a preset time period.

8. A data access device between heterogeneous file systems, characterized in that: include: a logic parsing module configured to receive a query statement, parse and generate a logic plan corresponding to the query statement, and determine from the logic plan the bucket identifiers of target data buckets in the database accessed by the query statement and a query timestamp for each target data bucket; A configuration query module is configured to query metadata information, determine a target distributed file system with a storage period that matches the target data bucket based on a query timestamp of the target data bucket, and obtain database configuration information corresponding to the target distributed file system storing the database, wherein the protocol types of the distributed file systems are different from each other and the data files of the data buckets generated by the database within the corresponding storage period are stored respectively according to different storage periods; a task construction module configured to construct a physical plan corresponding to the logical plan, wherein each target data bucket in the physical plan includes a corresponding query task, each query task includes a storage path of a target data file determined based on the database configuration information and a bucket identifier of the corresponding target data bucket and its query timestamp, and the storage path carries a protocol type identifier of the corresponding target distributed file system; The task execution module is configured to execute the corresponding query task for the target data file pointed to by the corresponding storage path through the client of the corresponding target distributed file system according to the protocol type identifier in the storage path of each query task in the physical plan, and construct the query results of each query task into query response information.

9. A data access device between heterogeneous file systems, comprising a central processing unit and a memory, characterized in that: The central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A non-volatile readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images