A data reading method, an electronic device, and a storage medium

Abstract

The application discloses a data reading method, an electronic device and a storage medium, and comprises the following steps: acquiring timestamps of all partitions of a target database; in response to any timestamp changing, acquiring a current file list of a target partition of the changed timestamp, wherein the timestamp changes when a file changes; comparing the current file list with a historical file list stored by a state backend to determine a target file that changes, and reading the target file according to a preset sorting mode. That is, in the application, the timestamps of all partitions are compared to determine the current file list of the partition whose timestamp changes, and then the current file list and the historical file list are compared to determine the target file that changes. The data changes in the partition can be effectively acquired for the processed partition and the unprocessed partition, and data loss is effectively prevented, and the accuracy of data reading is improved.

Description

Technical Field

[0001] This application relates to the technical field of data reading, and in particular to a data reading method, electronic device, and storage medium. Background Technology

[0002] Current data reading and processing, especially for data warehouses, typically relies on time-partitioned tables. Reading Hive partitioned tables (Hive is a distributed SQL engine, and Hive partitioned tables are a distributed table framework) uses three methods: partition name, partition time, and creation time. However, in the Flink (an open-source, high-performance, high-throughput, low-latency distributed computing engine) linker, these three methods only process the latest partition data once. After processing, it waits for new partitions to appear, and previously processed partitions are not checked again. Therefore, append-only and overwrite operations can lead to data loss, making it difficult to determine changes in partition data and affecting the accuracy of data reading. Summary of the Invention

[0003] The technical solution to the main technical problem addressed in this application is to provide a data reading method, electronic device, and storage medium that can effectively determine changes in partitioned data, thereby improving the accuracy of data reading.

[0004] To address the aforementioned technical problems, this application provides a data reading method applied to a data warehouse containing multiple databases, comprising: obtaining timestamps of all partitions of a target database; in response to a change in any of the timestamps, obtaining a current file list of the target partition with the changed timestamp, wherein the timestamps change when files are modified; comparing the current file list with a historical file list stored in the status backend to determine the modified target file, and reading the target file according to a preset sorting method.

[0005] In some embodiments, obtaining the timestamps of all partitions of the target database includes: periodically obtaining the current partition metadata information of all partitions of the target database from the metadata management center; and in response to the existence of corresponding current partition metadata information in the target database, obtaining the timestamp of each partition using the partition metadata information.

[0006] In some embodiments, the method further includes: after each data writing operation, the change in the timestamp is recorded and stored by the metadata management center when the metadata management center updates the metadata information.

[0007] In some embodiments, the step of obtaining the current file list of the target partition with the changed timestamp in response to a change in any of the timestamps includes: obtaining historical partition metadata information of all partitions stored in the status backend, and obtaining current partition metadata information of all partitions stored in the metadata management center; comparing the historical partition metadata information and the current partition metadata information to determine the partition corresponding to the discrepancy in the comparison result as the target partition, and obtaining the partition directories of all the target partitions; and obtaining the current file list of each target partition under the partition directory from the metadata management center.

[0008] In some embodiments, the method further includes: storing all non-empty current partition metadata information in the status backend; and, in response to the absence of corresponding partition metadata information in the status backend, using the current partition metadata information as historical partition metadata information.

[0009] In some embodiments, comparing the current file list with the historical file list stored in the status backend to determine the target file that has changed, and reading the target file according to a preset sorting method, includes: obtaining the historical file list about the target file stored in the status backend; comparing the historical file list with the current file list, and using the newly added file as the target file that has changed; determining the order of each partition using a partition discovery mode, and determining a preset sorting method using the order, and then reading the target file according to the preset sorting method.

[0010] In some embodiments, the method further includes: in response to the completion of reading the target file of the target partition, storing the current partition metadata information and the current file list of the target partition that have been read in the status backend.

[0011] In some embodiments, the method further includes: in response to the current file list queried in the changed target partition not matching the historical file list stored in the status backend, setting the target partition as overwritten and deleting the historical file information corresponding to the target file.

[0012] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide an electronic device, the electronic device including a memory and a processor coupled to the memory, the memory storing at least one computer program, which, when loaded and executed by the processor, is used to implement the method as described above.

[0013] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium having at least one program, which, when loaded and executed by a processor, is used to implement the method described above.

[0014] Unlike current technologies, the data reading method provided in this application is applied to a data warehouse containing multiple databases. The method includes: obtaining the timestamps of all partitions in the target database; in response to a change in any timestamp, obtaining the current file list of the target partition with the changed timestamp, where the timestamp changes when a file is modified; comparing the current file list with the historical file list stored in the status backend to determine the target file that has changed, and reading the target file according to a preset sorting method. In other words, this application compares the timestamps of all partitions to determine the current file list of the partitions with changed timestamps, and then compares the current file list with the historical file list to determine the target file that has changed. This method can effectively obtain changes in data within both processed and unprocessed partitions, thereby effectively preventing data loss and improving the accuracy of data reading. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating an embodiment of the data reading method in this application; Figure 2 This is a flowchart illustrating an embodiment of the data reading method in this application; Figure 3 This is a schematic diagram of the structure of an embodiment of the electronic device in this application; Figure 4 This is a schematic diagram of an embodiment of a computer-readable storage medium in this application. Detailed Implementation

[0016] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0018] Hive data sources are the de facto standard for modern enterprise data warehouses. Reading data from Hive for computation is one of the fundamental business logics in existing data mining scenarios. Furthermore, for large-scale data operations, Hive table structures are primarily time-partitioned. Flink, as a mainstream stream computing framework, provides solutions for streaming Hive partitioned tables in three ways: based on partition name, partition time, and creation time. Partition name is sorted lexicographically by partition directory name, checking for new partitions whose names are lexicographically larger than the current maximum lexicographical order. Partition time extracts the timestamp from the partition name, checking if any new directories with timestamps exceeding the current maximum have been added. Creation time logic is similar to partition time, but it checks the creation time of the partition directory. All three logics are implemented in the flink-hive-connector (a core component in Flink SQL used to connect to Hive data warehouses). However, this connector only processes the latest partition's data once; after processing, it waits for new partitions to appear and does not check or track previously processed partitions. In existing big data operations, it's common for historical partitions in Hive partition tables to be appended to or overwritten by upstream applications. Flink's practice of only checking the latest partitions leads to the loss of historical data. Furthermore, existing solutions require adding new components to inspect Hive metadata and modifying the FlinkHive connector to accommodate these new components. Consequently, append and overwrite operations result in data loss, making it difficult to determine changes in partition data and impacting the accuracy of data retrieval.

[0019] Therefore, a data reading method is provided that can effectively determine changes in partitioned data, thereby improving the accuracy of data reading.

[0020] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the data reading method in this application; it should be noted that, if there are substantial results, the method of this application does not necessarily reflect the actual data reading process. Figure 1 The sequence of processes shown is limited.

[0021] like Figure 1As shown, the data reading method of this application is applied to a data warehouse, which contains multiple databases and may include the following operations.

[0022] S10. Obtain the timestamps of all partitions in the target database.

[0023] In this context, a data warehouse refers to a subject-oriented, integrated, non-volatile, and time-varying collection of data. It can store multiple databases, which can be of the same or different types. The target database can be a specified or selected database among the multiple databases. Partitioning refers to dividing a database into multiple partitions, which can be stored and managed independently, but logically still belong to the same table. The timestamp refers to the partition key, which is the core basis for determining the partition to which data belongs. Using the timestamp as the core dimension, data is split into physically independent partitions according to time range, thereby adapting to the query, storage, and lifecycle management needs of time-series data.

[0024] Specifically, the selected database is used as the target database, and then all partitions of the target database and the timestamps of all partitions are obtained.

[0025] S20. In response to any timestamp change, obtain the current file list of the target partition whose timestamp has changed, wherein the timestamp changes when the file is modified.

[0026] The current file list refers to the list of files contained in the target partition whose timestamps have changed. When a file is modified, the timestamp of the partition containing the file also changes.

[0027] Specifically, timestamps are recorded in the metadata management center. If any timestamp in the metadata management center changes, it means that a file has been modified. Therefore, it is necessary to obtain the target partition with the changed timestamp, and then obtain the current file list of the target partition.

[0028] S30. Compare the current file list with the historical file list stored in the status backend to determine the target file that has been changed, and read the target file according to the preset sorting method.

[0029] In this context, the state backend refers to the component or storage layer in a distributed system responsible for storing, managing, and restoring runtime state. The real-time state data stored in the state backend (such as the stream processing results of Flink) can be written to the data warehouse through the ETL (Extract-Transform-Load, which is the process of extracting, cleaning, and transforming distributed data sources (databases, files, APIs, etc.) and loading them into the target system (data warehouse / data lake)) pipeline, becoming one of the data sources for data analysis (e.g., real-time sales data is first calculated in Flink to calculate the real-time total amount, and then written to the data warehouse for historical trend analysis). The historical file list refers to the previously stored file list about the target partition stored in the state backend.

[0030] Specifically, the current file list obtained by the cloud data management center is compared with the historical file list stored in the status backend to obtain the target file that has been changed. If there are multiple target files that have been changed, the target files are read according to a preset sorting method.

[0031] In this embodiment, the timestamps of all partitions are compared to determine the current file list of partitions whose timestamps have changed. Then, the current file list is compared with the historical file list to determine the target file that has changed. This method can effectively obtain the changes in data in both processed and unprocessed partitions, thereby effectively preventing data loss and improving the accuracy of data reading.

[0032] In some embodiments, obtaining the timestamps of all partitions of the target database may include the following operations.

[0033] Periodically retrieve the current partition metadata information of all partitions of the target database from the metadata management center; In response to the existence of corresponding current partition metadata information in the target database, the timestamp of each partition is obtained using the partition metadata information.

[0034] The metadata management center stores the currently acquired data for all databases, such as the partition information of each database, including the current partition metadata information; and the target database's current partition metadata information will only be acquired when the corresponding data is currently available.

[0035] Specifically, when a file is modified, the metadata management center retrieves the current partition metadata information of the database partition. To more clearly pinpoint the time of the change, it can be selected to periodically retrieve the current partition metadata information of all partitions of the selected target database from the metadata management center. Furthermore, for the target database that has current partition metadata information, the timestamp of each partition can be obtained using the current partition metadata information.

[0036] By periodically retrieving the corresponding current partition metadata information, the target file of the partition that has undergone changes can be identified more quickly and accurately, thereby reading the target file faster and improving reading efficiency.

[0037] Furthermore, after each data write, the change in the timestamp is recorded and stored by the metadata management center when updating the metadata information.

[0038] Here, "writing data" refers to the completion of writing data, and the timestamp changes every time data is written. Therefore, after each completion of data writing, the metadata information of newly written data needs to be updated in the metadata management center, and the corresponding timestamp change is also recorded in the metadata management center. Furthermore, the timestamp change is related to the updated metadata information.

[0039] Furthermore, in response to a change in any timestamp, retrieving the current file list of the target partition whose timestamp has changed can include the following operations: Retrieve historical partition metadata information for all partitions in the status backend storage, and retrieve current partition metadata information for all partitions in the metadata management center storage; By comparing historical partition metadata information with current partition metadata information, the partitions corresponding to discrepancies in the comparison results are identified as target partitions, and the partition directories of all target partitions are obtained. Retrieve the current file list for each target partition under the partition directory from the metadata management center.

[0040] The status backend stores historical partition metadata information for all partitions of each database, while the metadata management center stores current partition metadata information for all partitions of each database.

[0041] Therefore, all historical partition metadata information can be obtained from the status backend, and all current partition metadata information can be obtained from the metadata management center. Then, the historical partition metadata information and the current partition metadata information corresponding to each partition are compared to obtain the comparison results. If the comparison results match, it means that the files in the partitions with matching results have not been changed. If the comparison results do not match, it means that the files in the partitions with inconsistent results have been changed. Therefore, the partitions corresponding to inconsistent results are used as target partitions, and all target partitions are arranged into a directory, thus obtaining the partition directories for all target partitions. Then, based on the obtained partition directories, the current file list of each target partition under the partition directory is obtained from the metadata management center.

[0042] It is understandable that there can be one or more target partitions; and timestamps can have historical timestamps and current timestamps. A change in timestamps means that the current timestamp and the historical timestamp are inconsistent.

[0043] Furthermore, the following operations may also be included: Store all non-empty metadata information of the current partition into the status backend; If no corresponding partition metadata information is found in the status backend, the current partition metadata information will be used as the historical partition metadata information.

[0044] Not all partitions contain current partition metadata information. Therefore, for empty current partition metadata information (which could indicate a partition with no data), the empty current partition metadata information can be removed. For non-empty current partition metadata information, it is all stored in the status backend. The system then checks if the status backend contains the metadata information for each partition. If not, the obtained current partition metadata information is used as the historical partition metadata information for that partition. If it does, the current partition metadata information is compared with the corresponding historical partition metadata information.

[0045] It is understandable that partition metadata information includes at least: partition information and a list of files in the partition; partition information includes at least: partition name, partition creation time, partition update time, and a list of files in the partition.

[0046] In this embodiment, when the timestamp changes, the corresponding target partition is determined, and then the current file list of the target partition is obtained. This can effectively and quickly identify the partition where the files have changed as the target partition, providing a basis for subsequent comparison.

[0047] In some embodiments, comparing the current file list with the historical file list stored in the status backend to determine the target file that has changed, and reading the target file according to a preset sorting method, may include the following operations: Retrieve the list of historical files about the target file stored in the status backend; By comparing the historical file list with the current file list, the newly added file is used as the target file for the change; The partition discovery mode is used to determine the order of each partition, and the order is used to determine the preset sorting method, and then the target file is read according to the preset sorting method.

[0048] The status backend stores a historical file list of all files in all partitions. If not, the most recently obtained current file list is used as the historical file list.

[0049] Specifically, the partition where the timestamp changes is identified as the target partition. A list of historical files for the target file is obtained from the status backend. The current file list of the target partition is compared with the historical file list to identify newly added files as target files that have changed. Then, the partition discovery mode is used to determine the order in which each partition is read, and then the preset sorting method for each partition is determined. Finally, the corresponding target files are read according to the preset sorting method.

[0050] In some embodiments, files whose data has changed are also considered as target files that have been changed; if there are multiple target files, the partition order is first determined by the partition order, then the file order is determined by the file order within the partition, and then the partitions are read first by the partition order, and then the corresponding files are read by the file order.

[0051] It is understandable that partition discovery modes can include three user-configured modes, such as user mode, privileged mode, and global configuration mode.

[0052] In this embodiment, by comparing the current file list and the historical file list, it is possible to effectively determine which files have been changed, accurately and quickly identify the target file to which the changes were sent, and effectively improve reading efficiency.

[0053] In some embodiments, the following operations may also be included.

[0054] In response to the completion of reading the target file of the target partition, the current partition metadata information and the current file list of the target partition are stored in the status backend.

[0055] Specifically, for a target file that has been read, the current partition metadata information and current file list of the corresponding target partition are obtained. In order to make a comparison next time, the current partition metadata information and current file list of the target partition that has been read can be stored in the status backend.

[0056] Furthermore, in the next comparison, the current partition metadata information and current file list obtained and stored in the status backend can be used as the corresponding historical partition metadata information and historical file list.

[0057] In some embodiments, the following operations may also be included.

[0058] If no file matches the current file list found in the target partition that has been modified, and the historical file list stored in the status backend, then the target partition is set to be overwritten, and the historical file information corresponding to the target file is deleted.

[0059] Specifically, if the current file list does not match the corresponding historical file list in the status backend, it means that the current file list of the target partition has not appeared in the status backend, and thus it is set to overwrite the target partition; that is, all files in the target partition have been replaced. Therefore, it is also necessary to delete the historical file information corresponding to the target file.

[0060] For example, after reading all newly added files in a partition, the partition's metadata and file list information are added to the status, awaiting the next comparison. Implicitly, a comparison logic exists: if no file matches the file list retrieved from the modified partition against the file list stored in the status, the task indicates a partition overwrite operation was performed on the Hive table. In this case, the corresponding historical file information for that partition is deleted. The user can configure whether to accept delete messages to determine whether to send change messages downstream.

[0061] The problem this application aims to solve is that streaming reads of Hive partition tables in Flink tasks cannot track scenarios such as the addition or overwriting of partition data. Moreover, it can achieve real-time tracking of newly added partitions in the partition table and the resubmission of data in already committed partition tables without introducing additional components, using Flink's native capabilities.

[0062] To provide a clearer understanding, the principles of this application will be explained below.

[0063] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the data reading method in this application.

[0064] like Figure 2 As shown, the steps are as follows: First, the task starts and queries the metadata information of the target database; for example, it queries the metadata information of a specified target database from the metadata management center. Second, it determines whether the partition metadata information of the target database is empty; if empty, it waits for the check interval configured by the user for the linker and cyclically pulls the corresponding partition metadata information; if not empty, it proceeds to the third step. Third, it pulls the current partition metadata information, that is, it pulls all non-empty current partition metadata information of the target database and stores it as the status in the task's status backend. Fourth, it compares the metadata information, that is, it compares the current partition metadata information with the historical partition metadata information; if they match, it repeats the second step after waiting for the specified check interval; if they do not match, it determines the partition corresponding to the inconsistent comparison result as the target partition, obtains the partition directories of all the target partitions, and proceeds to the fifth step. Fifth, it pulls the current file list under the partition directory through the metadata management center. Sixth, it compares the current file list with the historical file list to obtain the target files that have changed; Seventh, it reads the target files according to the preset sorting method.

[0065] The specific operating logic is as follows: 1. Upon initial startup, the task queries the specified database's metadata information from hive-metastore (the metadata management center, one of the core components of Hive, responsible for storing and managing metadata (such as schema, location, and format) of Hive tables, partitions, and databases; it acts as a data directory, enabling Hive to parse queries and locate the actual data).

[0066] 2. Determine if the database has partition metadata information. If the partition is empty, wait for the check interval configured by the user for the connector and retrieve it in a loop.

[0067] 3. Retrieve all non-empty partition information from the specified database and store it as the status backend of the task. If the status backend already contains historical partition information, compare it with the historical partition information.

[0068] 4. After comparison, if the partition information is consistent with that of the local storage, repeat step 2 after waiting for the specified check interval.

[0069] 5. If the metadata is inconsistent with the local cache, the inconsistent partition directory is fetched, and a list of all files in the corresponding directory is obtained through HDFS.

[0070] 6. Compare the retrieved file list with the historical file list of the partition in the status, filter out newly added files, and read the data from each file in chronological order of creation time and send it downstream. The order between partitions is matched according to three user-configured modes. For example, if the partition name is used, partitions with newly added data are sorted lexicographically, while partitions without newly added data are not included. The other two modes are similar.

[0071] 7. After reading all newly added files for a partition, add the partition's metadata and file list to the status, awaiting the next comparison. Implicitly, if no file matches the file list in the status, the task indicates a partition overwrite operation on the Hive table. In this case, delete the corresponding partition's historical file information. The user can configure whether to accept delete messages to determine whether to send change messages downstream.

[0072] In this embodiment, by modifying the Flink Hive connector logic (a data interaction connector that acts as a data interaction bridge between Flink (a big data processing framework integrating stream and batch processing) and Apache Hive (a data warehouse tool based on Hadoop), achieving deep integration between Flink and the Hive data warehouse) and leveraging the capabilities of the Flink native framework, it is possible to track historical partition data in Hive without adding any new components. The linker source code is modified to periodically retrieve the update timestamps of each partition in the specified database from the metadata management center (Hive metastore). After each business operation writes data, it commits (submits a transaction) to update the metadata in the Hive metastore, and the metadata management center records the timestamp of this update. After retrieving the timestamps of all partitions, the linker internally uses Flink's state storage capability to store the retrieved partition information as state. Flink can use RocksDB (a high-performance storage engine) as the state backend by default. It compares the periodically retrieved information with historical states. When it detects that the update timestamp of one or more partitions has changed, it retrieves the file metadata of the changed partition from the state backend (HDFS, Hadoop Distributed File System), stores the file metadata as state, writes it to the state backend, compares it with the historical file metadata state, reads the newly added file content, and sends the data to downstream operators.

[0073] This application also provides an electronic device.

[0074] See Figure 3 , Figure 3 This is a schematic diagram of an embodiment of the electronic device in this application. The electronic device can perform the steps in the above method.

[0075] The electronic device 200 includes a memory 220, a processor 210 coupled to the memory, and at least one computer program stored in the memory 220 and executable on the processor 210. When the processor 210 loads and executes the at least one computer program, it implements the steps of the method for identifying the behavior of possessing dangerous goods described above. For related details, please refer to the detailed description in the above method; further elaboration will not be repeated here.

[0076] This application also includes a computer-readable storage medium.

[0077] Please see Figure 4 , Figure 4 This is a schematic diagram of an embodiment of a computer-readable storage medium in this application.

[0078] The computer-readable storage medium 300 stores at least one program 310, which, when loaded and executed by a processor, is used to implement the steps of the data reading method described above. For related details, please refer to the detailed description in the above method; it will not be repeated here.

[0079] The above scheme compares the timestamps of all partitions to determine the current file list of partitions where the timestamps have changed. Then, it compares the current file list with the historical file list to determine the target file that has been changed. This method can effectively obtain data changes in both processed and unprocessed partitions, thereby effectively preventing data loss and improving the accuracy of data reading.

[0080] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0082] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0083] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0084] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A data reading method, characterized in that, This is applied to a data warehouse, which contains multiple databases, including: Get the timestamps of all partitions in the target database; In response to any change in the timestamp, obtain the current file list of the target partition whose timestamp has changed, wherein the timestamp changes when a file is modified; The current file list is compared with the historical file list stored in the status backend to determine the target file that has changed, and the target file is read according to a preset sorting method.

2. The method according to claim 1, characterized in that, The process of obtaining the timestamps of all partitions in the target database includes: Periodically retrieve the current partition metadata information of all partitions of the target database from the metadata management center; In response to the existence of corresponding current partition metadata information in the target database, the timestamp of each partition is obtained using the current partition metadata information.

3. The method according to claim 2, characterized in that, Also includes: After each data is written, the change in the timestamp is recorded and stored by the metadata management center when the metadata management center updates the metadata information.

4. The method according to claim 1, characterized in that, In response to a change in any of the timestamps, the process of obtaining the current file list of the target partition with the changed timestamp includes: Obtain historical partition metadata information of all partitions stored in the status backend, and obtain current partition metadata information of all partitions stored in the metadata management center; By comparing the historical partition metadata information with the current partition metadata information, the partitions corresponding to the discrepancies in the comparison results are identified as target partitions, and the partition directories of all target partitions are obtained. Obtain the current file list for each target partition under the partition directory from the metadata management center.

5. The method according to claim 4, characterized in that, Also includes: Store all non-empty metadata information of the current partition into the status backend; If no corresponding partition metadata information is found in the status backend, the current partition metadata information is used as historical partition metadata information.

6. The method according to claim 1, characterized in that, The step of comparing the current file list with the historical file list stored in the status backend to determine the target file that has changed, and reading the target file according to a preset sorting method, includes: Obtain the list of historical files about the target file stored in the status backend; The historical file list and the current file list are compared, and the newly added file is used as the target file that has been changed; The partition discovery mode is used to determine the order of each partition, and the order is used to determine a preset sorting method, and then the target file is read according to the preset sorting method.

7. The method according to claim 1, characterized in that, Also includes: In response to the completion of reading the target file of the target partition, the current partition metadata information and the current file list of the target partition are stored in the status backend.

8. The method according to claim 1, characterized in that, Also includes: If no file is found in the current file list retrieved from the target partition that has been modified, and no file is found in the historical file list stored in the status backend, then the target partition is set to have been overwritten, and the historical file information corresponding to the target file is deleted.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor coupled to the memory, the memory storing at least one computer program, which, when loaded and executed by the processor, is used to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium has at least one program that, when loaded and executed by a processor, is used to implement the method as described in any one of claims 1-8.

Images