Data readiness detection method and device, equipment and storage medium

Abstract

The invention provides a data readiness detection method and device, equipment and a storage medium, and belongs to the field of data verifier.The method comprises the steps that to-be-called target data is obtained, and a metadata file corresponding to a data lake to which the target data belongs is obtained, the metadata file comprises metadata corresponding to each piece of data in the data lake; obtaining metadata of the target data from the metadata file, and obtaining preparation progress information of the target data from the metadata of the target data; and under the condition of determining that the preparation progress information meets a preset progress condition, determining that the target data is prepared. According to the method and the device, the efficiency and the accuracy of target data ready judgment are greatly improved, so that the data calling efficiency is greatly improved. The invention further relates to a block chain technology, and the metadata file can be stored in a block chain.

Description

Technical Field

[0001] The present application relates to the field of data verification technology, and in particular to a data readiness detection method, device, equipment and storage medium. Background Art

[0002] Hudi is an open source data lake tool used to manage data in large-scale data lakes. Hudi includes upstream and downstream parts in the data call process. The downstream part of the application will use batch-stream fusion scenarios, especially the downstream scheduling notification of the batch-stream scenario is particularly important. Currently, open source technology does not provide a direct technical readiness check solution, but only relies on the business to add a timestamp field to identify the data readiness. The downstream part consumes the field to start the spark or hive task to poll and query the business field, and then judge the data readiness status. This data readiness check method has the defects of low efficiency, high cost, and causing some downstream tasks to run untimely.

[0003] Therefore, how to detect whether the data to be called is ready is a problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the present application is to provide a data readiness detection method, device, equipment and storage medium, aiming to improve the accuracy and efficiency of data readiness detection.

[0005] In a first aspect, the present application provides a data readiness detection method, the data readiness detection method comprising the following steps:

[0006] Obtain target data to be called, and obtain a metadata file corresponding to the data lake to which the target data belongs, wherein the metadata file includes metadata corresponding to each data in the data lake;

[0007] Acquire metadata of the target data from the metadata file, and acquire preparation progress information of the target data from the metadata of the target data;

[0008] When it is determined that the preparation progress information satisfies a preset progress condition, it is determined that the target data is ready.

[0009] In a second aspect, the present application further provides a data readiness detection device, the data readiness detection device comprising an acquisition module and a determination module, wherein:

[0010] The acquisition module is used to acquire the target data to be called, and acquire the metadata file corresponding to the data lake to which the target data belongs, wherein the metadata file includes metadata corresponding to each data in the data lake;

[0011] The acquisition module is further used to acquire the metadata of the target data from the metadata file, and acquire the preparation progress information of the target data from the metadata of the target data;

[0012] The determining module is used to determine that the target data is ready when it is determined that the preparation progress information meets a preset progress condition.

[0013] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the data readiness detection method as described above are implemented.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the data readiness detection method as described above are implemented.

[0015] The present application provides a data readiness detection method, device, equipment and storage medium. The present application obtains the target data to be called, and obtains the metadata file corresponding to the data lake to which the target data belongs, and the metadata file includes the metadata corresponding to each data in the data lake; obtains the metadata of the target data from the metadata file, and obtains the preparation progress information of the target data from the metadata of the target data; and accurately determines that the target data is ready when it is determined that the preparation progress information meets the preset progress conditions. In the present application, the preparation progress information of the target data is obtained from the metadata of the target data, and it is judged whether the preset progress conditions are met based on the preparation progress information. When the preparation progress information meets the preset progress conditions, it can be accurately known that the target data is ready, which greatly improves the efficiency and accuracy of the judgment of the readiness of the target data, thereby greatly improving the efficiency of data calling. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 A flowchart of a data readiness detection method provided in an embodiment of the present application;

[0018] Figure 2 A flowchart of another data readiness detection method provided in an embodiment of the present application;

[0019] Figure 3 for Figure 1 A flowchart of sub-steps of a data readiness detection method;

[0020] Figure 4 for Figure 1 A flowchart of another sub-step of the data readiness detection method;

[0021] Figure 5 A flowchart of another data readiness detection method provided in an embodiment of the present application;

[0022] Figure 6 A schematic block diagram of a data readiness detection device provided in an embodiment of the present application;

[0023] Figure 7 A schematic block diagram of another data readiness detection device provided in an embodiment of the present application;

[0024] Figure 8 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0025] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0028] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0029] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0030] Hudi is an open source data lake tool used to manage data in large-scale data lakes. Hudi includes upstream and downstream parts in the data call process. The downstream part of the application will use batch-stream fusion scenarios, especially the downstream scheduling notification of the batch-stream scenario is particularly important. Currently, open source technology does not provide a direct technical readiness check solution, but only relies on the business to add a timestamp field to identify the data readiness. The downstream part consumes the field to start the spark or hive task to poll and query the business field, and then judge the data readiness status. This data readiness check method has the defects of low efficiency, high cost, and causing some downstream tasks to run untimely.

[0031] To solve the above problems, the present application provides a data readiness detection method, device, equipment and storage medium. The data readiness detection method includes: obtaining the target data to be called, and obtaining the metadata file corresponding to the data lake to which the target data belongs, the metadata file including the metadata corresponding to each data in the data lake; obtaining the metadata of the target data from the metadata file, and obtaining the preparation progress information of the target data from the metadata of the target data; and determining that the target data is ready when it is determined that the preparation progress information meets the preset progress condition.

[0032] The data readiness detection method can be applied to a computer device, which may be an electronic device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device.

[0033] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0034] Please refer to Figure 1 , Figure 1 A flowchart of a data readiness detection method provided in an embodiment of the present application.

[0035] like Figure 1 As shown, the data readiness detection method includes steps S101 to S103.

[0036] Step S101: Obtain target data to be called, and obtain a metadata file corresponding to a data lake to which the target data belongs, wherein the metadata file includes metadata corresponding to each data in the data lake.

[0037] Among them, the data lake is a centralized repository for managing large-scale data. The data lake can be set in a server, computer equipment, etc. The data lake stores metadata files of the data, and the metadata files include metadata corresponding to each data in the data lake.

[0038] In some embodiments, Figure 2 As shown, before step S101, steps S201 to S203 are also included.

[0039] Step S201: For each write data in the data lake, determine a first update time of the write data according to the execution time of each subtask of the write data.

[0040] The data written into the data lake is referred to as written data, for example, data 1, data 2, data 3, data 4 and data 5, where data 2 and data 5 are data written into the data lake, and data 2 and data 5 are respectively referred to as written data 2 and written data 5. The first update time is used to represent the execution time of the subtask closest to the current timestamp in the written data.

[0041] It should be noted that the write data includes multiple executed subtasks, each subtask includes the time of executing the task, and the time of executing each subtask is referred to as the execution time. For example, the write data includes subtask 1, subtask 2, subtask 3, subtask 4 and subtask 5, wherein the execution time of subtask 1 is 10:18 on May 10, the execution time of subtask 2 is 10:30 on May 10, the execution time of subtask 3 is 15:18 on May 10, the execution time of subtask 4 is 11:00 on May 10, and the execution time of subtask 5 is 9:08 on May 10.

[0042] In some embodiments, the execution time closest to the current timestamp is selected from the execution time of each subtask of writing data; the execution time closest to the current timestamp is determined as the first update time of writing data. By counting the execution time of each subtask, the first update time of writing data can be accurately determined.

[0043] Exemplarily, the write data includes subtask 1, subtask 2, subtask 3, subtask 4 and subtask 5, wherein the execution time of subtask 1 is 10:18 on May 10, the execution time of subtask 2 is 8:30 on May 10, the execution time of subtask 3 is 15:18 on May 10, the execution time of subtask 4 is 11:00 on May 10 and the execution time of subtask 5 is 9:08 on May 10; wherein, the execution time closest to the current timestamp is 15:18 on May 10, and the execution time closest to the current timestamp, 15:18 on May 10, is determined as the first update time of the write data.

[0044] Step S202: Determine a second update time for writing the data according to the completion time of each of the subtasks.

[0045] The second update time of the written data is the time when the update of the written data is completed.

[0046] In some embodiments, the completion time of each subtask is acquired to obtain multiple completion times, and the second update time of writing data is determined according to the multiple completion times. The second update time of writing data can be accurately determined by the completion time of each subtask.

[0047] In some embodiments, the second update time of writing data may be determined according to multiple completion times by determining the completion time of the last completed subtask as the second update time of writing data.

[0048] In some embodiments, the write data includes subtask 1, subtask 2, subtask 3, subtask 4 and subtask 5; the completion time of subtask 1 is 15:00 on May 10, the completion time of subtask 2 is 16:20 on May 10, the completion time of subtask 3 is 14:18 on May 10, the completion time of subtask 4 is 18:18 on May 10, and the completion time of subtask 5 is 12:18 on May 10. The completion time of subtask 4 is 18:18 on May 10, which is determined as the second update time of the write data.

[0049] Step S203: Use the first update time and the second update time of the write data as preparation progress information of the write data, and add them to the metadata of the write data in the metadata file.

[0050] The preparation progress information is used to represent the progress time of data preparation for writing data.

[0051] When the first update time and the second update time of the write data are obtained, the first update time and the second update time are used as the preparation progress information of the write data, and the preparation progress information is added to the metadata of the write data in the metadata file corresponding to the data lake, so as to update the preparation progress information of the write data, thereby improving the efficiency of data readiness detection of subsequent data calls.

[0052] In some embodiments, the target data to be called is retrieved and the metadata file corresponding to the target data is obtained from the data lake. The metadata file corresponding to the target data can be accurately queried from the data lake, which greatly improves the efficiency and accuracy of data readiness detection.

[0053] In some embodiments, the method of obtaining the metadata file corresponding to the target data from the data lake may be: obtaining a preset mapping relationship table between the target data and the metadata file, querying the metadata file matching the target data from the mapping relationship table, and obtaining the metadata file corresponding to the target data. The mapping relationship table is established in advance based on the target data and the metadata file, and the mapping relationship table can be set according to the time situation, which is not specifically limited in the embodiment of the present invention. The metadata file corresponding to the target data can be accurately queried through the mapping relationship table.

[0054] Step S102: Obtain metadata of the target data from the metadata file, and obtain preparation progress information of the target data from the metadata of the target data.

[0055] The preparation progress information includes a first update time and a second update time.

[0056] In some embodiments, metadata of the target data is obtained from the metadata file according to the data identifier of the target data, and preparation progress information of the target data is obtained from the metadata of the target data. The data identifier is used to identify a tag of the target data, and the data identifier can be set according to time conditions, which is not specifically limited in the embodiments of the present invention. Accurately obtaining metadata and preparation progress information of the target data from the metadata file can effectively improve the efficiency and accuracy of data readiness detection.

[0057] Step S103: When it is determined that the preparation progress information meets a preset progress condition, it is determined that the target data is ready.

[0058] Among them, the target data is ready, which means that the target data is in a state to be called, all data files are ready, and the downstream node can call the target data at any time, wherein the downstream node can be set according to actual conditions, and the embodiment of the present invention does not make specific limitations on this.

[0059] In some embodiments, such as Figure 3 As shown, step S103 includes sub-steps S1031 to S1033.

[0060] Sub-step S1031, obtaining the current time point, and determining the current time period of the current time point, wherein the time period is a time period divided into a time period.

[0061] The current time point is the time point when the instruction to call the target data is triggered. For example, if the time point when the instruction to call the target data is triggered is 10:00 on August 10, then the current time point is 10:00 on August 10.

[0062] It should be noted that the time period is a time period divided into a time period. The length of the time period division of the time period can be set according to the actual situation. The embodiment of the present invention does not specifically limit this. For example, the time period division length is one day, that is, one day is one time period; for another example, the time period division length is one week, that is, one week is one time period. Exemplarily, when the time period is one day, 00:00 to 23:59 on August 1 is one time period, and 00:00 to 23:59 on August 2 is one time period. For another example, when the time period is one week, 00:00 on Monday to 23:59 on Sunday is one time period.

[0063] Exemplarily, the time period is 1 day, the current time point is August 10, and the current time period of the current time point August 10 is the August 10 time period.

[0064] Exemplarily, the time period is 1 week, the current time point is August 10, and the current time point August 10 is in the 33rd week, so the current time period at the current moment is a 33-week time period.

[0065] Sub-step S1032: determine a first time period in which the first update time is located, and determine a second time period in which the second update time is located.

[0066] Exemplarily, the time period is 1 day, the first update time is 10:00 on August 10, and the first time period in which the first update time 10:00 on August 10 is the August 10 time period.

[0067] Exemplarily, the time period is 1 week, the first update time is August 10, and the first update time is August 10, which is in the 33rd week. Therefore, the first time period of the first update time is a 33-week time period.

[0068] Exemplarily, the time period is 1 day, the second update time is 10:00 on August 11, and the second time period in which the second update time 10:00 on August 11 is the August 11 time period.

[0069] Exemplarily, the time period is 1 week, the second update time is August 11, and the second update time is in the 33rd week. Therefore, the second time period in which the second update time is located is a 33-week time period.

[0070] Sub-step S1033: when the first time period and the second time period are not in the same time period as the current time period, determine that the target data is ready.

[0071] In some embodiments, when the first time period and the current time period are not in the same time period, and the second time period and the current time period are not in the same time period, it is determined that the target data is ready. By determining the first time period, the second time period and the current time period, it is possible to accurately know that the target data is ready, which greatly improves the efficiency and accuracy of data readiness detection.

[0072] Exemplarily, the time period is 1 day, the first time period is the August 10th time period, the second time period is the August 11th time period, and the current time period is the August 12th time period. It can be seen that the first time period August 10th time period and the second time period August 11th time period are not within 1 day of the same time period as the current time period August 12th time period, and it is determined that the target data is ready.

[0073] Exemplarily, the time period is 1 week, the first time period is the 32nd week time period, the second time period is the 32nd week time period, and the current time period is the 33rd week time period. It can be seen that the 32nd week time period of the first time period and the 32nd week time period of the second time period are not in the same time period of 1 week as the 33rd week time period of the current time period, and it is determined that the target data is ready.

[0074] In some embodiments, such as Figure 4 As shown, step S103 includes sub-step S1034.

[0075] Sub-step S1034: when the second time period and the current time period are not in the same time period and the first update time is empty, determine that the target data is ready.

[0076] In some embodiments, when the second time period and the current time period are not in the same time period and the first update time is empty, it is determined that the target data is ready. The data readiness of the target data can be accurately detected through the second time period and the first update time.

[0077] Exemplarily, the time period is 1 day, the second time period is the August 11 time period, the current time period is the August 12 time period, and the first update time is empty. Since the second time period August 11 time period and the current time period August 12 time period are not in the same time period, and the first update time is empty, it can be known that the target data is ready.

[0078] Exemplarily, the time period is 1 week, the second time period is the 32nd week time period, the current time period is the 33rd week time period, and the first update time is empty. Since the 32nd week time period of the second time period and the 33rd week time period of the current time period are not in the same time period of 1 week, and the first update time is empty, it is determined that the target data is ready.

[0079] The data readiness detection method provided in the above embodiment obtains the target data to be called, and obtains the metadata file corresponding to the data lake to which the target data belongs, the metadata file includes the metadata corresponding to each data in the data lake; obtains the metadata of the target data from the metadata file, and obtains the preparation progress information of the target data from the metadata of the target data; and accurately determines that the target data is ready when it is determined that the preparation progress information meets the preset progress conditions. In the present application, by obtaining the preparation progress information of the target data from the metadata of the target data, and judging whether the preset progress conditions are met based on the preparation progress information, when the preparation progress information meets the preset progress conditions, it can be accurately known that the target data is ready, which greatly improves the efficiency and accuracy of the judgment of the readiness of the target data, thereby greatly improving the efficiency of data calling.

[0080] See also Figure 5 , Figure 5 A flowchart of another data readiness detection method provided in an embodiment of the present application.

[0081] like Figure 5 As shown, the data readiness detection method includes steps S301 to S305.

[0082] Step S301: Obtain target data to be called, and obtain a metadata file corresponding to a data lake to which the target data belongs, wherein the metadata file includes metadata corresponding to each data in the data lake.

[0083] In some embodiments, the target data to be called is retrieved and the metadata file corresponding to the target data is obtained from the data lake. The metadata file corresponding to the target data can be accurately queried from the data lake, which greatly improves the efficiency and accuracy of data readiness detection.

[0084] In some embodiments, the method of obtaining the metadata file corresponding to the target data from the data lake may be: obtaining a preset mapping relationship table between the target data and the metadata file, querying the metadata file matching the target data from the mapping relationship table, and obtaining the metadata file corresponding to the target data. The mapping relationship table is established in advance based on the target data and the metadata file, and the mapping relationship table can be set according to the time situation, which is not specifically limited in the embodiment of the present invention. The metadata file corresponding to the target data can be accurately queried through the mapping relationship table.

[0085] Step S302: Obtain metadata of the target data from the metadata file, and obtain preparation progress information of the target data from the metadata of the target data.

[0086] In some embodiments, metadata of the target data is obtained from the metadata file according to the data identifier of the target data, and preparation progress information of the target data is obtained from the metadata of the target data. The data identifier is used to identify a tag of the target data, and the data identifier can be set according to time conditions, which is not specifically limited in the embodiments of the present invention. Accurately obtaining metadata and preparation progress information of the target data from the metadata file can effectively improve the efficiency and accuracy of data readiness detection.

[0087] Step S303: Acquire the current time point, and determine the time difference between the first update time and the current time point to obtain a first time difference.

[0088] In some embodiments, the current time point is obtained, and the time difference between the first update time and the current time point is calculated to obtain the first time difference. By calculating the time difference between the first update time and the current time point, the first time difference can be accurately obtained.

[0089] For example, the current time point is 10:00 on August 10, and the first update time is 8:00 on August 10. The time difference between the current time point 10:00 on August 10 and the first update time 8:00 on August 10 is calculated, and the first time difference is 2 hours.

[0090] Step S304: determine the time difference between the second update time and the current time point to obtain a second time difference.

[0091] In some embodiments, the time difference between the second update time and the current time point is calculated to obtain the second time difference. By calculating the time difference between the second update time and the current time point, the second time difference can be accurately obtained.

[0092] For example, the current time point is 10:00 on August 10, and the second update time is 9:00 on August 10. The time difference between the current time point 10:00 on August 10 and the second update time 9:00 on August 10 is calculated, and the second time difference is 1 hour.

[0093] Step S305: When the first time difference and the second time difference are both less than or equal to a preset time difference, determine that the target data is ready.

[0094] The preset time difference may be set according to actual conditions, and the embodiment of the present invention does not specifically limit this. For example, the preset time difference may be set to 5 hours.

[0095] In some embodiments, when the first time difference and the second time difference are both less than or equal to the preset time difference, it is determined that the target data is ready. The target data can be accurately known to be ready based on the first time difference, the second time difference and the preset time difference.

[0096] Exemplarily, the preset time difference is 6 hours, the first time difference is 3 hours and the second time difference is 2 hours. Therefore, the first time difference of 3 hours and the second time difference of 2 hours are both smaller than the preset time difference of 6 hours, and it is determined that the target data is ready.

[0097] In some embodiments, metadata of the target data is obtained from a metadata file, and preparation progress information of the target data is obtained from the metadata of the target data. When the preset progress condition is not satisfied according to the preparation progress information, it is determined that the data readiness test of the target data has not passed, and the number of tests of the target data is updated; when it is determined that the number of tests is less than or equal to the preset number of times, the step of obtaining metadata of the target data from the metadata file and obtaining preparation progress information of the target data from the metadata of the target data is returned to be executed; when it is determined that the number of tests is greater than or equal to the preset number of times, it is determined that the target data is not ready. The preset number of times can be set according to actual conditions, and the embodiment of the present invention does not specifically limit this. For example, the preset number of times can be set to 5 times. By testing the target data that has failed the data readiness test multiple times, the accuracy of the data readiness test can be improved.

[0098] In some embodiments, the method for determining that the preparation progress information does not meet the preset progress condition can be: when the first time difference and the second time difference are both greater than the preset time difference, and at least one of the first time period and the second time period is in the same time period as the current time period, it is determined that the preparation progress information does not meet the preset progress condition.

[0099] Exemplarily, when the first time difference and the second time difference are both greater than the preset time difference, and the first time period and the second time period are both in the same time period as the current time period, it is determined that the preparation progress information does not meet the preset progress condition.

[0100] The above-mentioned data readiness detection method obtains the preparation progress information of the target data from the metadata of the target data, and judges whether the preset progress conditions are met based on the preparation progress information. When the preparation progress information meets the preset progress conditions, it can accurately know that the target data is ready, which greatly improves the efficiency and accuracy of the target data readiness judgment, thereby greatly improving the efficiency of data calling.

[0101] See also Figure 6 , Figure 6 A schematic block diagram of a data readiness detection device provided in an embodiment of the present application.

[0102] like Figure 6 As shown, the data readiness detection device 400 includes an acquisition module 410 and a determination module 420, wherein:

[0103] The acquisition module 410 is used to acquire the target data to be called, and acquire the metadata file corresponding to the data lake to which the target data belongs, wherein the metadata file includes metadata corresponding to each data in the data lake;

[0104] The acquisition module 410 is further used to acquire the metadata of the target data from the metadata file, and acquire the preparation progress information of the target data from the metadata of the target data;

[0105] The determination module 420 is configured to determine that the target data is ready when it is determined that the preparation progress information satisfies a preset progress condition.

[0106] In some embodiments, the data readiness detection device 400 is further used for:

[0107] For each written data in the data lake, determining a first update time of the written data according to the execution time of each subtask of the written data;

[0108] Determining a second update time of the written data according to the completion time of each of the subtasks;

[0109] The first update time and the second update time of the write data are used as preparation progress information of the write data, and are added to the metadata of the write data in the metadata file.

[0110] In some embodiments, the data readiness detection device 400 is further used for:

[0111] Selecting the execution time closest to the current timestamp from the execution time of each subtask of writing data;

[0112] The execution time closest to the current timestamp is determined as the first update time of the written data.

[0113] In some embodiments, Figure 7 As shown, the data readiness detection device 500 includes an acquisition module 510, a generation module 520 and a determination module 530, wherein:

[0114] The acquisition module 510 is used to obtain the current time point

[0115] The generating module 520 is used to determine the time difference between the first update time and the current time point to obtain a first time difference;

[0116] The generating module 520 is further configured to determine a time difference between the second update time and the current time point to obtain a second time difference;

[0117] The determination module 530 is configured to determine that the target data is ready when the first time difference and the second time difference are both less than or equal to a preset time difference.

[0118] In some embodiments, the determining module 420 is further configured to:

[0119] Obtaining a current time point, and determining a current time period in which the current time point is located, wherein the time period is a time period divided into a time period;

[0120] Determine a first time period in which the first update time is located, and determine a second time period in which the second update time is located;

[0121] When the first time period and the second time period are not in the same time period as the current time period, it is determined that the target data is ready.

[0122] In some embodiments, the determining module 420 is further configured to:

[0123] In a case where the second time period and the current time period are not in the same time period and the first update time is empty, it is determined that the target data is ready.

[0124] In some embodiments, the data readiness detection device 400 is further used for:

[0125] When it is determined that the preparation progress information does not meet the preset progress condition, updating the detection times of the target data;

[0126] When it is determined that the number of detections is less than or equal to the preset number of times, returning to the step of acquiring metadata of the target data from the metadata file, and acquiring preparation progress information of the target data from the metadata of the target data;

[0127] When it is determined that the detection number is greater than or equal to the preset number, it is determined that the target data is not ready.

[0128] It should be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-mentioned data readiness detection device can refer to the corresponding process in the above-mentioned data readiness detection method embodiment, and will not be repeated here.

[0129] See also Figure 8 , Figure 8 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0130] like Figure 8 As shown, the computer device 600 includes a processor 602 and a memory 603 connected via a system bus 601 , wherein the memory may include a storage medium and an internal memory.

[0131] The storage medium may store a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor 602 may execute any one of the data readiness detection methods.

[0132] The processor 602 is used to provide computing and control capabilities and support the operation of the entire computer device.

[0133] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any data readiness detection method.

[0134] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] It should be understood that the processor 602 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0136] In one embodiment, the processor 602 is used to run a computer program stored in the memory to implement the following steps:

[0137] Obtain target data to be called, and obtain a metadata file corresponding to the data lake to which the target data belongs, wherein the metadata file includes metadata corresponding to each data in the data lake;

[0138] Acquire metadata of the target data from the metadata file, and acquire preparation progress information of the target data from the metadata of the target data;

[0139] When it is determined that the preparation progress information satisfies a preset progress condition, it is determined that the target data is ready.

[0140] In one embodiment, the processor 602 includes multiple subtasks in implementing the target data; before obtaining the target data to be called, it is also used to implement:

[0141] For each written data in the data lake, determining a first update time of the written data according to the execution time of each subtask of the written data;

[0142] Determining a second update time of the written data according to the completion time of each of the subtasks;

[0143] The first update time and the second update time of the write data are used as preparation progress information of the write data, and are added to the metadata of the write data in the metadata file.

[0144] In one embodiment, when the processor 602 determines the first update time of the write data according to the execution time of each subtask of the write data for each write data in the data lake, it is configured to implement:

[0145] Selecting the execution time closest to the current timestamp from the execution time of each subtask of writing data;

[0146] The execution time closest to the current timestamp is determined as the first update time of the written data.

[0147] In one embodiment, the processor 602 implements that the preparation progress information includes a first update time and a second update time; when it is determined that the preparation progress information satisfies a preset progress condition, when it is determined that the target data is ready, to implement:

[0148] Obtaining the current time point, and determining the time difference between the first update time and the current time point, to obtain a first time difference;

[0149] Determine the time difference between the second update time and the current time point to obtain a second time difference;

[0150] When the first time difference and the second time difference are both smaller than or equal to a preset time difference, it is determined that the target data is ready.

[0151] In one embodiment, the processor 602 implements that the preparation progress information includes a first update time and a second update time; when it is determined that the preparation progress information satisfies a preset progress condition, when it is determined that the target data is ready, to implement:

[0152] Obtaining a current time point, and determining a current time period in which the current time point is located, wherein the time period is a time period divided into a time period;

[0153] Determine a first time period in which the first update time is located, and determine a second time period in which the second update time is located;

[0154] When the first time period and the second time period are not in the same time period as the current time period, it is determined that the target data is ready.

[0155] In one embodiment, when the processor 602 determines that the target data is ready when determining that the preparation progress information satisfies a preset progress condition, it is configured to implement:

[0156] In a case where the second time period and the current time period are not in the same time period and the first update time is empty, it is determined that the target data is ready.

[0157] In one embodiment, after obtaining the metadata of the target data from the metadata file and obtaining the preparation progress information of the target data from the metadata of the target data, the processor 602 is further configured to:

[0158] When it is determined that the preparation progress information does not meet the preset progress condition, updating the detection times of the target data;

[0159] When it is determined that the number of detections is less than or equal to the preset number of times, returning to the step of acquiring metadata of the target data from the metadata file, and acquiring preparation progress information of the target data from the metadata of the target data;

[0160] When it is determined that the detection number is greater than or equal to the preset number, it is determined that the target data is not ready.

[0161] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the above description of the specific working process of the computer device can refer to the corresponding process in the aforementioned data readiness detection method embodiment, which will not be repeated here.

[0162] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the data readiness detection method of the present application.

[0163] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may be non-volatile or volatile. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0164] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0165] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0166] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0167] It should also be understood that the term "and / or" used in the present specification refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0168] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application.

Claims

1. A data readiness detection method, characterized in that: include: Obtain target data to be called, and obtain a metadata file corresponding to the data lake to which the target data belongs, wherein the metadata file includes metadata corresponding to each data in the data lake; Acquire metadata of the target data from the metadata file, and acquire preparation progress information of the target data from the metadata of the target data; When it is determined that the preparation progress information satisfies a preset progress condition, it is determined that the target data is ready.

2. The data readiness detection method according to claim 1, characterized in that: The target data includes a plurality of subtasks; Before obtaining the target data to be called, the method further includes: For each written data in the data lake, determining a first update time of the written data according to the execution time of each subtask of the written data; Determining a second update time of the written data according to the completion time of each of the subtasks; The first update time and the second update time of the write data are used as preparation progress information of the write data, and are added to the metadata of the write data in the metadata file.

3. The data readiness detection method according to claim 2, characterized in that: The step of determining, for each written data in the data lake, a first update time of the written data according to the execution time of each subtask of the written data, includes: Selecting the execution time closest to the current timestamp from the execution time of each subtask of writing data; The execution time closest to the current timestamp is determined as the first update time of the written data.

4. The data readiness detection method according to claim 1, characterized in that: The preparation progress information includes a first update time and a second update time; The step of determining that the target data is ready when it is determined that the preparation progress information satisfies a preset progress condition includes: Obtaining the current time point, and determining the time difference between the first update time and the current time point, to obtain a first time difference; Determine the time difference between the second update time and the current time point to obtain a second time difference; When the first time difference and the second time difference are both smaller than or equal to a preset time difference, it is determined that the target data is ready.

5. The data readiness detection method according to claim 1, characterized in that: The preparation progress information includes a first update time and a second update time; The step of determining that the target data is ready when it is determined that the preparation progress information satisfies a preset progress condition includes: Obtaining a current time point, and determining a current time period in which the current time point is located, wherein the time period is a time period divided into a time period; Determine a first time period in which the first update time is located, and determine a second time period in which the second update time is located; When the first time period and the second time period are not in the same time period as the current time period, it is determined that the target data is ready.

6. The data readiness detection method according to claim 5, characterized in that: The step of determining that the target data is ready when it is determined that the preparation progress information satisfies a preset progress condition includes: In a case where the second time period and the current time period are not in the same time period and the first update time is empty, it is determined that the target data is ready.

7. The data readiness detection method according to any one of claims 1 to 6, characterized in that: After acquiring the metadata of the target data from the metadata file and acquiring the preparation progress information of the target data from the metadata of the target data, the method further includes: When it is determined that the preparation progress information does not meet the preset progress condition, updating the detection times of the target data; When it is determined that the number of detections is less than or equal to the preset number of times, returning to the step of acquiring metadata of the target data from the metadata file, and acquiring preparation progress information of the target data from the metadata of the target data; When it is determined that the detection number is greater than or equal to the preset number, it is determined that the target data is not ready.

8. A data readiness detection device, characterized in that: The data readiness detection device comprises an acquisition module and a determination module, wherein: The acquisition module is used to acquire the target data to be called, and acquire the metadata file corresponding to the data lake to which the target data belongs, wherein the metadata file includes metadata corresponding to each data in the data lake; The acquisition module is further used to acquire the metadata of the target data from the metadata file, and acquire the preparation progress information of the target data from the metadata of the target data; The determining module is used to determine that the target data is ready when it is determined that the preparation progress information meets a preset progress condition.

9. A computer device, characterized in that: The computer device comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the data readiness detection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the data readiness detection method according to any one of claims 1 to 7 are implemented.

Images