Data processing method and device, equipment and storage medium

Abstract

The invention discloses a data processing method and device, equipment and a storage medium, and relates to the technical field of cloud computing. In the process of executing the first data processing process, the data verification process is called in parallel, and data verification is carried out on the actual processing amount and the theoretical processing amount in the preset time period before the preset verification time point in the process. When the verification result is that the difference value between the actual processing amount and the theoretical processing amount meets the preset correction condition, a second data processing process with the same processing logic as the first data processing process is called, data generated by the data source in the preset time period is processed, and a data correction result without abnormity is obtained; and utilizing the data correction result to cover the data processing result obtained by the first data processing process stored in the data lake bin, and correcting the stored data of the data lake bin. On the basis, the timeliness and efficiency of data verification and correction are improved, and normal operation of the data processing process is not affected.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and more specifically, to a data processing method, device, equipment and storage medium. Background Art

[0002] A data lake warehouse is a new data architecture that implements the data structure and data management functions of a data warehouse based on the low-cost storage of a data lake. It is also called "lake-warehouse integration", such as the data lake warehouse built on Iceberg. The data lake warehouse can meet the diverse needs of developers for data storage, processing, and analysis. It can be used to store data processing results of data processing processes. To ensure the correctness of the data stored in the data lake warehouse, ensuring the accuracy of the data in the data processing process and correcting faulty data are basic requirements.

[0003] In order to meet this demand, the usual practice is to discover data anomalies in the data processing process and the data lake warehouse based on abnormal alarms or inaccurate data processing results during the data processing process. Then, by manually proofreading the abnormal data and the data lake warehouse data, the data anomalies are confirmed, and targeted fault correction is performed. In this case, there are problems with the timeliness and efficiency of data verification and correction, which causes the data lake warehouse to be unable to operate normally for a long time. Summary of the invention

[0004] In view of the above problems, the present application provides a data processing method, device, equipment and storage medium to achieve the purpose of improving the timeliness and efficiency of data verification and correction. The specific scheme is as follows:

[0005] The first aspect of the present application provides a data processing method, comprising:

[0006] In the process of executing the first data processing process, when it is detected that a preset verification time point is reached, the data verification process is called in parallel to obtain the data volume of the data processed by the first data processing process in a preset time period before the preset verification time point as the actual processing volume, and obtain the theoretical processing volume of the first data processing process in the preset time period, and verify the actual processing volume based on the theoretical processing volume, wherein the theoretical processing volume is the data volume of the data generated by the data source in the preset time period;

[0007] When the verification result of the data verification process is that the difference between the actual processing amount and the theoretical processing amount meets the preset correction condition, all data generated by the data source within the preset time period are obtained, and a second data processing process is called to process all data generated by the data source within the preset time period to obtain a data correction result, and the processing logic of the second data processing process is the same as that of the first data processing process;

[0008] The data correction result is used to overwrite the data processing result of the first data processing process within the preset time period stored in the data lake warehouse.

[0009] In a possible implementation, using the data correction result to overwrite the data processing result of the first data processing process within the preset time period stored in the data lake warehouse includes:

[0010] Obtaining a partition granularity of the data lake warehouse, where the partition granularity is a granularity for dividing the storage space of the data lake warehouse according to time;

[0011] Based on the partition granularity, determining the target partition corresponding to the preset time period;

[0012] The data correction result is used to overwrite the data processing result of the first data processing process within the preset time period stored in the target partition of the data lake warehouse.

[0013] In a possible implementation, obtaining the theoretical processing amount of the first data processing process within the preset time period includes:

[0014] Obtaining, from an offline data storage system that pre-stores all data generated by the data source, at least one backup log stored in a storage directory corresponding to the preset time period, wherein the backup log includes all data generated by the data source between adjacent backup time points, and the storage directory in the offline data storage system is divided according to time;

[0015] The data volume of all data generated by the data source in the backup log within the preset time period is counted as the theoretical processing volume.

[0016] In a possible implementation, the process of pre-storing all data generated by the data source to the offline data storage system includes:

[0017] The time starts from the last backup time point and all data generated by the data source are recorded in real time;

[0018] When the timing reaches the preset backup interval, the current time is used as the backup time point, and all data generated by the data source recorded within the preset backup interval are compressed to obtain a backup log;

[0019] Determine a time period corresponding to the backup log based on the backup time point and the last backup time point;

[0020] Based on the time period, the backup log is stored in a preset directory of the offline data storage system, where the preset directory is a storage directory of the offline data storage system divided according to time.

[0021] In a possible implementation, when the actual processing amount and the theoretical processing amount are both total data amounts of data, verifying the actual processing amount based on the theoretical processing amount includes:

[0022] Determine the total data volume difference between the actual processing volume and the theoretical processing volume;

[0023] It is determined whether the total data volume difference satisfies a preset correction condition, wherein the preset correction condition at least includes: the total data volume difference exceeds a preset data volume threshold.

[0024] In a possible implementation, when the actual processing amount and the theoretical processing amount are both total data amounts of data, verifying the actual processing amount based on the theoretical processing amount includes:

[0025] Determine the total data volume difference between the actual processing volume and the theoretical processing volume;

[0026] It is determined whether the total data volume difference satisfies a preset correction condition, wherein the preset correction condition at least includes: the total data volume difference exceeds a preset data volume threshold.

[0027] In a possible implementation, when the verification result of the data verification process is that the difference between the actual processing amount and the theoretical processing amount meets the preset correction condition, the data processing method further includes: sending an alarm message indicating data processing abnormality to the front end.

[0028] A second aspect of the present application provides a data processing device, including:

[0029] a data verification unit, configured to, during the execution of a first data processing process, in parallel call a data verification process when detecting that a preset verification time point has been reached, so as to obtain, as an actual processing amount, a data volume of data processed by the first data processing process within a preset time period before the preset verification time point, and to obtain a theoretical processing volume of the first data processing process within the preset time period, and verify the actual processing volume based on the theoretical processing volume, wherein the theoretical processing volume is a data volume of data generated by a data source within the preset time period;

[0030] a data processing unit, configured to obtain all data generated by the data source within the preset time period, and call a second data processing process to process all data generated by the data source within the preset time period to obtain a data correction result, when the verification result of the data verification process is that the difference between the actual processing amount and the theoretical processing amount meets a preset correction condition, wherein the processing logic of the second data processing process is the same as that of the first data processing process;

[0031] A data correction unit is used to overwrite the data processing results of the first data processing process within the preset time period stored in the data lake warehouse with the data correction results.

[0032] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0033] The memory is used to store computer programs;

[0034] The processor is used to execute the computer program so that the electronic device can implement any one of the data processing methods.

[0035] In a fourth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the data processing methods described.

[0036] It can be seen from the above technical solutions that the data processing method provided by the embodiment of the present application, during the normal operation of the data processing process, at the preset verification time point, calls the data verification process in parallel, and verifies the actual processing volume and theoretical processing volume in the preset time period before the preset verification time point in the process, so as to verify whether the data processing process exists in the preset time period, and the abnormal situation caused by the lack of data to be processed. If it exists, the second data processing process with the same processing logic as the first data processing process is called again, and the data generated by the data source in the preset time period, that is, the complete data to be processed, is processed to obtain a data correction result without abnormalities, and the data correction result is used to overwrite the data processing result obtained by the first data processing process stored in the data lake warehouse, and the stored data in the data lake warehouse is corrected.

[0037] Compared with the method in the prior art of manually verifying and repairing data after the data processing process is completed, the present application calls other processes in parallel during the execution of the data processing process to complete the verification and correction of the data, thereby improving the timeliness and efficiency of data verification and correction without affecting the normal operation of the data processing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0039] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0040] Figure 2 A system architecture diagram for implementing a data processing method provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] The process of data processing is: obtaining / reading the data to be processed from the data source, processing the data to be processed according to the preset operation logic / processing logic, obtaining the data processing results, and writing the data processing results into the result table or database.

[0045] To avoid inaccurate data processing results, the existing technology will verify the correctness and accuracy of the data results after the data processing is completed. If the data results are inaccurate, the data processing process will be re-executed; or, the data processing process will be verified. Once an alarm of abnormal processing occurs, the data processing process will be stopped. After the abnormal processing failure is resolved, the data will be re-processed to make the data processing results written into the result table more accurate.

[0046] The applicant considers that discovering data anomalies through interruption of the data processing process or inaccurate data processing results will affect the normal operation of the data processing process, and the timeliness of data correction will be greatly reduced. Therefore, the applicant believes that the data processing process can be verified and corrected in parallel without affecting the data processing process. However, due to the different data processing logic / operation logic of different data processing processes, it is difficult to verify the process data. Therefore, the applicant believes that the data processed by the data processing process can be verified, that is, the data to be processed can be verified. It is understandable that once the data to be processed is abnormal, the data processing process and the data processing results must be inaccurate.

[0047] Based on the above-mentioned inventive concept, an embodiment of the present application provides a data processing method, which can be applied in a data processing system. The data processing system at least includes: a data source module, a data processing module and a data storage module. In the embodiment of the present application, the data storage module can be a data lake warehouse constructed by Iceberg, the data source module can include a data production server, etc., and the data processing module can be a data processing program, or a processor with data processing capabilities.

[0048] Reference Figure 1 , a flow chart of a data processing method provided in an embodiment of the present application is provided to illustrate the data processing method, and the specific process may include:

[0049] Step S110, during the execution of the first data processing process, when it is detected that a preset verification time point has been reached, a data verification process is called in parallel.

[0050] The preset verification time point is the time point for verifying the data processing process. A timer trigger program can be set in the first data processing process to trigger the data verification program at a preset time point. Alternatively, a preset verification time interval is set according to the required verification frequency of the data processing. For example, the preset verification time interval can be set to 1s, 1h, etc. According to the preset verification time interval and the time point of the last verification, the time point of the next verification is determined and used as the preset verification time point. When the preset verification time point is reached, the data verification program is triggered, and based on the following steps S120 and S130, the data between the time point of the last verification and the current preset verification time point is verified.

[0051] Step S120, obtaining the data volume processed by the first data processing process within a preset time period before the preset verification time point as the actual processing volume, and obtaining the theoretical processing volume of the first data processing process within the preset time period.

[0052] Step S130, verifying the actual processing capacity based on the theoretical processing capacity.

[0053] It can be understood that the amount of data consumed by the data processing process every second depends on the amount of data generated by the data source in that second. Therefore, the amount of data generated by the data source is the amount of data theoretically processed by the first data processing process, where "consumption" refers to the process of reading and processing from the data source.

[0054] In the embodiment of the present application, the amount of data actually processed by the first data processing process, i.e., the actual processing amount, is verified with the amount of data theoretically processed, i.e., the theoretical processing amount, to determine whether the first data processing process has data anomalies. The theoretical processing amount is the amount of data generated by the data source within a preset time period, and the preset time period can be the time period from the last verification time point to the current preset verification time point, so as to realize data verification in stages during the execution of the data processing process.

[0055] The method of obtaining the actual processing volume and the theoretical processing volume can be determined according to the length of the preset time period. If the preset time period is short, it can be almost regarded as a real-time verification, and the data verification process is called to count the data volume processed by the first data processing process in real time, and count the data volume generated by the data source in real time, and verify the two counted data volumes. If the length of the preset time period is long, and the data verification process cannot be triggered in real time for statistics, then in the embodiment of the present application, the data of the data processing process and the data source can be backed up to avoid data loss, resulting in the inability to count the data volume. After the data verification process is triggered, the backup files corresponding to the first data processing process and the data source are obtained, and the data volume within the preset time period is counted to obtain the actual processing volume and the theoretical data volume, and verify it. Optionally, the method of obtaining the actual processing volume and the theoretical data volume is not uniquely limited in the embodiment of the present application, and the method that can ensure that the statistical data is not lost and the data volume statistics are accurate can be adopted.

[0056] Furthermore, the actual processing volume and theoretical data volume obtained are verified. Optionally, the actual processing volume is compared to see if they are equal to the theoretical processing volume. If they are not equal, and the actual processing volume is less than the theoretical processing volume, it proves that there is a problem of missing data in the processing process of the first data processing process within the preset time period. The reason for the missing data may be that the first data processing process is unable to obtain complete data to be processed from the data source due to problems such as permission restrictions or program loopholes in the first data processing process, or data is omitted during the data processing process. Therefore, in order to ensure the correctness of the data processing results, the data results of the preset time period should be further corrected.

[0057] However, it is understandable that when the data generated by the data source is transmitted to the first data processing process, due to compression, format conversion, etc., the data processed by the first data processing process is the same as the data generated by the data source, but due to factors such as the data format, the data volume is different. Therefore, the embodiment of the present application can also set a preset data volume deviation value / threshold when verifying the actual processing volume and the theoretical processing volume. As long as the difference between the actual processing volume and the theoretical processing volume does not exceed the threshold, it can be determined that the data verification result is normal.

[0058] In one possible implementation, when both the theoretical processing amount and the actual processing amount are the number of rows of data, the process of verifying the actual processing amount based on the theoretical processing amount may include: determining the difference in the number of rows between the actual processing amount and the theoretical processing amount; judging whether the difference in the number of rows satisfies a preset correction condition, and the preset correction condition at least includes: the difference in the number of rows exceeds a preset row number threshold.

[0059] The number of rows of data refers to the number of records contained in a data set, and each row represents an independent entity or record. In this embodiment, the actual processing volume and the theoretical processing volume are both obtained by counting the number of rows of data, and each row of data may record multiple key data. If the number of rows is not equal, the resulting data processing error exceeds one thousandth, so the preset row number threshold can be set to 0. When the difference between the row data of the actual processing volume and the theoretical processing volume is greater than 0, it can be determined as data abnormality.

[0060] In another possible implementation, when the theoretical processing amount and the actual processing amount are both the total data amount of data, the process of verifying the actual processing amount based on the theoretical processing amount may include: determining the total data amount difference between the actual processing amount and the theoretical processing amount; judging whether the total data amount difference meets the preset correction conditions, and the preset correction conditions at least include: the total data amount difference exceeds the preset data amount threshold.

[0061] Data volume refers to the total amount of data stored in a database or data set, which can be comprehensively measured from multiple dimensions such as storage space, number of rows, and number of fields.

[0062] It is understandable that there is no particular restriction on the fields contained in a row of data, and therefore, the amount of data corresponding to a row of data is also different. The above-mentioned method of comparing the number of rows is a relatively coarse-grained verification method. In the embodiment of the present application, if the obtained theoretical processing volume and the actual processing volume both represent the total data volume, the verification accuracy is higher.

[0063] Similarly, by judging whether the total data volume difference between the theoretical processing volume and the actual processing volume exceeds a preset threshold, it is determined whether data anomalies occur in the first data processing process within a preset time period.

[0064] In an embodiment of the present application, the preset data volume threshold and the preset row number threshold in the preset correction condition can be set according to the accuracy requirements of the developer or the data processing system. If the accuracy requirements for the data processing results are high, the preset data volume threshold and the preset row number threshold can be set to 0. If a certain error is allowed in the data processing results, the preset quantity threshold and the preset row number threshold can be set to a value greater than 0 based on the allowed error. Based on this, it can be avoided that in the event of a statistical error in the data volume, the data in the data lake warehouse is frequently corrected, resulting in the data processing results of the first data processing process being unusable, thereby wasting data processing resources.

[0065] Step S140, when the verification result of the data verification process is that the difference between the actual processing amount and the theoretical processing amount meets the preset correction condition, all data generated by the data source within the preset time period are obtained.

[0066] Step S150, calling the second data processing process to process all data generated by the data source within a preset time period to obtain a data correction result.

[0067] The processing logic of the second data processing process is the same as that of the first data processing process.

[0068] When the verification result of the data verification process is that the difference between the actual processing volume and the theoretical processing volume meets the preset correction condition, that is, the verification result is data missing, an alarm message indicating data processing abnormality can be sent to the front end first, so that the developers can promptly repair the fault of the data processing system and avoid the same problem in subsequent data processing of the first data processing process.

[0069] Furthermore, the data processing result generated by the first data processing process when the data is missing is corrected. In order not to affect the normal processing of subsequent tasks of the first data processing process, the embodiment of the present application calls the second data processing process in parallel to correct the data processing result.

[0070] The second data processing process of the embodiment of the present application has the same data processing logic as the first data processing process. Therefore, all data generated by the data source within a preset time period is used as the data to be processed by the second data processing process, that is, when the data to be processed is guaranteed to be normal, the correctness of the data processing result of the second data processing process is ensured. In the embodiment of the present application, the data processing result obtained by the second data processing process is used for correction, which can be recorded as the data correction result.

[0071] Step S160: Use the data correction result to overwrite the data processing result of the first data processing process within a preset time period stored in the data lake warehouse.

[0072] The storage space of the data lake warehouse can be partitioned according to time granularity. Therefore, the data stored in the partition corresponding to the preset time period is the data processing result of the first data processing process within the preset time period. The data correction result covers the data processing result, and the abnormal data in the data lake warehouse can be corrected.

[0073] Optionally, the data in the data lake warehouse can be stored according to the time when it is written. In this case, each data corresponds to a timestamp representing the write event. Therefore, the data belonging to a preset time period is determined by the timestamp of each data. These data are the data processing results written into the data lake warehouse by the first data processing process within the preset time period.

[0074] In a possible implementation, using the data correction result to overwrite the data processing result of the first data processing process within a preset time period stored in the data lake warehouse includes: obtaining the partition granularity of the data lake warehouse, the partition granularity is the granularity of dividing the storage space of the data lake warehouse according to time; based on the partition granularity, determining the target partition corresponding to the preset time period; using the data correction result to overwrite the data processing result of the first data processing process within the preset time period stored in the target partition of the data lake warehouse.

[0075] The data lake warehouse can partition the storage space according to time. Each partition corresponds to a time period. According to the time period corresponding to each partition, the partition corresponding to the preset time period is determined. The data stored in the partition is the data processing result corresponding to the first data process in the preset time period.

[0076] However, there is a situation where the duration of the preset time period is less than the duration of the partition, then it is necessary to filter out the data stored in the partition and take the data belonging to the preset time period as the data processing result.

[0077] Therefore, in order to avoid the step of further screening the data and improve the efficiency of data correction, the embodiment of the present application can set the preset time period according to the partition granularity of the data lake warehouse. For example, the data lake warehouse stores data in days, and divides each day into 24 partitions according to hours. Each partition stores all the data processing results within this hour. Therefore, when performing data verification, the preset time period can be set to 1 hour, and the preset time point can be set to a time point such as 1:00, 2:00, etc.

[0078] Assuming that the data verification result of the preset time period 10:00-11:00 is data anomaly, all data generated by the data source in the time period 10:00-11:00 can be obtained, and data processing can be performed on it to obtain data correction results, and the data correction results can be used to cover all data stored in the corresponding partitions of 10:00-11:00 in the data lake warehouse. Based on this, the efficiency of data correction can be improved when replacing or correcting the data processing results, and the query and verification process of the data processing results can be avoided.

[0079] In summary, the data processing method provided by the embodiment of the present application, during the normal operation of the data processing process, at the preset verification time point, the data verification process is called in parallel, and the actual processing volume and theoretical processing volume in the preset time period before the preset verification time point in the process are verified to verify whether the data processing process exists in the preset time period, and the abnormal situation caused by the lack of data to be processed. If it exists, the second data processing process with the same processing logic as the first data processing process is called again, and the data generated by the data source in the preset time period, that is, the complete data to be processed, is processed to obtain a data correction result without abnormalities, and the data correction result is used to overwrite the data processing result obtained by the first data processing process stored in the data lake warehouse, and the stored data in the data lake warehouse is corrected.

[0080] Compared with the method in the prior art of manually verifying and repairing data after the data processing process is completed, the present application calls other processes in parallel during the execution of the data processing process to complete the verification and correction of the data, thereby improving the timeliness and efficiency of data verification and correction without affecting the normal operation of the data processing process.

[0081] Next, other possible implementations of the data processing method described in the embodiments of the present application are described.

[0082] In a possible implementation, step S120 obtains the theoretical processing volume of the first data processing process within a preset time period, including: obtaining at least one backup log stored in a storage directory corresponding to the preset time period from an offline data storage system that pre-stores all data generated by the data source, the backup log containing all data generated by the data source between adjacent backup time points, and the storage directory in the offline data storage system is divided according to time; and counting the data volume of all data generated by the data source within the preset time period in the backup log as the theoretical processing volume.

[0083] The embodiment of the present application obtains the data generated by the data source, and mainly saves the data generated by the data source by backing up the data generated by the data source, so as to avoid the loss of the original data, which makes it impossible to verify and correct.

[0084] The original data generated by the data source is stored in the offline data storage system in the form of a backup log, and considering that the embodiments of the present application use the time when the data was generated as the matching basis, the backup log is still stored by time in the offline data storage system, which is similar to the storage method of the data lake warehouse. Based on this, at least one corresponding backup log is determined from the offline data storage system according to the preset time period. It can be understood that the data contained in the backup log may just belong to the preset time period, but it is also possible that the data contained in the backup log contains data that is not in the preset time period. Therefore, when acquiring data, the data in the backup log that is not in the preset time period can be removed to ensure that the acquired data belongs to the data generated by the data source within the preset time period.

[0085] The data obtained from the backup log can be used for the second data processing process. The data generated by the data source is the most original data in the data processing process, and is used as the data to be processed. When there is no abnormality in the operation logic / processing logic of the second data processing process, the obtained data processing result is the correct result and can be used as the data correction result.

[0086] When calculating the theoretical processing volume, the data corresponding to the theoretical processing volume may not be taken from the backup log, but only the data corresponding to the theoretical processing volume may be calculated. The data corresponding to the theoretical processing volume may also be taken from the backup log in advance, and when the second data processing process is called, it can be executed as soon as possible without reading the offline data storage system again, thereby improving the efficiency of data correction.

[0087] In a possible implementation, the process of backing up data generated by a data source by an offline data storage system may include: starting timing from the last backup time point, and recording all data generated by the data source in real time; when the timing reaches a preset backup interval, taking the current time as the backup time point, and compressing all data generated by the data source recorded within the preset backup interval to obtain a backup log; determining a time period corresponding to the backup log based on the backup time point and the last backup time point; based on the time period, storing the backup log in a preset directory of the offline data storage system, the preset directory being a storage directory divided by the offline data storage system according to time.

[0088] The backup log / backup log file records all data generated by the data source and may contain multiple types of data, such as gateway access records, software application operation logs, software background developer logs, database change logs, etc. The data content can be determined according to the developer's verification requirements. For example, the data processing process only processes two types of data: user operation records and video playback records. When recording the data generated by the data source, all data of these two types, namely user operation records and video playback records, can be recorded.

[0089] The embodiment of the present application uses a scheduled backup method to back up the data generated by the data source. Every preset backup interval, the data generated by the recorded data source is backed up to generate a backup log.

[0090] Reference Figure 2 , a system architecture diagram for implementing a data processing method provided in an embodiment of the present application, wherein the data source is application server 1-n, and the generated data is divided into two parts, one of which is collected in real time as a Kafka message queue in the data processing process for processing by the data processing process, and the other is reserved for offline backup.

[0091] An offline transfer software is set up on each application server. The offline transfer software cuts the original log file to be read every 5 minutes. The result of the cutting is a new file and an old file. The content of the old file is all the data in the past 5 minutes, and the content of the new file is the data generated at a later time. The original log file is the file where the data source records the data generated by itself. The offline transfer software compresses the old file and then uploads it to the specified directory of HDFS to form a centralized backup.

[0092] The way the offline storage system stores backup logs is also based on time. Similar to the data lake warehouse, the main directory can be divided into units of days, such as 7-1, 7-2, etc., and 24 subdirectories are set under each main directory. Each subdirectory corresponds to one hour. Based on this, the time period corresponding to the backup log is stored in the corresponding subdirectory. For example, on July 5, the backup time point corresponding to the backup log and the last backup time point are 5:00 and 4:00, then the time period corresponding to the backup log is 4:00-5:00, and the storage directory of the backup log in the offline data storage system can be "7-5 / 4:00-5:00".

[0093] It can be understood that the embodiments of the present application uniformly use time as the basis for storing and acquiring data, which can improve the efficiency of data acquisition and correction, and the time granularity of the division can be selected according to the granularity used by the data storage system and the data lake warehouse. The division granularity of the embodiments of the present application is only for exemplary reference.

[0094] Further, refer to Figure 2 , which is an illustrative example of the practical application of the data processing method provided in the embodiments of the present application.

[0095] Figure 2The data lake warehouse in this example is a data lake warehouse built based on Iceberg. The data processing results of the data processing process are written into the Iceberg result table in the data lake warehouse. It can be understood that in order to simplify the attached figure, Figure 2 In the figure, “Iceberg result table” refers to “data lake warehouse”; the processing processes in the dotted boxes in the figure are data processing processes, box 1 represents the first data processing process, and box 2 represents the second data processing process.

[0096] The tasks undertaken by the data processing process in this example specifically refer to computing tasks based on the flink computing framework technology. The tasks are: reading data from the kafka message queue and performing a series of logical operations, such as removing duplicate data, content conversion, content association, summing, and averaging. The rules for the logical operations of each data processing process can be written by the developer and are not fixed. After the operation, the operation results are written into the iceberg result table, and the developer can also specify which table to write to.

[0097] by Figure 2 The data processing process 1 in the example is used as an example to illustrate that the processing tasks of the data processing process include: reading Kafka, logical calculation and writing to the Iceberg result table. Figure 2 In the process, Kafka cnnector is used to read the data to be processed in the Kafka message queue. Furthermore, the stream processing engine Flink is used to perform logical calculations and write the tasks of iceberg result table one by one on the read data to be processed, so as to realize the normal production of data warehouse by Flink stream tasks. Based on this, after the data processing process is executed, it will continue to run. The process will read new data from the data source without stopping. Each piece of data must be read once and can only be read once, and will not be read repeatedly.

[0098] Since the data processing process runs continuously, the amount of data consumed each second depends only on the amount of data generated by the data source. For example, if application servers 1-n generate 10 data, the data processing process should consume the latest 10 data. However, when a failure occurs in a certain link, such as a disk storing data in Kafka is broken, the data processing process only consumes 5 data, that is, the actual processing volume is 5 data. After verification with the theoretical processing volume of 10 data, it can be seen that data is missing in the data processing process, and the subsequent calculation results will inevitably be wrong.

[0099] When data is missing, call Figure 2The second data processing process in the process, or copy the first data processing process to obtain the second data processing process, and the data source data acquisition of the second data processing process should be switched from the Kafka message queue to the HDFS log file, from which the backup log file corresponding to the time period where the verification result is the data missing moment is obtained, and the Kafka-connector of the first data processing process is changed to File-connector, that is, the data reading mode of the second data processing process is switched from stream reading to batch reading. Because the backup log file backs up the complete original data, it does not need to be read one by one. Batch reading and batch operation can improve the efficiency of data correction.

[0100] Furthermore, because the first data processing process is a stream reading and stream running process, the data results generated are also written into the Iceberg result table one by one in an append-write manner, and the data results obtained by the second data processing process should replace the erroneous data results stored in the Iceberg result table. Therefore, the data results generated by the second data processing process are stored only in the Iceberg result table in a partition overlay manner.

[0101] It is understandable that Iceberg divides each day into 24 partitions, one partition for each hour. When the first data processing process runs normally, the data calculation results will be written to the corresponding partition according to the hour to which they belong. This method is the above-mentioned "append write". For example, the data calculation results generated by the first data processing process at 11:30 should be appended to the 11:00-12:00 partition in the Iceberg result table.

[0102] When the amount of data read at any time between 11:00 and 12:00 is less than the amount of data generated by the data source, all the data between 11:00 and 12:00 is obtained from the backup log, and the second data processing process is used to process the data to obtain the calculation results of all the data between 11:00 and 12:00. After the data calculation results are obtained, all the data in the 11:00-12:00 partition in the Iceberg result table is overwritten, i.e., "partition overwriting".

[0103] Based on this example, a software is developed to automatically back up the original data of production tasks and verify data consumption. When an anomaly is found, a repair task is automatically generated to correct the data without the need for human intervention.

[0104] The data processing device provided in the embodiment of the present application is described below. The data processing device described below and the data processing method described above can be referenced to each other.

[0105] First, combine Figure 3 , introduce the data processing device, such as Figure 3As shown, the data processing device may include:

[0106] The data verification unit 100 is used to, during the execution of the first data processing process, in parallel call the data verification process when detecting that a preset verification time point has been reached, so as to obtain the data volume of the data processed by the first data processing process in a preset time period before the preset verification time point as the actual processing volume, and to obtain the theoretical processing volume of the first data processing process in the preset time period, and verify the actual processing volume based on the theoretical processing volume, wherein the theoretical processing volume is the data volume of the data generated by the data source in the preset time period;

[0107] The data processing unit 200 is used to obtain all data generated by the data source within the preset time period when the verification result of the data verification process is that the difference between the actual processing amount and the theoretical processing amount meets the preset correction condition, and call a second data processing process to process all data generated by the data source within the preset time period to obtain a data correction result, and the processing logic of the second data processing process is the same as that of the first data processing process;

[0108] The data correction unit 300 is used to overwrite the data processing results of the first data processing process within the preset time period stored in the data lake warehouse with the data correction results.

[0109] In summary, in the process of normal operation of the data processing process, the embodiment of the present application calls the data verification process in parallel at the preset verification time point, and verifies the actual processing volume and theoretical processing volume in the preset time period before the preset verification time point in the process, so as to verify whether the data processing process exists in the preset time period, and the abnormal situation caused by the lack of data to be processed. If it exists, the second data processing process with the same processing logic as the first data processing process is called again, and the data generated by the data source in the preset time period, that is, the complete data to be processed, is processed to obtain a data correction result without abnormalities, and the data correction result is used to overwrite the data processing result obtained by the first data processing process stored in the data lake warehouse, and the stored data in the data lake warehouse is corrected.

[0110] Compared with the method in the prior art of manually verifying and repairing data after the data processing process is completed, the present application calls other processes in parallel during the execution of the data processing process to complete the verification and correction of the data, thereby improving the timeliness and efficiency of data verification and correction without affecting the normal operation of the data processing process.

[0111] In a possible implementation, the data correction unit includes:

[0112] A granularity acquisition subunit, used to acquire a partition granularity of the data lake warehouse, where the partition granularity is a granularity for dividing the storage space of the data lake warehouse according to time;

[0113] A partition determination subunit, configured to determine a target partition corresponding to the preset time period based on the partition granularity;

[0114] A data overwriting subunit is used to overwrite the data processing results of the first data processing process within the preset time period stored in the target partition of the data lake warehouse with the data correction results.

[0115] In a possible implementation, the data verification unit 100 includes:

[0116] A backup log acquisition subunit is used to acquire at least one backup log stored in a storage directory corresponding to the preset time period from an offline data storage system that pre-stores all data generated by the data source, wherein the backup log includes all data generated by the data source between adjacent backup time points, and the storage directory in the offline data storage system is divided according to time;

[0117] The data volume statistics subunit is used to count the data volume of all data generated by the data source in the backup log within the preset time period as the theoretical processing volume.

[0118] In a possible implementation, the functional unit for pre-storing all data generated by the data source to the offline data storage system may include:

[0119] A data recording unit, used to start timing from the last backup time point and record all data generated by the data source in real time;

[0120] A data compression unit, configured to, when the timing reaches a preset backup interval, use the current time as a backup time point, and compress all data generated by the data source recorded within the preset backup interval to obtain a backup log;

[0121] A time period determination unit, configured to determine a time period corresponding to the backup log based on the backup time point and the last backup time point;

[0122] The data storage unit is used to store the backup log in a preset directory of the offline data storage system based on the time period, and the preset directory is a storage directory of the offline data storage system divided according to time.

[0123] In a possible implementation, when the actual processing amount and the theoretical processing amount are both the number of rows of data, the data verification unit 100 includes:

[0124] A row difference determination subunit, used to determine the row quantity difference between the actual processing amount and the theoretical processing amount;

[0125] The row difference judgment subunit is used to judge whether the row quantity difference satisfies a preset correction condition, and the preset correction condition at least includes: the row quantity difference exceeds a preset row quantity threshold.

[0126] In a possible implementation, when the actual processing amount and the theoretical processing amount are both the total data amount of data, the data verification unit 100 includes:

[0127] A total difference determination subunit, used to determine the total data volume difference between the actual processing volume and the theoretical processing volume;

[0128] The total amount judgment subunit is used to judge whether the total data amount difference meets a preset correction condition, and the preset correction condition at least includes: the total data amount difference exceeds a preset data amount threshold.

[0129] In a possible implementation, the data processing device further includes:

[0130] The abnormal alarm unit is used to send an alarm message indicating data processing abnormality to the front end when the verification result of the data verification process is that the difference between the actual processing amount and the theoretical processing amount meets the preset correction condition.

[0131] The data processing device provided in the embodiment of the present application can be applied to electronic equipment for data processing.

[0132] Figure 4 A schematic diagram of the structure of an electronic device is shown. Figure 4 The structure of the electronic device may include: at least one processor 10, at least one memory 20, at least one communication bus 30 and at least one communication interface 40.

[0133] In the embodiment of the present application, the number of the processor 10 , the memory 20 , the communication bus 30 , and the communication interface 40 is at least one, and the processor 10 , the memory 20 , and the communication interface 40 communicate with each other through the communication bus 30 .

[0134] The processor 10 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0135] The memory 20 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory.

[0136] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement each processing flow in the aforementioned data processing method.

[0137] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the data processing methods described.

[0138] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0139] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0140] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that: include: In the process of executing the first data processing process, when it is detected that a preset verification time point is reached, the data verification process is called in parallel to obtain the data volume of the data processed by the first data processing process in a preset time period before the preset verification time point as the actual processing volume, and obtain the theoretical processing volume of the first data processing process in the preset time period, and verify the actual processing volume based on the theoretical processing volume, wherein the theoretical processing volume is the data volume of the data generated by the data source in the preset time period; When the verification result of the data verification process is that the difference between the actual processing amount and the theoretical processing amount meets the preset correction condition, all data generated by the data source within the preset time period are obtained, and a second data processing process is called to process all data generated by the data source within the preset time period to obtain a data correction result, and the processing logic of the second data processing process is the same as that of the first data processing process; The data correction result is used to overwrite the data processing result of the first data processing process within the preset time period stored in the data lake warehouse.

2. The data processing method according to claim 1, characterized in that: The using the data correction result to overwrite the data processing result of the first data processing process within the preset time period stored in the data lake warehouse includes: Obtaining a partition granularity of the data lake warehouse, where the partition granularity is a granularity for dividing the storage space of the data lake warehouse according to time; Based on the partition granularity, determining the target partition corresponding to the preset time period; The data correction result is used to overwrite the data processing result of the first data processing process within the preset time period stored in the target partition of the data lake warehouse.

3. The data processing method according to claim 1, characterized in that: The obtaining of the theoretical processing amount of the first data processing process within the preset time period includes: Obtaining, from an offline data storage system that pre-stores all data generated by the data source, at least one backup log stored in a storage directory corresponding to the preset time period, wherein the backup log includes all data generated by the data source between adjacent backup time points, and the storage directory in the offline data storage system is divided according to time; The data volume of all data generated by the data source in the backup log within the preset time period is counted as the theoretical processing volume.

4. The data processing method according to claim 3, characterized in that: The process of pre-storing all data generated by the data source to the offline data storage system includes: The time starts from the last backup time point and all data generated by the data source are recorded in real time; When the timing reaches the preset backup interval, the current time is used as the backup time point, and all data generated by the data source recorded within the preset backup interval are compressed to obtain a backup log; Determine a time period corresponding to the backup log based on the backup time point and the last backup time point; Based on the time period, the backup log is stored in a preset directory of the offline data storage system, where the preset directory is a storage directory of the offline data storage system divided according to time.

5. The data processing method according to claim 1, characterized in that: In the case where both the actual processing amount and the theoretical processing amount are the number of rows of data, the verifying the actual processing amount based on the theoretical processing amount includes: Determining a row quantity difference between the actual processing capacity and the theoretical processing capacity; It is determined whether the row quantity difference satisfies a preset correction condition, wherein the preset correction condition at least includes: the row quantity difference exceeds a preset row quantity threshold.

6. The data processing method according to claim 1, characterized in that: In the case where both the actual processing amount and the theoretical processing amount are the total data amount of data, the verifying the actual processing amount based on the theoretical processing amount includes: Determine the total data volume difference between the actual processing volume and the theoretical processing volume; It is determined whether the total data volume difference satisfies a preset correction condition, wherein the preset correction condition at least includes: the total data volume difference exceeds a preset data volume threshold.

7. The data processing method according to claim 1, characterized in that: When the verification result of the data verification process is that the difference between the actual processing amount and the theoretical processing amount meets the preset correction condition, the data processing method further includes: Send an alarm message to the front end indicating abnormal data processing.

8. A data processing device, characterized in that: include: a data verification unit, configured to, during the execution of a first data processing process, in parallel call a data verification process when detecting that a preset verification time point has been reached, so as to obtain, as an actual processing amount, a data volume of data processed by the first data processing process within a preset time period before the preset verification time point, and to obtain a theoretical processing volume of the first data processing process within the preset time period, and verify the actual processing volume based on the theoretical processing volume, wherein the theoretical processing volume is a data volume of data generated by a data source within the preset time period; a data processing unit, configured to obtain all data generated by the data source within the preset time period, and call a second data processing process to process all data generated by the data source within the preset time period to obtain a data correction result, when the verification result of the data verification process is that the difference between the actual processing amount and the theoretical processing amount meets a preset correction condition, wherein the processing logic of the second data processing process is the same as that of the first data processing process; A data correction unit is used to overwrite the data processing results of the first data processing process within the preset time period stored in the data lake warehouse with the data correction results.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the data processing method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the data processing method as described in any one of claims 1 to 7.

Images