Data processing and timeliness detection method, device and computer equipment

Abstract

The application relates to a data processing and timeliness detection method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: firstly, acquiring a first source data table and a first result data table that have been synchronized on a big data platform, wherein the first source data table and the first result data table are obtained by synchronizing real-time source data and real-time result data in a development environment at a fixed time; comparing the first source data table with the first result data table to obtain first difference data, and acquiring a second source data target value and a second result data target value corresponding to the first difference data in the development environment; comparing the second source data target value with the second result data target value to obtain second difference data; and finally, determining the timeliness of data processing in the development environment based on the second difference data. Meanwhile, the method can exclude the misjudgment of data caused by the untimely update of an offline table, and improve the accuracy of the detection result of the timeliness of data processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data, and in particular, to a data processing and timeliness detection method and device, computer equipment, a storage medium, and a computer program product. BACKGROUND

[0002] With the development of the field of big data, in order to achieve the complex business big data application requirements, data development often needs to be processed through multiple data processing links to obtain results, for example, through ETL technology for data development, wherein ETL is a process for describing data from the source end through extraction, transformation, and loading to the destination end. Through ETL technology, scattered, disorganized, and non-uniform data can be integrated together, so ETL technology has been widely used in the field of big data processing.

[0003] However, ETL technology involves multiple data links, and any delay in processing and timeliness of any link can cause data to lag, so the timeliness risk of ETL technology for data processing is significantly magnified, so how to ensure the timeliness and stability of data results in a complex ETL process has become a challenge in this field. SUMMARY

[0004] Therefore, it is necessary to provide a method, device, computer equipment, computer readable storage medium, and computer program product for detecting the timeliness of data processing to solve the technical problem of the above-mentioned data processing flow with high timeliness risk.

[0005] In a first aspect, the present application provides a data processing and timeliness detection method. The method comprises:

[0006] Obtaining a first source data table and a first result data table that have been synchronized on a big data platform; the data in the first result data table is obtained by executing a data processing flow on the data in the first source data table, and the first source data table and the first result data table are respectively obtained by synchronizing real-time source data and real-time result data in a development environment at regular intervals;

[0007] Comparing the first source data table and the first result data table to obtain first difference data;

[0008] Obtaining a second source data target value and a second result data target value corresponding to the first difference data in the development environment;

[0009] Comparing the second source data target value and the second result data target value to obtain second difference data;

[0010] Determining the timeliness of data processing in the development environment based on the second difference data.

[0011] In one of the embodiments, the first difference data is obtained by comparing the first source data table and the first result data table, including: obtaining a mapping relationship between data in the first source data table and data in the first result data table; according to the mapping relationship, comparing the first source data table and the first result data table to obtain target data in the first source data table which does not have mapped data in the first result data table; and determining the target data as the first difference data.

[0012] In one of the embodiments, the second source data target value and the second result data target value corresponding to the first difference data in the development environment are obtained, including: calling the first online interface to input the first difference data to obtain the second source data target value corresponding to the first difference data; and calling the second online interface to input the first difference data to obtain the second result data target value corresponding to the first difference data.

[0013] In one of the embodiments, the first online interface and the second online interface are multiple, and the method further includes: performing fusion processing on the source data returned by each first online interface to obtain the second source data target value; and performing fusion processing on the result data returned by each second online interface to obtain the second result data target value.

[0014] In one of the embodiments, the first difference data includes multiple data, and the method further includes: calling the first online interface to input the identifier of each data in the first difference data one by one to obtain the second source data target value corresponding to the first difference data; and calling the second online interface to input the identifier of each data in the first difference data one by one to obtain the second result data target value corresponding to the first difference data.

[0015] In one of the embodiments, the timeliness of data processing in the development environment is determined, including: obtaining the data amount of the second difference data; and determining the timeliness of data processing in the development environment according to a negative correlation relationship between the data amount and the timeliness.

[0016] In a second aspect, the application further provides a device for detecting the timeliness of data processing. The device includes:

[0017] A data table obtaining module is configured to obtain a first source data table and a first result data table currently synchronized on a big data platform; data in the first result data table is obtained by executing a data processing procedure on data in the first source data table, and the first source data table and the first result data table are respectively obtained by synchronizing real-time source data and real-time result data in a development environment at a fixed time;

[0018] a first data comparison module, configured to compare the first source data table and the first result data table to obtain first difference data;

[0019] a target value acquisition module, configured to acquire second source data target values and second result data target values corresponding to the first difference data in the development environment;

[0020] a second data comparison module, configured to compare the second source data target values and the second result data target values to obtain second difference data;

[0021] a timeliness confirmation module, configured to determine the timeliness of data processing in the development environment based on the second difference data.

[0022] In a third aspect, the present application also provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the detection of the timeliness of data processing of the method in any of the embodiments of the present application when executing the computer program.

[0023] In a fourth aspect, the present application also provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program implements the detection of the timeliness of data processing of the method in any of the embodiments of the present application when executed by a processor.

[0024] In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program, and the computer program implements the detection of the timeliness of data processing of the method in any of the embodiments of the present application when executed by a processor.

[0025] The data processing timeliness detection method, device, computer device, storage medium and computer program product have the following beneficial effects: when detecting data processing timeliness, the first data source table and the first result data table on the big data platform are obtained by synchronizing the data generated in the development environment in a timely manner, the data generated in the development environment is the data generated by executing the data processing process; the first difference data is obtained by comparing the first data source table and the first result data table; then the second data source target value and the second result data target value corresponding to the first difference data in the development environment are obtained, and the second difference data is obtained by comparing the second data source target value and the second result data target value. Since the data in the first result data table is obtained by executing the data processing process on the data in the first source data table, the update of the data in the first source data table is timely reflected in the first result data table, so that the processing timeliness of the data processing process can be determined by comparing the first source data table and the first result data table. However, since the data on the big data platform is synchronized from the development environment in a timely manner, there is an error with the real-time data in the development environment, so after determining the first difference data, the real-time second source data target value and the second result data target value corresponding to the first difference data in the development environment are also retrieved, and the processing timeliness of the data processing process is accurately determined by comparing the second source data target value and the second result data target value, the accurate problem data inconsistent with the real-time development environment is found out in a timely manner, the misjudgment problem data caused by the update of the offline table not being timely is excluded, and thus the business requirements are met. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 An application environment diagram of the data processing timeliness detection method in one embodiment;

[0027] Figure 2 A flowchart of the data processing timeliness detection method in one embodiment;

[0028] Figure 3 A flowchart of the data processing timeliness detection steps in one embodiment;

[0029] Figure 4 A structure block diagram of the data processing timeliness detection device in one embodiment;

[0030] Figure 5 An internal structure diagram of the computer device in one embodiment. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0032] The data processing and timeliness detection method provided by the embodiments of the present application can be applied to the application environment as shown in Figure 1 The application environment includes a server 102 and a terminal 104. The server 102 communicates with the terminal 104 through a network. A development environment can be deployed in the server 102, and a big data platform can be deployed in the terminal 104. In the application scenario of the present application, the server 102 obtains a first source data table and a first result data table that have been synchronized from the big data platform deployed in the terminal 104, and then compares the first source data table and the first result data table to obtain first difference data. Further, the server 102 obtains a second source data target value and a second result data target value corresponding to the first difference data in the development environment, and compares the second source data target value and the second result data target value to obtain second difference data. Finally, the server 102 determines the timeliness of data processing in the development environment based on the second difference data, and sends the obtained timeliness detection result to a monitoring terminal to display the timeliness detection result. The terminal 104 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 102 can be implemented by an independent server or a server cluster composed of multiple servers.

[0033] In one embodiment, as shown in Figure 2 , a data processing and timeliness detection method is provided. Taking the server 102 in Figure 1 as an example, the method includes the following steps:

[0034] Step 202: Obtain a first source data table and a first result data table that have been synchronized on a big data platform. The data in the first result data table is obtained by executing a data processing flow on the data in the first source data table. The first source data table and the first result data table are obtained by synchronizing real-time source data and real-time result data in a development environment at regular intervals.

[0035] The data processing flow represents an ETL flow.

[0036] It can be understood that the data in the development environment is real-time data, which can be stored in a real-time data table. The real-time data specifically includes source data and result data, and the result data is processing obtained by performing ETL processing on the source data. The detection of the ETL processing process and timeliness is specifically detecting whether the data fields between the result data and the source data in the development environment can be mapped or correspond, if yes, it indicates that the timeliness of the ETL processing is better, and the result data can be obtained by processing the new source data in time, otherwise, it indicates that the timeliness of the ETL processing is poor.

[0037] However, since the real-time data table in the development environment cannot be directly obtained for testing comparison or calculation, because it will affect the performance of online users. Therefore, it is necessary to build a big data platform, and synchronize the real-time data in the development environment to the offline data table of the big data platform at regular intervals, and detect the timeliness based on the offline data table.

[0038] Specifically, the source data table and the result data table currently synchronized from the development environment can be obtained from the big data platform, thereby obtaining a first source data table and a first result data table.

[0039] Step 204, comparing the first source data table and the first result data table to obtain first difference data.

[0040] Specifically, the first source data table and the first result data table can be compared, which can be realized by using a data comparison tool or writing a custom script. The comparison method can include field-level comparison, row-level comparison, or using a specific algorithm to detect differences. The difference data can be identified by marking, recording or exporting.

[0041] Step 206, obtaining second source data target values and second result data target values corresponding to the first difference data in the development environment.

[0042] It can be understood that because the data table on the big data platform is obtained by synchronizing the real-time data in the development environment at regular intervals, it is not real-time synchronization, so there will be errors between the data in the big data platform and the real-time data in the development environment. The error time is determined based on the synchronization frequency, for example, synchronization once every twenty-four hours, and the maximum error time is twenty-four hours. Therefore, the first difference data obtained by comparing the offline data table in the big data platform is not accurate, and some data may have been changed and updated, so it cannot be determined that the first difference data is really inconsistent in the real-time data table in the development environment. Therefore, after determining the first difference data, the application also obtains the value of the first difference data in the corresponding real-time data table in the development environment to determine whether it is really inconsistent.

[0043] In a specific implementation, the server can call the interface of the development environment to batch acquire the second source data target value and the second result data target value corresponding to the first difference data one by one. More specifically, a script for calling the development environment data can be acquired in advance, and the interface of the development environment is called through the script to acquire the second source data target value and the second result data target value.

[0044] In step 208, the second source data target value and the second result data target value are compared to obtain second difference data.

[0045] Specifically, the second source data target value and the second result data target value can be compared by writing SQL (Structured Query Language) instructions to obtain the second difference data.

[0046] Further, after comparing the second source data target value and the second result data target value to obtain the second difference data, it further includes: performing fault analysis on the second difference data, determining the fault reason of the second difference data, and generating a fault report based on the second difference data and the fault reason.

[0047] In step 210, the timeliness of data processing in the development environment is determined based on the second difference data.

[0048] Specifically, according to the characteristics and quantity of the second difference data, the server can use different methods to evaluate the timeliness of data processing. For example, real-time monitoring tools can be used to detect the delay time of data processing, or data quality evaluation tools can be used to analyze the accuracy and consistency of data processing. According to the evaluation result, corresponding measures can be taken to improve the timeliness of data processing, such as optimizing the data processing process, increasing resources or adjusting the data synchronization frequency, etc.

[0049] In the data processing and timeliness detection method, the server can obtain the first source data table and the first result data table from the source data and the result data in the development environment, obtain the first difference data by comparison, obtain the second source data target value and the second result data target value according to the first difference data, obtain the second difference data by comparison, and confirm the timeliness of data processing in the development environment according to the data amount of the second difference data. Since the data in the first result data table is obtained by executing the data processing procedure on the data in the first source data table, the update of the data in the first source data table will be reflected in the first result data table in time. Therefore, by comparing the first source data table and the first result data table, the processing timeliness of the data processing procedure can be determined. However, since the data on the big data platform is synchronized from the development environment at a fixed time, there is an error with the real-time data in the development environment. Therefore, after determining the first difference data, the second source data target value and the second result data target value corresponding to the first difference data in the development environment are also called, and the processing timeliness of the data processing procedure is accurately determined by comparing the second source data target value and the second result data target value, the real-time development environment is accurately determined, and the misjudgment problem data caused by the offline table update is excluded, thereby meeting the business requirements.

[0050] In one embodiment, in step 204, comparing the first source data table and the first result data table to obtain the first difference data can include:

[0051] Step 204a: Obtain the mapping relationship between the data in the first source data table and the data in the first result data table.

[0052] The mapping relationship refers to the relationship between the same fields or some other identifiers that can correspond to the data in the first source data table and the first result data table.

[0053] Step 204b: According to the mapping relationship, compare the first source data table and the first result data table to obtain target data in the first source data table that does not have mapping data in the first result data table.

[0054] Specifically, according to the mapping relationship between the first source data table and the first result data table, they are compared. Check if there is a situation where the data in the first source data table does not correspond to the data in the first result data table. The data in the first source data table that does not correspond to the data in the first result data table is the target data.

[0055] Step 204c: Determine the target data as the first difference data.

[0056] Specifically, the first difference data represents data that is different between the first source data table and the first result data table, and the first difference data can be caused by incomplete mapping relationships or other reasons. The first difference data can provide information about inconsistencies or errors between the first source data table and the first result data table.

[0057] In this embodiment, the first difference data is obtained through the first source data table and the first result data table, so that data quality problems can be discovered in a timely manner and the quality of the data can be improved.

[0058] Further, in one embodiment, obtaining, in step 206, the second source data target value and the second result data target value corresponding to the first difference data in the development environment can include the following steps: calling a first online interface to obtain the second source data target value corresponding to the first difference data by inputting the first difference data as a parameter; and calling a second online interface to obtain the second result data target value corresponding to the first difference data by inputting the first difference data as a parameter.

[0059] The first online interface and the second online interface are both application programming interfaces (APIs).

[0060] In one embodiment, the first online interface and the second online interface are multiple, and the method further includes: performing fusion processing on the source data returned by each first online interface to obtain the second source data target value; and performing fusion processing on the result data returned by each second online interface to obtain the second result data target value.

[0061] Specifically, fusion refers to summing up the values of the source data returned by each first online interface and the result data returned by each second online interface in the above embodiments by writing a SQL script, to obtain the second source data target value and the second result data target value.

[0062] In one embodiment, the first difference data includes multiple data, and the method further includes: calling the first online interface to input the identification of each data in the first difference data as a parameter one by one to obtain the second source data target value corresponding to the first difference data; and calling the second online interface to input the identification of each data in the first difference data as a parameter one by one to obtain the second result data target value corresponding to the first difference data.

[0063] Specifically, the above embodiments respectively call the first online interface and the second online interface and input the first difference data as a parameter, where the parameter can be a unique identification corresponding to the subject name, current or history, type, and finally the second source data target value and the second result data target value corresponding to the first difference data are obtained respectively.

[0064] Through the embodiment, a plurality of data in the first difference data set needs to be called one by one in batches, and therefore interface parameterization is used to automatically traverse each piece of data in the first difference data set one by one, so as to improve the accuracy of the second source data target value and the second result data target value.

[0065] Further, in one embodiment, determining the timeliness of data processing in the development environment can include the following steps: obtaining the data amount of the second difference data; and determining the timeliness of data processing in the development environment according to a negative correlation between the data amount and the timeliness.

[0066] Specifically, the first difference data is inconsistent with the source data before 24h, and the real-time situation of the data returned by the online real-time interface is still inconsistent, and the above data is the second difference data. According to the data amount of the second difference data, the timeliness of data processing in the development environment is determined. The more the second difference data is, the lower the timeliness of data processing in the development environment is, and vice versa. The less the second difference data is, the higher the timeliness of data processing in the development environment is.

[0067] Through the detection of the data amount of the second difference data, the timeliness of data processing can be known in time according to the above embodiment.

[0068] In order to better understand the above process of data processing timeliness, in combination with Figure 3 , a specific process of data processing timeliness of the present application is described in detail below, including the following steps:

[0069] S302, using the ETL process to synchronize the real-time source data and the real-time result data in the development environment at a fixed time, and synchronizing to the big data platform in real time, and recording the real-time source data and the real-time result data into the first source data table and the first result data table respectively.

[0070] S304, comparing the first source data table and the first result data table according to the data operation instruction, obtaining the first difference data as a data set A, and recording the first difference data into the test table one of suspected synchronization delay data.

[0071] S306, using the first source data table to write a data operation instruction to call the first online interface, and performing one-by-one parameter request on the first difference data in the data set A, to obtain the real-time online source data return target value, i.e. the second source data target value.

[0072] S308, using the first result data table to write a data operation instruction to call the second online interface, and performing one-by-one parameter request on the first difference data in the data set A, to obtain the real-time online result data return target value, i.e. the second result data target value.

[0073] S310, obtaining second difference data by comparing the second source data target value and the second result data target value according to the data operation instruction, and recording the second difference data into the test table two of the real-time synchronous delay data, and the data operation instruction can be set to run at a fixed time every day to more accurately and timely monitor the synchronization and timeliness of the second test table.

[0074] The data in the test table two is the data that is inconsistent between the first result data and the first source data in the test table one twenty-four hours ago, and the data is still inconsistent after being returned through the first online interface and the second online interface.

[0075] S312, determining the timeliness of data processing in the development environment based on the second difference data.

[0076] Through the above embodiment, the server can obtain the first source data table and the first result data table from the source data and the result data in the development environment, obtain the first difference data by comparison, obtain the second source data target value and the second result data target value according to the first difference data, obtain the second difference data after comparison, and confirm the timeliness of data processing in the development environment according to the data amount of the second difference data.

[0077] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0078] In one embodiment, as shown in Figure 4 Based on the same inventive concept, the embodiments of the present application also provide a data processing timeliness detection device for implementing the above-mentioned data processing timeliness detection method, which comprises a data table acquisition module 401, a first data comparison module 402, a target value acquisition module 403, a second data comparison module 404 and a timeliness confirmation module 405, wherein:

[0079] The data table acquisition module 401 is configured to acquire a first source data table and a first result data table that are currently synchronized on a big data platform; data in the first result data table is obtained by performing a data processing procedure on data in the first source data table; and the first source data table and the first result data table are respectively obtained by synchronizing real-time source data and real-time result data in a development environment at a fixed time.

[0080] The first data comparison module 402 is configured to compare the first source data table and the first result data table to obtain first difference data.

[0081] The target value acquisition module 403 is configured to acquire a second source data target value and a second result data target value corresponding to the first difference data in the development environment.

[0082] The second data comparison module 404 is configured to compare the second source data target value and the second result data target value to obtain second difference data.

[0083] The timeliness confirmation module 405 is configured to determine the timeliness of data processing in the development environment based on the second difference data.

[0084] In an embodiment, the data table acquisition module 401 is further configured to acquire a mapping relationship between data in the first source data table and data in the first result data table, and to determine the mapping relationship.

[0085] Correspondingly, the first data comparison module 402 is further configured to compare the first source data table and the first result data table to obtain target data in the first source data table that does not have mapped data in the first result data table, and to determine the target data as the first difference data.

[0086] In an embodiment, the target value acquisition module 403 further includes a first target value acquisition submodule configured to call a first online interface, input the first difference data as a parameter, obtain the second source data target value corresponding to the first difference data, and call a second online interface, input the first difference data as a parameter, and obtain the second result data target value corresponding to the first difference data.

[0087] In an embodiment, the target value acquisition module 403 further includes a second target value acquisition submodule configured to perform fusion processing on source data returned by each first online interface to obtain the second source data target value, and perform fusion processing on result data returned by each second online interface to obtain the second result data target value.

[0088] In one embodiment, the target value obtaining module 403 further includes a third target value obtaining sub-module, configured to call the first online interface, and input the identifier of each data in the first difference data one by one to obtain the second source data target value corresponding to the first difference data, and call the second online interface, and input the identifier of each data in the first difference data one by one to obtain the second result data target value corresponding to the first difference data.

[0089] In one embodiment, the second data comparison module 404 is further configured to obtain the data amount of the second difference data. The device further includes a timeliness confirmation module 405, configured to determine the timeliness of data processing in the development environment according to the negative correlation between the data amount and the timeliness.

[0090] The modules in the above data processing timeliness detection device can be implemented by software, hardware, or a combination thereof. The modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the modules.

[0091] In one embodiment, a computer device is provided, which can be a server, and the internal structure diagram thereof can be as shown in Figure 5 The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store data in the development environment. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a data processing timeliness detection method.

[0092] Those skilled in the art can understand that Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0093] In an embodiment, a computer readable storage medium is provided, and the computer readable storage medium has stored thereon a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0094] In an embodiment, a computer program product is provided, and the computer program product includes a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0095] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of the country and region.

[0096] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0097] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0098] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for detecting data processing and timeliness, characterized in that, The method comprises: obtaining a first source data table and a first result data table currently synchronized on a big data platform; data in the first result data table is obtained by performing a data processing procedure on data in the first source data table, and the first source data table and the first result data table are obtained by synchronizing real-time source data and real-time result data in a development environment at a fixed time respectively; comparing the first source data table and the first result data table to obtain first difference data; obtaining a second source data target value and a second result data target value corresponding to the first difference data in the development environment; comparing the second source data target value and the second result data target value to obtain second difference data; determining the timeliness of data processing in the development environment based on the second difference data.

2. The method of claim 1, wherein, The comparison of the first source data table and the first result data table to obtain the first difference data comprises: obtaining a mapping relationship between data in the first source data table and data in the first result data table; comparing the first source data table and the first result data table according to the mapping relationship to obtain target data in the first source data table that does not have mapped data in the first result data table; determining the target data as the first difference data.

3. The method of claim 1, wherein, The obtaining of the second source data target value and the second result data target value corresponding to the first difference data in the development environment comprises: calling a first online interface to input the first difference data to obtain a second source data target value corresponding to the first difference data; calling a second online interface to input the first difference data to obtain a second result data target value corresponding to the first difference data.

4. The method of claim 3, wherein, The first online interface and the second online interface have multiple interfaces, and the method further comprises: performing fusion processing on source data returned by each first online interface to obtain the second source data target value; performing fusion processing on result data returned by each second online interface to obtain the second result data target value.

5. The method of claim 3, wherein, The first difference data comprises multiple data, and the method further comprises: calling the first online interface to input the identifier of each data in the first difference data one by one to obtain a second source data target value corresponding to the first difference data; calling the second online interface to input the identifier of each data in the first difference data one by one to obtain a second result data target value corresponding to the first difference data.

6. The method of claim 1, wherein, The determination of the timeliness of data processing in the development environment comprises: obtaining the data amount of the second difference data; determining the timeliness of data processing in the development environment according to a negative correlation relationship between the data amount and the timeliness.

7. An apparatus for data processing and timeliness detection, characterized in that, The device comprises: The data table acquisition module is configured to acquire a first source data table and a first result data table that have been synchronized on a big data platform; data in the first result data table is obtained by performing a data processing procedure on data in the first source data table; and the first source data table and the first result data table are respectively obtained by synchronizing real-time source data and real-time result data in a development environment at a fixed time; The first data comparison module is configured to compare the first source data table and the first result data table to obtain first difference data. The target value acquisition module is configured to acquire a second source data target value and a second result data target value corresponding to the first difference data in the development environment. The second data comparison module is configured to compare the second source data target value and the second result data target value to obtain second difference data. The timeliness confirmation module is configured to determine the timeliness of data processing in the development environment based on the second difference data.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

Images