Data processing method, device, equipment and storage medium

Abstract

The present application discloses a data processing method, device, equipment and storage medium, and relates to the field of data processing technology. The method includes: receiving N first data files and a file configuration table of each first data file, the file configuration table including the first business date corresponding to M data result tables in the first data file, and the offset time corresponding to P data result tables in the M data result tables; determining the target business date of each data result table based on the first business date and the offset time; determining the data source table of the current job flow according to the job flow configuration table of the current job flow; searching the job flow configuration tables of all job flows according to the data source table, and determining multiple dependent job flows with the data source table as the data result table; and saving the execution data of the current job flow to the data result table of the current job flow when the target business dates of the data result tables corresponding to the multiple dependent job flows meet the preset conditions.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] In the data processing job flow, dependencies are set between jobs to ensure the causal logic of the business. Therefore, some jobs must be started after other related jobs are completed, that is, the start of a job is based on the completion of all its predecessor jobs. In the scenario of batch processing data at the end of the day, in order to ensure the accuracy of the business time of the data, the prerequisite can be to wait for the data of the corresponding business date (that is, the day) to be ready before starting to execute the data job and process the data of that day.

[0003] However, in actual execution, the data arriving on the same day may contain data from other days, such as data missing from the previous day. In this way, if the data from other days is also merged into the data of the same day, the data processing result will be inaccurate. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, an apparatus, a device and a storage medium, which can solve the technical problem of inaccurate data processing results caused by inaccurate business dates corresponding to data in the prior art.

[0005] The technical solution of this application is as follows:

[0006] In a first aspect, a data processing method is provided, comprising:

[0007] Receive N first data files and a file configuration table for each first data file, where the file configuration table includes a first business date corresponding to M data result tables in the first data file and an offset time corresponding to P data result tables in the M data result tables, where N, M, and P are positive integers and P is not greater than M;

[0008] Determine a target business date for each data result table based on the first business date and the offset time;

[0009] Determine the data source table of the current job flow according to the job flow configuration table of the current job flow;

[0010] Searching the job flow configuration tables of all job flows according to the data source table, and determining multiple dependent job flows that use the data source table as a data result table;

[0011] When the target business dates of the data result tables corresponding to the multiple dependent job flows all meet the preset conditions, the execution data of the current job flow is saved to the data result table of the current job flow.

[0012] In some embodiments, the job flow configuration table includes a loading mode of the data source table, and the loading mode is full loading or incremental loading; before saving the execution data of the current job flow to the data result table of the current job flow, the method further includes:

[0013] The data source table is loaded according to the loading method of the data source table to obtain the execution data of the current job flow.

[0014] In some embodiments, the job flow configuration table also includes the execution result of the job flow;

[0015] Load the data source table according to the loading method of the data source table to obtain the execution data of the current job flow, including:

[0016] The data source table is loaded according to the loading method of the data source table, and when the execution result of the job flow is successful, the execution data of the current job flow is obtained.

[0017] In some embodiments, before determining the data source table of the current job flow according to the job flow configuration table of the current job flow, the method further includes:

[0018] Based on the business data processing flow and the dependency relationship between multiple jobs, a job flow configuration table is set for multiple job flows.

[0019] In some embodiments, the preset condition is that the target business date of the data result table corresponding to the dependent job flow is equal to the business date of the current job flow;

[0020] After saving the execution data of the current job flow to the data result table of the current job flow, the method further includes:

[0021] Adjust the business date and execution time in the job flow configuration table of the current job flow. The execution time is the time when the execution data of the current job flow is saved to the data result table of the current job flow.

[0022] In a second aspect, a data processing device is provided, comprising:

[0023] A receiving module, configured to receive N first data files and a file configuration table of each first data file, wherein the file configuration table includes a first business date corresponding to M data result tables in the first data file, and an offset time corresponding to P data result tables in the M data result tables, where N, M, and P are positive integers, and P is not greater than M;

[0024] A first determination module, used to determine a target business date for each data result table based on the first business date and the offset time;

[0025] A second determination module, configured to determine a data source table of the current job flow according to the job flow configuration table of the current job flow;

[0026] A third determination module is used to search the job flow configuration tables of all job flows according to the data source table, and determine multiple dependent job flows that use the data source table as a data result table;

[0027] The storage module is used to save the execution data of the current job flow to the data result table of the current job flow when the target business dates of the data result tables corresponding to the multiple dependent job flows meet the preset conditions.

[0028] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the data processing method described in the first aspect.

[0029] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the data processing method described in the first aspect are implemented.

[0030] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0031] The data processing method provided by the embodiment of the present application receives N first data files and the file configuration table of each first data file, and then determines the target business date of each data result table based on the first business date and offset time in the file configuration table of the first data file; determines the data source table of the current job flow according to the job flow configuration table of the current job flow, and determines multiple dependent job flows with the data source table as the data result table according to the data source table; when the target business dates of the data result tables corresponding to the multiple dependent job flows meet the preset conditions, the execution data of the current job flow is saved to the data result table of the current job flow. Among them, the file configuration table includes the first business date corresponding to the M data result tables in the first data file, and the offset time corresponding to the P data result tables in the M data result tables. Since the target business date is determined based on the first business date and the offset time, that is, the inaccurate first business date is corrected by the offset time, so that the target business date is an accurate business date, which ensures the accuracy of the data source in the data processing process, thereby improving the accuracy of the data processing results.

[0032] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application, and do not constitute an improper limitation on the present application.

[0034] Figure 1 is a schematic diagram of a data processing system provided in an embodiment of the present application;

[0035] Figure 2 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0036] Figure 3 It is a flowchart of another data processing method provided in an embodiment of the present application;

[0037] Figure 4 It is a flowchart of another data processing method provided in an embodiment of the present application;

[0038] Figure 5 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0039] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make those of ordinary skill in the art better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are only examples consistent with some aspects of the present application as detailed in the attached claims.

[0042] Based on the background technology, it is known that in the prior art, the data arriving on the same day may contain data from other dates, such as data missing from the previous day. In this way, if the data from other dates are also merged into the data of the same day, the data processing result obtained will be inaccurate.

[0043] The technical field involved in the present application is a collaborative processing method for data from multiple data sources and a single batch of data from the same data source mixed with data from multiple business dates in a big data platform.

[0044] During the data processing process of the data platform, for the scenario of batch processing at the end of the day, in order to ensure the accuracy of the data business time, it is necessary to wait until the data content of the relevant data to be processed corresponding to the business date is ready before starting processing.

[0045] For example, in the asset management scenario, there will be multiple upstream systems providing transaction and valuation data to the data platform. Transaction data generally arrives on time every day, but due to its own particularity, valuation data sometimes takes multiple days to be processed and merged into one file before being provided to the data platform. When data merging is required, the prerequisite becomes waiting for the valuation data of all businesses on a certain business date to arrive, merging the data and then merging and counting with the corresponding transaction data to form report data that can be used by the business. In this scenario, the current solution only supports collaboration on the date of the file, and cannot identify the data content of the file for later collaboration, that is, the system will use the date corresponding to the file as the business date corresponding to all the data in the file. At this time, since the business date of the data in this file is different from the date of the file, it will cause errors in the data processing results.

[0046] In order to solve the above problems, the embodiment of the present application provides a data processing method, device, equipment and storage medium, which receives N first data files and the file configuration table of each first data file, and then determines the target business date of each data result table based on the first business date and offset time in the file configuration table of the first data file; determines the data source table of the current job flow according to the job flow configuration table of the current job flow, and determines multiple dependent job flows with the data source table as the data result table according to the data source table; when the target business dates of the data result tables corresponding to the multiple dependent job flows meet the preset conditions, the execution data of the current job flow is saved to the data result table of the current job flow. Among them, the file configuration table includes the first business date corresponding to the M data result tables in the first data file, and the offset time corresponding to the P data result tables in the M data result tables. Since the target business date is determined based on the first business date and the offset time, that is, the inaccurate first business date is corrected by the offset time, so that the target business date is an accurate business date, which ensures the accuracy of the data source in the data processing process, thereby improving the accuracy of the data processing results.

[0047] It should be understood that the data processing method provided in the embodiments of the present application is generally applied to a server or terminal device with data processing capabilities. The server may be an application server or a Web server. When deployed, the server may be a standalone server or a cluster server. The terminal device may be a computer, a mobile phone, a tablet or other electronic device with data processing capabilities.

[0048] The data processing method provided in the embodiment of the present application can also be applied to a data processing system.

[0049] like Figure 1 As shown, the data processing system includes a dependency configuration module 101, a file parsing module 102, a scheduling control module 103, and a database 104. The specific work content of each module is as follows:

[0050] The dependency configuration module 101 can be used to support technicians in configuring the dependency relationship of each job flow according to a specific structure, and supports the configuration of offset dependency on business dates.

[0051] The file parsing module 102 may be used to parse the data files provided by the upstream system, parse the data content of each data file to find out which business date data is included, and finally write the identifier of the data table and the business date involved into the database 104 .

[0052] The scheduling control module 103 can be used to parse the current configuration information of the database, determine which subsequent job flows can be started by the currently ready data, and execute related job flows.

[0053] In the data processing system in the embodiment of the present application, the file parsing module 102 can parse the data file provided by the upstream system, and obtain the accurate business date corresponding to each data table in the data file based on the parsed file, thereby ensuring the accuracy of the corresponding data source in the job flow and improving the accuracy of the data processing results.

[0054] In order to better understand the specific embodiments of the present application, the technical terms involved in the embodiments of the present application are first introduced.

[0055] Operation, a unit of applied analytical processing, is the process of processing data for a certain target result.

[0056] A job flow is an ordered set of jobs that can complete a business function. A job flow may also include only one job. The data processing method in the embodiment of the present application can be used to process data in a job flow.

[0057] The data result table can be a table recording the processing results output by the job flow corresponding to the upstream system after the data processing is completed. This data result table can be used as a data source table for the data processing system. In other words, there is a dependency relationship between the job flow corresponding to the upstream system and the job flow corresponding to the data processing system, and the job flow corresponding to the data processing system uses the data result table of the job flow corresponding to the upstream system as a data source table.

[0058] The data processing method provided in the embodiment of the present application is described in detail below by taking the above-mentioned data processing system executing the data processing method as an example.

[0059] Figure 2 A schematic diagram of a data processing method provided in an embodiment of the present application is shown. Figure 2 As shown, the method includes:

[0060] S210, receiving N first data files and a file configuration table of each first data file, wherein the file configuration table includes a first business date corresponding to M data result tables in the first data file, and an offset time corresponding to P data result tables in the M data result tables, where N, M, and P are positive integers, and P is not greater than M;

[0061] S220, determining a target business date for each data result table based on the first business date and the offset time;

[0062] S230, determining a data source table of the current job flow according to the job flow configuration table of the current job flow;

[0063] S240, searching the job flow configuration tables of all job flows according to the data source table, and determining a plurality of dependent job flows that use the data source table as a data result table;

[0064] S250: When the target business dates of the data result tables corresponding to the multiple dependent job flows all meet the preset conditions, the execution data of the current job flow is saved to the data result table of the current job flow.

[0065] The data processing method provided by the embodiment of the present application receives N first data files and the file configuration table of each first data file, and then determines the target business date of each data result table based on the first business date and offset time in the file configuration table of the first data file; determines the data source table of the current job flow according to the job flow configuration table of the current job flow, and determines multiple dependent job flows with the data source table as the data result table according to the data source table; when the target business dates of the data result tables corresponding to the multiple dependent job flows meet the preset conditions, the execution data of the current job flow is saved to the data result table of the current job flow. Among them, the file configuration table includes the first business date corresponding to the M data result tables in the first data file, and the offset time corresponding to the P data result tables in the M data result tables. Since the target business date is determined based on the first business date and the offset time, that is, the inaccurate first business date is corrected by the offset time, so that the target business date is an accurate business date, which ensures the accuracy of the data source in the data processing process, thereby improving the accuracy of the data processing results.

[0066] The above steps are described in detail below:

[0067] The first data file may be a data file received by the data processing system from an upstream system. Each data file may include multiple data result tables.

[0068] The job flow configuration table may be a table configured by the dependency configuration module 101 of the data processing system mentioned above. The job flow configuration table may record the data source table and data result table of the current job (the job flow corresponding to the data processing system), as well as the job flow it depends on (the job flow corresponding to the upstream system).

[0069] The job flow configuration table may also include the business date of the data source table required by the job flow. In S230, the data source table of the current job flow is determined according to the job flow configuration table of the current job flow, which can be determined according to the business date in the job flow configuration table and the identifier of the data source table, and the business date corresponding to the data source table is determined.

[0070] In addition, the job flow configuration table may further include an effective date of the job flow configuration table, which is used to indicate the effective time of the dependency relationship.

[0071] The job flow configuration table may also include batch numbers, which may be used to indicate the execution batches of specific jobs. Multiple batches may be executed in one day, or only one batch may be executed.

[0072] Here, the business date corresponding to the data source table in S230 may correspond to the target business date in S220. That is, the first business date is adjusted based on the offset time in S220 to obtain the target business date of each data result table. The time of the target business date is the accurate business date after adjustment compared to the first business date, which can improve the accuracy of the data processing result.

[0073] The offset time can be the number of days to adjust. For example, if the first business date corresponding to the data result table is September 26, and the offset time is 2 days backward, then the target business date after processing is September 28.

[0074] It should also be noted that the "data source table of the current job flow" in S230 may be a data result table of the job flow corresponding to the upstream system, that is, the data result table in the first data file received in S210.

[0075] In addition, there may be multiple dependent job flows corresponding to one job flow, that is, multiple data source tables may be processed to obtain a data result table. Therefore, the data processing system in the foregoing text may correspond to multiple upstream systems, and S240 searches the job flow configuration tables of all job flows according to the data source table, and multiple dependent job flows with the data source table as the data result table may be determined.

[0076] If the target business dates of the data result tables corresponding to the multiple dependent job flows found in S240 all meet the preset conditions, the current job flow can be executed, that is, the job flow corresponding to the data processing system is executed, and the execution data of the current job flow is saved in the data result table of the current job flow. In this way, the data processing system completes the current data processing job flow, and because the business date of the data source table used in the processing process is the correct business date, that is, the data source table is the correct data source table, thereby ensuring the accuracy of the data processing results.

[0077] In addition, it should be noted that in the embodiment of the present application, before sending the first data file, the upstream system also needs to configure a file configuration table for each first data file, and send the file configuration table together with the first data file to the data processing system.

[0078] In some embodiments, in order to improve the accuracy of data processing, the job flow configuration table may include a loading method of the data source table, and the loading method may be full loading or incremental loading. Figure 3 As shown, based on the above embodiment, before S250, the method may further include:

[0079] S310, loading the data source table according to the loading method of the data source table to obtain the execution data of the current job flow.

[0080] The loading mode can be full loading or incremental loading.

[0081] Specifically, full loading can be replacing the data source table corresponding to the previous business date with the data source table of the current business date, thereby obtaining the data table that needs to be processed by the current job flow, and then processing the data source table to obtain the execution data of the current job flow.

[0082] Incremental loading can be to add the data of the data source table of the current business date on the basis of the data source table corresponding to the previous business date, thereby obtaining the data table that needs to be processed by the current job flow, and then processing the data source table to obtain the execution data of the current job flow.

[0083] In some embodiments, the above job flow configuration table may also include the execution result of the job flow. After the above data loading process is completed, the execution result of the job flow in the job flow configuration table may be adjusted according to the loading situation. The execution result here may be successful execution or abnormal execution.

[0084] Based on this, the above S310 can specifically be loading the data source table according to the loading method of the data source table, and when the execution result of the job flow is successful, obtaining the execution data of the current job flow.

[0085] The data processing method provided in the embodiment of the present application configures the loading method of the data source table and the execution result of the job flow in the job flow configuration table. The data processing system can perform corresponding processing on the data source table according to the recording method, and determine whether the job flow is successfully executed based on the execution result of the job flow, thereby further ensuring the accuracy of the execution data and improving the accuracy of the data processing results.

[0086] In some embodiments, in order to improve data processing efficiency, such as Figure 4 As shown, based on the above embodiment, before S230, the method may further include:

[0087] S410, setting a job flow configuration table for multiple job flows based on the business data processing flow and the dependency relationship between multiple jobs.

[0088] The dependency configuration module in the data processing system can set the job flow configuration table for multiple job flows in advance.

[0089] Among them, the job flow configuration table can be configured according to the business data processing flow and the dependency relationship between multiple jobs. The dependency relationship here can be the execution order relationship between various jobs, as well as the data relationship. For example, a job that uses the data table of the current job as the data result table has a dependency relationship with the current job, that is, only after the job is completed and the data result table is obtained, the current job can use the data result table as the data source table to execute the current job.

[0090] The data result table is the intermediate result data generated during production.

[0091] In an embodiment of the present application, a job flow configuration table is set for multiple job flows in advance. By searching for a dependent job flow with a data source table of the current job flow as a data result table and executing the current job flow when the dependent job flow is successfully executed, the intermediate result data generated during production can be used as the data source, thereby reducing the complexity of job dependencies, reducing the generation of redundant data, saving computing resources, and improving scheduling efficiency.

[0092] In some embodiments, in order to further improve data processing efficiency, the preset condition in the above embodiments can be that the target business date of the data result table corresponding to the dependent job flow is equal to the business date of the current job flow. In this way, since the target business date in the above text is the accurate business date after the offset time adjustment, the data corresponding to the business date is also the accurate business data, and the accurate data processing result can be obtained by processing the data.

[0093] Based on this, after S250 saves the execution data of the current job flow to the data result table of the current job flow, the method may further include:

[0094] Adjust the business date and execution time in the job flow configuration table of the current job flow. The execution time is the time when the execution data of the current job flow is saved to the data result table of the current job flow.

[0095] Here, adjusting the business date may be adjusting the current business date to a business date corresponding to the next business processing cycle.

[0096] The execution time may be the time for processing the current job flow. The data processing system may identify the execution time and the business date, and further determine the business date of the data source table required by the job flow.

[0097] In an embodiment of the present application, after the execution data of the current job flow is saved to the data result table of the current job flow, the business date and execution time in the job flow configuration table can be adjusted and updated in a timely manner so that the data processing system will not repeatedly execute the data that has been processed, thereby saving computing resources and improving data processing efficiency.

[0098] It should be noted that the data processing method provided in the embodiment of the present application can also be executed by a data processing device, or a control module in the data processing device for executing the data processing method. In the embodiment of the present application, the data processing device provided in the embodiment of the present application is described by taking the data processing device executing the data processing method as an example.

[0099] Based on the same inventive concept, an embodiment of the present application also provides a data processing device.

[0100] Figure 5 A data processing device provided in an embodiment of the present application is shown. Figure 5 As shown, the data processing device 500 includes:

[0101] A receiving module 510 is used to receive N first data files and a file configuration table of each first data file, wherein the file configuration table includes a first business date corresponding to M data result tables in the first data file and an offset time corresponding to P data result tables in the M data result tables, where N, M, and P are positive integers and P is not greater than M.

[0102] A first determination module 520, configured to determine a target business date for each data result table based on the first business date and the offset time;

[0103] The second determination module 530 is used to determine the data source table of the current job flow according to the job flow configuration table of the current job flow;

[0104] The third determination module 540 is used to search the job flow configuration table of all job flows according to the data source table, and determine multiple dependent job flows that use the data source table as a data result table;

[0105] The storage module 550 is used to save the execution data of the current job flow to the data result table of the current job flow when the target business dates of the data result tables corresponding to the multiple dependent job flows meet the preset conditions.

[0106] In some embodiments, the job flow configuration table may include a loading mode of the data source table, and the loading mode may be full loading or incremental loading; the data processing device 500 may further include:

[0107] The loading module can be used to load the data source table according to the loading method of the data source table before saving the execution data of the current job flow to the data result table of the current job flow to obtain the execution data of the current job flow.

[0108] In some embodiments, the job flow configuration table may also include the execution result of the job flow;

[0109] The loading module can be specifically used to load the data source table according to the loading method of the data source table, and obtain the execution data of the current job flow when the execution result of the job flow is successful.

[0110] In some embodiments, the data processing device 500 may further include:

[0111] The configuration module can be used to set the job flow configuration table for multiple job flows based on the business data processing flow and the dependency relationship between multiple jobs before determining the data source table of the current job flow according to the job flow configuration table of the current job flow.

[0112] In some embodiments, the preset condition may be that the target business date of the data result table corresponding to the dependent job flow is equal to the business date of the current job flow; the data processing device 500 may further include:

[0113] The adjustment module can be used to adjust the business date and execution time in the job flow configuration table of the current job flow. The execution time is the time when the execution data of the current job flow is saved to the data result table of the current job flow.

[0114] The data processing device provided in the embodiment of the present application can be used to execute the data processing methods provided in the above-mentioned method embodiments. The implementation principles and technical effects are similar and will not be described here for the sake of simplicity.

[0115] Based on the same inventive concept, an embodiment of the present application also provides an electronic device.

[0116] Figure 6 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device may include a processor 601 and a memory 602 storing computer programs or instructions.

[0117] Specifically, the processor 601 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0118] The memory 602 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 602 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 602 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 602 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 602 is a non-volatile solid-state memory. In a specific embodiment, the memory 602 includes a read-only memory (ROM). In appropriate cases, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.

[0119] The processor 601 implements any one of the data processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 602 .

[0120] In one example, the electronic device may further include a communication interface 603 and a bus 610. Figure 6 As shown, the processor 601, the memory 602, and the communication interface 603 are connected via a bus 610 and communicate with each other.

[0121] The communication interface 603 is mainly used to implement communication between various modules, devices, units and / or devices in the embodiments of the present application.

[0122] Bus 610 includes hardware, software or both, and the parts of electronic equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 610 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0123] The electronic device can execute the data processing method in the embodiment of the present application, thereby realizing Figures 2 to 5 The data processing method and device described.

[0124] In addition, in combination with the base station fault detection method in the above embodiment, the embodiment of the present application can provide a readable storage medium for implementation. The readable storage medium stores program instructions; when the program instructions are executed by the processor, any one of the data processing methods in the above embodiment is implemented.

[0125] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0126] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0127] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0128] The above flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application describes various aspects of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of a computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. This processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of the boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0129] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A data processing method, characterized in that: include: Receive N first data files and a file configuration table of each of the first data files, wherein the file configuration table includes a first business date corresponding to M data result tables in the first data files, and an offset time corresponding to P data result tables in the M data result tables, where N, M, and P are positive integers and P is not greater than M; Determine a target business date for each data result table based on the first business date and the offset time; Determine the data source table of the current job flow according to the job flow configuration table of the current job flow; Searching the job flow configuration tables of all job flows according to the data source table, and determining a plurality of dependent job flows that use the data source table as a data result table; When the target business dates of the data result tables corresponding to the multiple dependent job flows all meet the preset conditions, the execution data of the current job flow is saved to the data result table of the current job flow.

2. The method according to claim 1, characterized in that The job flow configuration table includes a loading mode of a data source table, and the loading mode is full loading or incremental loading; before saving the execution data of the current job flow to the data result table of the current job flow, the method further includes: The data source table is loaded according to the loading method of the data source table to obtain the execution data of the current job flow.

3. The method according to claim 2, characterized in that The job flow configuration table also includes the execution result of the job flow; The step of loading the data source table according to the loading method of the data source table to obtain the execution data of the current job flow includes: The data source table is loaded according to the loading method of the data source table, and when the execution result of the job flow is successful, the execution data of the current job flow is obtained.

4. The method according to claim 1, characterized in that: Before determining the data source table of the current job flow according to the job flow configuration table of the current job flow, the method further includes: Based on the business data processing flow and the dependency relationship between multiple jobs, a job flow configuration table is set for multiple job flows.

5. The method according to claim 1, characterized in that The preset condition is that the target business date of the data result table corresponding to the dependent job flow is equal to the business date of the current job flow; After the execution data of the current job flow is saved in the data result table of the current job flow, the method further includes: The business date and execution time in the job flow configuration table of the current job flow are adjusted, where the execution time is the time when the execution data of the current job flow is saved to the data result table of the current job flow.

6. A data processing device, characterized in that: include: A receiving module, configured to receive N first data files and a file configuration table of each of the first data files, wherein the file configuration table includes a first business date corresponding to M data result tables in the first data file, and an offset time corresponding to P data result tables in the M data result tables, where N, M, and P are positive integers, and P is not greater than M; A first determination module, configured to determine a target business date for each data result table based on the first business date and the offset time; A second determination module, configured to determine a data source table of the current job flow according to the job flow configuration table of the current job flow; A third determination module is used to search the job flow configuration tables of all job flows according to the data source table, and determine a plurality of dependent job flows that use the data source table as a data result table; The storage module is used to save the execution data of the current job flow to the data result table of the current job flow when the target business dates of the data result tables corresponding to the multiple dependent job flows all meet the preset conditions.

7. The device according to claim 6, characterized in that The job flow configuration table includes a loading method of the data source table, and the loading method is full loading or incremental loading; the device also includes: The loading module is used to load the data source table according to the loading method of the data source table before saving the execution data of the current job flow to the data result table of the current job flow to obtain the execution data of the current job flow.

8. The device according to claim 7, characterized in that The job flow configuration table also includes the execution result of the job flow; The loading module is specifically configured to load the data source table according to the loading method of the data source table, and obtain the execution data of the current job flow when the execution result of the job flow is successful.

9. The device according to claim 6, characterized in that The device also includes: The configuration module is used to set the job flow configuration table for multiple job flows based on the business data processing flow and the dependency relationship between multiple jobs before determining the data source table of the current job flow according to the job flow configuration table of the current job flow.

10. The device according to claim 6, characterized in that The preset condition is that the target business date of the data result table corresponding to the dependent job flow is equal to the business date of the current job flow; the device also includes: The adjustment module is used to adjust the business date and execution time in the job flow configuration table of the current job flow, where the execution time is the time when the execution data of the current job flow is saved to the data result table of the current job flow.

11. An electronic device, characterized in that: It comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the data processing method as described in any one of claims 1 to 5.

12. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of the data processing method according to any one of claims 1 to 5 are implemented.

Images