Data processing method and device, equipment, storage medium and program product

Abstract

The invention discloses a data processing method and device, equipment, a storage medium and a program product, and relates to the technical field of data processing.The data processing method comprises the steps that source data files in a data warehouse are processed in parallel through a first data processing link and a second data processing link; and under the condition that a target database receives any one of a first data processing result of the first data processing link and a second data processing result of the second data link, executing a preset data processing operation on the received data processing result. The data processing time can be shortened, and the data processing efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, equipment, storage medium and program product. Background Art

[0002] In the data processing process, the data warehouse plays a key role in data aggregation and processing. After the source data file is generated, the data warehouse converts it into the data structure required by the business system and performs the warehouse operation.

[0003] However, the data processing process relies on the data warehouse for data aggregation and processing, which results in long processing task execution time and fails to meet business expectations. Summary of the invention

[0004] The main purpose of this application is to provide a data processing method, device, equipment, storage medium and program product, aiming to solve the technical problem that data warehouses aggregate and process data, resulting in long processing task execution time.

[0005] To achieve the above objectives, the present application proposes a data processing method, comprising:

[0006] Processing source data files in the data warehouse in parallel using the first data processing link and the second data processing link;

[0007] When the target database receives any one of the first data processing result of the first data processing link and the second data processing result of the second data link, a preset data processing operation is performed on the received data processing result.

[0008] In one embodiment, the first data link is an internal data processing platform of the data warehouse performing data processing on the source data;

[0009] The second data processing link is that an external data processing platform performs data processing on the source data, wherein the external data processing platform is communicatively connected with the data warehouse.

[0010] In one embodiment, the external data processing platform is a Spark data processing platform.

[0011] In one embodiment, when the target database receives any one of the first data processing result of the first data processing link and the second data processing result of the second data link, after performing a preset data processing operation on any one of the received data, the method includes:

[0012] When the target database receives the other of the first data result and the second data result, checking whether the first data result and the second data result are consistent;

[0013] In case of inconsistency, the preset data processing operation is terminated.

[0014] In one embodiment, after performing a preset data processing operation on the received data processing result, the method includes:

[0015] Generate the current processing cycle field in the data processing summary table;

[0016] In the case where the generation of the current processing cycle field is detected in the current processing cycle, the occupation state of the unique index of the data processing channel is detected;

[0017] When the occupied state is unoccupied, a preset data processing operation is performed on the received data processing result through the unique index of the data processing channel, and the state identifier of the unique index channel is updated to the occupied state.

[0018] In one embodiment, the step of performing a preset data processing operation on the received data processing result includes:

[0019] Determine whether the actual duration of performing the preset data processing operation on the received data processing result is greater than the preset duration;

[0020] When the actual duration is longer than the preset duration, an abnormal situation alarm message is output.

[0021] In addition, to achieve the above-mentioned purpose, the present application also proposes a data processing device, the data processing device comprising:

[0022] A first processing module, configured to process source data files in the data warehouse in parallel using a first data processing link and a second data processing link;

[0023] The second processing module is used to perform a preset data processing operation on the received data processing result when the target database receives any one of the first data processing result of the first data processing link and the second data processing result of the second data link.

[0024] In addition, to achieve the above objectives, the present application also proposes a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method described above.

[0025] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0026] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0027] One or more technical solutions proposed in this application have at least the following technical effects:

[0028] Compared with the related art, in which the source data files are aggregated and processed by the data warehouse, the present application uses the first data link and the second data link in parallel to process the source data files in the data warehouse, which not only avoids excessive reliance on the data warehouse, but also makes full use of computing resources, reduces the total time of data processing, and enhances the stability and timeliness of the data processing link. At the same time, the first data processing link and the second data processing link are used in a dual-link parallel operation mode. When the target database receives either the first data processing result or the second data processing result, the preset data processing operation is performed on the received data processing result, which can use data faster, improve data availability, and increase business response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 A flow chart for the end-of-day data processing in the financial industry;

[0032] Figure 2 This is a diagram of the seven levels of relationships at the end of the day;

[0033] Figure 3 A flowchart of the first embodiment of the data processing method of the present application is provided;

[0034] Figure 4 This is a schematic diagram of the process of the first data link and the second data link in parallel processing of source data files in this application;

[0035] Figure 5 A schematic diagram of the process flow of processing source data files by the second data processing link of the present application;

[0036] Figure 6 This is a schematic diagram of the extraction, aggregation, and migration process of the personal loan big data fusion platform for this application;

[0037] Figure 7 This is a schematic diagram of the module structure of the data processing device according to an embodiment of the present application;

[0038] Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the data processing method in the embodiment of the present application.

[0039] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0040] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0041] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0042] The main solution of the embodiment of the present application is: use the first data processing link and the second data processing link to parallelly process the source data files in the data warehouse; when the target database receives any one of the first data processing result of the first data processing link and the second data processing result of the second data link, perform a preset data processing operation on the received data processing result.

[0043] In this embodiment, for the convenience of description, the following description is made with the data processing device as the execution subject.

[0044] In the data processing process, since the system relies on many basic data files, the data volume is large, and the processing logic is complex, the acquisition and processing of data source files are handed over to the data warehouse. Figure 1 As shown, Figure 1 This is a flow chart for end-of-day data processing in the financial industry.

[0045] After the core basic data file, i.e. the source data file, is generated, the data warehouse processes the source data file into the data structure required by the business system and removes it from the warehouse. After the data warehouse removes the processed source data file, the processed source data file is then subjected to data extraction and ETL (Extract, Transform, Load) data migration operations via the exchange platform, and the data is migrated to the business system database through the online application ATFA (Automated Task Flow Automation). After the migration operation is completed, the financial transaction cloud task scheduling platform will be automatically triggered to prepare for the execution of the end-of-day task. The task scheduling platform starts the end-of-day application ATF (Automated Task Flow)_DED (Daily End-of-Day)_A, and pulls up several subtasks in turn to complete the core business logic. Finally, DAP (Data Access Protocol) recognizes that data reflow begins after the end of the day for subsequent system use.

[0046] However, when processing data through the above process, there are the following defects:

[0047] 1. Data processing takes a long time, and the end-of-day efficiency cannot meet business expectations

[0048] After the end-of-day trial operation, the operation and maintenance personnel monitored and observed its operation and found that the average end-of-day completion time was 05:30, and the end time was delayed to 08:00 due to occasional abnormalities, which seriously exceeded business expectations. Through the analysis of the entire end-of-day process, it was concluded that the key factor affecting the end time was the data warehouse data processing module. Since data processing in the warehouse depends on the execution of several scheduling tasks, the execution efficiency of the scheduling tasks is closely related to the task priority and the number of tasks in the queue. After multiple consultations with the data warehouse team to adjust the task priority, the execution efficiency still could not meet business requirements.

[0049] 2. Processing in the data warehouse is highly dependent and link reliability is poor

[0050] like Figure 2 As shown, Figure 2 The following is a seven-layer relationship diagram of the end of day (the green nodes are data processing nodes). If any of the nodes is delayed or abnormal, the end of day will not be completed as scheduled, and the link reliability is poor. In addition, the development and management rights of most of the operations in the figure are not owned by the team, and the cost of problem location and maintenance in abnormal situations is extremely high.

[0051] 3. Low security and lack of fault degradation strategy

[0052] The basis for the smooth execution of the end of the day is that the warehouse processing is timely and accurate, and the financial transaction task scheduling platform can normally schedule the end of the day application. If any one of them fails, the end of the day cannot be completed. Considering the extreme case where the infrastructure failure causes the link to be unable to proceed, in order to avoid major business impact, a downgrade strategy is essential.

[0053] Therefore, when data aggregation and processing are performed through the above data processing flow, there are problems such as long processing task execution time and poor link stability, which cannot meet business expectations.

[0054] The present application provides a solution. Compared with the related art, in which the source data files are aggregated and processed by the data warehouse, the present application uses the first data link and the second data link in parallel to process the source data files in the data warehouse, which not only avoids excessive reliance on the data warehouse, but also makes full use of computing resources, reduces the total time of data processing, and enhances the stability and timeliness of the data processing link. At the same time, the first data processing link and the second data processing link are used in a dual-link parallel operation mode. When the target database receives either the first data processing result or the second data processing result, the preset data processing operation is performed on the received data processing result, which can use data faster, improve data availability, and increase business response speed.

[0055] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a data processing device, etc. The following takes a data processing device as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the present application embodiment provides a data processing method, referring to Figure 3 , Figure 3 This is a flowchart of the first embodiment of the data processing method of the present application.

[0057] In this embodiment, the data processing method includes steps S10 to S20:

[0058] Step S10: Process the source data files in the data warehouse in parallel using the first data processing link and the second data processing link.

[0059] Step S20: When the target database receives any one of the first data processing result of the first data processing link and the second data processing result of the second data link, a preset data processing operation is performed on the received data processing result.

[0060] In this embodiment, the first data link is an internal data processing platform of the data warehouse that processes the source data. The second data processing link is an external data processing platform that processes the source data, wherein the external data processing platform is in communication connection with the data warehouse. The external data processing platform is a Spark data processing platform.

[0061] The source data file can be data collected from various data sources (such as databases, log files, sensor data, external APIs, etc.) without being cleaned, transformed or integrated. The target database can be an Oracle database.

[0062] The preset data processing operation may be that after the target database receives either the first data processing result or the second data processing result, it starts to execute the end-of-day task by online application of the ATFA external trigger task scheduling platform, such as updating the data processing results in the scheduling platform using the first data processing result or the second data processing result.

[0063] The first data processing result may include the data volume of the source data file processed by the first data processing link, the source data file type, the result file after the source data file is extracted, converted, and loaded, etc. The second data processing result may include the data volume of the source data file processed by the second data processing link, the data file type, the result file after the source data file is extracted, converted, and loaded, etc.

[0064] Specifically, refer to Figure 4 As shown, Figure 4 The figure is a flow chart of the parallel processing of source data files by the first data link and the second data link. For the basic data files, i.e., the source data files, the first data processing link and the second data processing link extract, convert, load, migrate, etc. the source data files in parallel. When the first data processing link is used to process the source data files in the data warehouse, the internal data processing platform of the data warehouse extracts, converts, loads, etc. the source data files, i.e., the data warehouse internal job data aggregation. After the internal data processing platform of the data warehouse extracts, converts, loads, etc. the source data files to obtain the first data processing result, the first data processing result can be migrated to the Oracle database, i.e., the ETL data migration is performed through the ETL data link to migrate the first data processing result to the Oracle database.

[0065] When using the second data processing link to process the source data files in the data warehouse, in order to avoid data delay caused by a large number of data warehouse operations, the tasks of data extraction, aggregation, migration, etc. can be handed over to the big data fusion platform for processing, such as the personal loan big data fusion platform, refer to Figure 5 As shown, Figure 5 A schematic diagram of a flow chart of processing a source data file for a second data processing link.

[0066] Specifically, after the source data file is generated, the source data file is first exported through the exchange platform, and then the personal loan big data fusion platform is used to perform data extraction, aggregation, and migration tasks to obtain the second data processing result. Figure 6 As shown, Figure 6 This is a flow chart of extraction, aggregation, and migration of the personal loan big data fusion platform. For the extraction operation, the source data file is obtained from the data exchange platform, that is, Figure 6 Aggregation operation, that is, data processing is performed on the results of the extraction operation to generate data processing results, that is, Figure 6 The migration job is to migrate the data generated by the aggregation job from the HIVE table of the big data platform to the business database. Figure 6 After obtaining the second data processing result, the second data processing result can be stored in HIVE (honeycomb data warehouse), that is, the external data processing platform processes the source data. After storing the second data processing result in HIVE, the second data processing result can be migrated from HIVE to the Oracle database through the data output task.

[0067] It is understandable that when the second data link is used to process the source data file, the duration of the corresponding job task when the extraction node, aggregation node or migration node is executed may exceed the specified duration. For example, the amount of data processed may exceed expectations, resulting in extended processing time, or network delays or failures may also affect data transmission and processing speed, resulting in extended processing time. In order to improve the efficiency and timeliness of data processing, further, as an optional implementation method, when any node in the second data processing link exceeds the specified duration, a prompt message is output to the operation and maintenance personnel to ensure that the operation and maintenance personnel can respond to the problem quickly, such as automatically notifying the operation and maintenance personnel by greeting, calling, etc.

[0068] When the target database receives either the first data processing result of the first data processing link or the second data processing result of the second data link, it can start executing the end-of-day task by online application of the ATFA external trigger task scheduling platform, such as using the first data processing result or the second data processing result to update the data in the financial transaction cloud task scheduling platform.

[0069] It should be noted that when the end-of-day task is started by the online application of the ATFA external trigger task scheduling platform, the end-of-day task depends on the task scheduling platform, which is responsible for scheduling and container resource management. In order to provide system robustness and ensure that the end-of-day task can be executed as scheduled in the event of infrastructure failure, further, as an optional implementation method, step S20 specifically includes:

[0070] Step S201, determining whether the actual duration of performing a preset data processing operation on a received data processing result is greater than a preset duration.

[0071] Step S202: when the actual duration is greater than the preset duration, output abnormal situation alarm information.

[0072] Specifically, when the end-of-day task is started to be executed through the online application ATFA external trigger task scheduling platform, the online application ATFA can identify whether the end-of-day task is completed within the specified time. When the online application ATFA identifies that the end-of-day task is not completed within the specified time, it can output abnormal situation alarm information to remind the operation and maintenance personnel to check the abnormal situation, that is, to determine whether the actual time of performing the preset data processing operation on the received data processing result is greater than the preset time. If the actual time is greater than the preset time, the abnormal situation alarm information is output.

[0073] After outputting the abnormal situation alarm information, the operation and maintenance personnel need to carefully evaluate the current system status after receiving the abnormal situation alarm information and determine whether it is necessary to initiate a downgrade strategy, that is, whether to migrate the real-time source data files to the target database.

[0074] It is understandable that, due to the huge amount of data to be migrated, in order to ensure the accuracy of the migrated data, so that when the data is abnormal, it is possible to intervene in advance and quickly locate and solve the problem, further, as an optional implementation, after step S20, it also includes:

[0075] Step S21A: when the target database receives the other of the first data result and the second data result, check whether the first data result is consistent with the second data result.

[0076] Step S22A: in case of inconsistency, terminate the execution of the preset data processing operation.

[0077] Specifically, after receiving the first data processing result and the second data processing result, the target database can write the first data processing result as ETL data (data processed by Extract, Transform, Load) into the first data table corresponding to the first data processing link, and write the second data processing result as Spark data (data processed by a big data processing framework such as Apache Spark) into the second data table corresponding to the second data processing link, wherein the first data table and the second data table are independent of each other, and according to the received first data processing result and the second data processing result and the date of the preprocessing operation on the first data processing result or the second data processing result, the date corresponding to the first data result and the second data processing result is generated in the data processing summary table. After receiving the first data processing result and the second data processing result belonging to the same date, the target database can check whether the first data processing result is consistent with the second data processing result, such as whether the total amount of data processed by the first data processing link is consistent with the total amount of data processed by the second data processing link, and whether the detail fields in the source data file processed by the first data processing link are consistent with the detail fields in the source data file processed by the second data processing link. As shown in Table 1.

[0078] Table 1

[0079]

[0080]

[0081] When the first data processing result is inconsistent with the second data processing result, the preset data processing operation is terminated, and the abnormal data, such as the type of abnormal data, the cause of the abnormality, etc., are confirmed. After the abnormal data is determined, the abnormal data can be used as abnormal alarm information and the abnormal alarm information is output. After receiving the abnormal alarm information, the operation and maintenance personnel can review the abnormal data. After the operation and maintenance personnel review the system inspection results and find that they are correct, they can modify the trigger flag to the correct data link and re-trigger the end of the day again, so as to achieve the purpose of emergency switching of the data source, and finally complete the end of the day operation on time and accurately.

[0082] In this embodiment, when the target database receives the other of the first data result and the second data result, it checks whether the first data result and the second data result are consistent, that is, through the mutual verification of dual-link data, it can not only ensure the accuracy of the end-of-day data, but also intervene in advance, locate and solve the problem in the case of data anomalies. In addition, if a data problem is found during the comparison process, the preset data processing operation can be terminated in time to stop the loss as much as possible to avoid causing more business impact, and the operation and maintenance personnel can be notified to handle it.

[0083] It should be noted that, in order to avoid repeated processing or data conflicts caused by triggering multiple tasks at the same time, further, after step S20, the following steps are further included:

[0084] Step S21B, generating a current processing cycle field in the data processing summary table.

[0085] Step S22B, when the generation of the current processing cycle field is detected in the current processing cycle, the occupation status of the unique index of the data processing channel is detected.

[0086] Step S23B, when the occupied state is unoccupied, a preset data processing operation is performed on the received data processing result through the unique index of the data processing channel, and the state identifier of the unique index channel is updated to the occupied state.

[0087] In this embodiment, the current processing period field may be the date of performing a preprocessing operation on the first data processing result or the second data processing result. The unique index may be a field or a combination of fields used to uniquely identify a certain date data in the target database.

[0088] Specifically, referring to Table 1, when the target database receives either the first data processing result or the second data processing result, it can generate a current processing cycle field in the data processing summary table, such as 2022.02.17. After the current processing cycle field is generated in the data processing summary table, the occupancy status of the unique index of the data processing channel can be detected. When the occupancy status is unoccupied, the preset data processing operation is performed on the received data processing result through the unique index of the data processing channel, and the status identifier of the unique index channel is updated to the occupied state, so as to avoid the target database from performing the preset data processing operation on the received data processing result when receiving the other of the first data processing result and the second data processing result, thereby avoiding secondary triggering and causing abnormal business logic.

[0089] In this embodiment, when the occupancy status of the unique index of the data processing channel of the current processing cycle field corresponding to the data processing result is unoccupied, the preset data processing operation is performed on the received data processing result through the unique index of the data processing channel, that is, the unique index is used to ensure that each data processing result is processed only once, thereby reducing the risk of data duplication or conflict, thereby improving the accuracy of the data.

[0090] It is worth mentioning that when the data warehouse processes the source data file, there will be a problem that the data processing takes too long. The main reason for this problem is that before the data warehouse extracts, converts, loads and processes the source data file, the data warehouse first needs to obtain the source data from multiple source databases and generate the source data file based on the source data. Since the source data for generating the source data file may need to be obtained from multiple source databases, and the correlation between some source databases and the source data file is not large, it will take a lot of time to obtain the source data for generating the source data file from these source databases, which will cause the data warehouse to process the source data file. In order to solve this problem and improve the efficiency of business execution, as an optional real-time method, when the data warehouse obtains the source data of the source data file from multiple source databases, it first determines the correlation between the source data and each source database, and deletes the source database whose correlation is lower than the preset threshold.

[0091] For example, as mentioned above, the average end-of-day completion time during the trial run was 05:30, and some even ended at 08:00, far exceeding the expected end time. After confirming that the cause of this phenomenon was that the data warehouse took too long to process data, we checked the execution efficiency of all related jobs again and found that 50% of the days of some dependent external jobs ended at more than 04:30, and the execution efficiency was much lower than the average time of other jobs, which was 04:10. Therefore, it can be clearly seen that this job was the key factor causing the long delay at the end of the day. After analyzing the code and evaluating the amount of data and the scope of impact, it was decided to delete this dependency and modify it to a real-time interface call, which advanced the execution end time by about 36 minutes.

[0092] In this embodiment, compared with the related art, in which the source data files are aggregated and processed by the data warehouse, the present application uses the first data link and the second data link in parallel to process the source data files in the data warehouse, which not only avoids excessive dependence on the data warehouse, but also makes full use of computing resources, reduces the total time of data processing, and enhances the stability and timeliness of the data processing link. At the same time, the first data processing link and the second data processing link are used in a dual-link parallel operation mode. When the target database receives either the first data processing result or the second data processing result, the preset data processing operation is performed on the received data processing result, which can use data faster, improve data availability, and increase business response speed.

[0093] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0094] This application also provides a data processing device, please refer to Figure 7 , the data processing device comprises:

[0095] A first processing module, configured to process source data files in the data warehouse in parallel using a first data processing link and a second data processing link;

[0096] The second processing module is used to perform a preset data processing operation on the received data processing result when the target database receives any one of the first data processing result of the first data processing link and the second data processing result of the second data link.

[0097] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem that the data warehouse aggregates and processes data, resulting in a long processing task execution time. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0098] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned embodiment 1.

[0099] Reference below Figure 8 , which shows a schematic diagram of the structure of a data processing device suitable for implementing an embodiment of the present application. The data processing device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The data processing data warehouse shown performs data aggregation and processing, resulting in a long execution time for the processing task. The device is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0100] like Figure 8As shown, the data processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0101] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0102] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem that the data warehouse aggregates and processes data, resulting in a long processing task execution time. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and the other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0103] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0104] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0105] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.

[0106] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0107] The computer-readable storage medium may be included in a data processing device, or may exist independently without being incorporated into a data processing device.

[0108] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a data processing device, the data processing device: utilizes a first data processing link and a second data processing link to process source data files in a data warehouse in parallel; and when a target database receives any one of a first data processing result of the first data processing link and a second data processing result of the second data link, performs a preset data processing operation on the received data processing result.

[0109] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0110] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0112] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problem that the data warehouse aggregates and processes data, resulting in a long processing task execution time. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the data processing method provided by the above-mentioned embodiment, and will not be repeated here.

[0113] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data processing method when executed by a processor.

[0114] The computer program product provided by the present application can solve the technical problem that the data warehouse aggregates and processes data, resulting in a long processing task execution time. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and will not be repeated here.

[0115] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The data processing method comprises: Processing source data files in the data warehouse in parallel using the first data processing link and the second data processing link; When the target database receives any one of the first data processing result of the first data processing link and the second data processing result of the second data link, a preset data processing operation is performed on the received data processing result.

2. The data processing method according to claim 1, characterized in that: The first data link performs data processing on the source data via an internal data processing platform of the data warehouse; The second data processing link is a data processing platform that performs data processing on the source data, wherein the external data processing platform is communicatively connected to the data warehouse.

3. The data processing method according to claim 2, characterized in that: The external data processing platform is the Spark data processing platform.

4. The data processing method according to claim 3, characterized in that: When the target database receives any one of the first data processing result of the first data processing link and the second data processing result of the second data link, after performing a preset data processing operation on the received any one of the first data processing result and the second data processing result of the second data link, the method comprises: When the target database receives the other of the first data result and the second data result, checking whether the first data result is consistent with the second data result; In case of inconsistency, the execution of the preset data processing operation is terminated.

5. The data processing method according to claim 1, characterized in that: After performing the preset data processing operation on the received data processing result, the method includes: Generate the current processing cycle field in the data processing summary table; In the case where the generation of the current processing cycle field is detected in the current processing cycle, detecting the occupation state of the unique index of the data processing channel; When the occupied state is unoccupied, a preset data processing operation is performed on the received data processing result through the unique index of the data processing channel, and the state identifier of the unique index channel is updated to the occupied state.

6. The data processing method according to claim 4, characterized in that: The step of performing a preset data processing operation on the received data processing result comprises: Determine whether the actual duration of performing the preset data processing operation on the received data processing result is greater than the preset duration; When the actual duration is longer than the preset duration, an abnormal situation alarm message is output.

7. A data processing device, characterized in that: The data processing device comprises: A first processing module, configured to process source data files in the data warehouse in parallel using a first data processing link and a second data processing link; The second processing module is used to perform a preset data processing operation on the received data processing result when the target database receives any one of the first data processing result of the first data processing link and the second data processing result of the second data link.

8. A data processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

Images