Data processing method, system and computer readable storage medium

Abstract

The present invention discloses a data processing method, comprising: when a batch system detects an execution instruction, if it determines that the data to be processed corresponding to the execution instruction is big data and that the execution information corresponding to the execution instruction is correct, the execution information is sent to a big data platform, and a task record corresponding to the execution instruction is inserted into a database, and the task status of the task record is monitored; when the big data platform receives the execution information, it runs the processing task corresponding to the execution information, and when a first processing result corresponding to the processing task is obtained, the first processing result is written to the database, and the task status of the task record is updated in the database; if the batch system detects that the task status of the task record is completed, the first processing result is read from the database and output. The present invention also discloses a data processing system and a computer-readable storage medium. The present invention aims to increase the processing speed of data.

Description

Technical Field

[0001] The present invention relates to the technical field of financial technology (Fintech), and in particular to a data processing method, system, and computer-readable storage medium. Background Art

[0002] In recent years, with the continuous development of financial technology (Fintech), particularly internet finance, data processing technology has been introduced into the daily services of banks and other financial institutions. Due to their business needs, banks and other financial institutions generate a large amount of diverse data every day. This data often has certain value, such as determining user needs and adjusting business directions through data analysis. Therefore, banks and other financial institutions need to process this data, such as classification, statistics, and migration.

[0003] In the existing technology, banks and other financial institutions often use batch processing to process data. The application scenario of existing batch processing is to use Spring Batch (a lightweight and complete batch processing framework) alone to read large amounts of data from data sources such as databases, files or queues, and then process the data in batches according to business rules. For example, data can be divided into batches according to time sequence and then processed in batches.

[0004] However, when faced with situations where the data volume reaches big data levels, the data dimensions are multiple, and the calculations are complex, SpringBatch has obvious disadvantages such as being time-consuming. Summary of the Invention

[0005] The main purpose of the present invention is to provide a data processing method, system and computer-readable storage medium, aiming to improve the speed of processing data.

[0006] To achieve the above object, the present invention provides a data processing method, which includes the following steps:

[0007] When the batch system detects an operation instruction, if it is determined that the data to be processed corresponding to the operation instruction is big data, it determines whether the execution information corresponding to the operation instruction is correct, wherein the big data is data with a data volume not less than a preset data threshold;

[0008] When the batch system determines that the execution information is correct, it sends the execution information to the big data platform, inserts the task record corresponding to the running instruction into the database, and monitors the task status of the task record;

[0009] Upon receiving the execution information, the big data platform executes the processing task corresponding to the execution information, and upon obtaining a first processing result corresponding to the processing task, writes the first processing result into the database, and updates the task status of the task record in the database;

[0010] If the batch system detects that the task status of the task record is completed, the batch system reads the first processing result from the database and outputs it.

[0011] Preferably, when the batch system detects an operation instruction, the data processing method further includes:

[0012] If it is determined that the data to be processed is non-big data, the batch system reads the data to be processed, calls the first processing function corresponding to the data to be processed to process the data to obtain a corresponding second processing result, and outputs the second processing result.

[0013] Preferably, the step of determining whether the execution information corresponding to the running instruction is correct includes:

[0014] The batch system determines whether there is a processing task corresponding to the execution information in the big data platform;

[0015] If so, the batch system determines that the execution information is correct.

[0016] Preferably, when the big data platform receives the execution information, the step of running the processing task corresponding to the execution information includes:

[0017] When receiving the execution information, the big data platform reads the task information and parameter set of the execution information and determines the processing task corresponding to the task information;

[0018] The big data platform runs the processing task based on the parameter set, wherein running the processing task includes reading the data to be processed according to the parameter set, and calling the second processing function corresponding to the data to be processed to process the data to be processed to obtain the corresponding first processing result.

[0019] Preferably, after the step of writing the first processing result corresponding to the processing task into the database and updating the task status of the task record in the database when the first processing result corresponding to the processing task is obtained, the data processing method further includes:

[0020] If the batch system detects that the task status of the task record is processing failure, it sends the execution information to the big data platform again, and at the same time, updates the task status of the task record to processing, and monitors the task status;

[0021] If the batch system detects that the task status of the task record is completed, the batch system reads the corresponding third processing result from the database and outputs it.

[0022] Preferably, if the batch system detects that the task status of the task record is processing failure, the execution information is sent to the big data platform again, and at the same time, the task status of the task record is updated to processing, and after the step of monitoring the task status, the data processing method further includes:

[0023] If the batch system detects that the task status of the task record is processing failure, it obtains log information of the first processing result and the third processing result, and outputs error information including the log information.

[0024] Preferably, if the batch system detects that the task status of the task record is completed, the step of reading the first processing result from the database and outputting it includes:

[0025] If the batch system detects that the task status of the task record is completed, reading the first processing result from the database and determining the purpose of the first processing result;

[0026] The batch system determines a processing method corresponding to the purpose, processes the first processing result using the processing method, and outputs the processed first processing result.

[0027] Preferably, when the batch system detects an operation instruction, if it is determined that the data to be processed corresponding to the operation instruction is big data, then after the step of determining whether the execution information corresponding to the operation instruction is correct, the data processing method further includes:

[0028] When the batch system determines that the execution information is wrong, it initializes the batch system and obtains pre-configured initial execution information as the current execution information, and sends the execution information corresponding to the running instruction to the big data platform, inserts the task record corresponding to the running instruction into the database, and monitors the task status of the task record.

[0029] In addition, to achieve the above-mentioned purpose, the present invention also provides a data processing system, which includes: a memory, a processor, and a data processing program stored in the memory and executable on the processor, and the data processing program implements the steps of the data processing method described above when executed by the processor.

[0030] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a data processing program is stored. When the data processing program is executed by a processor, the steps of the data processing method described above are implemented.

[0031] The data processing method proposed by the present invention is as follows: when a batch system detects a run instruction, if it determines that the data to be processed corresponding to the run instruction is big data, it determines whether the execution information corresponding to the run instruction is correct, wherein the big data is data with a data volume not less than a preset data threshold; when the batch system determines that the execution information is correct, it sends the execution information to the big data platform, inserts the task record corresponding to the run instruction into the database, and monitors the task status of the task record; when the big data platform receives the execution information, it runs the processing task corresponding to the execution information, and when obtaining the first processing result corresponding to the processing task, writes the first processing result into the database and updates the task status of the task record in the database; if the batch system detects that the task status of the task record is completed, it reads the first processing result from the database and outputs it. The present invention places the processing task of big data on a big data platform with a faster processing speed and then places the processing result in the database. The batch system only needs to read the processing result directly from the database without processing the processing task, thereby effectively improving the speed of processing data. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the data processing system structure of the hardware operating environment involved in the embodiment of the present invention;

[0033] Figure 2 Schematic diagram of the flow of the first embodiment of the data processing method of the present invention.

[0034] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0035] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0036] like Figure 1 As shown, Figure 1 It is a schematic diagram of the system structure of the hardware operating environment involved in the embodiment of the present invention.

[0037] like Figure 1As shown, the system may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0038] Those skilled in the art will understand that Figure 1 The system structure shown in the figure does not constitute a limitation of the data processing system, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0039] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module and a data processing program.

[0040] Among them, the operating system is a program that manages and controls the data processing system and software resources, and supports the operation of the network communication module, user interface module, data processing program and other programs or software; the network communication module is used to manage and control the network interface 1002; the user interface module is used to manage and control the user interface 1003.

[0041] exist Figure 1 In the data processing system shown, the data processing system calls the data processing program stored in the memory 1005 through the processor 1001 and executes the operations in each embodiment of the following data processing method.

[0042] Based on the above hardware structure, an embodiment of the data processing method of the present invention is proposed.

[0043] Reference Figure 2 , Figure 2 This is a flow chart of a first embodiment of a data processing method according to the present invention, wherein the method comprises:

[0044] Step S10: When the batch system detects an operation instruction, if it is determined that the data to be processed corresponding to the operation instruction is big data, it determines whether the execution information corresponding to the operation instruction is correct, wherein the big data is data with a data volume not less than a preset data threshold;

[0045] Step S20: When the batch system determines that the execution information is correct, it sends the execution information corresponding to the running instruction to the big data platform, inserts the task record corresponding to the running instruction into the database, and monitors the task status of the task record;

[0046] Step S30: upon receiving the execution information, the big data platform executes the processing task corresponding to the execution information, and upon obtaining a first processing result corresponding to the processing task, writes the first processing result into the database, and updates the task status of the task record in the database;

[0047] Step S40: If the batch system detects that the task status of the task record is completed, the first processing result is read from the database and output.

[0048] The data processing method of this embodiment is applied to the data processing system of financial institutions such as wealth management institutions or banks. The data processing system includes a batch system, a BDP big data platform and a DB database. Among them, the batch system is preferably SpringBatch (a lightweight and complete batch processing framework designed to help enterprises build robust and efficient batch processing applications) including a message middleware. The batch system and the big data platform are connected through the message middleware and the database. During the business implementation process, due to the strong processing capability of the big data platform, data that requires complex calculations or occupies a high amount of computing resources will be placed on the big data platform for processing, while simple data or data that occupies fewer computing resources will be processed by the batch system itself.

[0049] When the data processing system of this embodiment is started, it receives the running instruction through the batch system. When the batch system receives the running instruction, if it determines that the corresponding data to be processed is big data and the execution information corresponding to the running instruction is correct, it sends the execution information to the big data platform through the message middleware and inserts a task record in the task polling table created in the database. The big data platform includes a listener, which is responsible for receiving the execution information sent by the batch system and pulling up the corresponding processing task, that is, the Hive computing task. Hive refers to a data warehouse tool based on Hadoop, which can map structured data files to a database table and provide simple SQL query functions. It can convert SQL statements into MapReduce tasks for execution. Hadoop refers to a distributed system infrastructure. MapReduce is a programming model for parallel computing of large-scale data sets (greater than 1TB). Due to the existing tools used, the specific processing process is not described in detail here. After the Hive computing task is executed, the corresponding processing results are written to the database, and the pre-created Hive UDF function (user-defined function, a user-defined function refers to a function customized in the Hive data tool) updates the task status of the task record in the database, while the batch system monitors the task status of the task record in the database. When it determines that the task status is completed, it directly reads the processing result in the database. This makes the entire process of processing big data no longer need to be carried out in the batch system. Due to the stronger processing capability of the big data platform, the data processing speed is faster.

[0050] The following describes each step in detail:

[0051] Step S10, when the batch system detects an operation instruction, if it is determined that the data to be processed corresponding to the operation instruction is big data, it determines whether the execution information corresponding to the operation instruction is correct, wherein the big data is data with a data volume not less than a preset data threshold.

[0052] In this embodiment, when the batch system detects the run instruction, it starts the Job batch task. The Job contains three steps, namely Step 1, BdpTaskStep and Step 2. After the Job is started, the three steps are executed in sequence. Among them, Step 1 is optional. If Step 1 exists, Step 1 specifically performs parameter verification or data preprocessing.

[0053] When verifying parameters, Step 1 specifically verifies whether the execution information of the executor BatchTaskCommunicat in BdpTaskStep is correct. BatchTaskCommunicat is an object that inherits Spring Batch Tasklet and is encapsulated based on the task information created by the big data platform and the corresponding parameter set. The method for verifying whether the execution information is correct is to compare the execution information with the execution information created by the big data platform to determine whether the two are consistent; or to determine whether there is a processing task corresponding to the execution information in the big data platform, so that the big data platform can start the corresponding processing task when receiving the execution information.

[0054] When doing data preprocessing, Step 1 specifically preprocesses the data to be processed obtained by the batch system from data sources such as folders, mainly to verify whether the data is valid, and specifically determine whether the obtained data to be processed conforms to the preset format. If not, it is filtered.

[0055] After executing Step 1, the Job continues to execute BdpTaskStep, which is a big data platform task step. Therefore, before this step, the batch system must first determine whether the data to be processed corresponding to the running instruction is big data, where big data is data with a data volume not less than a preset data threshold, such as 1TB. The method for determining whether it is big data can be to determine that the data to be processed is big data if there is a label of the data to be processed in the preset user-defined form, or to determine whether the data to be processed is big data based on the data volume of the data to be processed marked in the running instruction. In specific implementation, it is generally up to the user to define which data is big data, and save the label of the data to be processed that is big data in the user-defined form. Therefore, the batch system only needs to determine whether the data to be processed is big data. Whether it is user-defined big data can be determined by querying the user-defined form to determine whether the current data to be processed is big data. If there is a label corresponding to the data to be processed in the user-defined form, it is big data, otherwise, it is not. Alternatively, the user divides the sources of the data to be processed in advance, and determines whether the current data to be processed is big data by determining the source of the data to be processed. Specifically, the source of the data to be processed can be determined by determining the file directory location where the data to be processed is located, and then, determine whether the source of the current data to be processed is the preset big data source, so as to determine whether the data to be processed is big data. For example, if the data under file directory A are all big data, then it is only necessary to determine whether the data to be processed comes from A. If it comes from A, the current data to be processed is big data, otherwise, it is not, etc.

[0056] If the batch system determines that the current data to be processed is big data, it determines that the current data to be processed needs to be processed by the big data platform and needs to execute BdpTaskStep. Since in this step, BatchTaskCommunicat needs to send execution information to the big data platform through the message middleware, the batch system needs to first determine whether the execution information corresponding to the run instruction is correct.

[0057] Specifically, the step of determining whether the execution information corresponding to the running instruction is correct includes:

[0058] Step a, the batch system determines whether there is a processing task corresponding to the execution information in the big data platform;

[0059] In step b, if the execution information exists, the batch system determines that the execution information is correct.

[0060] In this step, the process of batch system verifying whether the execution information is correct includes: determining whether there is a processing task corresponding to the execution information in the big data platform, which can be determined specifically by the task ID, that is, the execution information contains the task ID created by the big data platform. If it exists, it is determined that the current execution information is correct. If it does not exist, it means that even if the execution information is sent to the big data platform, the corresponding processing task cannot be pulled up, and the current execution information is determined to be incorrect. If it is incorrect, an error message is output.

[0061] In step S20, when the batch system determines that the execution information is correct, it sends the execution information to the big data platform, inserts the task record corresponding to the running instruction into the database, and monitors the task status of the task record.

[0062] In this embodiment, the batch system sends the execution information to the big data platform only when it determines that the execution information is correct, and inserts a task record into the task polling table of the database, and monitors the task status of the task record in real time. That is, in this step, the execution information is sent to the big data platform only when the data to be processed is big data and the execution information is correct. Conversely, the big data platform only processes big data. The monitoring process can be real-time or timed. In this embodiment, it is preferred to monitor the task status of the task record in real time, and the task status includes processing, processing completed, and processing failed. It can be understood that when the task record is first inserted, the task status of the task record is processing.

[0063] In this step, the big data processing task is handed over to the big data platform, and the batch system itself only needs to monitor the task status of the task records in the database.

[0064] Furthermore, the data processing method further includes:

[0065] In step c, if it is determined that the data to be processed is non-big data, the batch system reads the data to be processed, calls the first processing function corresponding to the data to be processed to process the data to obtain a corresponding second processing result, and outputs the second processing result.

[0066] In this step, if the batch system determines that the data to be processed is non-big data, it skips BdpTaskStep and executes Step 2, reading the data to be processed from the data source corresponding to the execution instruction. It then calls the first processing function corresponding to the data to process the data to obtain the corresponding second processing result, and outputs the second processing result. The first processing function is a user-defined function used to process the data to be processed. That is, the first processing function is determined according to user needs. For example, if the user wants to obtain the statistical results of the data to be processed, the first processing function is a user-defined statistical function. For example, if the user wants to know the bank's business income for a month, they issue the corresponding execution instruction. The batch system then obtains the data to be processed based on the execution instruction, calls the corresponding statistical function to process the data to obtain the processing result.

[0067] Furthermore, the data processing method further includes:

[0068] When the batch system determines that the execution information is wrong, it initializes the batch system and obtains pre-configured initial execution information as the current execution information, and sends the execution information corresponding to the running instruction to the big data platform, inserts the task record corresponding to the running instruction into the database, and monitors the task status of the task record.

[0069] In this step, if the batch system determines that the execution information is wrong, the batch system is initialized and restored to the initial configuration stage, thereby obtaining the pre-configured initial execution information, and using the initial execution information as the current execution information. The execution information corresponding to the running instruction is then sent to the big data platform, and the task record corresponding to the running instruction is inserted into the database, and the task status of the task record is monitored, as well as subsequent steps.

[0070] It should be noted that the initial execution information is the user creating the corresponding processing task in the big data platform in advance, and obtaining the corresponding task ID and the parameter set for running the processing task, so that it is generated and configured into the batch system based on the task ID and parameter set, and serves as the initialization parameter of the batch system. Therefore, during the data processing process, if the execution information is incorrect due to Trojan viruses, such as the execution information is maliciously tampered with by Trojan viruses, the initial execution information can be obtained by initializing the batch system, thereby ensuring that the data processing process is not blocked.

[0071] In step S30, when the big data platform receives the execution information, it runs the processing task corresponding to the execution information, and when it obtains the first processing result corresponding to the processing task, it writes the first processing result into the database and updates the task status of the task record in the database.

[0072] In this embodiment, when the big data platform receives the execution information sent by the batch system through the listener, it runs the processing task corresponding to the execution information. That is, the execution information and the processing task correspond one-to-one in the big data platform. When the big data platform receives the execution information, it can pull up the corresponding processing task and then execute the processing task to obtain the corresponding first processing result. When the first processing result is obtained, the first processing result is written into the database, and the task status of the task record is updated in the database. If the processing is successful, the task status of the task record is updated to processing completed, etc.

[0073] Furthermore, when the big data platform receives the execution information, the step of running the processing task corresponding to the execution information includes:

[0074] Step d: upon receiving the execution information, the big data platform reads the task information and parameter set of the execution information and determines the processing task corresponding to the task information;

[0075] In this step, when the big data platform receives the execution information, it reads the task information and parameter set of the execution information, that is, the execution information includes the task information and parameter set, and then determines the corresponding processing task through the task information. In this embodiment, the task information refers to taskId, which is the task ID created on the BDP platform, and the parameter set refers to paramMap, which is the parameter set required for the processing task to run and is configured in advance by the user.

[0076] In step e, the big data platform runs the processing task based on the parameter set, wherein running the processing task includes reading the data to be processed according to the parameter set, and calling the second processing function corresponding to the data to be processed to process the data to be processed to obtain the corresponding first processing result.

[0077] According to the parameter set, the data to be processed is read from the data source corresponding to the running instruction, and the second processing function corresponding to the data to be processed is called to process the data to be processed, thereby obtaining a first processing result, wherein the second processing function is the same as the first processing function.

[0078] Step S40: If the batch system detects that the task status of the task record is completed, the first processing result is read from the database and output.

[0079] In this embodiment, the batch system monitors the task status of the task records in the database in real time. If the batch system detects that the task status is completed, it means that the big data platform has written the first processing result into the database. Then, the first processing result is directly read from the database and output.

[0080] That is, in this step, Step 2 of the Job directly reads the first processing result from the database and outputs it, without further processing the data to be processed.

[0081] When the batch system of this embodiment detects a run instruction, it sends the execution information corresponding to the run instruction to the big data platform, inserts the task record corresponding to the run instruction into the database, and monitors the task status of the task record; wherein, when the big data platform receives the execution information, it runs the processing task corresponding to the execution information, and when obtaining the first processing result corresponding to the processing task, it writes the first processing result into the database, and updates the task status of the task record in the database; if the batch system detects that the task status of the task record is completed, it reads the first processing result from the database and outputs it. The present invention places the processing task on a big data platform with a faster processing speed, and then places the processing result in the database. The batch system only needs to read the processing result directly from the database without processing the processing task, thereby effectively improving the speed of processing data.

[0082] Furthermore, based on the first embodiment of the data processing method of the present invention, a second embodiment of the data processing method of the present invention is proposed.

[0083] The second embodiment of the data processing method differs from the first embodiment of the data processing method in that the data processing method further includes:

[0084] Step f: If the batch system detects that the task status of the task record is processing failure, the execution information is sent to the big data platform again, and at the same time, the task status of the task record is updated to processing, and the task status is monitored;

[0085] Step g: If the batch system detects that the task status of the task record is completed, the corresponding third processing result is read from the database and output.

[0086] If the batch system of this embodiment detects that the task status of the task record is processing failure, that is, the big data platform has not obtained the correct processing result, the big data platform needs to reprocess it in the hope of obtaining the correct processing result.

[0087] The following describes each step in detail:

[0088] Step f: If the batch system detects that the task status of the task record is processing failure, the execution information is sent to the big data platform again. At the same time, the task status of the task record is updated to processing, and the task status is monitored.

[0089] In this embodiment, if the batch system detects that the task status of the task record in the database is processing failure, it means that the big data platform has an error in the processing process and has not obtained the correct processing result. The big data platform needs to reprocess it. Specifically, the batch system will send the execution information to the big data platform again, and update the task status of the task record in the database from processing failure to processing, and monitor the task status again.

[0090] It can be understood that after receiving the execution information, the big data platform performs the same processing process as the above embodiment, which will not be repeated here. If the big data platform finally obtains the third processing result, the third processing result will also be written into the database, and the task status of the task record will be updated to processing completed.

[0091] Step g: If the batch system detects that the task status of the task record is completed, the corresponding third processing result is read from the database and output.

[0092] In this embodiment, if the batch system detects that the task status of the task record in the database is completed after sending the execution information again, the corresponding third processing result is directly read from the database and output.

[0093] Furthermore, after step f, the data processing method further includes:

[0094] Step h: If the batch system detects that the task status of the task record is processing failure, log information of the first processing result and the third processing result is obtained, and error information including the log information is output.

[0095] In this embodiment, if the batch system detects that the task status of the task record in the database is a processing failure, that is, the big data platform fails to process again, the log information of the first processing result and the third processing result is obtained, and an error message containing the log information is output for manual error correction and maintenance, where the first processing result and the third processing result may be consistent.

[0096] Understandably, when the batch system first detects that a task record in the database has a processing failure status, it can also directly report an error to notify the relevant operations and maintenance personnel to perform error correction and maintenance. In actual implementation, the batch system only reports an error after the big data platform fails to process three times.

[0097] When the batch system of this embodiment detects a processing failure, it notifies the big data platform to reprocess in the hope of obtaining the corresponding processing result. If it still fails, an error is reported, and manual error correction and maintenance are performed to realize intelligent data processing. Moreover, if an unexpected error occurs, such as the inability to read the data to be processed from the data source, the accident can be avoided by reprocessing without the need for manual intervention, thereby improving the speed of data processing.

[0098] Furthermore, based on the first and second embodiments of the data processing method of the present invention, a third embodiment of the data processing method of the present invention is proposed.

[0099] The third embodiment of the data processing method differs from the first and second embodiments of the data processing method in that step S30 includes:

[0100] Step i: if the batch system detects that the task status of the task record is completed, reading the first processing result from the database and determining the purpose of the first processing result;

[0101] In step j, the batch system determines a processing method corresponding to the purpose, processes the first processing result using the processing method, and outputs the processed first processing result.

[0102] In Step 2 of this embodiment, after reading the first processing result, the purpose of the first processing result is further determined, and the first processing result is processed using a corresponding processing method so that the processing result is more in line with actual requirements.

[0103] The following describes each step in detail:

[0104] In step i, if the batch system detects that the task status of the task record is completed, the batch system reads the first processing result from the database and determines the purpose of the first processing result.

[0105] In this embodiment, if the batch system detects that the task status of a task record in the database is "processed completed," it reads the first processing result from the database and determines the purpose of the first processing result. Specifically, a mapping relationship is pre-established between the data to be processed and the purpose of the first processing result. For example, if the data to be processed is account information, the purpose of the first processing result is statistics; if the data to be processed is customer conversation information, the purpose of the first processing result is analysis, etc. Therefore, the purpose of the first processing result can be determined based on the data to be processed.

[0106] In step j, the batch system determines a processing method corresponding to the purpose, processes the first processing result using the processing method, and outputs the processed first processing result.

[0107] In this embodiment, the batch system selects a corresponding processing method based on the determined purpose and processes the first processing result. If the purpose of the first processing result is statistics, the processing method of generating a report is selected, that is, the first processing result is generated into a report according to a preset report format, such as a table. If the purpose of the first processing result is analysis, the processing method of exporting to a file is selected, that is, the first processing result is exported to a file according to a preset export method, such as batch export. The purpose of the first processing result corresponds to the processing method one-to-one. Finally, the processed first processing result is output.

[0108] It is understandable that the processing method may be no processing. For example, if the purpose of the first processing result is display, the processing method adopted is no processing, and the first processing result is directly output for display.

[0109] After the batch system of this embodiment reads the processing results from the database, it reprocesses the processing results, which is similar to the "rough processing" performed by the big data platform and the "fine processing" performed by the batch system. The processing tasks are placed on the big data platform for processing, and the batch system directly reads the processing results and then processes them, effectively improving the speed of processing data.

[0110] The present invention also provides a computer-readable storage medium.

[0111] The computer-readable storage medium of the present invention stores a data processing program, which implements the steps of the data processing method described above when executed by a processor.

[0112] The method implemented when the data processing program running on the processor is executed can refer to the various embodiments of the data processing method of the present invention, and will not be described in detail here.

[0113] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0114] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0115] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0116] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data processing method, characterized in that: The data processing method comprises the following steps: When the batch system detects an operation instruction, if it is determined that the data to be processed corresponding to the operation instruction is big data, it determines whether the execution information corresponding to the operation instruction is correct, wherein the big data is data with a data volume not less than a preset data threshold; When the batch system determines that the execution information is correct, it sends the execution information to the big data platform, inserts the task record corresponding to the running instruction into the database, and monitors the task status of the task record; Upon receiving the execution information, the big data platform executes the processing task corresponding to the execution information, and upon obtaining a first processing result corresponding to the processing task, writes the first processing result into the database, and updates the task status of the task record in the database; If the batch system detects that the task status of the task record is completed, the batch system reads the first processing result from the database and outputs it.

2. The data processing method according to claim 1, wherein: The data processing method further includes: When the batch system detects the running instruction, if it is determined that the data to be processed is non-big data, the batch system reads the data to be processed, calls the first processing function corresponding to the data to be processed to process the data to be processed, so as to obtain the corresponding second processing result, and outputs the second processing result.

3. The data processing method according to claim 1, wherein: The step of determining whether the execution information corresponding to the running instruction is correct includes: The batch system determines whether there is a processing task corresponding to the execution information in the big data platform; If so, the batch system determines that the execution information is correct.

4. The data processing method according to claim 1, wherein: When the big data platform receives the execution information, the steps of running the processing task corresponding to the execution information include: When receiving the execution information, the big data platform reads the task information and parameter set of the execution information and determines the processing task corresponding to the task information; The big data platform runs the processing task based on the parameter set, wherein running the processing task includes reading the data to be processed according to the parameter set, and calling the second processing function corresponding to the data to be processed to process the data to be processed to obtain the corresponding first processing result.

5. The data processing method according to claim 1, wherein: After the step of writing the first processing result corresponding to the processing task into the database and updating the task status of the task record in the database when the first processing result corresponding to the processing task is obtained, the data processing method further includes: If the batch system detects that the task status of the task record is processing failure, it sends the execution information to the big data platform again, and at the same time, updates the task status of the task record to processing, and monitors the task status; If the batch system detects that the task status of the task record is completed, the batch system reads the corresponding third processing result from the database and outputs it.

6. The data processing method according to claim 5, wherein: If the batch system detects that the task status of the task record is processing failure, the execution information is sent to the big data platform again, and at the same time, the task status of the task record is updated to processing, and after the step of monitoring the task status, the data processing method further includes: If the batch system detects that the task status of the task record is processing failure, it obtains log information of the first processing result and the third processing result, and outputs error information including the log information.

7. The data processing method according to any one of claims 1 to 6, wherein: If the batch system detects that the task status of the task record is completed, the step of reading the first processing result from the database and outputting it includes: If the batch system detects that the task status of the task record is completed, reading the first processing result from the database and determining the purpose of the first processing result; The batch system determines a processing method corresponding to the purpose, processes the first processing result using the processing method, and outputs the processed first processing result.

8. The data processing method according to claim 1, wherein: When the batch system detects the running instruction, if it is determined that the data to be processed corresponding to the running instruction is big data, after the step of determining whether the execution information corresponding to the running instruction is correct, the data processing method further includes: When the batch system determines that the execution information is wrong, it initializes the batch system and obtains pre-configured initial execution information as the current execution information, and sends the execution information corresponding to the running instruction to the big data platform, inserts the task record corresponding to the running instruction into the database, and monitors the task status of the task record.

9. A data processing system, characterized in that: The data processing system includes: a memory, a processor, and a data processing program stored in the memory and executable on the processor. When the data processing program is executed by the processor, the steps of the data processing method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a data processing program, which implements the steps of the data processing method according to any one of claims 1 to 8 when executed by a processor.

Images