Data processing system, method, electronic device and storage medium

Abstract

The present application provides a data processing system, method, electronic device and storage medium, the system comprising: a data warehouse and a computing engine; the computing engine being deployed in an independent cluster; the computing engine, in response to receiving a target processing task, obtaining to-be-processed data from the data warehouse, batch-processing the to-be-processed data based on configuration data corresponding to the to-be-processed data, and obtaining a batch processing result; after completing the target processing task, the computing engine merges at least two sub-files to obtain a target file, and transmits the target file to the data warehouse, the sub-file being obtained by packaging a preset number of batch results when the computing engine determines that the number of batch processing results is greater than a preset number, and the target file adopts a structured storage format; after receiving the target file, the data warehouse stores the batch processing results in the target file.

Description

Technical Field

[0001] The present application relates to the fields of data processing and cloud computing, and in particular to a data processing system, method, electronic device, and storage medium. Background Art

[0002] Currently, the bank's impairment system requires a monthly impairment provision for all assets on the application database side (the computing engine is located on the same cluster as the application database, sharing computing and storage resources). This provision process is performed on a detailed, item-by-item basis, resulting in a large data volume and lengthy batch runs. A single period of detailed data contains approximately 20-30 million items, and the entire monthly batch run takes approximately 2.5-3 hours. Due to high requirements for data quality and accuracy at the end of the month, multiple reruns may be required, and three reruns can take 7-9 hours. This results in insufficient computing resources in the application database, low batch run efficiency, and high operational and maintenance costs. Furthermore, the batch run time is excessive, and the completed data needs to be stored in the application database, placing significant storage pressure on the application database. Summary of the Invention

[0003] The present application provides a data processing system, method, electronic device and storage medium.

[0004] In one aspect, an embodiment of the present application provides a data processing system, the system comprising: a data warehouse and a computing engine; the computing engine is deployed in an independent cluster;

[0005] In response to receiving the target processing task, the computing engine obtains the data to be processed from the data warehouse, and performs batch processing on the data to be processed based on the configuration data corresponding to the data to be processed to obtain a batch processing result;

[0006] After the target processing task is completed, the computing engine merges at least two sub-files to obtain a target file, and transmits the target file to the data warehouse, wherein the sub-file is obtained by the computing engine by packaging a preset number of batch processing results when it determines that the number of batch processing results is greater than a preset number, and the target file adopts a structured storage format;

[0007] After receiving the target file, the data warehouse stores the batch processing result in the target file.

[0008] Wherein, the system further includes an application database:

[0009] In response to receiving the target processing task, the computing engine reads a configuration data table from the application database and stores the configuration data table into a memory.

[0010] In which, the computing engine includes a storage device. When the computing engine determines that at least one target information has been received and the computing engine is not executing a processing task, the computing engine reads the update information corresponding to the target information from the application database, and updates the configuration data table stored in the storage device based on the update information. The target information indicates that the configuration data table in the application database has changed, and the configuration data table includes multiple configuration data.

[0011] In which, the computing engine obtains at least one target identifier from the data to be processed, and based on the at least one target identifier, obtains configuration data corresponding to the data to be processed from the configuration data table in the memory, wherein the target identifier includes at least a business identifier, and the business identifier represents the business type to which the data to be processed belongs.

[0012] Wherein, the data warehouse further includes at least one storage node;

[0013] The computing engine transmits the target file to a target storage node, where the target storage node is determined from the at least one storage node;

[0014] After receiving the target file, the target storage node imports the batch processing result in the target file into a database, updates the data in the target data table of the database, and refreshes the data to make the updated data in the target data table of the database effective.

[0015] wherein, after the target processing task is completed, the computing engine generates a report based on the batch processing result and sends it to the application database;

[0016] After receiving the report, the application database stores the report and displays it to the user.

[0017] Another aspect of the present application provides a data processing method, which is applied to a data processing system, wherein the system includes a data warehouse and a computing engine, and the computing engine is deployed in an independent cluster. The method includes:

[0018] In response to receiving the target processing task, obtaining the data to be processed from the data warehouse;

[0019] performing batch processing on the data to be processed based on configuration data corresponding to the data to be processed to obtain a batch processing result;

[0020] After the target processing task is completed, at least two sub-files are merged to obtain a target file, and the target file is transmitted to the data warehouse so that the data warehouse stores the batch processing results in the target file after receiving the target file. The sub-file is obtained by packaging the preset number of batch processing results when it is determined that the number of batch processing results is greater than the preset number. The merged target file adopts a structured storage format.

[0021] The system further includes an application database, and the method further includes:

[0022] In response to receiving the target processing task, reading a configuration data table from the application database;

[0023] The configuration data table is stored in a memory.

[0024] In another aspect, the present application provides an electronic device, comprising:

[0025] a processor, a memory for storing instructions executable by the processor;

[0026] The processor is used to read the executable instructions from the memory and execute the instructions to implement the data processing method.

[0027] On the other hand, the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the data processing method.

[0028] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other objects, features and advantages of the exemplary embodiments of the present application will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present application are shown in an illustrative and non-limiting manner, in which:

[0030] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts.

[0031] Figure 1 A schematic structural diagram of a data processing system according to an embodiment of the present application is shown;

[0032] Figure 2 A schematic structural diagram of a data processing system according to another embodiment of the present application is shown;

[0033] Figure 3A schematic structural diagram of a data processing system according to another embodiment of the present application is shown;

[0034] Figure 4 A flow chart showing a data processing method according to an embodiment of the present application is shown;

[0035] Figure 5 A flow chart showing a data processing method according to another embodiment of the present application is shown;

[0036] Figure 6 A flow chart showing a data processing method according to another embodiment of the present application is shown;

[0037] Figure 7 A flow chart showing a data processing method according to another embodiment of the present application is shown;

[0038] Figure 8 A flow chart showing a data processing method according to another embodiment of the present application is shown;

[0039] Figure 9 A flow chart showing a data processing method according to another embodiment of the present application is shown;

[0040] Figure 10 A schematic diagram of the structure of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0041] In order to make the purpose, features, and advantages of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of this application.

[0042] In order to improve batch running efficiency, reduce operation and maintenance costs, shorten batch running time, and reduce storage pressure on application databases, an embodiment of the present application provides a data processing system, such as Figure 1 As shown, the system 100 includes: a data warehouse 101 and a computing engine 102; the computing engine 102 is deployed in an independent cluster.

[0043] In this embodiment, data warehouse 101 may utilize Hive (a data warehouse) or Spark SQL (a data warehouse), and computing engine 102 may utilize Hadoop MapReduce (a computing engine) or Apache Spark (a computing engine). In other embodiments, data warehouse 101 may utilize any other data warehouse, and computing engine 102 may utilize any other computing engine. The device that hosts computing engine 102 is located in an independent cluster. In other embodiments, computing engine 102 may also be located in a cluster adjacent to data warehouse 101.

[0044] In response to receiving the target processing task, the computing engine 102 obtains the data to be processed from the data warehouse 101 , and performs batch processing on the data to be processed based on the configuration data corresponding to the data to be processed to obtain a batch processing result.

[0045] In this embodiment, the target processing task is issued by the user on the application database (front-end). The target processing task can be a processing task that requires batch processing, such as batch data impairment processing, cross-system data fusion, and AI model training. In other implementations, the target processing task can be any other processing task that requires batch processing.

[0046] After receiving the target processing task, calculation engine 102 reads the corresponding data to be processed from data warehouse 101 based on the target processing task, obtains the configuration data corresponding to the data to be processed, and then performs batch processing on the data to obtain the batch processing results based on the configuration data. When the target processing task is to perform batch impairment processing on data, the configuration data may include impairment parameters (including general codes, a list of impairment items, various adjustment coefficients, etc.) and calculation engine parameters (including judgment rule parameters, measurement rule parameters, product measurement model configuration table, etc.).

[0047] After the target processing task is completed, the computing engine 102 merges at least two sub-files to obtain a target file, and transmits the target file to the data warehouse 101. The sub-files are obtained by the computing engine 102 packaging a preset number of batch processing results when it is determined that the number of batch processing results is greater than a preset number. The merged target file adopts a structured storage format.

[0048] In this embodiment, when the number of batch processing results obtained by the computing engine 102 is greater than a preset number, the computing engine 102 packages these preset number of batch processing results into sub-files, and after the target processing task is completed, merges all sub-files to obtain the target file, and finally transfers the target file to the data warehouse 101 for storage.

[0049] For example, when executing a batch processing task, computing engine 102 generates a parquet file (a file format) based on the paginated batch processing results. Each batch of the results is stored locally (approximately 7,000 records per batch). After completing the target processing task, computing engine 102 starts a new process to merge the parquet files generated by the previous pagination into a new standard parquet file. Computing engine 102 uploads the merged parquet file to data warehouse 101 for storage.

[0050] The target file uses a structured storage format.

[0051] For example, the target file adopts a binary structured storage format, and the header includes fixed-length file metadata: a file identification header (including information such as identifying the file type), the number of batch processing results, the main body stores a preset number of batch processing results in a preset order, and the tail includes an index table, which stores a preset number of records in sequence. Each record contains the offset and data length of the corresponding batch processing result, realizing fast random access.

[0052] After receiving the target file, the data warehouse 101 stores the batch processing result in the target file.

[0053] In the above solution, by deploying computing engine 102 in an independent cluster and constructing it into data processing system 100 with data warehouse 101, dual decoupling from the application database's computing and storage resources is achieved. Because computing engine 102 operates independently in a dedicated hardware resource environment, it can fully utilize the cluster's computing resources to execute batch processing tasks, avoiding the lack of computing resources caused by sharing computing resources with the application database. This significantly improves the efficiency of data batch processing and shortens the execution time of batch processing. At the same time, batch processing results are directly stored in data warehouse 101, completely eliminating the continuous occupation of application database storage resources and significantly reducing the growth rate of data stored in the application database. The architecture of data processing system 100 systematically solves the performance bottleneck problem of the coupled deployment model through resource isolation and data flow reconstruction, improving batch processing efficiency while ensuring the stability of the front-end business system. Furthermore, during the execution of the target processing task, computing engine 102 no longer transmits processing results one by one in the existing method. Instead, it packages multiple batch processing results into sub-files according to a preset number. After the target processing task is completed, it merges them into a complete target file and sends them to data warehouse 101. This significantly reduces the number of data transfers, mitigates the risk of network fluctuations caused by frequent small file transfers, and reduces the write workload on Data Warehouse 101. The merged target files are stored in a structured format, preserving data integrity while avoiding the problem of fragmented files taking up excessive storage space. Through the collaborative design of merged transmission and centralized storage, the system achieves a balance between processing speed and resource utilization while ensuring data accuracy.

[0054] In an example of the present application, a data processing system is also provided. Figure 2 As shown, the system 100 further includes an application database 103 .

[0055] In this embodiment, the application database 103 may be Oracle (a database), MySQL (a database), SQL Server (a database), OceanBase (a database), etc. In other implementations, the application database 103 may be any other database.

[0056] In response to receiving the target processing task, the computing engine 102 reads the configuration data table from the application database 103 and stores the configuration data table into a memory.

[0057] In this embodiment, since the application database 103 is a front-end database, the user can modify the configuration data corresponding to each processing task through front-end operations. Therefore, the configuration data table is stored in the application database 103.

[0058] After receiving the target processing task, the computing engine 102 reads the configuration data table from the application database 103 and stores the configuration data table into the memory.

[0059] Because users need to modify the configuration data corresponding to each processing task on the front end, the configuration data table needs to be stored in the application database 103. In the prior art, a period (such as one or two days) is typically set, and then the configuration data table is extracted from the application database 103 via the bus. The data is then loaded into the data warehouse 101 the next day. Therefore, the configuration data modified by the user on the front end cannot take effect in the data warehouse 101 in real time. In the above solution, when the computing engine 102 receives the target processing task, it reads the latest configuration data table from the application database 103 in real time and stores it in memory, allowing the configuration data modified by the user on the front end to take effect in real time. This solution eliminates the delay problem of traditional periodic synchronization, ensures that the computing engine 102 always executes batch processing tasks based on the latest configuration data, and improves the system's responsiveness to dynamic adjustments and the real-time performance of processing tasks.

[0060] In an example of the present application, a data processing system is also provided. Figure 2 As shown, the computing engine includes a storage device 1021. When the computing engine 102 determines that at least one target information has been received and the computing engine 102 is not performing a processing task, the update information corresponding to the target information is read from the application database 103, and the configuration data table stored in the memory is updated based on the update information. The target information indicates that the configuration data table in the application database 103 has changed, and the configuration data table includes multiple configuration data.

[0061] In this embodiment, the computing engine 102 no longer fully updates the configuration data table when receiving a target processing task. Instead, when receiving at least one target information sent by the application database 103 and the computing engine 102 is not executing a processing task, the configuration data table stored in the storage device 1021 of the computing engine 102 is incrementally updated.

[0062] The target information is the information generated by the application database 103 and sent to the computing engine 102 after the user modifies the configuration data table in the application database 103 through the front end, so as to notify the computing engine 102 that the configuration data table in the application database 103 has changed and the configuration data table in the storage device 1021 of the computing engine 102 needs to be updated.

[0063] Upon receiving at least one target information from application database 103, computing engine 102 determines whether computing engine 102 is currently executing a processing task. If so, computing engine 102 continues to wait. If not, computing engine 102 reads update information corresponding to the target information from application database 103 based on the target information, and then updates the configuration data table in storage device 1021 of computing engine 102 based on the update information.

[0064] It should be pointed out that the target information can also directly include update information. Then, when the computing engine 102 receives at least one target information sent by the application database 103 and the computing engine 102 is not executing a processing task, the configuration data table in the storage device 1021 can be directly updated based on the update information in the target information without consuming additional communication time.

[0065] In the above-mentioned solution, when the user modifies the configuration data on the front end, causing the configuration data table in the application database 103 to change, the application database 103 will automatically generate target information and send it to the computing engine 102. After receiving at least one piece of target information, once the computing engine 102 determines that the computing engine 102 is in a state of not executing a processing task, it obtains the change information from the application database 103 based on the target information, and synchronously updates the configuration data table in the storage device 1021 based on the updated information. This avoids the limitation of the traditional solution that must rely on a fixed period to update the configuration data table, and at the same time will not interfere with the running tasks due to real-time reading. Every adjustment made by the user to the configuration on the front end can take effect quickly when the computing engine 102 is idle, which not only ensures the stability of the task execution process, but also realizes the precise connection between the configuration data change and the response of the computing engine 102, significantly improving the flexibility and efficiency of the system in dynamically adapting to business needs.

[0066] In an example of the present application, a data processing system is also provided. Figure 2 As shown, the computing engine 102 obtains at least one target identifier from the data to be processed, and obtains configuration data corresponding to the data to be processed from the configuration data table in the memory based on the at least one target identifier, and the target identifier includes at least a business identifier, and the business identifier represents the business type to which the data to be processed belongs.

[0067] The computing engine 102 obtains at least one target identifier from the data to be processed, and obtains configuration data corresponding to the data to be processed from a configuration data table in the memory based on the target identifier.

[0068] The target identifier includes at least a business identifier, which indicates the business type of the data to be processed. For example, the business identifiers include impairment processing, real-time fusion, and model training. When the business identifier is impairment processing, the business type of the data to be processed is batch data impairment processing. When the business identifier is real-time fusion, the business type of the data to be processed is cross-system data fusion. When the business identifier is model training, the business type of the data to be processed is AI model training.

[0069] For different business types, the target identifier may also include other identifiers.

[0070] For example, when the business type is batch data impairment processing, the target identifier may also include an asset identifier, which may be corporate loans, credit card receivables, fixed assets, etc.

[0071] For another example, when the business type is cross-system data fusion, the target identifier can also include a data source identifier and a purpose identifier. The data source identifier can be a core system, an external API (Application Programming Interface), a third-party database, etc., and the purpose identifier can be risk rating, customer profiling, anti-fraud, etc.

[0072] For another example, when the business type is AI model training, the target identifier can also include a scenario identifier, which can be credit scoring, image quality inspection, sales forecast, etc.

[0073] In the above scheme, by using the target identifier carried in the data to be processed, the corresponding configuration data is read from the configuration data table pre-loaded in the memory, which can achieve accurate adaptation to different business scenarios. The business identifier serves as the core classification basis, so that multiple types of data can be processed simultaneously in the same batch of processing tasks. The computing engine 102 loads the corresponding configuration data according to the business type of the data to be processed and switches the computing logic, avoiding the tedious operations of pre-manual classification or multiple scheduling required in traditional schemes. Through the intelligent routing mechanism based on the target identifier, the system can still maintain efficient and stable processing efficiency when facing complex business mixed processing requirements.

[0074] In an example of the present application, a data processing system is also provided. Figure 3 As shown, the data warehouse 101 also includes at least one storage node 1011 .

[0075] The computing engine 102 transmits the target file to a target storage node, where the target storage node is determined from the at least one storage node 1011 .

[0076] The computing engine 102 may determine the target storage node based on the current load status of each storage node 1011 in the data warehouse.

[0077] After receiving the target file, the target storage node imports the batch processing result in the target file into a database, updates the data in the target data table of the database, and refreshes the data to make the updated data in the target data table of the database effective.

[0078] For example, compute engine 102 establishes communication with Hive using the Hive connection tool and executes the "Load Data Inpath" command to import the Parquet file on the target storage node into the specified table in the Hive database. Next, the compute engine establishes communication with Impala (the Impala process running on the query node) using the Impala connection tool and executes the "Data Refresh" command, making the newly added or updated data in the Hive database immediately effective on the Impala node (the query node).

[0079] In the above solution, the efficiency and reliability of the data storage and validation process are further optimized by introducing a dynamic allocation mechanism for storage nodes. The system can determine the optimal storage node to receive the packaged files of the batch processing results based on the current load status, avoiding processing delays caused by excessive pressure on a single storage node. After receiving the file, the target storage node writes the results to the database in batches, which significantly reduces the frequency of database operations compared to updating one by one, while maintaining the stability of the data table structure. After the write is completed, the data refresh mechanism is used to ensure that the update results are effective immediately and are not perceived by external services, effectively shortening the interval from data storage to business availability. Through node selection and batch processing, the execution efficiency of large-scale data updates is improved while maintaining data consistency.

[0080] In an example of the present application, a data processing system is also provided. Figure 2 As shown, after the target processing task is completed, the computing engine 102 generates a report based on the batch processing result and sends it to the application database 103.

[0081] After the target processing task is completed, the computing engine 102 generates a report based on all the batch processing results obtained and sends it to the application database 103. The report can represent the overall information of the business related to the target processing task.

[0082] For example, when the business type is batch data impairment processing, the report can include impairment details (such as the asset impairment amount, impairment ratio, etc.), risk distribution information (such as the asset risk level), execution verification information (such as the deviation between the execution results and expectations), impact assessment information (such as the impact of the impairment on departments and products), and other data.

[0083] For another example, when the business type is cross-system data fusion, the report can include data quality information (such as the integrity, consistency, accuracy of the fused data), field mapping information (such as the field mapping table between different systems), association analysis information (such as the impact of the newly added data dimensions after fusion on the business), processing efficiency information (such as data loading time, data fusion time, etc.) and other data.

[0084] For another example, when the business type is AI model training, the report can include performance indicator information (such as the accuracy and recall rate of the training set and the validation set), training monitoring information (such as the loss function convergence curve, training time, resource utilization, etc.) and other data.

[0085] After receiving the report, the application database 103 stores the report and displays it to the user.

[0086] After receiving the report, the application database 103 stores the report, and visualizes the report through a preset template or other means, and displays it to the user.

[0087] In the above scheme, after the target processing task is completed, the computing engine 102 generates a report based on all batch processing results and transmits it to the application database 103 for storage, without the need for manual export or secondary processing. After receiving the report, the application database 103 can not only retain the original records for a long time, but also visualize them through preset templates and other methods and display them to users. Users can immediately access the report to understand the relevant information of the target processing task. This ensures the consistency of data from processing to presentation, reduces delays and errors in the intermediate links, and avoids the performance loss caused by frequent access to the data warehouse 101 by external systems. While ensuring data security, it significantly improves the timeliness and ease of use of result delivery.

[0088] In order to improve batch running efficiency, reduce operation and maintenance costs, shorten batch running time, and reduce storage pressure on application databases, an embodiment of the present application provides a data processing method, which is applied to a data processing system, the system including a data warehouse and a computing engine, and the computing engine is deployed in an independent cluster, such as Figure 4 As shown, the method includes:

[0089] Step 201: In response to receiving a target processing task, obtain data to be processed from the data warehouse.

[0090] After receiving the target processing task, the computing engine reads the corresponding data to be processed from the data warehouse based on the target processing task.

[0091] Step 202: batch-process the data to be processed based on configuration data corresponding to the data to be processed to obtain a batch processing result.

[0092] Configuration data corresponding to the data to be processed is obtained, and then batch processing is performed on the data to be processed based on the configuration data to obtain a batch processing result.

[0093] In step 203, after the target processing task is completed, at least two sub-files are merged to obtain a target file, and the target file is transmitted to the data warehouse so that the data warehouse stores the batch processing results in the target file after receiving the target file. The sub-file is obtained by packaging the preset number of batch processing results when it is determined that the number of batch processing results is greater than the preset number. The merged target file adopts a structured storage format.

[0094] In an example of the present application, a data processing method is also provided, wherein the system further includes an application database, such as Figure 5 As shown, the method further includes:

[0095] Step 301: In response to receiving a target processing task, read a configuration data table from the application database.

[0096] Users can issue target processing tasks on the front end. After the computing engine receives the target processing tasks issued by the user, it reads the configuration data table from the application database.

[0097] Step 302: Store the configuration data table into memory.

[0098] The configuration data table is stored in the memory, and the corresponding configuration data can be read from the memory later to perform the target processing task.

[0099] In an example of the present application, a data processing method is also provided, wherein the computing engine includes a storage device such as Figure 6 As shown, the method further includes:

[0100] Step 401: Determine that at least one target information is received and the computing engine is not executing a processing task, then read update information corresponding to the target information from the application database, where the target information indicates that a configuration data table in the application database has changed.

[0101] When a user modifies a configuration data table in the application database through front-end operations, the application database sends target information to the computing engine. After receiving at least one target information, the computing engine determines whether it is currently executing a processing task. If so, it continues to wait. If not, it reads the updated information corresponding to the target information from the application database.

[0102] Step 402: Update the configuration data table stored in the storage device based on the update information, where the configuration data table includes a plurality of configuration data.

[0103] Incrementally update the configuration data table stored in memory based on the updated information.

[0104] In an example of the present application, a data processing method is also provided, such as Figure 7 As shown, the method further includes:

[0105] Step 501: Acquire at least one target identifier from the data to be processed, wherein the target identifier at least includes a service identifier, and the service identifier represents the service type to which the data to be processed belongs.

[0106] At least one target identifier is obtained from the data to be processed.

[0107] Step 502: Acquire configuration data corresponding to the data to be processed from the configuration data table in the memory based on the at least one target identifier.

[0108] The configuration data corresponding to the data to be processed is obtained from the configuration data table in the memory based on the target identifier.

[0109] In an example of the present application, a data processing method is also provided, such as Figure 8 As shown, the method further includes:

[0110] Step 601: The target file is transmitted to a target storage node, so that the target storage node imports the batch processing result in the target file into a database after receiving the target file. The target storage node is determined from the at least one storage node.

[0111] The current load status of multiple storage nodes in the data warehouse is obtained, and the optimal storage node is determined from these storage nodes as the target storage node based on the current load status. The optimal storage node may be the storage node with the lowest load or not performing other tasks.

[0112] After the target storage node is determined, the target file is transferred to the target storage node.

[0113] After receiving the target file, the target storage node imports the batch processing results in the target file into the database for storage.

[0114] Step 602: Send a refresh instruction to the target storage node, so that the target storage node refreshes data based on the data in the target data table of the database.

[0115] The computing engine sends a refresh instruction to the target storage node.

[0116] After receiving the refresh instruction, the target storage node refreshes the data based on the data in the target data table of the database. The target data table is the data table imported from the batch processing results in the target file.

[0117] In an example of the present application, a data processing method is also provided, such as Figure 9 As shown, the method further includes:

[0118] Step 701: After the target processing task is completed, a report is generated based on the batch processing result.

[0119] After the target processing task is completed, the computing engine generates a report corresponding to the target processing task business type based on all the obtained batch processing results.

[0120] Step 702: Send the report to the application database, so that the application database stores the report after receiving it and displays it to the user.

[0121] Send the report to the application database.

[0122] After receiving the report, the application database stores the report and visualizes the report through preset templates and other methods to display it to the user.

[0123] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0124] Figure 10 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0125] like Figure 10As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. Computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0126] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0127] The computing unit 801 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the data processing method. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the data processing method by any other suitable means (e.g., via firmware).

[0128] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0130] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0133] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0134] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0135] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the present disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0136] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A data processing system, comprising: Data warehouse and computing engine; The computing engine is deployed in an independent cluster; In response to receiving the target processing task, the computing engine obtains the data to be processed from the data warehouse, and performs batch processing on the data to be processed based on the configuration data corresponding to the data to be processed to obtain a batch processing result; After the target processing task is completed, the computing engine merges at least two sub-files to obtain a target file, and transmits the target file to the data warehouse, wherein the sub-file is obtained by the computing engine by packaging a preset number of batch processing results when it determines that the number of batch processing results is greater than a preset number, and the target file adopts a structured storage format; After receiving the target file, the data warehouse stores the batch processing result in the target file; The system further comprises an application database: In response to receiving the target processing task, the computing engine reads the configuration data table from the application database and stores the configuration data table into a memory; The computing engine includes a storage device. When the computing engine determines that at least one target information has been received and the computing engine is not executing a processing task, the computing engine reads update information corresponding to the target information from the application database, and updates the configuration data table stored in the storage device based on the update information. The target information indicates that a configuration data table in the application database has changed, and the configuration data table includes multiple configuration data.

2. According to the system of claim 1, the computing engine obtains at least one target identifier from the data to be processed, and obtains configuration data corresponding to the data to be processed from the configuration data table in the memory based on the at least one target identifier, and the target identifier includes at least a business identifier, and the business identifier represents the business type to which the data to be processed belongs.

3. The system according to claim 1, wherein the data warehouse further comprises at least one storage node; The computing engine transmits the target file to a target storage node, where the target storage node is determined from the at least one storage node; After receiving the target file, the target storage node imports the batch processing result in the target file into a database, updates the data in the target data table of the database, and refreshes the data to make the updated data in the target data table of the database effective.

4. The system according to claim 1, wherein after the target processing task is completed, the computing engine generates a report based on the batch processing results and sends the report to the application database; After receiving the report, the application database stores the report and displays it to the user.

5. A data processing method, applied to a data processing system comprising a data warehouse and a computing engine, wherein the computing engine is deployed in an independent cluster, comprising: In response to receiving the target processing task, obtaining the data to be processed from the data warehouse; performing batch processing on the data to be processed based on configuration data corresponding to the data to be processed to obtain a batch processing result; After the target processing task is completed, at least two sub-files are merged to obtain a target file, and the target file is transmitted to the data warehouse, so that the data warehouse stores the batch processing results in the target file after receiving the target file, the sub-file is obtained by packaging the preset number of batch processing results when it is determined that the number of batch processing results is greater than a preset number, and the target file adopts a structured storage format; The system further includes an application database, and the method further includes: In response to receiving the target processing task, reading a configuration data table from the application database; Storing the configuration data table in memory; The computing engine includes a storage device, and the method further includes: determining that at least one target information is received and the computing engine is not executing a processing task, reading update information corresponding to the target information from the application database, the target information indicating that a configuration data table in the application database is changed; The configuration data table stored in the storage device is updated based on the update information, where the configuration data table includes a plurality of configuration data.

6. An electronic device comprising: a processor, a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the data processing method according to claim 5. 7 . A computer-readable storage medium storing a computer program for executing the data processing method according to claim 5 .

Images