Batch processing implementation method and apparatus

Abstract

This invention discloses a batch processing method and apparatus, applied in the field of big data technology. The method includes: identifying the data source in the batch processing requirement; determining the batch common mechanism type based on the data source; determining the required batch processing time for the data volume in the batch processing requirement; matching the window of the required batch processing time for the data volume with the overall batch processing, where the overall batch is the sum of the data volume in the batch processing requirement and previously existing batch data; initiating batch processing based on the matched required batch processing time for the data volume; and generating a batch processing template corresponding to the batch common mechanism type. This invention enables intelligent design for batch development while ensuring the stability of the batch architecture, and also contributes to the standardization of the system's batch mechanism.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a method and apparatus for batch processing. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] For mature banking systems, there are various ways to implement batch processing. Whether it is launching a new batch processing function or optimizing performance, after receiving a batch processing requirement, it is an urgent problem to solve the problem of obtaining a highly automated and flexible and reliable batch implementation solution without affecting the stability of the existing overall batch processing. Summary of the Invention

[0004] This invention provides a batch processing implementation method to obtain a highly automated and flexible batch processing solution without affecting the stability of existing overall batch processing. The method includes:

[0005] Identify the data sources in batch processing requirements;

[0006] Based on the data source, determine the type of batch public mechanism;

[0007] The required batch processing time for the data volume is determined based on the data volume in the batch processing requirement. The window of the required batch processing time for the data volume is matched with the overall batch processing, where the overall batch is the sum of the data volume in the batch processing requirement and the previously existing batch data.

[0008] Based on the required batch processing time for the matched data volume, initiate batch processing and generate batch processing templates corresponding to the batch common mechanism types.

[0009] This invention also provides a batch processing implementation apparatus to obtain a highly automated and flexible batch processing solution without affecting the stability of existing overall batch processing. The apparatus includes:

[0010] The identification module is used to identify the data source in batch processing requirements;

[0011] The public mechanism classification module is used to determine the batch public mechanism type based on the data source.

[0012] The job time window determination module is used to determine the job time required for batch processing of the data volume based on the data volume in the batch processing requirement, and to match the window of the job time required for batch processing of the data volume with the overall batch processing, wherein the overall batch is the sum of the data volume in the batch processing requirement and the previously existing batch data.

[0013] The template module is used to start batch processing based on the required batch processing time of the matched data volume and generate batch processing templates corresponding to the batch common mechanism types.

[0014] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described batch processing method.

[0015] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described batch processing method.

[0016] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described batch processing method.

[0017] Compared to existing technologies that require expert analysis and discussion of technical solutions, where different experts have different analysis and design standards, making unified management and maintenance difficult, this invention identifies the data source in batch processing requirements; determines the batch common mechanism type based on the data source; determines the required batch processing time based on the data volume in the batch processing requirements; matches the window of the required batch processing time with the overall batch processing, where the overall batch is the sum of the data volume in the batch processing requirements and previously existing batch data; and initiates batch processing based on the required batch processing time of the matched data volume, generating a batch processing template corresponding to the batch common mechanism type. In this process, this invention starts from the data source, determines the corresponding batch processing template according to the batch common mechanism type, and then automatically executes batch processing through analysis. This enables intelligent design for batch development while ensuring the stability of the batch architecture, and also contributes to the standardization of the system's batch mechanism. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0019] Figure 1 This is a flowchart of the batch processing implementation method in an embodiment of the present invention;

[0020] Figure 2 This is a flowchart illustrating the data classification process in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of the batch common mechanism type in an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of the batch processing implementation device proposed in an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0025] Figure 1 The flowchart of the batch processing implementation method in this embodiment of the invention includes:

[0026] Step 101: Identify the data sources in the batch processing requirements;

[0027] Step 102: Determine the batch public mechanism type based on the data source.

[0028] Step 103: Determine the batch processing time required for the data volume based on the data volume in the batch processing requirement, and match the window of the batch processing time required for the data volume with the overall batch processing, wherein the overall batch is the sum of the data volume in the batch processing requirement and the previously existing batch data.

[0029] Step 104: Based on the required batch processing time for the matched data volume, start batch processing and generate the batch processing template corresponding to the batch common mechanism type.

[0030] In step 101, the data sources in the batch processing requirements are identified.

[0031] In the specific implementation process, based on the background of the requirements, the data source is determined to be entity data or transaction detail data inside or outside the system, thereby achieving the filtering of data source types.

[0032] In step 102, the batch public mechanism type is determined based on the data source.

[0033] In one embodiment, determining the batch public mechanism type based on the data source includes:

[0034] Based on the data source, determine the data category for batch processing;

[0035] Based on the data classification, determine the type of batch public mechanism.

[0036] like Figure 2 The above is a data classification flowchart in an embodiment of the present invention. In one embodiment, determining the data classification of batch-processed data based on the data source includes:

[0037] Step 201: When the data source is internal to the system, determine that the data is classified as entity data, and includes one or any combination of account subject, customer information and invoices;

[0038] Step 202: When the data source is internal to the system, determine the data classification as transaction detail data, which includes one or any combination of contract information generated online and transaction flow data.

[0039] Step 203: When the data source is external to the system, determine that the data is classified as entity data, and includes one or any combination of external related accounts and customer information;

[0040] Step 204: When the data source is external to the system, determine the data classification as transaction detail data, which includes one or any combination of external system inbound details and transaction flow data.

[0041] like Figure 3 The diagram illustrates the batch common mechanism type in an embodiment of the present invention. In one embodiment, the batch common mechanism type is determined based on the data classification, including:

[0042] Step 301: When batch processing transaction detail data online, determine the batch common mechanism type as online batch common mechanism;

[0043] Step 302: When batch processing transaction detail data in night mode, determine the batch public mechanism type as night batch public mechanism;

[0044] Step 303: When batch processing transaction detail data in daytime online mode, determine the batch public mechanism type as daytime batch public mechanism;

[0045] Step 304: When batch processing entity data with low business timeliness, determine the batch common mechanism type as P2 filtering P1 batch common mechanism;

[0046] Step 305: When batch processing entity class data without data modification, determine the batch common mechanism type as data collection common mechanism.

[0047] In the specific implementation process, after identifying the data source, a second subdivision is carried out to determine which of the five types of batch public mechanisms it belongs to: online batch public mechanism, nighttime batch public mechanism, daytime batch public mechanism, P2 screening P1 batch public mechanism, and data collection batch public mechanism. It can be expanded on this basis.

[0048] The following is a specific embodiment illustrating the process of selecting the batch common mechanism type in this embodiment of the invention:

[0049] When the data is classified as transaction details, it is determined whether there are reusable batch online transactions. If there are, the batch online transaction interface can be called to complete the batch processing in the manner of the online batch common mechanism. If there are no, the P2 filtering P1 batch common mechanism is used.

[0050] When the data is classified as entity data, a pure batch processing method is used to complete the batch processing;

[0051] Taking updating customer information as an example, considering the large volume of customer information, low timeliness requirements, inability to maintain customer information through counter transactions, and the lack of conflict between customer information updated by external systems, the P2 filtering P1 batch public mechanism is selected. In this case, the P1 environment is the production operation environment, and the P2 environment is the data collection and backup environment.

[0052] In step 103, the batch processing time required for the data volume in the batch processing requirement is determined based on the data volume in the batch processing requirement. The window of the batch processing time required for the data volume is matched with the overall batch processing, where the overall batch is the sum of the data volume in the batch processing requirement and the previously existing batch data.

[0053] In one embodiment, matching the window of the time required for batch processing of the data volume with the overall batch processing includes:

[0054] The time period from the completion of job A (which requires batch processing of data) to the start of job B is defined as the time period for the matched batch processing jobs.

[0055] In the specific implementation process, after determining the type of batch public mechanism, the job duration is estimated based on the amount of data in the batch processing requirements. The job time window required for the amount of data is matched with the overall batch job schedule, the serial access point is found, and the job time window is determined.

[0056] In step 104, based on the required batch processing time for the matched data volume, batch processing is initiated, and a batch processing template corresponding to the batch common mechanism type is generated.

[0057] In one embodiment, based on the required batch processing time for the matched data volume, batch processing is initiated, and a batch processing template corresponding to the batch common mechanism type is generated. The method further includes:

[0058] Batch processing templates are stored in a template library, and these templates are used to automatically execute batch processing.

[0059] In the specific implementation process, taking the update of customer information data as an example, the batch public mechanism type is determined to be P2 filtering P1 batch public mechanism according to the data classification of customer information data. The batch processing is performed in accordance with the batch processing template corresponding to the template library according to the P2 filtering P1 batch public mechanism, and the batch processing results are generated and returned to the external system.

[0060] In summary, compared with existing technologies that require expert analysis and discussion of technical solutions, where different experts have different analysis and design standards, making unified management and maintenance difficult, the embodiments of the present invention identify the data source in the batch processing requirements; determine the batch common mechanism type based on the data source; determine the batch processing time required for the data volume in the batch processing requirements; match the window of the batch processing time required for the data volume with the overall batch processing, where the overall batch is the sum of the data volume in the batch processing requirements and the previously existing batch data; and start batch processing based on the batch processing time required for the matched data volume, generating a batch processing template corresponding to the batch common mechanism type. In the above process, the present invention starts from the data source, determines the corresponding batch processing template according to the batch common mechanism type, and then automatically executes batch processing through analysis. This enables intelligent design for batch development while ensuring the stability of the batch architecture, and also contributes to the standardization of the system's batch mechanism.

[0061] This invention also proposes a batch processing implementation device, the principle of which is similar to the batch processing implementation method, and will not be described in detail here. Figure 4 The batch processing implementation apparatus proposed in the embodiments of the present invention includes:

[0062] The identification module 401 is used to identify the data source in batch processing requirements;

[0063] The public mechanism classification module 402 is used to determine the batch public mechanism type based on the data source.

[0064] The job time window determination module 403 is used to determine the job time required for batch processing of the data volume based on the data volume in the batch processing requirement, and to match the window of the job time required for batch processing of the data volume with the overall batch processing, wherein the overall batch is the sum of the data volume in the batch processing requirement and the previously existing batch data.

[0065] The template module 404 is used to start batch processing based on the required batch processing time of the matched data volume and generate batch processing templates corresponding to the batch common mechanism types.

[0066] In the specific implementation process, the identification module 401 is used to determine the data source and identify the data type as either internal or external to the system, or as either entity data or transaction detail data.

[0067] In one embodiment, the public mechanism classification module 402 is specifically used for:

[0068] Based on the data source, determine the data category for batch processing;

[0069] Based on the data classification, determine the type of batch public mechanism.

[0070] In one embodiment, the public mechanism classification module 402 is further specifically used for:

[0071] When the data source is internal to the system, the data is classified as entity data, and includes one or any combination of account holders, customer information, and invoices;

[0072] When the data source is internal to the system, the data is classified as transaction detail data, which includes one or any combination of contract information generated online and transaction flow data.

[0073] When the data source is external to the system, the data is classified as entity data, and includes one or any combination of external related accounts and customer information;

[0074] When the data source is external to the system, the data is classified as transaction detail data, which includes one or any combination of external system inbound details and transaction flow data.

[0075] In one embodiment, the public mechanism classification module 402 is further specifically used for:

[0076] When performing batch processing on online transaction detail data, the batch common mechanism type is determined to be online batch common mechanism;

[0077] When batch processing entity data in night mode, the batch common mechanism type is determined to be the night batch common mechanism;

[0078] When batch processing transaction detail data under daytime online mode, the batch public mechanism type is determined to be daytime batch public mechanism;

[0079] When batch processing entity data with low business timeliness, the batch common mechanism type is determined to be P2 filtering P1 common mechanism.

[0080] When batch processing entity class data that has not been modified, the batch common mechanism type is determined to be the data collection common mechanism.

[0081] In one embodiment, the job time window determination module 403 is specifically used for:

[0082] The time period from the completion of job A (which requires batch processing of data) to the start of job B is defined as the time period for the matched batch processing jobs.

[0083] In one embodiment, the template module 404 is specifically used for:

[0084] Batch processing templates are stored in a template library, and these templates are used to automatically execute batch processing.

[0085] In summary, compared with existing technologies that require expert analysis and discussion of technical solutions, where different experts have different analysis and design standards, making unified management and maintenance difficult, the embodiments of the present invention identify the data source in the batch processing requirements; determine the batch common mechanism type based on the data source; determine the batch processing time required for the data volume in the batch processing requirements; match the window of the batch processing time required for the data volume with the overall batch processing, where the overall batch is the sum of the data volume in the batch processing requirements and the previously existing batch data; and start batch processing based on the batch processing time required for the matched data volume, generating a batch processing template corresponding to the batch common mechanism type. In the above process, the present invention starts from the data source, determines the corresponding batch processing template according to the batch common mechanism type, and then automatically executes batch processing through analysis. This enables intelligent design for batch development while ensuring the stability of the batch architecture, and also contributes to the standardization of the system's batch mechanism.

[0086] This invention also provides a computer device. Figure 5 This is a schematic diagram of a computer device in an embodiment of the present invention. The computer device 500 includes a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 530, it implements the aforementioned blockchain-based coupon payment method. This embodiment of the present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the aforementioned parameter localization method.

[0087] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described batch processing method.

[0088] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described batch processing method.

[0089] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0091] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0092] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0093] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A batch processing implementation method characterized by, include: Identify the data sources in batch processing requirements; Based on the data source, determine the type of batch public mechanism; The required batch processing time for the data volume is determined based on the data volume in the batch processing requirement. The window of the required batch processing time for the data volume is matched with the overall batch processing, where the overall batch is the sum of the data volume in the batch processing requirement and the previously existing batch data. Based on the required batch processing time for the matched data volume, start batch processing and generate batch processing templates corresponding to the batch common mechanism types. Based on the aforementioned data sources, determine the types of batch public mechanisms, including: Based on the data source, determine the data category for batch processing; Based on the data classification, determine the type of batch public mechanism; Based on the data source, determine the data classification for batch processing, including: When the data source is internal to the system, the data is classified as entity data, and includes one or any combination of account holders, customer information, and invoices; When the data source is internal to the system, the data is classified as transaction detail data, which includes one or any combination of contract information generated online and transaction flow data. When the data source is external to the system, the data is classified as entity data, and includes one or any combination of external related accounts and customer information; When the data source is external to the system, the data is classified as transaction detail data, which includes one or any combination of external system inbound details and transaction flow data; Based on the data classification, the types of batch public mechanisms are determined, including: When performing batch processing on online transaction detail data, the batch common mechanism type is determined to be online batch common mechanism; When batch processing transaction detail data in night mode, the batch public mechanism type is determined to be the night batch public mechanism; When batch processing transaction detail data under daytime online mode, the batch public mechanism type is determined to be daytime batch public mechanism; When batch processing entity data with low business timeliness, the batch common mechanism type is determined to be P2 filtering P1 batch common mechanism. When batch processing entity data that has not been modified, the batch common mechanism type is determined to be the data collection common mechanism; Match the window of time required for batch processing of the data volume with the overall batch processing, including: The time period from the completion of job A (which requires batch processing of data) to the start of job B is defined as the time period for the matched batch processing jobs.

2. The method of claim 1, wherein, After generating the batch processing template corresponding to the batch common mechanism type, it also includes: Batch processing templates are stored in a template library, and these templates are used to automatically execute batch processing.

3. A batch processing apparatus, characterized in that, include: The identification module is used to identify the data source in batch processing requirements; The public mechanism classification module is used to determine the batch public mechanism type based on the data source. The job time window determination module is used to determine the job time required for batch processing of the data volume based on the data volume in the batch processing requirement, and to match the window of the job time required for batch processing of the data volume with the overall batch processing, wherein the overall batch is the sum of the data volume in the batch processing requirement and the previously existing batch data. A template module is built to start batch processing based on the job time required for batch processing of the matched data volume, and to generate batch processing templates corresponding to the batch common mechanism types. The public mechanism classification module is specifically used for: Based on the data source, determine the data category for batch processing; Based on the data classification, determine the type of batch public mechanism; The public mechanism classification module is specifically used for: When the data source is internal to the system, the data is classified as entity data, and includes one or any combination of account holders, customer information, and invoices; When the data source is internal to the system, the data is classified as transaction detail data, which includes one or any combination of contract information generated online and transaction flow data. When the data source is external to the system, the data is classified as entity data, and includes one or any combination of external related accounts and customer information; When the data source is external to the system, the data is classified as transaction detail data, which includes one or any combination of external system inbound details and transaction flow data; The public mechanism classification module is specifically used for: When performing batch processing on online transaction detail data, the batch common mechanism type is determined to be online batch common mechanism; When batch processing transaction detail data in night mode, the batch public mechanism type is determined to be the night batch public mechanism; When batch processing transaction detail data under daytime online mode, the batch public mechanism type is determined to be daytime batch public mechanism; When batch processing entity data with low business timeliness, the batch common mechanism type is determined to be P2 filtering P1 batch common mechanism. When batch processing entity data that has not been modified, the batch common mechanism type is determined to be the data collection common mechanism; The job time window determination module is specifically used for: The time period from the completion of job A (which requires batch processing of data) to the start of job B is defined as the time period for the matched batch processing jobs.

4. The apparatus as described in claim 3, characterized in that, The template module is also used for: Batch processing templates are stored in a template library, and these templates are used to automatically execute batch processing.

5. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 2.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 2.

7. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 2.

Images