Batch processing method for complex data model and electronic equipment

Abstract

The invention relates to a batch processing method for a complex data model and electronic equipment, the method is based on Spring Batch framework extension, a domain interface used for bearing all core business logics in one step is defined, a domain implementation class is created based on the domain interface, and the domain implementation class comprises creating a master domain instance and a slave domain instance; according to the relation among the Domain Job, the Domain Flow and the Domain Step and the JobExecution Listener instance, a batch processing task instance in the Spring Batch is created, and dynamic construction of a Spring Batch original task corresponding to a complex data model in a memory is achieved; according to the method, a partition data list is generated by running a partition method of a main domain instance, data is automatically distributed to a plurality of parallel partitions by a Spring Batch framework, each partition is processed by an independent slave domain instance, and processing of corresponding business logic of a batch processing task is realized. Compared with the prior art, the method has the advantages of supporting a flexible configuration operation process of a complex data model and parallel computing of a memory level and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and relates to a batch processing method for a complex data model and an electronic device. BACKGROUND

[0002] In addition to high performance and high reliability, batch processing systems in the financial field and the like have characteristics that are significantly different from those of other industries: high model complexity (involving multi-layer nested logic and mathematical formulas), strict calculation accuracy requirements (floating point precision and risk exposure hedging need to be handled). Taking the "margin calculation" scenario in the financial futures industry as an example: after calculating the margin of a client's portfolio, risk hedging needs to be performed for different portfolio combinations (such as cross-asset, cross-tenor position offsetting), and through multi-layer traversal and complex mathematical formulas, the net risk exposure after hedging is obtained, and then the final margin amount is determined. Such calculations not only rely on multi-dimensional association of data, but also need to complete multiple iterations and verifications in a single batch processing, which puts extremely high requirements on the flexibility and computing power of the system.

[0003] Current mainstream batch processing frameworks can provide basic capabilities, but cannot fully adapt to the special needs of complex data models: Hadoop / Spark: With the core advantage of distributed computing, it is good at processing massive data simple operators (such as map, flatMap, reduce, filter, group by), but its operator function is single, and cannot support the complex business logic required by complex data models (such as multi-layer nested traversal, dynamic risk hedging rules, intermediate state transmission of multi-stage calculation results, etc.).

[0004] Spring Batch (Java mainstream batch processing framework): Supports developers to write complex batch processing logic, but its internal implementation of Step only focuses on "sharded data batch processing" (fixed batch size is used to split input data, and each piece is processed sequentially), and does not support repeated traversal and multi-stage cross calculation on full data by default (such as the need to backtrack the association between client positions and risk factors multiple times in margin calculation), and lacks direct adaptation ability for "complex data models" (such as the multi-dimensional association structure of clients-positions-contracts-risk factors).

[0005] A search revealed Chinese invention patent application publication number CN 118014506 A, which discloses a method, apparatus, computer device, and storage medium for batch processing of financial transactions. The method includes: dividing the business process of a financial transaction into multiple batch tasks; defining multiple batch tasks within the Spring Batch framework; then, determining the execution order and preconditions for each batch task according to the business process; configuring the execution order of each batch task in the framework configuration according to the execution order; and inserting conditional judgment logic before each batch task to determine whether the business data of the financial transaction meets the corresponding preconditions. When the accounting date arrives, the business data of the financial transaction is processed in batches. This existing patent application has the problem of being unsuitable for multi-stage cross-calculations in complex data models.

[0006] How to achieve batch processing of in-memory parallel computing of complex data models has become a technical problem that needs to be solved. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art by providing a batch processing method and electronic device for complex data models.

[0008] The objective of this invention can be achieved through the following technical solutions: According to one aspect of the present invention, a batch processing method for complex data models is provided. The method is based on an extension of the Spring Batch framework, defines a domain interface for carrying all core business logic of a step, and creates a domain implementation class based on the domain interface, wherein the domain implementation class includes creating a main domain instance and a slave domain instance. Based on the domain interface, predefine DomainJob, DomainFlow, DomainStep and their interrelationships for each task ID, and create a JobExecutionListener instance based on the information on DomainJob. Based on the relationship between DomainJob, DomainFlow, and DomainStep, and the JobExecutionListener instance, a batch processing task instance in Spring Batch is created to realize the dynamic construction of Spring Batch native tasks corresponding to complex data models in memory; The partitioning method of the main domain instance generates a list of partitioned data. The Spring Batch framework automatically distributes the data to multiple parallel partitions. Each partition is processed by an independent slave domain instance, realizing the processing of the business logic corresponding to the batch processing task.

[0009] As a preferred technical solution, input annotations and output annotations are marked on instances of the domain implementation class; The attributes of the input annotations in the main domain instance maintain the full amount of data, and ensure that only one copy is maintained in memory; The attributes of the input annotations from the domain instance only maintain the data for the current partition, and maintain one copy for each partition.

[0010] As a preferred technical solution, after dynamically constructing the task, the task execution process begins, which specifically includes: Pre-processing for task execution: Based on input annotations, load input data from all domain implementation classes of the current task into the data cache container object of the framework execution context object in parallel; Step execution: Used to execute each process in the task; And post-task processing: After all tasks are completed, record and print the job execution log.

[0011] As a preferred technical solution, the pre-processing steps for task execution include: Read all domain implementation classes of the current task in the framework context object, and obtain the attributes and metadata of the classes marked with input annotations; Filter out all properties whose loading mode is Job runtime; Based on the data source type attribute of the input annotation, which indicates parallel loading of input data, the parallel-loaded input data is stored in the data cache container object of the framework execution context object.

[0012] As a preferred technical solution, the process of the above steps includes: 2-5-1, Pre-processing for step execution: Based on the input annotation metadata information in the domain implementation class corresponding to the current step, perform global initialization of the main domain instance; 2-5-2, Partitioning Algorithm Execution: Execute the partitioning algorithm in the current step and pass the partitioned data to each partition; 2-5-3, Partition Instance Creation: Create a slave domain instance for the current partition, and inject the partition data of the current partition into the corresponding slave domain instance after splitting it; 2-5-4, Business Logic Calculation: Run user-defined business logic from a domain instance and assign the final calculation result to the attribute of the annotation output.

[0013] As a preferred technical solution, the partitioning algorithm execution process includes: running the partitioning method in the main domain instance to generate a partitioned data list, creating an execution context object for each partition, splitting the partitioned data list and putting it into the partitioning context according to the business logic order, and submitting the partitioning context to the Spring Batch framework; The process of creating the partition instance includes: creating a slave domain instance for the current partition, obtaining the partition data of the current partition from the partition context, performing partition initialization in the slave domain instance, obtaining the attributes marked with input annotations through the master domain instance, and injecting them into the slave domain instance after splitting the partition data of the current partition.

[0014] As a preferred technical solution, the DomainJob is used to store the metadata of the entire batch processing job and uses the task ID as its unique primary key; a DomainJob includes multiple DomainFlows and their execution order, and a DomainFlow manages the execution order of multiple DomainSteps.

[0015] As a preferred technical solution, the dynamic construction process of the task in memory includes: Read the DomainJob, DomainFlow, DomainStep and their interrelationships from the database based on the task ID; For each DomainStep, create a domain subtask implementation class and a main domain instance; Create JobExecutionListener and Partitioner instances based on the main domain instance; Create a DomainStep instance based on the domain subtask implementation class, JobExecutionListener instance, and Partitioner instance; Create a DomainFlow instance based on the relationship between DomainFlow and DomainStep; Based on the relationship between DomainJob, DomainFlow, and DomainStep, and by creating task instances in Spring Batch using JobExecutionListener instances, dynamic task construction is achieved.

[0016] As a preferred technical solution, different DomainFlows under a DomainJob are executed in order of order; if the order is the same, they are executed in parallel. Each DomainStep stores the configuration information for steps in a single, independent batch task, and they are executed in the order specified by the order attribute.

[0017] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described thereon.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1) This invention generates a partitioned data list (such as customer ID, member ID) through the partitioning method of MasterDomain, and the Spring Batch framework automatically distributes the data to multiple parallel partitions. Each partition is processed by an independent SlaveDomain instance, which elegantly realizes the full loading, sharding and database persistence of data, and solves the problem that the native Spring Batch framework does not support full loading. Therefore, the batch processing method of this invention can ensure that the data of a shard required for complex calculations is fully loaded into memory before calculation, thus realizing memory-level parallel computing (without relying on a distributed cluster), which greatly improves the execution efficiency of complex models (such as multi-layer risk hedging calculation); at the same time, it supports partition-level fault tolerance (the failure of a single partition does not affect other partitions), enhancing the robustness of the system.

[0019] 2) This invention defines a Domain interface, parses the predefined DomainJob / DomainFlow / DomainStep relationships based on the Domain interface, and creates native Spring Batch components (such as Job / Step / Flow). It dynamically constructs complete batch processing task instances at runtime (without hard-coding the Job structure), supports flexible configuration of job processes for complex data models (such as adjusting the step order and adding / removing partition strategies according to business needs), and can adapt to new batch processing scenarios without recompiling the code, significantly improving system scalability and maintenance efficiency.

[0020] 3) This invention decouples input data, intermediate results, output data and business logic through Domain interfaces and input / output annotations. Developers only need to focus on writing the core algorithm and do not need to deal with underlying details such as data loading, partition management and persistence. This supports flexible iteration of complex algorithms, accelerates the response speed to business needs, lowers the threshold for financial business development and improves code readability and maintainability. Attached Figure Description

[0021] Figure 1 A flowchart illustrating the system initialization process; Figure 2 This is a flowchart illustrating the task execution process; Figure 3 This is a flowchart illustrating the steps involved in the process. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] This invention, based on the extensibility capabilities (Tasklet interface) of the Spring Batch framework, addresses the core needs of fields such as finance for "complex data models + high-performance parallel computing." Through a three-layer architecture design—dynamic job construction, in-memory parallel partitioning, and domain-oriented data abstraction—it develops a batch processing system that supports complex algorithm writing, flexible full-data traversal, and high reliability and concurrency. The aim is to solve the following problems: 1) How to shield the complexity of underlying data input and output so that developers can focus on the development of business logic (such as margin algorithm).

[0024] 2) How to improve the execution efficiency of complex models through in-memory parallel computing without relying on distributed clusters.

[0025] 3) How to support dynamic and repeated traversal of the full data and the transfer of multi-stage calculation results (such as the reuse of intermediate results in risk hedging).

[0026] This invention is not only applicable to complex data models in the financial field, but also to the fields of healthcare, intelligent transportation, and energy, such as precision medicine in healthcare and grid dispatching in the energy field.

[0027] This embodiment relates to a batch processing method for complex data models. First, the technical terms involved in the technical solution are explained, as shown in Table 1.

[0028] Table 1 To understand the role of each method in the Domain interface, it is necessary to analyze it in conjunction with the execution lifecycle of Step in the Spring Batch framework, the partitioning mechanism, and the roles of components such as Listener and Partitioner: 1. beforeAll Timing of invocation: Triggered by StepExecutionListener during the global pre-execution phase of Step execution.

[0029] Function: To complete global initialization at the Step level. For example: loading resources shared by the entire Step (such as global configuration files, public database connection pools); initializing cross-partition context information; recording the global state before Step execution (such as total number of tasks, startup time).

[0030] 2. partition Timing of invocation: Triggered by the Partitioner implementation class during the partitioning phase of Step.

[0031] Function: Defines the rules for partitioning data / tasks. This includes: splitting large batches of data to be processed into multiple "partitions" (e.g., by data ID range or time interval); returning the number of partitions and a unique identifier for each partition (for subsequent partition-level processing).

[0032] 3. beforeEach Timing of invocation: Triggered in the single partition pre-processing stage of Step, before each partition begins processing.

[0033] Function: To complete the initialization at the current partition level. For example: initializing resources exclusive to the current partition (such as database transactions and temporary files within the partition); parsing parameters of the current partition (such as partition ID and data range) and injecting them into the business logic; and recording the local state of the current partition before execution (such as the amount of data to be processed within the partition and the start time).

[0034] 4. process Timing of invocation: Triggered during the single-partition execution phase of Step, after each partition is initialized.

[0035] Function: It carries the core business logic of the current partition. For example: reading input data of the current partition (such as pulling from the database, file, or message queue); performing business processing on the data (such as data transformation, rule validation, and aggregation calculation); and writing the processed results (such as writing to the database or generating output files). It is the core method for each partition to "do work" and determines how the current partition achieves its business goals.

[0036] 5. afterEach Timing of invocation: Triggered in the post-processing stage of a single partition in Step, after each partition has completed its process.

[0037] Function: To perform cleanup and cleanup at the current partition level. For example: close resources exclusively used by the current partition (such as releasing database connections and deleting temporary files); analyze the execution results of the current partition (such as the number of successful / failed records and the execution time); and record logs or exception information after the execution of the current partition (to facilitate troubleshooting).

[0038] 6. afterAll Timing of invocation: Triggered during the global post-execution phase of Step, after all partitions have been completed, by StepExecutionListener.

[0039] Function: To complete global cleanup and summarization at the Step level. For example: summarizing the execution results of all partitions (such as total number of successes, total number of failures, and overall execution time); releasing resources shared by the entire Step (such as global locks and public connection pools); and recording the final status of Step execution (such as marking it as "completed" or "failed" and writing it to the audit log).

[0040] In this invention, the method is based on the Spring Batch framework extension. The overall process is divided into system initialization and job execution. The job execution includes step execution. By dynamically constructing jobs and performing in-memory parallel computing, the method achieves efficient processing of complex data models.

[0041] The system initialization process is as follows: Figure 1 As shown, this includes system self-checks and domain metadata loading: 1-1, System self-check: Implements a self-check function after system startup. The specific steps are as follows: 1-1-1. Check the database connection (verify data source availability); 1-1-2. Detect the communication status of dependent third-party components (such as message queues and caching services); 1-1-3. Load and verify configuration parameters (such as parallelism threshold, timeout, and log level) to ensure configuration integrity.

[0042] 1-2, Loading Domain metadata, which involves loading the implementation classes that implement the Domain interface (i.e., Domain implementation classes) and the attributes annotated with @Input and @Output. Specifically, this process is as follows: 1-2-1. Create a globally unique FrameworkContext object (memory singleton pattern); 1-2-2. Scan all implementation classes of the Domain interface (excluding abstract classes) using reflection, and read all properties marked with @Input and @Output in the implementation classes and their parent classes; 1-2-3. Parse annotation metadata (such as data source type dataSrc, loading mode loadMode, lifecycle disposeAt), and store the attribute and annotation information in the framework context object FrameworkContext to provide a basis for subsequent data binding and lifecycle management.

[0043] The task execution process is as follows: Figure 2 The process includes, in sequence, context initialization, dynamic task construction and invocation, pre-processing for task execution, step execution, post-processing for task execution, and context persistence.

[0044] 2-1, Context Initialization, is used to initialize the context objects used during task execution. The specific process is as follows: 2-1-1, Instantiate the FrameworkExecutionContext object; 2-1-2, Instantiate the UserExecutionContext object; 2-1-3. Wrap the FrameworkContext object created in the domain metadata loading step 1-2 and the two context objects (instantiated FrameworkExecutionContext object and UserExecutionContext object) in steps 2-1-1 and 2-1-2 into a single context object for use in subsequent Job runtime.

[0045] 2-2, Dynamic Task Construction and Invocation, used to implement the dynamic construction of Spring Batch's native Job objects in memory, specifically handled as follows: 2-2-1, Read the domain step information DomainJob, domain step flow information DomainFlow, domain task information DomainStep, and their interrelationships from the database based on the task ID (JobId); 2-2-2, Create a domain subtask implementation class (DomainTasklet) based on the domain task information of each domainStep. 2-2-3, create a MasterDomain instance for each DomainStep task information, and create a StepExecutionListener instance and a Partitioner instance based on this MasterDomain instance; 2-2-4. Create a DomainStep instance in Spring Batch based on the DomainTasklet, StepExecutionListener, and Partitioner instances; 2-2-5, Create a DomainFlow instance in Spring Batch based on the relationship between DomainFlow and DomainStep, such as... Figure 2 The system consists of Flow1, Flow2, and Flow3. Flow1 includes Step 1 and Step 2, which are executed sequentially, while Flow2 only contains Step 3. Flow1 and Flow2 can be started and executed simultaneously. Once both have completed their respective steps, the execution of Flow3 is triggered, and Step 4 is executed sequentially in Flow3.

[0046] 2-2-6, Create a JobExecutionListener instance based on the information on DomainJob; 2-2-7. Based on the relationship between DomainJob, DomainFlow, and DomainStep, and by creating a task (Job) instance in Spring Batch using the JobExecutionListener instance, dynamic construction of Jobs is achieved. 2-2-8, Call the relevant Spring Batch interfaces to run the Job created in the previous step.

[0047] This invention enables dynamic construction of Spring Batch Jobs through configuration. Therefore, if a batch processing task encounters an error, only the configuration needs to be modified to change the Job's processing flow, thereby achieving the function of resuming execution from breakpoints.

[0048] 2-3, Pre-processing for task execution, used to implement the call to the beforeJob method in the JobExecutionListener implementation class, specifically: 2-3-1, Read all the domain implementation classes of the current Job in the FrameworkContext, and obtain the attributes and metadata information of the classes annotated with @Input; 2-3-2, filter out all attributes where loadMode is set to Job runtime; 2-3-3, based on the data source type dataSrc attribute of the @Input annotation, which indicates parallel loading of input data, the parallel-loaded input data is stored in the data cache container object of the FrameworkExecutionContext object.

[0049] 2-4. After the pre-processing of the task, the step (i.e. the execution flow) will be executed. The execution flow of each step will be described in detail later.

[0050] 2-6, Post-task processing, used to call the afterJob method in the JobExecutionListener implementation class, specifically: 2-6-1, Wait for all tasks in the asynchronous thread pool within the FrameworkExecutionContext object to complete (ensuring no omissions). 2-6-2, Record and print the job execution log (including time consumption, success / failure partition statistics, etc.); 2-7, Context Persistence, is used to persist the log objects of Job execution in the FrameworkExecutionContext object. Specifically, it synchronously updates the execution status of the Job in the database, while asynchronously persisting the execution information of each partition and asynchronous task execution information during the running process.

[0051] Steps 2-5, see the flowchart. Figure 3 ,include: 2-5-1, Step execution preprocessing, specifically the call to the beforeStep method in the StepExecutionListener implementation class: Read the metadata information of the @Input annotation in the Domain implementation class corresponding to the current Step from the FrameworkContext object; If LoadMode is a property of the Job runtime, the data loaded during the Job runtime is retrieved directly from the data cache container object in the FrameworkExecutionContext; If LoadMode is a property for Step runtime, then data is read from the data source in real time based on dataSrc; Execute the beforeAll method in the MasterDomain instance; 2-5-2, Partition Algorithm Execution: This step executes the partitioning algorithm of the current step and passes the partitioned data to each partition. Specifically: Running the partition method in the MasterDomain instance generates a partition data list partitionData. In the financial technology field, partitionData can be a member number or a customer number. Create an execution context object for each partition, then split the partition data list and put it into the partition context according to the business logic order; Submit the partition context to the Spring Batch framework.

[0052] 2-5-3, Partition Instance Creation: This step creates the SlaveDomain instance and loads its attribute data. The specific steps are as follows: Create a SlaveDomain instance for the current partition; Retrieve the partition data (partitionData) of the current partition from the partition context in the previous step; Executes the user-defined beforeEach method in the SlaveDomain instance, taking UserExecutionContext and partitionData as parameters. The beforeEach method is an overridable callback method that executes before the @Input attribute is split. The specific implementation depends on the business logic. If not overridden, it is an empty method by default.

[0053] Retrieve the attributes annotated with @Input from the MasterDomain instance, split them according to the partitionData of the current partition, and inject the split data into the SlaveDomain.

[0054] 2-5-4, Business Logic Calculation: This step implements the execution of user-defined business logic, specifically as follows: The `process` method of `SlaveDomain` is called (where developers write their core business logic). At this point, `SlaveDomain` contains all the input data required for the current partition (such as all positions held by a client and historical risk factors). Developers can repeatedly iterate through instance attributes (such as iterating through positions → contracts → risk factors) and perform complex mathematical calculations (such as portfolio margin calculations). The final calculation result is then assigned to the attribute annotated with `@Output` (such as the client's margin amount).

[0055] By decoupling input data, intermediate results, output data, and business logic (the SlaveDomain.process() method), developers only need to focus on writing core algorithms (such as margin calculation rules) without having to deal with underlying details such as data loading, partition management, and persistence. For example, when modifying hedging rules, only the process() method needs to be adjusted, thus accelerating the response speed to business requirements.

[0056] 2-5-5, Output Persistence: This step implements the persistence of attributes with the @Output annotation in SlaveDomain instances.

[0057] The specific steps are as follows: The output data is written to a data cache container object in the FrameworkExecutionContext so that it can be used as input for the next step; If the isAsync property of the @Output annotation is false, the output data is persisted directly; otherwise, the output data is wrapped into an asynchronous task and handed over to the asynchronous thread pool object in the FrameworkExecutionContext for processing. Execute the user-implemented afterEach method in the SlaveDomain instance.

[0058] 2-5-6, Step Execution Post-Processing: This step mainly handles the post-processing work after the current Step has been executed, specifically: Execute the user-implemented afterAll method in the MasterDomain instance; Clean up invalid data in the data cache container object in FrameworkExecutionContext: Read the metadata information of the @Input and @Output annotations in FrameworkContext. If the value of the disposeAt attribute on the annotation is consistent with the class name of the current MasterDomain, then clean up the data corresponding to the current attribute to minimize memory overhead. This embodiment also relates to a batch processing system for complex data models. The system is based on the Spring Batch framework and includes a system initialization module and a task execution module. It achieves efficient processing of complex data models by dynamically constructing tasks (jobs) and performing in-memory parallel computing.

[0059] The system initialization module is used for system self-checks and domain metadata loading. Domain metadata loading involves loading the implementation classes of the domain interfaces (i.e., Domain implementation classes) and their attributes annotated with `@Input` and `@Output`. This includes: creating a globally unique framework context object; scanning all implementation classes of the domain interfaces via reflection, reading all attributes marked with `@Input` and `@Output` annotations in the implementation classes and their parent classes; parsing annotation metadata (such as data source type `dataSrc`, loading mode `loadMode`, and lifecycle `disposeAt`), and storing the attribute and annotation information in the framework context object to provide a basis for subsequent data binding and lifecycle management.

[0060] like Figure 2 The task execution module includes a context initialization submodule, a task dynamic construction and invocation submodule, a task pre-processing submodule, a step execution submodule, a task post-processing submodule, and a context persistence submodule.

[0061] The context initialization submodule is used to initialize the context objects used during task execution.

[0062] The Task Dynamic Construction and Invocation submodule is used to implement the dynamic construction of Spring Batch's native Job objects in memory.

[0063] The task pre-processing submodule is used to call the beforeJob method in the JobExecutionListener implementation class and store the parallel-loaded input data into the data cache container object of the framework execution context object.

[0064] The step execution submodule is used to execute the various processes of the step.

[0065] The task execution post-processing submodule is used to call the afterJob method in the JobExecutionListener implementation class. After all tasks have been executed, the job execution log is recorded and printed.

[0066] The context persistence submodule is used to persist the log objects of Job execution in the framework execution context object. If LoadMode is a property of the Job runtime, it directly retrieves the data loaded during Job runtime from the data cache container object in FrameworkExecutionContext. If LoadMode is a property for Step runtime, then data is read from the data source in real time based on dataSrc; Execute the beforeAll method in the MasterDomain instance.

[0067] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0068] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0069] The processing unit performs the various methods and processes described above. For example, in some embodiments, the methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute the methods by any other suitable means (e.g., by means of firmware).

[0070] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0071] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0072] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0073] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A batch processing method for complex data models, characterized in that, This method is based on the Spring Batch framework extension, defines a domain interface to carry all the core business logic of a step, and creates a domain implementation class based on the domain interface, wherein the domain implementation class includes creating a main domain instance and a slave domain instance; Based on the domain interface, predefine DomainJob, DomainFlow, DomainStep and their interrelationships for each task ID, and create a JobExecutionListener instance based on the information on DomainJob. Based on the relationship between DomainJob, DomainFlow, and DomainStep, and the JobExecutionListener instance, a batch processing task instance in Spring Batch is created to realize the dynamic construction of Spring Batch native tasks corresponding to complex data models in memory; The partitioning method of the main domain instance generates a list of partitioned data. The Spring Batch framework automatically distributes the data to multiple parallel partitions. Each partition is processed by an independent slave domain instance, realizing the processing of the business logic corresponding to the batch processing task.

2. The batch processing method for complex data models according to claim 1, characterized in that, Add input and output annotations to instances of the domain implementation class; The attributes of the input annotations in the main domain instance maintain the full amount of data, and ensure that only one copy is maintained in memory; The attributes of the input annotations from the domain instance only maintain the data for the current partition, and maintain one copy for each partition.

3. The batch processing method for complex data models according to claim 2, characterized in that, After dynamically constructing the task, the task execution process begins, which includes: Pre-processing for task execution: Based on input annotations, load input data from all domain implementation classes of the current task into the data cache container object of the framework execution context object in parallel; Step execution: Used to execute each process in the task; And post-task processing: After all tasks are completed, record and print the job execution log.

4. The batch processing method for complex data models according to claim 3, characterized in that, The pre-processing steps for task execution include: Read all domain implementation classes of the current task in the framework context object, and obtain the attributes and metadata of the classes marked with input annotations; Filter out all properties whose loading mode is Job runtime; Based on the data source type attribute of the input annotation, which indicates parallel loading of input data, the parallel-loaded input data is stored in the data cache container object of the framework execution context object.

5. A batch processing method for complex data models according to claim 3, characterized in that, The process of performing the steps includes: 2-5-1, Pre-processing for step execution: Based on the input annotation metadata information in the domain implementation class corresponding to the current step, perform global initialization of the main domain instance; 2-5-2, Partitioning Algorithm Execution: Execute the partitioning algorithm in the current step and pass the partitioned data to each partition; 2-5-3, Partition Instance Creation: Create a slave domain instance for the current partition, and inject the partition data of the current partition into the corresponding slave domain instance after splitting it; 2-5-4, Business Logic Calculation: Run user-defined business logic from a domain instance and assign the final calculation result to the attribute of the annotation output.

6. A batch processing method for complex data models according to claim 5, characterized in that, The partitioning algorithm execution process includes: running the partitioning method in the main domain instance to generate a partitioned data list, creating an execution context object for each partition, splitting the partitioned data list and putting it into the partitioned context according to the business logic order, and submitting the partitioned context to the Spring Batch framework; The process of creating the partition instance includes: creating a slave domain instance for the current partition, obtaining the partition data of the current partition from the partition context, performing partition initialization in the slave domain instance, obtaining the attributes marked with input annotations through the master domain instance, and injecting them into the slave domain instance after splitting the partition data of the current partition.

7. The batch processing method for complex data models according to claim 1, characterized in that, The DomainJob is used to store metadata for the entire batch job and uses the task ID as its unique primary key. A DomainJob includes multiple DomainFlows and their execution order, and a DomainFlow manages the execution order of multiple DomainSteps.

8. A batch processing method for complex data models according to claim 1, characterized in that, The dynamic construction process of a task in memory includes: Read the DomainJob, DomainFlow, DomainStep and their interrelationships from the database based on the task ID; For each DomainStep, create a domain subtask implementation class and a main domain instance; Create JobExecutionListener and Partitioner instances based on the main domain instance; Create a DomainStep instance based on the domain subtask implementation class, JobExecutionListener instance, and Partitioner instance; Create a DomainFlow instance based on the relationship between DomainFlow and DomainStep; Based on the relationship between DomainJob, DomainFlow, and DomainStep, and by creating task instances in Spring Batch using JobExecutionListener instances, dynamic task construction is achieved.

9. A batch processing method for complex data models according to claim 1, characterized in that, Different DomainFlows under a DomainJob are executed in order of order; if the order is the same, they are executed in parallel. Each DomainStep stores the configuration information for steps in a single, independent batch task, and they are executed in the order specified by the order attribute.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 9.

Images