Data transfer method

Abstract

The invention discloses a data transfer method, relates to the technical field of data processing, and aims to solve the problem that thread security is difficult to guarantee in a multi-thread concurrent processing mode, and the data transfer method comprises the following steps: S1, initializing a resource configuration related class, instantiating a conversion object, instantiating a reading object, initializing an operation parameter, and initializing the resource related class; s2, judging whether the total number of the to-be-processed records exceeds a limit or not; judging whether all read records have been processed or not; s3, obtaining a writing business object according to the reading record; s4, processing the write service object, and obtaining a task write service object set; s5, processing the task write business object set, and executing the task; according to the data transfer method, services and technologies can be separated, development difficulty increase caused by interleaving of the services and the technologies can be avoided, universal logic can be processed, safety and performance problems can be solved, development difficulty can be reduced, development efficiency can be improved, and thread safety and resource leakage problems can be avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data carry-over method. Background Art

[0002] In many business scenarios, when the performance of serial processing of business records one by one does not meet the business requirements, batch processing is usually used to improve efficiency. If it still cannot meet the business requirements, multi-threaded concurrent processing is needed to further improve performance. Although the use of multi-threading technology can improve efficiency, the problem of ensuring thread safety makes development more difficult, including the reasonable use of thread pools and control of the size of memory used. These non-business technical issues are difficult for general developers to master, and various thread safety and resource leakage issues are inevitable.

[0003] Data collection, conversion, and transmission is a complex process. In order to meet the different demands of business and technology, developers need to focus on business and technical issues at the same time. The inability to separate business and technical concerns leads to the interweaving of business logic and technical code, greatly increasing the complexity of the code, increasing the difficulty of maintenance and expansion, resulting in poor code quality and security risks.

[0004] For example, there is a Chinese patent with publication number CN107315752B, which involves a data carry-over method and system, and discloses a data carry-over method, including: configuring data source information and carry-over conditions of the data to be carried forward, wherein the carry-over conditions include the starting primary key ID and the ending primary key ID of the data table to be carried forward; sharding the data table to obtain multiple data shards; and using multi-threaded asynchronous forwarding in sequence to achieve simple configuration and convenient carry-over, greatly simplifying the operational procedures of carry-over and backup, and improving the efficiency of carry-over and backup. However, the Chinese patent with publication number CN107315752B does not separate business and technology, is difficult to develop, and is prone to thread safety and resource leakage issues. Summary of the invention

[0005] In order to solve the problem that it is difficult to ensure thread safety in multi-threaded concurrent processing, the present invention proposes a data carry-over method, which can not only separate business and technology, avoid the increase in development difficulty caused by the interweaving of business and technology, but also process common logic and solve security and performance problems, reduce development difficulty, improve development efficiency, and avoid the occurrence of thread safety and resource leakage problems.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a data carry-over method, comprising: Step S1, initialize resource configuration related classes, instantiate conversion objects, instantiate reading objects, initialize job parameters, and initialize resource related classes; Step S2, determine whether the total number of records to be processed exceeds the limit; determine whether all read records have been processed; Step S3, obtaining a write business object according to the read record; Step S4, process the write business object and obtain the task write business object set; Step S5, process the task, write the business object set, and execute the task.

[0007] In this technical solution, general data processing and conversion are implemented through abstract classes, and the business logic of unique data in specific business scenarios is processed by the implementation class, which can separate business and technology, reduce development difficulty, and improve development efficiency; and solve the problem of ensuring thread safety in the process of multi-threaded concurrent processing of business records, and prevent the occurrence of resource leakage problems.

[0008] Preferably, instantiating the conversion object in step S1 includes: Step S11, defining a read business object and a write business object; Step S12, realizing the conversion from the read business object to the write business object.

[0009] Preferably, instantiating the read object in step S1 includes: Step S13, instantiating the query object; Step S14, customizing the query expansion object according to business needs; Step S15, attaching the query expansion object to the query object, and attaching the query object to the read object.

[0010] Preferably, step S13 includes: Step S131, set the query conditions for querying records; for example, the record creation time is greater than or equal to the start time and less than the end time; Step S132, set the maximum number of records for full query; for example, 1 million records; Step S133, setting the number of paging query records; for example, 24,000 records.

[0011] Preferably, step S14 includes: Step S141, when the query condition of the query object cannot meet the demand, a custom query extension object is defined, and the properties of the query extension object are defined according to business needs; for example, the status of the record is normal, the level of the record is advanced, etc.; Step S142, reading the attributes of the query object and the query extension object as query conditions.

[0012] Preferably, determining whether the total number of records to be processed exceeds the limit in step S2 includes: obtaining the total number of records to be processed of the job, when the total number of records to be processed of the job is greater than the maximum number of records for the full query, the total number of records to be processed exceeds the limit, abnormally exiting and resetting the total number of records to be processed; otherwise, continuing.

[0013] If the total number of records to be processed exceeds the limit, it indicates that you need to narrow the query scope and reduce the number of records to be queried. This is because querying too many records requires a lot of system resources, and system resources are limited. It is very likely that the job will fail due to insufficient system resources. You need to adjust the total number of records to be processed to a reasonable range based on actual conditions.

[0014] Preferably, determining whether all read records have been processed in step S2 includes: obtaining the number of records that have been processed, determining whether the number of records that have been processed is equal to the total number of records to be processed, if the number of records that have been processed is equal to the total number of records to be processed, the job is completed; if the number of records that have been processed is not equal to the total number of records to be processed, it means that there are still records to be processed, and the records are read in pages.

[0015] Preferably, step S3 comprises: Step S31, converting the read record into a read business object; Step S32, converting the read business object into the write business object, converting the fields of the record into the corresponding attributes of the read business object, and setting the attributes of the read business object as the mapping attributes of the write business object;.

[0016] Preferably, step S4 comprises: Step S41, initializing the job and setting the maximum number of write business objects that the job can process; Step S42, determining whether there are enough write business objects, if yes, then process all the write business objects, otherwise process the remaining write business objects to be processed; Step S43, instantiate the write business object to obtain a write business object set; Step S44, splitting the write business object set according to the maximum number of write business objects that can be processed by the task to obtain several task write business object sets.

[0017] In this technical solution, the number of objects in each write business object set is the maximum number of write business objects, and the number of objects in the minimum write business object set may be smaller than the maximum number of write business objects.

[0018] Preferably, step S5 comprises: Step S51, process each write business object set in turn, Step S52, determine whether there are still tasks to write business object sets to be processed, if yes, initialize the tasks and execute the tasks, otherwise determine whether all read records have been processed.

[0019] The beneficial effects of the present invention are: (1) Separating business from technology, implementing general data processing and conversion through abstract classes, and implementing business logic for processing unique data in specific business scenarios through implementation classes, can reduce development difficulty and improve development efficiency; (2) Ensure thread safety and prevent various thread safety and resource leakage issues. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The present invention is a flow chart of a data carry-over method.

[0021] Figure 2 It is a class diagram of a data carry-over method of the present invention.

[0022] Figure markings: C1, output implementation class; C2, output; C3, task implementation class; C4, task; C5, work; C6, work implementation class; C7, query; C8, input; C9, input implementation class; C10, job; C11, conversion; C12, job implementation class; C13, conversion implementation class; C14, extended query; C15, read business object; C16, write business object. DETAILED DESCRIPTION

[0023] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation method described herein is only an optimal embodiment of the present invention, which is only used to explain the present invention and does not limit the scope of protection of the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0024] Example 1 This embodiment provides a data carry-over method. Figure 1 , mainly includes the following steps.

[0025] Step S1, initialize resource configuration related classes, instantiate conversion objects, instantiate reading objects, initialize job parameters, and initialize resource related classes.

[0026] In this embodiment, resource configuration related classes include reading data source configuration, writing data source configuration, and thread pool configuration, and resource related classes include reading data source, writing data source, and thread pool.

[0027] It can be understood that in a data transfer method of this embodiment, the read data source configuration, the write data source configuration, and the thread pool configuration are first initialized, and then the conversion object is instantiated, the read object is instantiated, the job parameters are initialized, and then the read data source, the write data source, and the thread pool are initialized.

[0028] In this embodiment, the data source includes a source data source and a target data source. After the data source configuration is written, the source data source and the target data source are created according to the data source configuration.

[0029] The specific process of initializing the thread pool in S1 is mainly to customize the thread pool, including implementing the AutoCloseable interface, customizing the thread factory, and customizing the thread pool tool class.

[0030] Specifically, the AutoCloseable interface is implemented through a custom thread pool, and resources are released in the implementation of the interface method.

[0031] Customize the thread factory, set business-meaningful names for threads, and implement custom and meaningful thread name prefixes to distinguish and identify threads created by different thread pools, facilitating subsequent analysis and problem location.

[0032] Customize the thread pool tool class, set a reasonable resource shortage response strategy, and ensure that the number of concurrent execution threads meets expectations.

[0033] For example, if the number of concurrent threads is 4, the main thread will apply for 4 child threads from the thread pool and arrange for the 4 child threads to execute tasks. When arranging the fifth task, it is found that all threads in the thread pool are executing tasks. At this time, the main thread claims a task to execute. This ensures that the main thread and all child threads are executing tasks, does not waste thread resources, maximizes resource utilization, and provides the best performance.

[0034] The specific process of instantiating the conversion object in S1 includes the following sub-steps.

[0035] Step S11, defining a read service object and a write service object.

[0036] Step S12, instantiate the conversion from the read business object to the write business object.

[0037] It should be noted that the read business object and the write business object are the same object, including the following attributes.

[0038] The attribute id is of numeric type and can store the content of the app_id field of the record.

[0039] The attribute msg, of type text, can store the content of the message field of the record.

[0040] The attribute updateTime, of type time, can store the content of the update_time field of the record.

[0041] If the read business object and the write business object are different objects, you need to convert the mapped attributes and implement the corresponding business logic according to business needs.

[0042] In the technical solution of step S1 in this embodiment, the fields of the conversion record are the attributes of the corresponding read business object, and the attributes of the read business object are set as the mapping attributes of the write business object.

[0043] The specific process of instantiating the read object in S1 includes the following sub-steps.

[0044] Step S13, instantiate the query object; the specific process of S13 includes the following sub-steps.

[0045] Step S131, setting query conditions for querying records.

[0046] In this embodiment, the query condition is set to that the record creation time is greater than or equal to the start time and less than the end time.

[0047] Step S132, setting the maximum number of records for full query.

[0048] In this embodiment, the maximum number of records for a full query is set to 1 million records.

[0049] Step S133, setting the number of paging query records.

[0050] In this embodiment, the number of paging query records is set to 24,000 records.

[0051] Step S14: Customize the query extension object according to business needs.

[0052] Specifically, when the query condition of the query object cannot meet the demand, a customized query extension object is created, and the properties of the query extension object are defined according to business needs; and the properties of the query object and the query extension object are read as the query condition.

[0053] For example, in some implementations, the query condition may be set as "the status of the record is normal, and the level of the record is advanced".

[0054] Step S15, attaching the query expansion object to the query object, and attaching the query object to the read object.

[0055] Step S2, determine whether the total number of records to be processed exceeds the limit; determine whether all read records have been processed; release resources.

[0056] The specific process of determining whether the total number of records to be processed exceeds the limit in step S2 includes: obtaining the total number of records to be processed of the job. When the total number of records to be processed of the job is greater than the maximum number of records for the full query, the total number of records to be processed exceeds the limit, abnormally exiting and resetting the total number of records to be processed; otherwise, continuing.

[0057] It should be noted that if the total number of records to be processed exceeds the limit, it indicates that the query scope needs to be narrowed and the number of full query records needs to be reduced; because querying too many records requires a large amount of system resources, and system resources are limited, it is very likely that the job will fail due to insufficient system resources; the total number of records to be processed needs to be adjusted to a reasonable range based on actual conditions.

[0058] The specific process of determining whether all read records have been processed in step S2 includes the following contents.

[0059] First, get the number of records that have been processed, and then determine whether the number of records that have been processed is equal to the total number of records to be processed.

[0060] If the number of records processed is equal to the total number of records to be processed, the job is complete.

[0061] If the number of processed records is not equal to the total number of records to be processed, it means there are still records to be processed, and the records are read in pages.

[0062] It should be noted that reading records in pages means reading one page at a time. For example, if there are 1000 records on each page, after reading the 1000 records on the first page, the 1000 records on the second page are read, and so on, until the last page is read.

[0063] In the technical solution for step S2 of this embodiment, releasing resources includes closing the data source and closing the thread pool. After the job is processed, the data source and the thread pool are safely closed to release resources, avoid memory leaks, and ensure that the program continues to run healthily and stably.

[0064] In this embodiment, the try-with-resources statement is used to manage resources, making resource release more concise.

[0065] Specifically, when the try block is executed, whether it ends normally or an exception occurs, these resources will be automatically closed without explicitly calling the close() method, thereby ensuring the safe and reliable release of the requested resources.

[0066] According to the configuration of concurrency parameters, locking is used to ensure that the main thread waits for all the child threads submitted by it to complete before continuing to execute.

[0067] By locking, when the main thread is idle, it will wait for all the child threads that it has submitted to run tasks to complete execution before continuing to execute. This ensures that before the main thread processes a new batch of tasks, all the child threads that the main thread has submitted to run tasks have completed execution.

[0068] Step S3, obtaining a write business object according to the read record; this step mainly includes the following multiple sub-steps.

[0069] Step S31, converting the read record into a read business object.

[0070] Step S32, converting the read business object into a write business object.

[0071] In the technical solution for step S3 of this embodiment, the conversion object class is called to realize the conversion from the read business object to the write business object, the fields of the conversion record are the attributes of the corresponding read business object, and the attributes of the read business object are set as the mapping attributes of the write business object.

[0072] Step S4, process the write business object and obtain the task write business object set; the specific process of step S4 includes the following sub-steps.

[0073] Step S41, initialize the job and set the maximum number of write business objects that the job can process.

[0074] Step S42, determine whether there are enough write business objects, if yes, process all the write business objects, otherwise process the remaining write business objects to be processed.

[0075] It should be noted that all write business objects represent the maximum number of write business objects.

[0076] Step S43, instantiate the write business object to obtain a write business object set.

[0077] It can be understood that the number of objects in each write business object set is the maximum number of write business objects, and the number of objects in the smallest write business object set may be smaller than the maximum number of write business objects.

[0078] Step S44, splitting the write business object set according to the maximum number of write business objects that can be processed by the task to obtain several task write business object sets.

[0079] Step S5, process the task, write the business object set, and execute the task.

[0080] Specifically, each write business object set is processed in turn to determine whether there is a task write business object set to be processed. If yes, the task is initialized and executed. Otherwise, it is determined again whether all read records have been processed.

[0081] The following is a detailed description of the custom class, the static structure of the class, and the relationship between the classes in a data carry-over method of this embodiment. Figure 2 .

[0082] The custom QueryExt (extended query) class is used to customize query conditions, instantiate QueryExt objects and initialize extended query conditions.

[0083] Instantiate the Query class, introduce the QueryExt class object, and define the query conditions and query configuration.

[0084] Customize ReaderBO (read business object) class to receive read data.

[0085] Custom WriterBO (write business object) class, used to convert to write data.

[0086] Customize the DBTransform class (Transform implementation class), inherit the Transform (transformation) abstract class, introduce custom ReaderBO class objects and custom WriterBO class objects, and complete the conversion from ReaderBO class objects to WriterBO class objects.

[0087] Customize DBReader class (Reader implementation class), inherit Reader (input) abstract class, implement abstract methods, define query and count query SQL statements. Introduce custom ReaderBO class object Query class object.

[0088] Customize the DBWriter class (Writer implementation class), inherit the Writer (output) abstract class, implement abstract methods, define insert SQL statements, and introduce custom WriterBO class objects.

[0089] Customize the DBTask class (Task implementation class), inherit the Task abstract class, implement abstract methods, reference the Writer abstract class, and introduce a custom DBWriter class object.

[0090] Customize the DBJob class (Job implementation class), inherit the Job (work) abstract class, implement abstract methods, reference the Task abstract class, and introduce custom DBTask class objects. Customize the DBWork class (Work implementation class), inherit the Work (job) abstract class, implement abstract methods, reference the Reader abstract class, the Job abstract class, the Transform abstract class, reference custom ReaderBO, custom WriterBO class, custom QueryExt class, and introduce custom DBJob class objects.

[0091] In this embodiment, general data processing and conversion are implemented through abstract classes, and the business logic of the class processing the unique data of specific business scenarios is implemented. The general and specific business logic codes are separated, allowing business developers to focus only on the logic of specific businesses, thereby reducing development difficulty and improving development efficiency.

[0092] The data carry-over method in this embodiment can process common logic and solve security and performance issues, thereby ensuring the robustness and high performance of the software.

[0093] Example 2 This embodiment provides a data carry-over method, which mainly includes the following steps.

[0094] Step S1, initialize resource configuration related classes, instantiate conversion objects, instantiate reading objects, initialize job parameters, and initialize resource related classes.

[0095] In this embodiment, resource configuration related classes include reading data source configuration, writing data source configuration, and thread pool configuration, and resource related classes include reading data source, writing data source, and thread pool.

[0096] It can be understood that in a data transfer method of this embodiment, the read data source configuration, the write data source configuration, and the thread pool configuration are first initialized, and then the conversion object is instantiated, the read object is instantiated, the job parameters are initialized, and then the read data source, the write data source, and the thread pool are initialized.

[0097] In this embodiment, the data source includes a source data source and a target data source. After the data source configuration is written, the source data source and the target data source are created according to the data source configuration.

[0098] The specific process of initializing the thread pool in S1 is mainly to customize the thread pool, including implementing the AutoCloseable interface, customizing the thread factory, and customizing the thread pool tool class.

[0099] Specifically, the AutoCloseable interface is implemented through a custom thread pool, and resources are released in the implementation of the interface method.

[0100] Customize the thread factory, set business-meaningful names for threads, and implement custom and meaningful thread name prefixes to distinguish and identify threads created by different thread pools, facilitating subsequent analysis and problem location.

[0101] Customize the thread pool tool class, set a reasonable resource shortage response strategy, and ensure that the number of concurrent execution threads meets expectations.

[0102] The specific process of instantiating the conversion object in S1 includes the following sub-steps.

[0103] Step S11, defining a read service object and a write service object.

[0104] Step S12, instantiate the conversion from the read business object to the write business object.

[0105] It should be noted that the read business object and the write business object are the same object, including the following attributes.

[0106] The attribute id is of numeric type and can store the content of the app_id field of the record.

[0107] The attribute msg, of type text, can store the content of the message field of the record.

[0108] The attribute updateTime, of type time, can store the content of the update_time field of the record.

[0109] If the read business object and the write business object are different objects, you need to convert the mapped attributes and implement the corresponding business logic according to business needs.

[0110] In the technical solution of step S1 in this embodiment, the fields of the conversion record are the attributes of the corresponding read business object, and the attributes of the read business object are set as the mapping attributes of the write business object.

[0111] The specific process of instantiating the read object in S1 includes the following sub-steps.

[0112] Step S13, instantiate the query object; the specific process of S13 includes the following sub-steps.

[0113] Step S131, setting query conditions for querying records.

[0114] For example, the query condition can be set to that the record creation time is greater than or equal to the start time and less than the end time.

[0115] Step S132, setting the maximum number of records for full query.

[0116] In this embodiment, the maximum number of records for a full query is set to 2 million records.

[0117] Step S133, setting the number of paging query records.

[0118] In this embodiment, the number of paging query records is set to 40,000 records.

[0119] Step S14: Customize the query extension object according to business needs.

[0120] Specifically, when the query condition of the query object cannot meet the demand, a customized query extension object is created, and the properties of the query extension object are defined according to business needs; and the properties of the query object and the query extension object are read as the query condition.

[0121] Step S15, attaching the query expansion object to the query object, and attaching the query object to the read object.

[0122] Step S2, determine whether the total number of records to be processed exceeds the limit; determine whether all read records have been processed; release resources.

[0123] The specific process of determining whether the total number of records to be processed exceeds the limit in step S2 includes: obtaining the total number of records to be processed of the job. When the total number of records to be processed of the job is greater than the maximum number of records for the full query, the total number of records to be processed exceeds the limit, abnormally exiting and resetting the total number of records to be processed; otherwise, continuing.

[0124] It should be noted that if the total number of records to be processed exceeds the limit, it indicates that the query scope needs to be narrowed and the number of full query records needs to be reduced; because querying too many records requires a large amount of system resources, and system resources are limited, it is very likely that the job will fail due to insufficient system resources; the total number of records to be processed needs to be adjusted to a reasonable range based on actual conditions.

[0125] The specific process of determining whether all read records have been processed in step S2 includes the following contents.

[0126] First, get the number of records that have been processed, and then determine whether the number of records that have been processed is equal to the total number of records to be processed.

[0127] If the number of records processed is equal to the total number of records to be processed, the job is complete.

[0128] If the number of processed records is not equal to the total number of records to be processed, it means there are still records to be processed, and the records are read in pages.

[0129] It should be noted that reading records in pages means reading one page at a time. For example, if there are 1000 records on each page, after reading the 1000 records on the first page, the 1000 records on the second page are read, and so on, until the last page is read.

[0130] In the technical solution for step S2 of this embodiment, releasing resources includes closing the data source and closing the thread pool. After the job is processed, the data source and the thread pool are safely closed to release resources, avoid memory leaks, and ensure that the program continues to run healthily and stably.

[0131] In this embodiment, the try-with-resources statement is used to manage resources, making resource release more concise.

[0132] Specifically, when the try block is executed, whether it ends normally or an exception occurs, these resources will be automatically closed without explicitly calling the close() method, thereby ensuring the safe and reliable release of the requested resources.

[0133] According to the configuration of concurrency parameters, locking is used to ensure that the main thread waits for all the child threads submitted by it to complete before continuing to execute.

[0134] By locking, when the main thread is idle, it will wait for all the child threads that it has submitted to run tasks to complete execution before continuing to execute. This ensures that before the main thread processes a new batch of tasks, all the child threads that the main thread has submitted to run tasks have completed execution.

[0135] Step S3, obtaining a write business object according to the read record; this step mainly includes the following multiple sub-steps.

[0136] Step S31, converting the read record into a read business object.

[0137] Step S32, converting the read business object into a write business object.

[0138] In the technical solution for step S3 of this embodiment, the conversion object class is called to realize the conversion from the read business object to the write business object, the fields of the conversion record are the attributes of the corresponding read business object, and the attributes of the read business object are set as the mapping attributes of the write business object.

[0139] Step S4, process the write business object and obtain the task write business object set; the specific process of step S4 includes the following sub-steps.

[0140] Step S41, initialize the job and set the maximum number of write business objects that the job can process.

[0141] Step S42, determine whether there are enough write business objects, if yes, process all the write business objects, otherwise process the remaining write business objects to be processed.

[0142] It should be noted that all write business objects represent the maximum number of write business objects.

[0143] Step S43, instantiate the write business object to obtain a write business object set.

[0144] It can be understood that the number of objects in each write business object set is the maximum number of write business objects, and the number of objects in the smallest write business object set may be smaller than the maximum number of write business objects.

[0145] Step S44, splitting the write business object set according to the maximum number of write business objects that can be processed by the task to obtain several task write business object sets.

[0146] Step S5, process the task, write the business object set, and execute the task.

[0147] Specifically, each write business object set is processed in turn to determine whether there is a task write business object set to be processed. If yes, the task is initialized and executed. Otherwise, it is determined again whether all read records have been processed.

[0148] Example 3 This embodiment provides a data carry-over tool, which adopts the above-mentioned data carry-over method.

[0149] By using a data transfer tool in this embodiment, it is possible to solve technical problems through the data transfer tool, solve business problems through extensions, and dynamically adjust technical parameters through various configurations to solve business and technical problems at the same time. This can lower the threshold for business development, allowing business personnel to focus only on business development, while at the same time using multi-threading technology to improve performance.

[0150] A data transfer tool in this embodiment provides SDK, relies only on JDK8, and adopts a template mode. Developers only need to implement the abstract methods of abstract classes to complete the writing of business codes. Through the properties of various configuration objects, they can flexibly control various key indicators such as the number of threads and the amount of batch data, and develop codes suitable for their own business scenarios.

Claims

1. A data carry-over method, characterized in that: include: S1, initialize resource configuration related classes, instantiate conversion objects, instantiate reading objects, initialize job parameters, and initialize resource related classes; S2, determine whether the total number of records to be processed exceeds the limit; determine whether all read records have been processed; release resources; S3, obtain the write business object based on the read record; S4, process the write business object and obtain the task write business object set; S5, process the task, write the business object set, and execute the task.

2. A data carry-over method according to claim 1, characterized in that: Instantiating the conversion object in step S1 includes: S11, define read business objects and write business objects; S12, instantiate the conversion from the read business object to the write business object.

3. A data carry-over method according to claim 1, characterized in that: Instantiating the read object in step S1 includes: S13, instantiate the query object; S14, customize query extension objects according to business needs; S15, attaching the query expansion object to the query object, and attaching the query object to the read object.

4. A data carry-over method according to claim 3, characterized in that: Step S13 includes: S131, setting query conditions for querying records; S132, setting the maximum number of records for full query; S133, setting the number of paging query records.

5. A data carry-over method according to claim 4, characterized in that: Step S14 includes: when the query condition of the query object cannot meet the demand, customizing the query extension object, defining the properties of the query extension object according to business needs; reading the properties of the query object and the query extension object as the query condition.

6. A data carry-over method according to claim 1, characterized in that: In step S2, determining whether the total number of records to be processed exceeds the limit includes: obtaining the total number of records to be processed of the job. When the total number of records to be processed of the job is greater than the maximum number of records for the full query, the total number of records to be processed exceeds the limit, abnormally exiting and resetting the total number of records to be processed; otherwise, continuing.

7. A data carry-over method according to claim 1, characterized in that: In step S2, judging whether all read records have been processed includes: obtaining the number of records that have been processed, judging whether the number of records that have been processed is equal to the total number of records to be processed, if the number of records that have been processed is equal to the total number of records to be processed, the job is completed; if the number of records that have been processed is not equal to the total number of records to be processed, reading records in pages.

8. A data carry-over method according to claim 1, characterized in that: Step S3 includes: S31, converting the read record into a read business object; S32, converting the read business object into a write business object.

9. A data carry-over method according to claim 1, characterized in that: Step S4 includes: S41, initializing the work and setting the maximum number of write business objects that the work can process; S42, determining whether there are enough write business objects, if yes, then processing all the write business objects, otherwise processing the remaining write business objects to be processed; S43, instantiate a write business object to obtain a write business object set; S44, splitting the write business object set according to the maximum number of write business objects that can be processed by the task to obtain several task write business object sets.

10. A data carry-over method according to any one of claims 1 to 9, characterized in that: Step S5 includes: processing each write business object set in turn, determining whether there is a task write business object set to be processed, if so, initializing the task, executing the task, otherwise determining whether all read records have been processed.

Images