A data carryover method

Abstract

The application discloses a data closing method, relates to the technical field of data processing, and aims to solve the problem that thread safety cannot be guaranteed in a multi-thread concurrent processing mode, and comprises the following steps: S1, initializing a resource configuration related class, instantiating a conversion object, instantiating a reading object, initializing a job parameter, and initializing a resource related class; S2, judging whether the total number of records to be processed exceeds a limit; judging whether all reading records have been processed; S3, obtaining a write business object according to the reading records; S4, processing the write business object, and obtaining a task write business object set; and S5, processing the task write business object set, and executing a task. The data closing method can separate business and technology, avoid the increase of development difficulty caused by the interweaving of business and technology, process general logic, solve safety and performance problems, reduce development difficulty, improve development efficiency, and avoid the occurrence of thread safety and resource leakage problems.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data transfer method. BACKGROUND

[0002] In many business scenarios, when the performance of the serial processing mode of processing business records one by one cannot meet the business requirements, a batch processing mode is usually used to improve efficiency, and if the business requirements cannot be met, a multi-thread concurrent processing mode is used to further improve performance. Although the multi-thread technology can improve efficiency, the problem of ensuring thread safety greatly increases the development difficulty, including the use of thread pool, the control of the size of the used memory, and other business-unrelated technical problems that are difficult for general developers to master, and various thread safety and resource leakage problems are inevitable.

[0003] Data acquisition, conversion and transmission are a complex process. In order to meet different business and technical demands, developers need to pay attention to both business and technical problems, and the separation of business and technical concerns leads to the intermingling of business logic and technical code, greatly increasing the complexity of the code, increasing the difficulty of maintenance and expansion, resulting in poor code quality and security risks.

[0004] For example, a Chinese patent with publication number CN107315752B relates to a data transfer method and system, which discloses a data transfer method, including: configuring data source information and transfer conditions of data to be transferred, wherein the transfer conditions include the start primary key ID and the end primary key ID of the data table to be transferred; dividing the data table into multiple data shards; and using multi-thread asynchronous to transfer the multiple data shards one by one, which realizes simple configuration, convenient transfer, greatly simplifies the operation process of transfer and backup, and improves the efficiency of transfer and backup. However, the Chinese patent with publication number CN107315752B does not separate business and technology, which increases the development difficulty and is prone to thread safety and resource leakage problems. SUMMARY

[0005] In order to solve the problem of difficult thread safety in multi-thread concurrent processing mode, the present application proposes a data transfer method, which can separate business and technology, avoid the intermingling of business and technology to increase development difficulty, handle general logic and solve security and performance problems, reduce development difficulty, improve development efficiency, and avoid thread safety and resource leakage problems.

[0006] In order to achieve the above purpose, the present application adopts the following technical solution: a data transfer method, comprising:

[0007] Step S1, initializing resource configuration related classes, instantiating conversion objects, instantiating reading objects, initializing job parameters, initializing resource related classes;

[0008] Step S2, judging whether the total number of records to be processed exceeds the limit; judging whether all reading records have been processed;

[0009] Step S3, obtaining a write business object according to the reading record;

[0010] Step S4, processing the write business object to obtain a task write business object set;

[0011] Step S5, processing the task write business object set to execute the task.

[0012] In the technical solution, the abstract class is used to realize general data processing and conversion, and the implementation class is used to process the business logic of specific data in a specific business scenario, so that the business and technology can be separated, the development difficulty is reduced, and the development efficiency is improved. The problem of ensuring thread safety in the process of processing business records in multiple threads is solved, and the occurrence of resource leakage is prevented.

[0013] Preferably, the instantiation of the conversion object in step S1 comprises:

[0014] Step S11, defining a read business object and a write business object;

[0015] Step S12, realizing conversion of the read business object to the write business object.

[0016] Preferably, the instantiation of the reading object in step S1 comprises:

[0017] Step S13, instantiating a query object;

[0018] Step S14, customizing a query extension object according to business needs;

[0019] Step S15, attaching the query extension object to the query object, and attaching the query object to the reading object.

[0020] Preferably, step S13 comprises:

[0021] Step S131, setting a query condition of the query record; for example, the record creation time is greater than or equal to the start time and less than the end time; step S132, setting the maximum number of full query records; for example, 1 million records;

[0022] Step S133, setting the number of records for page query; for example, 24,000 records.

[0023] Preferably, step S14 comprises:

[0024] Step S141, when the query condition of the query object cannot meet the requirement, the custom query extension object is expanded, and the attribute of the query extension object is defined according to the business requirement; for example, the state of the record is normal, and the level of the record is senior.

[0025] Step S142, the attributes of the query object and the query extension object are read as the query condition.

[0026] Preferably, the judgment of whether the total number of the records to be processed exceeds the limit in step S2 comprises: obtaining the total number of the records to be processed of the job, when the total number of the records to be processed of the job is greater than the maximum number of records of the full query, the total number of the records to be processed exceeds the limit, and the job is exited abnormally and the total number of the records to be processed is reset; otherwise, the job continues.

[0027] If the total number of the records to be processed has exceeded the limit, it indicates that the query range needs to be narrowed and the number of records of the full query needs to be reduced; because too many records need to occupy a large amount of system resources, and the system resources are limited, the job may fail to run due to insufficient system resources; the total number of the records to be processed needs to be adjusted to a reasonable range according to the actual situation.

[0028] Preferably, the judgment of whether all the read records have been processed in step S2 comprises: obtaining the number of the processed records, judging whether the number of the processed records is equal to the total number of the records to be processed, if the number of the processed records is equal to the total number of the records to be processed, the job is completed; if the number of the processed records is not equal to the total number of the records to be processed, it indicates that there are still records to be processed, and the records are read in batches.

[0029] Preferably, step S3 comprises:

[0030] Step S31, converting the read record into a read business object;

[0031] Step S32, converting the read business object into a write business object; converting the fields of the record into the attributes of the corresponding read business object, and setting the attributes of the read business object to be read as the mapping attributes of the write business object.

[0032] Preferably, step S4 comprises:

[0033] Step S41, initializing the work, and setting the maximum number of write business objects that the work can process;

[0034] Step S42, judging whether there are enough write business objects, if yes, the work processes all the write business objects, otherwise, the work processes the remaining write business objects to be processed;

[0035] Step S43, instantiating the write business object to obtain a write business object set;

[0036] Step S44, splitting the write business object set according to the maximum number of write business objects that the work can process, to obtain a plurality of task write business object sets.

[0037] In the technical solution, the number of objects in each write business object set is the maximum number of write business objects, and the number of objects in the smallest write business object set can be less than the maximum number of write business objects.

[0038] Preferably, step S5 comprises:

[0039] Step S51, processing each write business object set in turn,

[0040] Step S52, determining whether there are still task write business object sets to be processed, if yes, initializing the task and executing the task, and if not, determining whether all read records have been processed.

[0041] The beneficial effects of the present application are:

[0042] (1) separating business and technology, implementing general data processing and conversion through abstract classes, and processing the business logic of specific data in specific business scenarios through implementation classes, which can reduce development difficulty and improve development efficiency;

[0043] (2) ensuring thread safety and preventing various thread safety and resource leakage problems. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a flowchart of a data closing method of the present application.

[0045] Figure 2 is a class diagram of a data closing method of the present application.

[0046] Reference signs: C1, output implementation class; C2, output; C3, task implementation class; C4, task; C5, work; C6, work implementation class; C7, query; C8, input; C9, input implementation class; C10, job; C11, conversion; C12, job implementation class; C13, conversion implementation class; C14, extended query; C15, read business object; C16, write business object. DETAILED DESCRIPTION

[0047] To make the purpose, technical solutions and advantages of the present application clearer and more apparent, the present application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only the best mode of the present application, which are used to explain the present application and do not limit the protection scope of the present application. All other embodiments obtained by those skilled in the art without making creative efforts fall within the protection scope of the present application.

[0048] Example 1

[0049] The embodiment provides a data closing method, referring to Figure 1 , mainly comprising the following steps.

[0050] Step S1, initializing resource configuration related classes, instantiating conversion objects, instantiating reading objects, initializing job parameters, and initializing resource related classes.

[0051] In the embodiment, the resource configuration related classes include reading data source configuration, writing data source configuration, and thread pool configuration, and the resource related classes include reading data source, writing data source, and thread pool.

[0052] It can be understood that in the data closing method of the embodiment, the reading data source configuration is initialized, the writing data source configuration is initialized, and the thread pool configuration is initialized, then the conversion objects are instantiated, the reading objects are instantiated, the job parameters are initialized, and then the reading data source is initialized, the writing data source is initialized, and the thread pool is initialized.

[0053] In the embodiment, the data source includes a source data source and a target data source, and after the writing data source configuration, the source data source and the target data source are created according to the configuration of the data source.

[0054] For the specific process of initializing the thread pool in S1, the thread pool is customized, including implementing the AutoCloseable interface, customizing the thread factory, and customizing the thread pool tool class.

[0055] Specifically, the AutoCloseable interface is implemented through the customized thread pool, and resources are released in the implementation of the interface method.

[0056] The thread factory is customized, the thread is set with a name having business meaning, the thread name prefix having meaning is implemented, and the thread name prefix is used for distinguishing and identifying threads created by different thread pools, so that subsequent analysis and positioning problems are facilitated.

[0057] The thread pool tool class is customized, a reasonable resource shortage coping strategy is set, and the number of concurrent execution threads is ensured to meet the expectation.

[0058] For example, assuming that the number of concurrent threads is 4, the main thread will apply for 4 sub-threads from the thread pool, and arrange 4 sub-threads to execute tasks, when arranging to execute the fifth task, it is found that all threads of the thread pool are executing tasks, at this time the main thread claims a task for execution by itself. In this way, the main thread and all sub-threads are ensured to execute tasks, thread resources are not wasted, resources are maximized, and the best performance is provided.

[0059] For the specific process of instantiating the conversion object in S1, the following several sub-steps are included.

[0060] Step S11, define read business objects and write business objects.

[0061] Step S12, instantiate the conversion of read business objects to write business objects.

[0062] It should be noted that the read business object and the write business object are the same object, including the following several attributes.

[0063] The attribute id is of the numerical type, which can store the content of the app_id field of the record.

[0064] The attribute msg is of the text type, which can store the content of the message field of the record.

[0065] The attribute updateTime is of the time type, which can store the content of the update_time field of the record.

[0066] If the read business object and the write business object are different objects, the mapped attributes need to be converted, and the corresponding business logic is realized according to the business needs.

[0067] In the technical solution of the present embodiment with respect to step S1, the fields of the converted record are the attributes of the corresponding read business object, and the attributes of the read read business object are set as the mapped attributes of the write business object.

[0068] For the specific process of instantiating the read object in S1, it includes the following several sub-steps.

[0069] Step S13, instantiate the query object; for the specific process of S13, it includes the following several sub-steps.

[0070] Step S131, set the query condition of the query record.

[0071] In the present embodiment, the query condition is set to the record creation time greater than or equal to the start time and less than the end time.

[0072] Step S132, set the maximum number of full query records.

[0073] In the present embodiment, the maximum number of full query records is set to 1 million records.

[0074] Step S133, set the number of paging query records.

[0075] In the present embodiment, the number of paging query records is set to 24,000 records.

[0076] Step S14, according to the business needs, customize the query expansion object.

[0077] Specifically, when the query conditions of the query object cannot meet the requirements, a custom query extension object is defined, and the attributes of the query extension object are defined according to business needs; the attributes of the query object and the query extension object are read as query conditions.

[0078] For example, in some implementations, the query criteria can be set to "the record's status is normal and the record's level is high".

[0079] Step S15: Attach the query extension object to the query object and attach the query object to the read object.

[0080] Step S2: Determine if the total number of records to be processed exceeds the limit; determine if all read records have been processed; release resources.

[0081] The specific process for determining whether the total number of records to be processed exceeds the limit in step S2 includes: obtaining the total number of records to be processed for the job; if the total number of records to be processed for the job is greater than the maximum number of records for full query, the total number of records to be processed exceeds the limit, the process exits abnormally and resets the total number of records to be processed; otherwise, the process continues.

[0082] It should be noted that if the total number of records to be processed has exceeded the limit, it indicates that the query scope needs to be narrowed and the number of records to be queried in full needs to be reduced. This is because querying too many records will consume a lot of system resources, and system resources are limited. It is very likely that the job will fail due to insufficient system resources. The total number of records to be processed needs to be adjusted to a reasonable range according to the actual situation.

[0083] The specific process for determining whether all read records have been processed in step S2 includes the following:

[0084] First, obtain the number of records that have been processed, and then determine whether the number of records that have been processed is equal to the total number of records to be processed.

[0085] If the number of records processed equals the total number of records to be processed, the job is complete.

[0086] If the number of records already processed is not equal to the total number of records to be processed, it means there are still records to be processed, and records are read in pages.

[0087] It should be noted that paginated record reading means reading one page at a time. For example, if each page has 1,000 records, then after reading the first page of 1,000 records, the second page of 1,000 records will be read, and so on, until the last page is read.

[0088] In the technical solution of the embodiment about step S2, releasing the resources includes closing the data source and closing the thread pool, and after the job processing is completed, the data source and the thread pool are safely closed, the resources are released, memory leakage is avoided, and the program is ensured to continuously run healthily and stably.

[0089] In the embodiment, the try-with-resources statement is used to manage the resources, so that the resource release is more concise.

[0090] Specifically, after the execution of the try block is completed, whether the execution is normally ended or an exception occurs, the resources are automatically closed, and the close() method does not need to be called explicitly, so that the safe and reliable release of the applied resources is ensured.

[0091] According to the configuration of the concurrency parameter, it is ensured by the lock that the main thread waits for all the submitted sub-threads to be completed before continuing to execute.

[0092] Through the lock, when the main thread is idle, it waits for all the sub-threads that are submitted to run the tasks to be completed before continuing to execute. Thus, it is ensured that before the main thread processes a new batch of tasks, the sub-threads that are submitted by the main thread to run the tasks have all been executed.

[0093] Step S3, obtaining a write business object according to the read record; for this step, the following multiple sub-steps are mainly included.

[0094] Step S31, converting the read record into a read business object.

[0095] Step S32, converting the read business object into a write business object.

[0096] In the technical solution of the embodiment about step S3, the conversion object class is called to implement the conversion of the read business object into the write business object, the fields of the conversion record are the attributes of the corresponding read business object, and the attributes of the read business object are set as the mapped attributes of the write business object.

[0097] Step S4, processing the write business object to obtain a task write business object set; for the specific process of step S4, the following several sub-steps are included.

[0098] Step S41, initializing the work and setting the maximum number of write business objects that the work can process.

[0099] Step S42, determining whether there are enough write business objects, if yes, the work processes all the write business objects, and if not, the work processes the remaining write business objects to be processed.

[0100] It should be noted that all the write business objects represent the maximum number of write business objects.

[0101] Step S43, instantiating the write business objects to obtain a write business object set.

[0102] It can be understood that the number of objects in each write business object set is the maximum write business object number, and the number of objects in the smallest write business object set can be less than the maximum write business object number.

[0103] Step S44, splitting the write business object set according to the maximum write business object number that can be processed by the worker to obtain a plurality of task write business object sets.

[0104] Step S5, processing the task write business object set to execute the task.

[0105] Specifically, each write business object set is processed in turn to determine whether there is a task write business object set to be processed, if yes, initializing the task and executing the task, otherwise, determining whether all read records have been processed.

[0106] Next, a data clearing method of the embodiment is described in detail with reference to the self-defined class, the static structure of the class, and the relationship between the classes. Figure 2 .

[0107] The QueryExt (extended query) class is defined to customize the query condition, and the QueryExt object is instantiated and the extended query condition is initialized.

[0108] The Query (query) class is instantiated to introduce the QueryExt class object, define the query condition and the query configuration.

[0109] The ReaderBO (read business object) class is defined to receive the read data.

[0110] The WriterBO (write business object) class is defined to convert into the write data.

[0111] The DBTransform class (Transform implementation class) is defined to inherit the Transform (conversion) abstract class, introduce the ReaderBO class object and the WriterBO class object, and complete the conversion from the ReaderBO class object to the WriterBO class object.

[0112] The DBReader class (Reader implementation class) is defined to inherit the Reader (input) abstract class, implement the abstract method, define the query and count query SQL statements, and introduce the ReaderBO class object and the Query class object.

[0113] Customize DBWriter class (Writer implementation class), inherit Writer (output) abstract class, implement abstract method, define insert SQL statement, introduce custom WriterBO class object.

[0114] Customize DBTask class (Task implementation class), inherit Task (task) abstract class, implement abstract method, reference Writer abstract class, introduce custom DBWriter class object.

[0115] Customize DBJob class (Job implementation class), inherit Job (work) abstract class, implement abstract method, reference Task abstract class, introduce custom DBTask class object, customize DBWork class (Work implementation class), inherit Work (work) abstract class, implement abstract method, reference Reader abstract class, Job abstract class, Transform abstract class, reference custom ReaderBO, custom WriterBO class, custom QueryExt class, introduce custom DBJob class object.

[0116] In the embodiment, the abstract class is used to implement general data processing and conversion, the implementation class is used to process specific business logic of specific data of business scene, the general and specific business logic codes are separated, the business developer only focuses on the specific business logic, and thus the development difficulty is reduced and the development efficiency is improved.

[0117] The data settlement method can process general logic and solve security and performance problems, and guarantees robustness and high performance of software.

[0118] Embodiment 2

[0119] The embodiment provides a data settlement method, mainly including the following steps.

[0120] In step S1, resource configuration related classes are initialized, a conversion object is instantiated, a reading object is instantiated, job parameters are initialized, and resource related classes are initialized.

[0121] In the embodiment, the resource configuration related classes include reading data source configuration, writing data source configuration and thread pool configuration, and the resource related classes include reading data source, writing data source and thread pool.

[0122] It can be understood that in the data settlement method, the reading data source configuration is initialized, the writing data source configuration is initialized, and the thread pool configuration is initialized, then the conversion object is instantiated, the reading object is instantiated, the job parameters are initialized, and then the reading data source is initialized, the writing data source is initialized, and the thread pool is initialized.

[0123] In the embodiment, the data source includes a source data source and a target data source, and after the write data source is configured, the source data source and the target data source are created according to the configuration of the data source.

[0124] For the specific process of initializing the thread pool in S1, the thread pool is customized, including implementing the AutoCloseable interface, customizing the thread factory, and customizing the thread pool tool class.

[0125] Specifically, by customizing the thread pool, the AutoCloseable interface is implemented, and resources are released in the implementation of the interface method.

[0126] The thread factory is customized, the thread is set with a name having business meaning, the thread name prefix is customized to have meaning, and the threads created by different thread pools are distinguished and identified for subsequent analysis and positioning of problems.

[0127] The thread pool tool class is customized, a reasonable resource shortage response strategy is set, and the number of concurrent execution threads is ensured to meet the expectation.

[0128] For the specific process of instantiating the conversion object in S1, the following sub-steps are included.

[0129] Step S11, defining a read business object and a write business object.

[0130] Step S12, instantiating the conversion of the read business object to the write business object.

[0131] It should be noted that the read business object and the write business object are the same object, including the following attributes.

[0132] The attribute id is of a numerical type and can store the content of the app_id field of the record.

[0133] The attribute msg is of a text type and can store the content of the message field of the record.

[0134] The attribute updateTime is of a time type and can store the content of the update_time field of the record.

[0135] If the read business object and the write business object are different objects, the attributes of the conversion mapping need to be converted, and the corresponding business logic is implemented according to the business needs.

[0136] In the technical solution of the embodiment regarding step S1, the fields of the conversion record are the attributes of the corresponding read business object, and the attributes of the read business object are set as the mapping attributes of the write business object.

[0137] For the specific process of instantiating the read object in S1, the following sub-steps are included.

[0138] Step S13, instantiating a query object; for the specific process of S13, including several sub-steps as follows.

[0139] Step S131, setting the query condition of the query record.

[0140] For example, the query condition can be set as the record creation time greater than or equal to the start time and less than the end time.

[0141] Step S132, setting the maximum record number of full query.

[0142] In this embodiment, the maximum record number of full query is set as 2 million records.

[0143] Step S133, setting the paging query record number.

[0144] In this embodiment, the paging query record number is set as 40 thousand records.

[0145] Step S14, customizing the query extension object according to the business needs.

[0146] Specifically, when the query condition of the query object cannot meet the needs, the query extension object is customized, the attributes of the query extension object are defined according to the business needs, and the attributes of the query object and the query extension object are read as the query condition.

[0147] Step S15, attaching the query extension object to the query object, and attaching the query object to the read object.

[0148] Step S2, judging whether the total number of records to be processed exceeds the limit; judging whether all read records have been processed; releasing resources.

[0149] For the specific process of judging whether the total number of records to be processed exceeds the limit in step S2, including: obtaining the total number of records to be processed of the job, when the total number of records to be processed of the job is greater than the maximum record number of full query, the total number of records to be processed exceeds the limit, and the process is exited abnormally and the total number of records to be processed is reset; otherwise, continue.

[0150] It should be noted that if the total number of records to be processed has exceeded the limit, it indicates that the query range needs to be narrowed and the full query record number needs to be reduced; because too many query records need to occupy a large amount of system resources, and the system resources are limited, it is likely that the job will fail due to insufficient system resources; the total number of records to be processed needs to be adjusted to a reasonable range according to the actual situation.

[0151] For the specific process of judging whether all read records have been processed in step S2, including the following contents.

[0152] First, the number of records already processed is obtained, and then it is judged whether the number of records already processed is equal to the total number of records to be processed.

[0153] If the number of records already processed is equal to the total number of records to be processed, the job is completed.

[0154] If the number of records already processed is not equal to the total number of records to be processed, it means that there are still records to be processed, and the records are read in pages.

[0155] It should be noted that the meaning of reading records in pages is to read one page at a time, for example, there are 1000 records in each page, after reading 1000 records in the first page, 1000 records in the second page are read, and so on, until the last page is read.

[0156] In the technical solution of the present embodiment with respect to step S2, releasing resources includes closing the data source and closing the thread pool, and after the job processing is completed, the data source and the thread pool are safely closed, the resources are released, memory leakage is avoided, and the program is ensured to run healthily and stably.

[0157] In the present embodiment, the try-with-resources statement is used to manage resources, so that the resource release is more concise.

[0158] Specifically, after the execution of the try block is completed, whether it is normally ended or an exception occurs, these resources will be automatically closed, without the need for explicit call of the close() method, thereby ensuring the safe and reliable release of the applied resources.

[0159] According to the configuration of the concurrency parameter, it is guaranteed by the lock that the main thread waits for all the submitted sub-threads to end before continuing to execute.

[0160] Through the lock, when the main thread is idle, it will wait for all the sub-threads running the tasks submitted by itself to be executed before continuing to execute. Thus, it is ensured that before the main thread processes a new batch of tasks, the sub-threads running the tasks submitted by the main thread have all been executed.

[0161] Step S3, obtaining a write business object according to the read record; for this step, the following multiple sub-steps are mainly included.

[0162] Step S31, converting the read record into a read business object.

[0163] Step S32, converting the read business object into a write business object.

[0164] In the technical solution of the present embodiment with respect to step S3, the conversion object class is called to realize the conversion of the read business object to the write business object, the fields of the converted record are the attributes of the corresponding read business object, and the attributes of the read read business object are set as the mapped attributes of the write business object.

[0165] Step S4, processing write business objects, obtaining task write business object sets; for the specific process of step S4, including the following several sub-steps.

[0166] Step S41, initializing work, setting the maximum number of write business objects that the work can process.

[0167] Step S42, judging whether there are enough write business objects, yes, the work processes all write business objects, otherwise, processing the remaining write business objects to be processed.

[0168] It should be noted that all write business objects represent the maximum number of write business objects.

[0169] Step S43, instantiating write business objects, obtaining write business object sets.

[0170] It can be understood that the number of objects in each write business object set is the maximum number of write business objects, and the number of objects in the smallest write business object set may be less than the maximum number of write business objects.

[0171] Step S44, splitting the write business object set according to the maximum number of write business objects that the work can process, obtaining several task write business object sets.

[0172] Step S5, processing task write business object sets, executing tasks.

[0173] Specifically, each write business object set is processed in turn, and it is judged whether there are still task write business object sets to be processed, yes, initializing the task, executing the task, otherwise, re-judging whether all read records have been processed.

[0174] Embodiment 3

[0175] The embodiment provides a data closing tool adopting the data closing method.

[0176] Using the data closing tool of the embodiment, the technical problem can be solved through the data closing tool, the business problem can be solved through expansion, various configurations can be used to dynamically adjust technical parameters to solve business and technical problems, which can reduce the threshold of business development, so that business personnel only need to focus on business development, and at the same time, the performance can be improved by using multi-threading technology.

[0177] The data closing tool of the embodiment provides an SDK, only relies on JDK8, adopts a template mode, a developer only needs to implement an abstract method of an abstract class to complete the writing of business code, various configuration object properties are used to flexibly control the number of threads, batch data volume and various key indicators, and a code suitable for a business scene is developed.

Claims

1. A data roll-over method, characterized by, Comprise: S1, initialize resource configuration related class, instantiate conversion object, instantiate read object, initialize job parameter, initialize resource related class; S2, judge whether the total number of records to be processed exceeds the limit; judge whether all read records have been processed; Release resources; S3, get write business object according to read record, comprising: converting read record to read business object, and then converting read business object to write business object; S4, process write business object, get task write business object set; S5, process task write business object set, execute task.

2. The data carryover method of claim 1, wherein, The instantiation of conversion object in step S1 comprises: S11, define read business object and write business object; S12, instantiate conversion from read business object to write business object.

3. The data carryover method of claim 1, wherein, The instantiation of read object in step S1 comprises: S13, instantiate query object; S14, customize query extension object according to business needs; S15, attach query extension object to query object, and attach query object to read object.

4. The data carryover method of claim 3, wherein, Step S13 comprises: S131, set query condition of query record; S132, set maximum number of full query records; S133, set number of paging query records.

5. The data carryover method of claim 4, wherein, Step S14 comprises: when the query condition of the query object cannot meet the needs, customize the query extension object, define the attributes of the query extension object according to business needs; read the attributes of the query object and the query extension object as query conditions.

6. The data carryover method of claim 1, wherein, The judgment of whether the total number of records to be processed exceeds the limit in step S2 comprises: obtaining the total number of records to be processed of the job, when the total number of records to be processed of the job is greater than the maximum number of full query records, the total number of records to be processed exceeds the limit, exit abnormally and reset the total number of records to be processed; otherwise, continue.

7. The data carryover method of claim 1, wherein, The judgment of whether all read records have been processed in step S2 comprises: obtaining the number of records that have been processed, judging whether the number of records that have been processed is equal to the total number of records to be processed, if the number of records that have been processed is equal to the total number of records to be processed, the job is completed; if the number of records that have been processed is not equal to the total number of records to be processed, read records in batches.

8. The data carryover method of claim 1, wherein, Step S4 comprises: S41, initialize work, set the maximum number of write business objects that the work can process; S42, judge whether there are enough write business objects, yes, the work processes all write business objects, otherwise, process the remaining write business objects to be processed; S43, instantiate write business object, get write business object set; S44, split the write business object set according to the maximum number of write business objects that the work can process, to get several task write business object sets.

9. A data carryover method according to any one of claims 1 to 8, wherein, Step S5 comprises: processing each write business object set in turn, judging whether there are task write business object sets to be processed, yes, initializing task, executing task, otherwise, judging whether all read records have been processed.

Images