Data processing framework implementation method, device, equipment and computer storage medium

Abstract

The present invention discloses a method, device, equipment and computer storage medium for implementing a data processing framework, and relates to the technical field of financial technology. The method includes: creating a target sequence number generator; creating a target sequence number generator factory class, inheriting the default sequence number generator factory class of Spring Batch, and rewriting the getIncrementer method inherited in the target sequence number generator factory class to replace the MySQL sequence number generator in the inherited getIncrementer method with the target sequence number generator; creating a target configurator class, inheriting the default configurator class of Spring Batch, and rewriting the createJobRepository method inherited in the target configurator class to specify the sequence number generator factory class as the target sequence number generator factory class. The present invention can avoid the problem of sequence number generation failure to realize running Spring Batch on TiDB.

Description

Technical Field

[0001] The present invention relates to the technical field of financial technology (Fintech), and in particular to a method, device, equipment and computer storage medium for implementing a data processing framework. Background Art

[0002] With the development of computer technology, more and more technologies are being applied in the financial field. The traditional financial industry is gradually transforming into financial technology (Fintech). However, due to the security and real-time requirements of the financial industry, higher requirements are also placed on technology.

[0003] At present, the task processing systems of Internet banks are usually faced with massive amounts of data to be processed. In response to this, the industry usually runs the data processing framework Spring Batch (batch processing framework) on the database for processing. However, traditional relational databases have poor horizontal scalability. When encountering a large number of tasks and massive amounts of data to be processed, the database is likely to become a bottleneck. TiDB, as a new distributed database that combines the characteristics of traditional relational databases and non-relational databases, can support unlimited horizontal expansion and is also compatible with MySQL protocols and syntax. Therefore, using TiDB to process a large number of tasks and massive amounts of data can completely solve the bottleneck problem of the database.

[0004] However, when running Spring Batch on TiDB, in the process of generating serial numbers, Spring Batch will use the corresponding table and execute related commands on the table to obtain the serial number. Since TiDB's lock type is optimistic, when multiple threads perform update operations on the same record of the same table, conflicts are very likely to occur when submitting transactions, resulting in failure to generate serial numbers and affecting the normal operation of Spring Batch. Therefore, how to avoid the problem of serial number generation failure and run Spring Batch on TiDB is an urgent problem to be solved. Summary of the invention

[0005] The main purpose of the present invention is to provide a method, device, equipment and computer storage medium for implementing a data processing framework, aiming to avoid the problem of serial number generation failure so as to implement Spring Batch running on TiDB.

[0006] To achieve the above object, the present invention provides a method for implementing a data processing framework, the method for implementing the data processing framework comprising:

[0007] Create a target serial number generator based on a preset method;

[0008] Create a target sequence number generator factory class, inherit the default sequence number generator factory class of the Spring Batch framework, and rewrite the inherited getIncrementer method in the target sequence number generator factory class to replace the MySQL sequence number generator in the inherited getIncrementer method with the target sequence number generator;

[0009] Create a target configurator class, inherit the default configurator class of the Spring Batch framework, and rewrite the inherited createJobRepository method in the target configurator class to specify the serial number generator factory class as the target serial number generator factory class.

[0010] Optionally, the step of creating a target sequence number generator based on a preset method includes:

[0011] Create a target sequence number generator, inherit the MySQL sequence number generator of the Spring Batch framework, and rewrite the inherited getNextKey method in the target sequence number generator to cyclically call the getNextKey method of the MySQL sequence number generator.

[0012] Optionally, after the step of creating a target sequence number generator that inherits the MySQL sequence number generator of the Spring Batch framework, the step further includes:

[0013] Rewrite the inherited getCacheSize method in the target serial number generator to set the serial number cache quantity to a preset value, wherein the preset value is greater than 1.

[0014] Optionally, the step of creating a target sequence number generator based on a preset method includes:

[0015] A distributed cache is created, the distributed cache being used to generate sequence numbers to serve as a target sequence number generator.

[0016] Optionally, the implementation method of the data processing framework further includes:

[0017] A preset annotation is added to the target configurator class to instruct the Spring Batch framework to place the target configurator class into the operating environment to replace the default configurator class.

[0018] In addition, to achieve the above-mentioned purpose, the present invention further provides a device for implementing a data processing framework, and the device for implementing the data processing framework includes:

[0019] A first creation module, used to create a target sequence number generator based on a preset method;

[0020] The second creation module is used to create a target sequence number generator factory class, inherit the default sequence number generator factory class of the Spring Batch framework, and rewrite the inherited getIncrementer method in the target sequence number generator factory class to replace the MySQL sequence number generator in the inherited getIncrementer method with the target sequence number generator;

[0021] The third creation module is used to create a target configurator class, inherit the default configurator class of the Spring Batch framework, and rewrite the createJobRepository method inherited in the target configurator class to specify the serial number generator factory class as the target serial number generator factory class.

[0022] Optionally, the first creation module includes:

[0023] The first creation unit is used to create a target sequence number generator, inherit the MySQL sequence number generator of the Spring Batch framework, and rewrite the inherited getNextKey method in the target sequence number generator to cyclically call the getNextKey method of the MySQL sequence number generator.

[0024] Optionally, the first creation module further includes:

[0025] A method rewriting unit is used to rewrite the getCacheSize method inherited in the target serial number generator to set the serial number cache quantity to a preset value, wherein the preset value is greater than 1.

[0026] In addition, to achieve the above-mentioned purpose, the present invention also provides an implementation device of a data processing framework, and the implementation device of the data processing framework includes: a memory, a processor, and an implementation program of the data processing framework stored in the memory and executable on the processor, and when the implementation program of the data processing framework is executed by the processor, the steps of the implementation method of the data processing framework as described above are implemented.

[0027] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which an implementation program of a data processing framework is stored, and when the implementation program of the data processing framework is executed by a processor, the steps of the implementation method of the data processing framework as described above are implemented.

[0028] The present invention provides an implementation method, device, equipment and computer storage medium of a data processing framework. A target sequence number generator is obtained by creation based on a preset method, and then a target sequence number generator factory class is created, which inherits a default sequence number generator factory class of a SpringBatch framework, and a getIncrementer method inherited in the target sequence number generator factory class is rewritten to replace a MySQL sequence number generator in the inherited getIncrementer method with the target sequence number generator; finally, a target configurator class is created, which inherits a default configurator class of a Spring Batch framework, and a createJobRepository method inherited in the target configurator class is rewritten to specify the sequence number generator factory class as the target sequence number generator factory class. The present invention reconstructs a serial number generator based on the TiDB database (i.e., the target serial number generator), a target configurator class, and a target serial number generator factory class through inheritance and rewriting, and places them into the Spring Batch running environment, replacing the original MySQL serial number generator. When the Spring Batch framework runs on the TiDB database, it ignores the original MySQL serial number generator and uses the target serial number generator, thereby avoiding the problem of serial number generation failure and realizing the running of the Spring Batch framework on TiDB. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention;

[0030] Figure 2 It is a flowchart of a first embodiment of a method for implementing a data processing framework of the present invention;

[0031] Figure 3 The diagram is a startup principle and operation principle diagram of the Spring Batch framework involved in the first embodiment of the present invention;

[0032] Figure 4 This is a functional module diagram of the first embodiment of the device for implementing the data processing framework of the present invention.

[0033] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0034] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0035] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0036] The implementation device of the data processing framework of the embodiment of the present invention may be a smart phone, or a terminal device such as a PC (Personal Computer), a tablet computer, or a portable computer.

[0037] like Figure 1 As shown, the implementation device of the data processing framework may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0038] Those skilled in the art will understand that Figure 1 The implementation device structure of the data processing framework shown in the figure does not constitute a limitation on the implementation device of the data processing framework, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0039] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, and an implementation program of a data processing framework.

[0040] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and perform data communication with the backend server; the user interface 1003 is mainly used to connect to the client and perform data communication with the client; and the processor 1001 can be used to call the implementation program of the data processing framework stored in the memory 1005 and perform the following operations:

[0041] Create a target serial number generator based on a preset method;

[0042] Create a target sequence number generator factory class, inherit the default sequence number generator factory class of the Spring Batch framework, and rewrite the inherited getIncrementer method in the target sequence number generator factory class to replace the MySQL sequence number generator in the inherited getIncrementer method with the target sequence number generator;

[0043] Create a target configurator class, inherit the default configurator class of the Spring Batch framework, and rewrite the inherited createJobRepository method in the target configurator class to specify the serial number generator factory class as the target serial number generator factory class.

[0044] Further, the processor 1001 may call an implementation program of the data processing framework stored in the memory 1005, and further perform the following operations:

[0045] Create a target sequence number generator, inherit the MySQL sequence number generator of the Spring Batch framework, and rewrite the inherited getNextKey method in the target sequence number generator to cyclically call the getNextKey method of the MySQL sequence number generator.

[0046] Further, the processor 1001 may call an implementation program of the data processing framework stored in the memory 1005, and further perform the following operations:

[0047] Rewrite the inherited getCacheSize method in the target serial number generator to set the serial number cache quantity to a preset value, wherein the preset value is greater than 1.

[0048] Further, the processor 1001 may call an implementation program of the data processing framework stored in the memory 1005, and further perform the following operations:

[0049] A distributed cache is created, the distributed cache being used to generate sequence numbers to serve as a target sequence number generator.

[0050] Further, the processor 1001 may call an implementation program of the data processing framework stored in the memory 1005, and further perform the following operations:

[0051] A preset annotation is added to the target configurator class to instruct the Spring Batch framework to place the target configurator class into the operating environment to replace the default configurator class.

[0052] Based on the above hardware structure, various embodiments of the implementation method of the data processing framework of the present invention are proposed.

[0053] The invention provides a method for realizing a data processing framework.

[0054] Reference Figure 2 , Figure 2 It is a flowchart of the first embodiment of the method for implementing the data processing framework of the present invention.

[0055] In this embodiment, the implementation method of the data processing framework includes:

[0056] Step S10, creating a target sequence number generator based on a preset method;

[0057] In the prior art, although the MySQL protocol and syntax can be used to run Spring Batch on TiDB, if the MySQL protocol is used, when generating sequence numbers, Spring Batch will use the following three tables: BATCH_JOB_SEQ, BATCH_JOB_EXECUTION_SEQ, BATCH_STEP_EXECUTION_SEQ; and execute the following two commands on these three tables to obtain the sequence numbers: update<TABLE_NAME> set ID = last_insert_id (ID + 1); select last_insert_id (). However, since the lock type of TiDB is optimistic, when multiple threads perform update operations on the same record of the same table, conflicts are very likely to occur when submitting transactions, resulting in failure to generate sequence numbers and affecting the normal operation of Spring Batch.

[0058] To solve the above problems, this application is based on the Spring Batch specification and the optimistic locking feature of TiDB. Through inheritance and rewriting, a serial number generator based on the TiDB database (i.e., target serial number generator), a target configurator class, and a target serial number generator factory class are reconstructed and placed in the Spring Batch running environment, replacing the original MySQL serial number generator. When the Spring Batch framework runs on the TiDB database, it ignores the original MySQL serial number generator and uses the target serial number generator, thereby avoiding the problem of serial number generation failure and realizing the running of the Spring Batch framework on TiDB.

[0059] The implementation method of the data processing framework of this embodiment is implemented by the implementation device of the data processing framework. The target sequence number generator can be first created based on the preset method. The generation method of the target sequence number generator includes: 1) creating a target sequence number generator, inheriting the MySQL sequence number generator of the Spring Batch framework, and rewriting the inherited getNextKey method in the target sequence number generator to cyclically call the getNextKey method of the MySQL sequence number generator; 2) creating a distributed cache, which is used to generate sequence numbers as the target sequence number generator. The specific creation process can refer to the second and third embodiments described below, which will not be repeated here.

[0060] Step S20, creating a target sequence number generator factory class, inheriting the default sequence number generator factory class of the Spring Batch framework, and rewriting the inherited getIncrementer method in the target sequence number generator factory class to replace the MySQL sequence number generator in the inherited getIncrementer method with the target sequence number generator;

[0061] In this embodiment, Figure 3 The system startup principle and operation principle shown in the figure are as follows: When the application system including the Spring Batch framework is started, the Spring Batch framework will first do a series of initialization configuration work. By default, the initialization configuration work will be performed by the default configurator DefaultBatchConfiguerer preset by Spring Batch. This series of initialization configuration work includes steps such as setting the data source and creating a job storage repository (JobRepository). The job storage repository is used to store various metadata of various jobs when Spring Batch is running. Obviously, this involves the generation of metadata serial numbers. Therefore, Spring Batch presets a default serial number generator factory DefaultDataFieldMaxValueIncrementerFactory, which will create a corresponding serial number generator according to the type of data source. For example, if the data source is MySQL, a preset MySQL serial number generator MySQLMaxValueIncrementer is created. When the initialization configuration work is completed, when the system is running, if Spring Batch wants to store new metadata in the warehouse, it will obtain a latest ID from the serial number generator created above according to the metadata type and attach it to the metadata, and then store the metadata in the job storage warehouse.

[0062] Based on the startup and operation principles of the above Spring Batch framework, after creating a new target serial number generator for TiDB, it is also necessary to modify the default serial number generator factory class DefaultDataFieldMaxValueIncrementerFactory and the default configurator class DefaultBatchConfiguerer, so that the Spring Batch framework can be run on TiDB.

[0063] Therefore, after creating the target sequence number generator, first create the target sequence number generator factory class (remembered as TiDBDataFieldMaxValueIncrementerFactory), inheriting the default sequence number generator factory class of the Spring Batch framework (that is, Figure 3 DefaultDataFieldMaxValueIncrementerFactory in the target sequence number generator factory class), that is, the target sequence number generator factory class inherits from the default sequence number generator factory class. Then, the inherited getIncrementer method in the target sequence number generator factory class is rewritten to replace the MySQL sequence number generator in the inherited getIncrementer method with the target sequence number generator. Among them, the getIncrementer method is used to call the sequence number generator.

[0064] Step S30, creating a target configurator class, inheriting the default configurator class of the Spring Batch framework, and rewriting the inherited createJobRepository method in the target configurator class to specify the serial number generator factory class as the target serial number generator factory class.

[0065] Finally, create the target configurator class (referred to as TiDBBatchConfigurer) and inherit the default configurator class of the Spring Batch framework (i.e. Figure 3 In other words, copy the createJobRepository() method code of DefaultBatchConfigurer in its entirety and set the factory class of the serial number generator to the target factory class TiDBDataFiledMaxValueIncrementerFactory. The createJobRepository method is used to create a job repository.

[0066] Furthermore, after the step of "creating a target configurator class that inherits the default configurator class of the Spring Batch framework", the implementation method of the data processing framework further includes:

[0067] A preset annotation is added to the target configurator class to instruct the Spring Batch framework to place the target configurator class into the operating environment to replace the default configurator class.

[0068] Add a preset annotation to the target configurator class to instruct the Spring Batch framework to place the target configurator class in the runtime environment to replace the default configurator class. The preset annotation is @Component. By adding the preset annotation @Component to the target configurator class TiDBBatchConfigurer, the created target configurator class TiDBBatchConfigurer can be placed in the Spring Batch runtime environment to replace the default configurator class DefautBatchConfigurer.

[0069] An embodiment of the present invention provides an implementation method of a data processing framework, which obtains a target sequence number generator by creating it based on a preset method, then creates a target sequence number generator factory class, inherits a default sequence number generator factory class of a Spring Batch framework, and rewrites the inherited getIncrementer method in the target sequence number generator factory class to replace the MySQL sequence number generator in the inherited getIncrementer method with the target sequence number generator; finally, creates a target configurator class, inherits the default configurator class of the Spring Batch framework, and rewrites the inherited createJobRepository method in the target configurator class to specify the sequence number generator factory class as the target sequence number generator factory class. The embodiment of the present invention reconstructs a serial number generator based on the TiDB database (i.e., the target serial number generator), a target configurator class, and a target serial number generator factory class through inheritance and rewriting, and places them into the Spring Batch running environment, replacing the original MySQL serial number generator. When the Spring Batch framework runs on the TiDB database, it ignores the original MySQL serial number generator and uses the target serial number generator, thereby avoiding the problem of serial number generation failure and enabling the Spring Batch framework to run on TiDB.

[0070] Furthermore, based on the above-mentioned first embodiment, a second embodiment of a method for implementing the data processing framework of the present invention is proposed.

[0071] In this embodiment, step S10 may include:

[0072] Create a target sequence number generator, inherit the MySQL sequence number generator of the Spring Batch framework, and rewrite the inherited getNextKey method in the target sequence number generator to cyclically call the getNextKey method of the MySQL sequence number generator.

[0073] In this embodiment, as one of the methods for creating a target sequence number generator, the creation process is as follows: Create a target sequence number generator, denoted as TiDBMaxValueIncrementer, which inherits the MySQL sequence number generator of the Spring Batch framework (i.e. Figure 3 MySQLMaxValueIncrementer in TiDB), that is, TiDBMaxValueIncrementer is inherited from MySQLMaxValueIncrementer. Then, rewrite the inherited getNextKey method in the target sequence number generator to call the getNextKey method of the MySQL sequence number generator in a loop. Among them, the getNextKey() method is used to obtain the next sequence number. This is because: TiDB's lock type is an optimistic lock. When multiple threads execute update commands on the same record of the same table, conflicts may occur when committing transactions, resulting in failure of sequence number generation. By rewriting the getNextKey() method, the parent class's getNextKey() method is called in a loop. If the parent class getNextKey() method does not throw a transaction conflict exception, it means that the sequence number generation is successful; otherwise, call the parent class's getNextKey() method again until it succeeds.

[0074] This embodiment creates a target serial number generator TiDBMaxValueIncrementer to inherit the MySQL serial number generator MySQLMaxValueIncrementer of the Spring Batch framework, and cyclically calls the getNextKey() method of the parent class in the subclass. This can avoid transaction conflicts in the default MySQL serial number generator when running the Spring Batch framework on TiDB, resulting in serial number generation failure and affecting the normal operation of the Spring Batch framework.

[0075] Furthermore, after the step of "creating a target sequence number generator that inherits the MySQL sequence number generator of the Spring Batch framework", the implementation method of the data processing framework may further include:

[0076] Rewrite the inherited getCacheSize method in the target serial number generator to set the serial number cache quantity to a preset value, wherein the preset value is greater than 1.

[0077] Since only one serial number is cached in the getCacheSize() method of the MySQLMaxValueIncrementer of the MySQL serial number generator, each time the serial number is obtained, the database must be accessed, which will affect the performance of the database and the processing efficiency. Therefore, in this embodiment, after creating the target serial number generator and inheriting the MySQL serial number generator of the Spring Batch framework, the inherited getCacheSize() method in the target serial number generator is rewritten to set the serial number cache quantity to a preset value, wherein the getCacheSize() method is used to obtain the cache size, and the preset value is greater than 1, and can be set to a value such as 100-10000 according to actual needs, and is not specifically limited here. In other words, by rewriting the getCacheSize() method, the serial number data cached in the target serial number generator is a preset value greater than 1, which can avoid accessing the database every time the getNextKey() method is called, which affects the performance and improves efficiency.

[0078] Furthermore, based on the above-mentioned first embodiment, a third embodiment of a method for implementing the data processing framework of the present invention is proposed.

[0079] In this embodiment, step S10 may further include:

[0080] A distributed cache is created, the distributed cache being used to generate sequence numbers to serve as a target sequence number generator.

[0081] In this embodiment, as another method for creating a target sequence number generator, the creation process is as follows: create a distributed cache, which is used to generate a sequence number as the target sequence number generator. The distributed cache can be Redis (Remote Dictionary Server, a distributed memory database), Memcached (a distributed high-speed cache system), etc. The created distributed cache only needs to comply with the interface specification of the sequence number generator and can generate a unique sequence number.

[0082] In this embodiment, a distributed cache is created to replace the MySQL sequence number generator MySQLMaxValueIncrementer, so that when the Spring Batch framework runs on TiDB, the original MySQL sequence number generator is ignored and the newly created target sequence number generator (i.e., distributed cache) is used instead. This avoids transaction conflicts in the default MySQL sequence number generator when running the Spring Batch framework on TiDB, resulting in sequence number generation failure and affecting the normal operation of the Spring Batch framework.

[0083] The invention also provides a device for realizing the data processing framework.

[0084] Reference Figure 4 , Figure 4 This is a functional module diagram of the first embodiment of the device for implementing the data processing framework of the present invention.

[0085] like Figure 4 As shown, the implementation device of the data processing framework includes:

[0086] A first creation module 10, used to create a target sequence number generator based on a preset method;

[0087] The second creation module 20 is used to create a target sequence number generator factory class, inherit the default sequence number generator factory class of the Spring Batch framework, and rewrite the inherited getIncrementer method in the target sequence number generator factory class to replace the MySQL sequence number generator in the inherited getIncrementer method with the target sequence number generator;

[0088] The third creation module 30 is used to create a target configurator class, inherit the default configurator class of the Spring Batch framework, and rewrite the inherited createJobRepository method in the target configurator class to specify the serial number generator factory class as the target serial number generator factory class.

[0089] Furthermore, the first creation module 10 includes:

[0090] The first creation unit is used to create a target sequence number generator, inherit the MySQL sequence number generator of the Spring Batch framework, and rewrite the inherited getNextKey method in the target sequence number generator to cyclically call the getNextKey method of the MySQL sequence number generator.

[0091] Furthermore, the first creation module 10 further includes:

[0092] A method rewriting unit is used to rewrite the getCacheSize method inherited in the target serial number generator to set the serial number cache quantity to a preset value, wherein the preset value is greater than 1.

[0093] Furthermore, the first creation module 10 further includes:

[0094] The second creation unit is used to create a distributed cache, and the distributed cache is used to generate a sequence number to serve as a target sequence number generator.

[0095] Furthermore, the implementation device of the data processing framework also includes:

[0096] An annotation adding module is used to add preset annotations to the target configurator class to instruct the SpringBatch framework to place the target configurator class into the running environment to replace the default configurator class.

[0097] Among them, the functional implementation of each module in the implementation device of the above-mentioned data processing framework corresponds to the various steps in the implementation method embodiment of the above-mentioned data processing framework, and its functions and implementation processes are no longer repeated here one by one.

[0098] The present invention also provides a computer storage medium on which is stored an implementation program of a data processing framework, and when the implementation program of the data processing framework is executed by a processor, the steps of the implementation method of the data processing framework as described in any of the above embodiments are implemented.

[0099] The specific embodiments of the computer storage medium of the present invention are basically the same as the embodiments of the implementation method of the above-mentioned data processing framework, and will not be described in detail here.

[0100] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0101] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0102] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0103] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for implementing a data processing framework, characterized in that: The implementation method of the data processing framework includes: A target sequence number generator is created based on a preset method, including creating a target sequence number generator, inheriting a MySQL sequence number generator of a SpringBatch framework, and rewriting the inherited getNextKey method in the target sequence number generator to call the getNextKey method of the MySQL sequence number generator in a loop, and rewriting the inherited getCacheSize method in the target sequence number generator to set the number of sequence number caches to a preset value, wherein the preset value is greater than 1; or, creating a distributed cache, wherein the distributed cache is used to generate sequence numbers as a target sequence number generator; Create a target sequence number generator factory class, inherit the default sequence number generator factory class of the Spring Batch framework, and rewrite the inherited getIncrementer method in the target sequence number generator factory class to replace the MySQL sequence number generator in the inherited getIncrementer method with the target sequence number generator; Create a target configurator class, inherit the default configurator class of the Spring Batch framework, and rewrite the inherited createJobRepository method in the target configurator class to specify the serial number generator factory class as the target serial number generator factory class.

2. The method for implementing the data processing framework according to claim 1, characterized in that: The implementation method of the data processing framework also includes: A preset annotation is added to the target configurator class to instruct the Spring Batch framework to place the target configurator class into the operating environment to replace the default configurator class.

3. A device for implementing a data processing framework, characterized in that: The implementation device of the data processing framework includes: A first creation module, used to create a target sequence number generator based on a preset method; The second creation module is used to create a target sequence number generator factory class, inherit the default sequence number generator factory class of the Spring Batch framework, and rewrite the inherited getIncrementer method in the target sequence number generator factory class to replace the MySQL sequence number generator in the inherited getIncrementer method with the target sequence number generator; The third creation module is used to create a target configurator class, inherit the default configurator class of the Spring Batch framework, and rewrite the createJobRepository method inherited in the target configurator class to specify the serial number generator factory class as the target serial number generator factory class; The first creation module includes: a first creation unit, used to create a target sequence number generator, inherit the MySQL sequence number generator of the Spring Batch framework, and rewrite the getNextKey method inherited in the target sequence number generator to cyclically call the getNextKey method of the MySQL sequence number generator; a method rewriting unit, used to rewrite the getCacheSize method inherited in the target sequence number generator to set the sequence number cache quantity to a preset value, wherein the preset value is greater than 1; Or, the first creation module includes: a second creation unit, used to create a distributed cache, and the distributed cache is used to generate a serial number to serve as a target serial number generator.

4. A device for implementing a data processing framework, characterized in that: The implementation device of the data processing framework includes: a memory, a processor, and an implementation program of the data processing framework stored in the memory and executable on the processor. When the implementation program of the data processing framework is executed by the processor, the steps of the implementation method of the data processing framework as described in claim 1 or 2 are implemented.

5. A computer storage medium, characterized in that: The computer storage medium stores an implementation program of the data processing framework, and when the implementation program of the data processing framework is executed by the processor, the steps of the implementation method of the data processing framework as claimed in claim 1 or 2 are implemented.

Images