Micro-service-based distributed batch processing method, system and registry center

Abstract

The application discloses a kind of distributed batch processing method, system and registration center based on microservice, applied to distributed technical field, which comprises: service consumer initiates batch processing request to service registration center when detecting that preset batch processing task trigger condition is satisfied;Service registration center determines the current multiple instructions corresponding to the identification of the current batch processing service according to the identification of the current batch processing service and the relationship between the identification of the multiple batch processing services and the multiple instructions stored in advance;Determine the multiple microservices for processing the current batch processing task according to the service identification of each instruction;According to the data area of each instruction, the corresponding microservice instance is deserialized;By calling microservice instance, the calling instruction is sent to the machine where multiple microservices are distributed for execution.The application can improve the work efficiency of batch processing program running on a single machine and solve the problem of non-scalability.

Description

Technical Field

[0001] This invention relates to the field of distributed technology, and in particular to a distributed batch processing method, system, and registry center based on microservices. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] Currently, traditional batch processing programs typically consist of several business actions executed sequentially, such as data extraction, cleaning, transformation, and loading. These steps usually run on a single machine. When dealing with large amounts of data, single machines are limited by hardware resources such as CPU, memory, and disk, resulting in long running times and difficulties in scaling computing power. For a batch payment transfer business, the batch program reads each record in the payment transfer file in a loop, cleans and transforms each record, extracts the business data, calls the payment transfer processing logic, and completes a payment transfer business. When the payment transfer file contains millions or tens of millions of records, even with a multi-threaded mechanism, single-machine processing time often takes several hours or even more than ten hours. Summary of the Invention

[0004] This invention provides a microservice-based distributed batch processing method to address the problems of long runtime and lack of scalability of batch processing programs on a single machine. The method includes:

[0005] When a service consumer detects that a preset batch processing task trigger condition has been met, it initiates a batch processing request to the service registry center; the batch processing request includes the identifier of the batch processing service that processes the current batch processing task.

[0006] The service registry determines the current batch processing service identifier and the relationships between multiple pre-stored batch processing service identifiers and multiple instructions to identify the current batch processing service identifier and the current multiple instructions. Based on the service identifier of each instruction, it determines multiple microservices to process the current batch processing task. Based on the identifier and data area of ​​each instruction, it deserializes microservice instances. It then calls the microservices, distributing the services across multiple machines for execution. Each instruction includes a service identifier and a data area, and the multiple instructions corresponding to each batch processing service identifier are pre-processed and generated according to the corresponding type of batch processing service processing program.

[0007] The service consumer generates a call instruction based on the identifiers of multiple microservice distributions and sends the call instruction to the machines of the multiple microservice distributions;

[0008] The batch processing task is processed by multiple microservices distributed across machines within the service provider that receive the invocation instruction.

[0009] This invention also provides a microservice-based distributed batch processing method. This method, applied to a service registry, addresses the problems of long runtime and lack of scalability of batch processing programs on a single machine. The method includes:

[0010] Based on the identifier of the current batch processing service and the relationship between the identifiers of multiple pre-stored batch processing services and multiple instructions, the current multiple instructions corresponding to the identifier of the current batch processing service are determined; each instruction includes a service identifier and a data area, and the multiple instructions corresponding to the identifier of each batch processing service are pre-processed and generated according to the corresponding type of batch processing service processing program;

[0011] Based on the service identifier of each current instruction, determine the multiple microservices that will handle the current batch processing task;

[0012] Based on the data area of ​​each current instruction, a microservice instance is deserialized;

[0013] This invention also provides a microservice-based distributed batch processing system to solve the problems of long single-machine running time and lack of scalability in batch processing programs. The system includes:

[0014] A service consumer initiates a batch processing request to the service registry when a preset batch processing task trigger condition is detected; it deserializes microservice instances through the data area, thereby sending the call instruction to multiple machines distributed across the microservices; the batch processing request includes the identifier of the batch processing service that processes the current batch processing task.

[0015] The service registry is used to determine the current batch processing service identifier and the relationships between the identifiers of the current batch processing service and multiple instructions stored in advance; to determine the multiple microservices that process the current batch processing task based on the service identifier of each instruction; and to determine the multiple microservices based on the data area of ​​each instruction. Each instruction includes a service identifier and a data area, and the multiple instructions corresponding to the identifier of each batch processing service are pre-processed and generated according to the corresponding type of batch processing service processing program.

[0016] Multiple microservices distributed across machines within the service provider that receive the invocation instruction are used to process the batch processing task.

[0017] This invention also provides a microservice-based distributed batch processing service registry, comprising:

[0018] The instruction determination unit is used to determine the current multiple instructions corresponding to the identifier of the current batch processing service based on the identifier of the current batch processing service and the relationship between the identifiers of multiple batch processing services and multiple instructions stored in advance; each instruction includes a service identifier and a data area, and the multiple instructions corresponding to the identifier of each batch processing service are pre-processed and generated according to the corresponding type of batch processing service processing program;

[0019] Registration Unit: Multiple service providers can register their services to a registration unit.

[0020] Storage unit, service registry with data area;

[0021] The microservice determination unit is used to determine multiple microservices that will process the current batch processing task based on the service identifier of each current instruction.

[0022] The sending unit is used to send the machine identifiers of multiple microservices to the service consumer;

[0023] The scheduling unit is used for scheduling timed tasks, initiating instruction calls as a service consumer, and handling instruction status transitions and failure / exception handling.

[0024] The encryption / decryption unit is used to encrypt and decrypt sensitive data in the data area.

[0025] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described distributed batch processing method based on microservices.

[0026] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned microservice-based distributed batch processing method.

[0027] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described microservice-based distributed batch processing method.

[0028] In this embodiment of the invention, a distributed batch processing scheme based on microservices proceeds as follows: When a service consumer detects that a preset batch processing task trigger condition is met, it initiates a batch processing request to a service registry center; the batch processing request includes an identifier of the batch processing service that processes the current batch processing task; the service registry center determines the current multiple instructions corresponding to the identifier of the current batch processing service based on the identifier of the current batch processing service and the relationship between the identifiers of multiple batch processing services and multiple instructions stored in advance; it determines multiple microservices that process the current batch processing task based on the service identifier of each current instruction; it deserializes a microservice instance based on the data area of ​​each current instruction; and it invokes the microservice instance through a sending unit; each instruction includes a service identifier and a data area, and the multiple instructions corresponding to the identifier of each batch processing service are pre-processed according to the corresponding type of batch processing service processing program to generate a service consumer that generates an invocation instruction based on the machine identifiers of the multiple microservices and sends the invocation instruction to the machines of the multiple microservices; the machines of the multiple microservices that receive the invocation instruction in the service provider process the batch processing task. In the above process, the embodiments of the present invention enable the service consumer to initiate a batch processing request to the service registry when it detects that the preset batch processing task triggering conditions are met. The service registry determines the current batch processing service identifier and the relationship between the identifiers of multiple batch processing services and multiple instructions stored in advance. Based on the service identifier of each instruction, it determines multiple microservices to process the current batch processing task, thereby improving the efficiency of batch processing task processing and improving the problem of non-scalability. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0030] Figure 1 This is a structural diagram of the distributed batch processing method, system, and registration center based on microservices in this embodiment of the invention;

[0031] Figure 2 This is a schematic diagram of the distributed batch processing method, system, and registration center based on microservices in this embodiment of the invention.

[0032] Figure 3 This is a flowchart of a microservice-based distributed batch processing method in an embodiment of the present invention;

[0033] Figure 4This is a flowchart illustrating the distributed batch processing method, system, and service call of the registry center based on microservices in this embodiment of the invention.

[0034] Figure 5 This is a flowchart of a microservice-based distributed batch processing method applied to a service registry center in an embodiment of the present invention;

[0035] Figure 6 This is a schematic diagram of a microservice-based distributed batch processing system in an embodiment of the present invention;

[0036] Figure 7 This is a schematic diagram of a distributed batch processing service registry based on microservices in an embodiment of the present invention;

[0037] Figure 8 This is a schematic diagram of a distributed batch processing method based on microservices in an embodiment of the present invention;

[0038] Figure 9 This is a schematic diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0040] Before introducing the embodiments of the present invention, the terminology involved in the embodiments of the present invention will be introduced first.

[0041] 1. ETL: It is an abbreviation for Extract-Transform-Load, which describes the process of extracting, transforming, and loading data from a source to a target system.

[0042] 2. Standalone program: A program that can only run on a single computer.

[0043] 3. Batch processing: also known as batch processing, batch jobs, or batch programs; as the name suggests, batch processing involves performing a series of operations on a single object, such as extracting, transforming, loading, calculating, and outputting data from a data file.

[0044] 4. Task: The basic unit of work performed by a computer, which is one or more sequences of instructions processed by a control program.

[0045] 5. Instructions: The instructions and commands that direct a computer to work. A program is a series of instructions arranged in a certain order. Instructions can process input data and return the results.

[0046] 6. Distributed system: Built on a computer network, it consists of multiple independently operating computers, presenting a unified whole to the user.

[0047] 7. Distributed computing: Using distributed systems to solve computational problems, a problem is divided into several tasks, and each task can be completed by one or more computers.

[0048] 8. Microservices: A service-oriented distributed architecture style that advocates dividing large applications into a series of small services, each focusing on its own single business function, running in an independent process, and having clear service boundaries; using lightweight communication mechanisms (HTTP / RPC) to achieve interoperability and collaboration.

[0049] 9. Microservice framework: Provides infrastructure functions for microservice operation, such as service discovery, service registration, service invocation, service communication, and service execution. At the same time, the microservice framework provides service load balancing to deploy multiple instances of a service on multiple machines. The microservice framework makes it easier for developers to develop and deploy microservice programs; commonly used microservice frameworks include Spring Cloud and Dubbo.

[0050] 10. Main control service: Controls the execution of the main service according to the parameter configuration.

[0051] 11. Main Service: A specific batch processing task. The main service consists of one or more instruction services.

[0052] 12. Instruction Service: The service that executes instructions. It is the smallest unit of execution in distributed computing and can be executed on one or more machines.

[0053] 13. Multithreading: A thread is a program segment that can run independently. Multithreading refers to the implementation of multiple threads executing concurrently through software or hardware. The operating system divides the processor's running time into several time periods and then allocates these time periods to each thread for execution. However, at any given moment, only one program is running on the processor.

[0054] 14. Concurrent processing: refers to a situation where multiple programs are in the process of starting and running within a certain time period. The operating system divides the processor's running time into several time periods and then allocates these time periods to each program for execution. However, at any given moment, only one program is running on the processor, while the other programs are in a suspended state.

[0055] 15. Serialization: The process of converting an object into bytes or a string is called object serialization.

[0056] 16. Deserialization: The process of restoring bytes or strings back to an object is called object deserialization.

[0057] Figure 1 This is a structural diagram of the distributed batch processing method, system, and registry center based on microservices in this embodiment of the invention. Figure 2 This is a schematic diagram of the distributed batch processing method, system, and registry center based on microservices in this embodiment of the invention. It can be seen that in this embodiment, when a service consumer detects that a preset batch processing task trigger condition is met, it initiates a batch processing request to the service registry center. The service registry center determines the current batch processing service identifier and the relationships between multiple pre-stored batch processing service identifiers and multiple instructions to determine the current multiple instructions corresponding to the current batch processing service identifier. Based on the service identifier of each current instruction, it determines multiple microservices to process the current batch processing task, thereby improving the efficiency of the batch processing program running on a single machine and addressing the issue of scalability.

[0058] Figure 3 This is a flowchart of a microservice-based distributed batch processing method in an embodiment of the present invention, including:

[0059] Step 301: When the service consumer detects that the preset batch processing task triggering conditions are met, it initiates a batch processing request to the service registry center; the batch processing request includes the identifier of the batch processing service that processes the current batch processing task.

[0060] Step 302: The service registry determines the current batch processing service identifier corresponding to the current batch processing service identifier based on the identifier of the current batch processing service and the relationship between the identifiers of multiple batch processing services and multiple instructions stored in advance; it determines multiple microservices to process the current batch processing task based on the service identifier of each instruction; and it deserializes the microservice instance based on the data area of ​​each instruction. Each instruction includes a service identifier and a data area, and the multiple instructions corresponding to the identifier of each batch processing service are pre-processed and generated according to the corresponding type of batch processing service processing program.

[0061] Step 303: The service consumer calls the microservice instance and distributes the instructions across multiple microservice machines for execution;

[0062] Step 304: The microservice that the service provider receives the invocation instruction processes the batch processing task.

[0063] The distributed batch processing method based on microservices provided in this embodiment of the invention operates as follows: Service consumer ( Figure 1When the application layer (mentioned in the text) detects that the preset batch processing task triggering conditions are met, it initiates a batch processing request to the service registry center. The batch processing request includes the identifier of the batch processing service that processes the current batch processing task. The service registry center determines the current batch processing service identifier corresponding to the current batch processing service identifier based on the identifier of the current batch processing service and the relationship between the identifiers of multiple batch processing services and multiple instructions stored in advance (this step is the content executed by the main control service mentioned below, determining the invocation of the main service, such as the batch payment main service). Based on the service identifier of each current instruction, it determines the multiple microservices that process the current batch processing task. Based on the data area of ​​each current instruction, it determines the machine identifiers distributed among the multiple microservices (this step is the content executed by the instruction service mentioned below). It sends the machine identifiers distributed among the multiple microservices to the service consumer. Each instruction includes a service identifier and a data area. The multiple instructions corresponding to the identifier of each batch processing service are pre-processed and generated according to the corresponding type of batch processing service processing program. The service consumer (corresponding to Figure 1 The service layer in the process generates invocation instructions based on the machine identifiers of multiple microservices and sends the invocation instructions to the machines distributed across the microservices. The machines in the service provider that receive the invocation instructions process the batch processing tasks. In this process, the embodiment of the invention, when a service consumer detects that a preset batch processing task trigger condition is met, initiates a batch processing request to the service registry. The service registry, based on the identifier of the current batch processing service and the pre-stored relationship between the identifiers of multiple batch processing services and multiple instructions, determines the current multiple instructions corresponding to the identifier of the current batch processing service. Based on the service identifier of each current instruction, it determines the multiple microservices that will process the current batch processing task, thereby improving the efficiency of batch processing programs running on a single machine and addressing the issue of scalability.

[0064] This invention employs microservice distributed computing technology. First, a batch processing problem is extracted into several services and instructions. The services perform business actions and save the results as instructions (each instruction consists of a data area and a service identifier). These instructions can be parsed by the instruction service in the batch scheduling module. Based on the service identifier, the service instance is located in the service registry. Through serialization and deserialization, the data area is transformed into the program's data structure, which is then passed as parameters to the service, thereby completing the execution of the instructions.

[0065] Each step is explained in detail below:

[0066] In step 301, when the service consumer detects that the preset batch processing task triggering conditions are met, it initiates a batch processing request to the service registry center; the batch processing request includes the identifier of the batch processing service that processes the current batch processing task.

[0067] In one embodiment, such as Figure 2 As shown by the first two arrows, the above-mentioned microservice-based distributed batch processing method may further include: preprocessing each type of batch processing service to generate an identifier for each batch processing service and its corresponding multiple instructions according to the following method:

[0068] Obtain the batch processing procedure for each type of batch processing service;

[0069] Each type of batch processing service is converted into a batch processing service; each batch processing service corresponds to multiple instructions.

[0070] Each batch processing service is broken down into multiple microservices; these microservices are distributed and deployed on multiple machines at the service provider, with each microservice corresponding to one instruction.

[0071] The identifier of each batch of processing services and its corresponding multiple instructions are stored in the service registry center.

[0072] In one embodiment, the service registry has a data area for storing instruction data, including the instruction identifier and instruction data (including status, business data, timestamps, etc.). The service registry may have one or more data areas, and the data can be distributed and stored on different partitions or machines. At the same time, in order to protect data security, sensitive data in the data area can be encrypted and decrypted by an encryption and decryption unit, such as: sensitive data of the service registry (instruction identifier, microservice name, IP address, system username, password, etc.), sensitive data of the data area (username, home address, telephone number, bank card number, etc.).

[0073] In step 302, the service registry determines the current batch processing service identifier corresponding to the current batch processing service identifier based on the identifier of the current batch processing service and the relationship between the identifiers of multiple batch processing services and multiple instructions stored in advance; it determines multiple microservices to process the current batch processing task based on the service identifier of each current instruction; and it deserializes microservice instances based on the data area of ​​each current instruction; each instruction includes a service identifier and a data area, and the multiple instructions corresponding to the identifier of each batch processing service are pre-processed and generated according to the corresponding type of batch processing service processing program.

[0074] In one embodiment, based on the data area of ​​each current instruction, the data area is transformed into the program's data structure through serialization and deserialization to generate a microservice instance.

[0075] In step 303, the service consumer generates a call instruction based on the machine identifiers of the multiple microservice distributions and sends the call instruction to the machines of the multiple microservice distributions.

[0076] In one embodiment, the service consumer generates a call instruction based on the machine identifiers of the multiple microservices and sends the call instruction to the machines of the multiple microservices. This includes: the service consumer determines whether to send the call instruction to the machines of the multiple microservices based on the number of multiple microservices and the current running status of each machine in the service consumer.

[0077] In step 304, the batch processing task is processed by machines distributed across multiple microservices in the service provider that have received the invocation instruction.

[0078] In a specific embodiment, in a microservice architecture, services and instructions are executed by a single computer in the cluster, and all services and instructions are distributed across all machines in the cluster for execution, thereby solving the problems of long batch processing time and single point of failure on a single machine. For example, in a batch payment transfer business, the process is first extracted into a payment transfer data processing service, a payment transfer instruction, and a payment transfer service. The payment transfer data processing service reads each record in the payment transfer file in a loop, cleans and transforms each record, and converts the data into a payment transfer instruction (consisting of a data area and a payment transfer service identifier). The instruction service in the batch scheduling module finds the payment transfer service based on the service identifier of the instruction, converts the data area into a program data structure as a parameter to call the payment transfer service, and completes a payment transfer business.

[0079] Figure 4 This is a flowchart illustrating the distributed batch processing method, system, and service call of the registry center based on microservices in this embodiment of the invention. Specifically, the distributed batch scheduling method based on microservices includes:

[0080] The batch scheduling module is extracted into three components: a master control service, a master service, and a command service. Multiple instances of each service are registered through the service registry of the microservice framework for distributed processing.

[0081] 1. The task scheduling module is the triggering and calling mechanism of the batch scheduling module. Taking time scheduling as an example: it executes a timed trigger to call the main control service once every minute.

[0082] 2. Batch Scheduling Module: The main control service reads parameters and determines which main services to execute based on the parameter settings and the current service running status, and concurrently calls the main services that meet the conditions.

[0083] 3. The main service executes, reads parameters, and decides whether to execute the instruction service based on the parameter settings and the current service running status. Based on the concurrency parameters, it concurrently calls the instruction service in an asynchronous manner.

[0084] 4. Command service execution: Read and parse the command, find the service instance in the service registry based on the service identifier, transform the data area into a microservice instance through deserialization, and complete the execution of the command by calling the microservice.

[0085] Figure 5 This is a flowchart of a microservice-based distributed batch processing method applied to a service registry center in an embodiment of the present invention, such as... Figure 5 As shown, the method includes the following steps:

[0086] Step 501: Based on the identifier of the current batch processing service and the relationship between the identifiers of multiple batch processing services and multiple instructions stored in advance, determine the current multiple instructions corresponding to the identifier of the current batch processing service; each instruction includes a service identifier and a data area, and the multiple instructions corresponding to the identifier of each batch processing service are pre-processed and generated according to the corresponding type of batch processing service processing program.

[0087] Step 502: Determine multiple microservices to process the current batch processing task based on the service identifier of each current instruction.

[0088] Step 503: Deserialize the microservice instance based on the data in the data area of ​​each current instruction;

[0089] Step 504: Invoke the microservice instance and send the instructions to different service provider machines for execution.

[0090] In a specific embodiment, traditional batch processing is abstracted and extracted to obtain services and instructions, wherein:

[0091] 1. Service: A single business function. In a microservice architecture, a large application is divided into a series of small services.

[0092] 2. Batch Service: A single batch business function that processes data into instructions and saves them through steps such as extraction, cleaning, transformation, and loading; Batch Service can be invoked and executed by the instruction service in the batch service scheduling module.

[0093] 3. Command: Consists of service identifier + data area:

[0094] a) Data area: This can be a piece of XML or JSON to describe data. Through the serialization and deserialization mechanism of the programming language, this XML or JSON can be converted into an implementation class of the programming language and called for execution.

[0095] b) Service Identifier: A string can uniquely identify a service in the system. In a microservice framework, the service identifier and service instance are registered with the service registry. One service identifier can correspond to multiple instances distributed on different machines. Other microservices can find an implementation of the service in the registry through the service identifier and make a call, thereby completing the distributed service call.

[0096] The instruction can be parsed by the instruction service in the batch scheduling module. Based on the service identifier, the service instance is found in the service registry. The data area is transformed into a microservice instance through serialization and deserialization, and the instance is called to complete the execution of the instruction.

[0097] This invention decouples traditional batch processing programs by extracting services and instructions. By registering multiple instances of a service in a microservice registry, it transforms a single-machine program into distributed computing (deploying a service across multiple machines, finding an instance in the registry using the service identifier, and then invoking the service).

[0098] 4. Concurrent execution of instructions: This invention uses a microservice asynchronous callback mechanism for concurrent execution. The main service reads the parameter of the number of concurrent requests, controls the number of concurrent requests, and uses an asynchronous method to call up the instruction service.

[0099] 5. Instruction storage: Using database partitioning technology, an instruction table is divided into several partitions to ensure instruction reading speed. For example, an instruction table with ten million records can be partitioned into one hundred partitions, with each partition containing only one hundred thousand records, greatly improving access speed. Alternatively, database sharding technology can be used to store the instruction table in multiple databases; or in-memory technology can be used to directly store the instruction package in a distributed memory cluster.

[0100] Meanwhile, this invention also uses database partitions as concurrency. First, the main service is configured according to the number of partitions in the instruction table, and the main control service in the batch scheduling module starts the main service according to the concurrency parameters.

[0101] This invention also proposes a microservice-based distributed batch processing system, the principle of which is similar to the microservice-based distributed batch processing method, and will not be described in detail here. Figure 6 This is a schematic diagram of a microservice-based distributed batch processing system according to an embodiment of the present invention. The system includes:

[0102] Service consumer 601 is used to initiate a batch processing request to the service registry when a preset batch processing task trigger condition is detected; generate a call instruction based on the machine identifiers of multiple microservices and send the call instruction to the machines of multiple microservices; the batch processing request includes the identifier of the batch processing service that processes the current batch processing task.

[0103] Service registration center 602 is used to determine the current multiple instructions corresponding to the identifier of the current batch processing service based on the identifier of the current batch processing service and the relationship between the identifiers of multiple batch processing services and multiple instructions stored in advance; determine multiple microservices for processing the current batch processing task based on the service identifier of each current instruction; and deserialize microservice instances based on the data area of ​​each current instruction; each instruction includes a service identifier and a data area, and the multiple instructions corresponding to the identifier of each batch processing service are pre-processed and generated according to the corresponding type of batch processing service processing program;

[0104] The service provider 603 receives the invocation instruction and processes the batch processing task.

[0105] In one embodiment, a preprocessing device is further included for preprocessing each type of batch processing service procedure to generate an identifier for each batch processing service and its corresponding multiple instructions according to the following method:

[0106] Obtain the batch processing procedure for each type of batch processing service;

[0107] Each type of batch processing service is converted into a batch processing service; each batch processing service corresponds to multiple instructions.

[0108] Each batch processing service is broken down into multiple microservices; these microservices are distributed and deployed on multiple machines at the service provider, with each microservice corresponding to one instruction.

[0109] The identifier of each batch of processing services and its corresponding multiple instructions are stored in the service registry center.

[0110] In one embodiment, the service registry 602 is specifically used to: determine the current multiple instructions corresponding to the identifier of the current batch processing service based on the identifier of the current batch processing service and the relationship between the identifiers of multiple batch processing services and multiple instructions stored in advance; determine multiple microservices for processing the current batch processing task based on the service identifier of each current instruction; deserialize microservice instances based on the data area of ​​each current instruction; and call microservice instances to distribute instructions to multiple machines for execution.

[0111] In one embodiment, the service consumer 601 is specifically configured to: determine whether to send the invocation instruction to the machines distributed across the multiple microservices based on the number of multiple microservices and the current running status of each machine in the service consumer.

[0112] This invention also proposes a microservice-based distributed batch processing service registry, the principle of which is similar to the microservice-based distributed batch processing method, and will not be described in detail here. Figure 7This is a schematic diagram of a microservice-based distributed batch processing service registry in an embodiment of the present invention. The registry includes:

[0113] The instruction determination unit 701 is used to determine the current multiple instructions corresponding to the identifier of the current batch processing service based on the identifier of the current batch processing service and the relationship between the identifiers of multiple batch processing services and multiple instructions stored in advance; each instruction includes a service identifier and a data area, and the multiple instructions corresponding to the identifier of each batch processing service are pre-processed and generated according to the corresponding type of batch processing service processing program;

[0114] Registration unit 702 allows multiple service providers to register their services with the registration unit.

[0115] Storage unit 703, the service registry center has a data area;

[0116] The microservice determination unit 704 is used to determine multiple microservices that will process the current batch processing task based on the service identifier of each current instruction.

[0117] Sending unit 705 is used to send the machine identifiers of multiple microservices to the service consumer;

[0118] The scheduling unit 706 is used for scheduled tasks, initiating instruction calls as a service consumer, and handling instruction status transitions and failure / exception handling.

[0119] The encryption / decryption unit 707 is used to encrypt and decrypt sensitive data in the data area.

[0120] In one embodiment, the microservice determination unit 702 is specifically used to: transform the data area into a microservice instance through serialization and deserialization.

[0121] In specific embodiments, such as Figure 8 As shown, the distributed batch processing method based on microservices includes the following modules:

[0122] 1. The task scheduling module 801 is used to proactively invoke the main control service in the distributed batch scheduling module according to certain logic. That is, it initiates a batch processing request when the preset batch processing task triggering conditions are met. For example: based on a time expression, the main control service in the distributed batch scheduling module is invoked at regular intervals; based on message triggering, the listening program invokes the main control service upon receiving a message; based on manual triggering, business personnel invoke the main control service by clicking on the interface. This task scheduling module implements the above step 301.

[0123] 2. Batch Service Scheduling Module 802: Used to provide master control service, master service, and instruction service.

[0124] a) Main control service: Reads configuration parameters, current running status, and concurrency parameters to determine whether to start the main service. If the conditions are not met, the main control service will exit directly and wait for the next call. If the conditions are met, the main control service will start the main service and determine how many main services to run at the same time based on the concurrency parameters.

[0125] b) Main service: Reads configuration parameters, current running status, and concurrency parameters to determine whether to invoke the instruction service, and controls how many instruction services to run simultaneously based on the concurrency parameters;

[0126] c) Command Service: Parses the command, finds the service instance in the service registry based on the service identifier, and converts the data area into the program's data structure through serialization and deserialization to use as parameters to invoke the service instance and complete the execution of the command.

[0127] The batch service scheduling module described above implements steps 302-304.

[0128] 3. Batch Module 803: This module executes batch business steps and saves the results as instructions. Each batch processing procedure is extracted as a service, and multiple instances are deployed based on a microservice framework. This task scheduling module implements the steps described above of pre-processing each type of batch processing service to generate an identifier for each batch processing service and its corresponding multiple instructions.

[0129] In specific embodiments, the microservice-based distributed batch processing method includes:

[0130] By abstracting a batch processing program into batch services and instructions, a batch program that executes sequentially on a single machine is transformed into a microservice that can be processed in a distributed manner.

[0131] 1. Batch Service: A single batch business function that processes data into instructions and saves them through a series of extraction, transformation, and loading steps.

[0132] 2. Instruction: An instruction is an executable program unit consisting of a data area and a service identifier; instructions can be parsed and executed by the instruction service.

[0133] a) Data area: This can be an XML or JSON string used to describe data. Through the serialization and deserialization mechanisms of programming languages, this XML or JSON can be converted into an implementation class of the programming language to execute business logic processing.

[0134] b) Service Identifier: A string can uniquely identify a service in the system. In a microservice framework, the service identifier and service instance are registered with the service registry. One service identifier can correspond to multiple instances distributed on different machines. Other microservices can find an implementation of the service in the registry through the service identifier and make a call, thereby completing the distributed service call.

[0135] The embodiments of this invention employ a microservice-based distributed solution. Through the service registry mechanism of microservices, it automatically completes service subscription and publication, service routing, cluster fault tolerance, multi-instance deployment, and service invocation. However, existing technologies have the following drawbacks:

[0136] 1. Based on socket communication, instructions are sent to different machines for distributed execution. This method involves complex communication and response processing between servers. In addition, socket communication requires both parties to agree on the message format for each transaction, and both parties parse the message according to the agreed message format. This results in low development efficiency and difficult system maintenance. Both parties also need to maintain communication addresses and ports, making configuration management and maintenance difficult.

[0137] 2. Remote access technologies based on programming languages, such as RPC calls and RMI calls; this method shields the underlying communication format of sockets, provides more advanced object serialization and deserialization, and does not require agreement on message format or parsing of messages; however, both communicating parties still need to maintain communication addresses and ports, making configuration, management, and maintenance difficult; as the number of machines increases, there is no automatic service subscription and publishing mechanism, resulting in poor service reliability.

[0138] 3. Multi-threaded solutions: Traditional batch processing programs often use multi-threading to process processing tasks in parallel. However, multi-threading is limited by machine resources such as CPU, memory, and disk I / O. Once the machine resource bottleneck is reached, it cannot be expanded, and the performance is limited.

[0139] 4. Traditional batch scheduling frameworks, such as Spring Task and Quartz, can also perform cluster scheduling. If a node in the cluster fails, the task can be transferred to another machine. However, they do not support distributed jobs. Batch jobs can only be processed on a single machine, which is limited by the resources of a single machine and has a performance bottleneck.

[0140] In summary, in the method proposed in this embodiment of the invention, when a service consumer detects that a preset batch processing task triggering condition is met, it initiates a batch processing request to the service registry. The batch processing request includes an identifier of the batch processing service that processes the current batch processing task. The service registry determines the current multiple instructions corresponding to the identifier of the current batch processing service based on the identifier of the current batch processing service and the relationship between the identifiers of multiple batch processing services and multiple instructions stored in advance. Based on the service identifier of each current instruction, it determines multiple microservices that process the current batch processing task. Based on the data area of ​​each current instruction, it determines the machine identifiers of the distribution of the multiple microservices. The machine identifiers of the distribution of the multiple microservices are sent to the service consumer. Each instruction includes a service identifier and a data area. The multiple instructions corresponding to the identifier of each batch processing service are pre-processed according to the corresponding type of batch processing service processing program. The service consumer generates a calling instruction based on the machine identifiers of the distribution of the multiple microservices and sends the calling instruction to the machines of the distribution of the multiple microservices. The machines of the distribution of the multiple microservices in the service provider that receive the calling instruction process the batch processing task. In the above process, the embodiments of the present invention initiate a batch processing request to the service registry when the service consumer detects that the preset batch processing task triggering conditions are met. The service registry determines the current batch processing service identifier and the relationship between the identifiers of multiple batch processing services and multiple instructions stored in advance. Based on the service identifier of each instruction, it determines multiple microservices to process the current batch processing task, thereby solving the problem that batch processing programs have long running time on a single machine and cannot be scaled.

[0141] This invention also provides a computer device. Figure 9 This is a schematic diagram of a computer device in an embodiment of the present invention. The computer device 900 includes a memory 910, a processor 920, and a computer program 930 stored in the memory 910 and executable on the processor 920. When the processor 920 executes the computer program 930, it implements the aforementioned microservice-based distributed batch processing method. This embodiment of the present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the aforementioned microservice-based distributed batch processing method.

[0142] The beneficial effects of the embodiments of the present invention are as follows:

[0143] 1. High performance: Microservice-based distributed batch processing distributes batch processing tasks to each point in the distributed cluster for parallel processing, breaking through the resource limitations of a single machine in high concurrency and possessing better parallel processing capabilities.

[0144] 2. High reliability: Based on microservices, distributed batch processing distributes batch processing tasks to each point in the distributed cluster, avoiding single points of failure. If one batch server goes down, other machines can automatically take over its tasks and continue to complete the batch scheduling tasks.

[0145] 3. High cost-effectiveness, based on microservices for distributed batch processing, the distributed cluster uses PC servers, and the machine price is much lower than that of mainframes and minicomputers;

[0146] 4. High scalability, distributed cluster: computing power is increased by adding PC servers, which is low-cost and highly scalable, while mainframes or minicomputers cannot expand computing power when they reach the single-machine resource bottleneck.

[0147] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned microservice-based distributed batch processing method.

[0148] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described microservice-based distributed batch processing method.

[0149] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0153] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A distributed batch processing method based on microservices, characterized in that, include: When a service consumer detects that a preset batch processing task trigger condition has been met, it initiates a batch processing request to the service registry center; the batch processing request includes the identifier of the batch processing service that processes the current batch processing task. The service registry determines the current batch service identifier and the current multiple instructions corresponding to the current batch service identifier based on the identifier of the current batch service and the relationship between the identifiers of multiple batch services and multiple instructions stored in advance. Based on the service identifier of each current instruction, determine the multiple microservices that will handle the current batch processing task; Based on the data area of ​​each current instruction, a microservice instance is deserialized; calling the microservice instance can distribute the instructions to multiple machines for execution; each instruction includes a service identifier and a data area, and multiple instructions corresponding to the identifier of each batch processing service are pre-processed and generated according to the corresponding type of batch processing service processing program; Service consumers send instructions to multiple distributed machines for execution by invoking microservice instances; Upon receiving the invocation instruction, the service provider processes the batch task on machines distributed across multiple microservice instances.

2. The method as described in claim 1, characterized in that, The service registry has a data area for storing instruction data, including instruction identifiers and instruction data (including status, business data, timestamps, etc.). The service registry can have one or more data areas, and the data can be distributed across different partitions or machines. In order to protect data security, sensitive data in the data area can be encrypted and decrypted through encryption and decryption units, such as sensitive data in the service registry (instruction identifier, microservice name, IP address, system username, password, etc.) and sensitive data in the data area (username, home address, telephone number, bank card number, etc.).

3. The method as described in claim 1, characterized in that, It also includes: pre-processing each type of batch processing service procedure to generate an identifier for each batch processing service and its corresponding multiple instructions according to the following method: Obtain the batch processing procedure for each type of batch processing service; Each type of batch processing service is converted into a batch processing service; each batch processing service corresponds to multiple instructions. Each batch processing service is broken down into multiple microservices; Multiple microservices are deployed in a distributed manner on multiple machines at the service provider, and each microservice corresponds to one instruction; The identifier of each batch of processing services and its corresponding multiple instructions are stored in the service registry center.

4. The method as described in claim 1, characterized in that, Based on the data area of ​​each current instruction, the microservice instance is determined by serializing and deserializing the data area into the program's data structure.

5. The method as described in claim 1, characterized in that, Multiple service providers register microservices with the service registry. Service consumers obtain microservices from the service registry based on the instruction identifier, deserialize microservice instances based on the instruction data area, and send the call instructions to multiple machines distributed across the microservices for execution through the sending unit. This includes: the service consumer determining whether to send the call instructions to multiple machines distributed across the microservices based on the number of microservices and the current running status of each machine in the service consumer.

6. A distributed batch processing method based on microservices, characterized in that, This method is applied to a service registry and includes: Based on the identifier of the current batch processing service and the relationship between the identifiers of multiple pre-stored batch processing services and multiple instructions, the current multiple instructions corresponding to the identifier of the current batch processing service are determined; each instruction includes a service identifier and a data area, and the multiple instructions corresponding to the identifier of each batch processing service are pre-processed and generated according to the corresponding type of batch processing service processing program; Based on the service identifier of each current instruction, determine the multiple microservices that will handle the current batch processing task; Deserialize the microservice instance based on the data area of ​​each current instruction; Service consumers can distribute instructions across different machines for execution by calling this microservice.

7. The method as described in claim 6, characterized in that, Based on the data area of ​​each current instruction, the data area is transformed into a data structure of the program through serialization and deserialization to instantiate a microservice instance.

8. A microservice-based distributed batch processing system, characterized in that, include: Service consumers are used to initiate batch processing requests to the service registry when a preset batch processing task trigger condition is detected. The microservice instance is deserialized in the data area, thereby sending the call command to multiple machines distributed across the microservices; the batch request includes the identifier of the batch service that processes the current batch task. The service registry is used to determine the current batch service identifier and the current multiple instructions corresponding to the current batch service identifier, based on the identifier of the current batch service and the relationship between the identifiers of multiple batch services and multiple instructions stored in advance. Based on the service identifier of each current instruction, determine the multiple microservices that will handle the current batch processing task; Based on the data area of ​​each current instruction, determine the multiple microservices; Each instruction includes a service identifier and a data area. Multiple instructions corresponding to the identifier of each batch processing service are pre-processed and generated according to the corresponding type of batch processing service processing program. Multiple microservices distributed across machines within the service provider that receive the invocation instruction are used to process the batch processing task.

9. The system as described in claim 8, characterized in that, It also includes a preprocessing unit for preprocessing each type of batch processing service procedure to generate an identifier for each batch processing service and its corresponding multiple instructions in the following manner: Obtain the batch processing procedure for each type of batch processing service; Each type of batch processing service is converted into a batch processing service; each batch processing service corresponds to multiple instructions. Each batch processing service is broken down into multiple microservices; Multiple microservices are deployed in a distributed manner on multiple machines at the service provider, and each microservice corresponds to one instruction; The identifier of each batch of processing services and its corresponding multiple instructions are stored in the service registry center.

10. The system as described in claim 8, characterized in that, The service registry is specifically used to: transform the data area into the program's data structure through serialization and deserialization to determine microservices.

11. The system as described in claim 8, characterized in that, The service consumer is specifically used to determine whether to send the call instruction to the machines distributed across the multiple microservices, based on the number of microservices and the current running status of each machine in the service consumer.

12. A distributed batch processing service registry based on microservices, characterized in that, include: The instruction determination unit is used to determine the current multiple instructions corresponding to the identifier of the current batch processing service based on the identifier of the current batch processing service and the relationship between the identifiers of multiple batch processing services and multiple instructions stored in advance; each instruction includes a service identifier and a data area, and the multiple instructions corresponding to the identifier of each batch processing service are pre-processed and generated according to the corresponding type of batch processing service processing program; Registration Unit: Multiple service providers can register their services to a registration unit. Storage unit, service registry with data area; The microservice determination unit is used to determine multiple microservices that will process the current batch processing task based on the service identifier of each current instruction. The sending unit is used to send the machine identifiers of multiple microservices to the service consumer; The scheduling unit is used for scheduling timed tasks, initiating instruction calls as a service consumer, and handling instruction status transitions and failure / exception handling. The encryption / decryption unit is used to encrypt and decrypt sensitive data in the data area.

13. The service registration center as described in claim 12, characterized in that, A microservice instance is defined by a data area (one or more) that is serialized and deserialized into a program's data structure. At the same time, multiple service providers can register their services with the service registry, and service consumers can find the service providers through the service registry.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

16. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images