A multi-task allocation batch processing method and system based on pre-calculation

Abstract

The present invention relates to a multi-task allocation batch processing method and system based on pre-calculation, including identifying whether a user request belongs to an online request or a batch processing request, and adjusting the task allocation for batch processing using a pre-calculation method according to the type, using job resource pre-calculation to provide a job pre-allocation strategy to improve job allocation efficiency; using dynamic configuration to complete real-time monitoring of application jobs, role conversion, application online and offline, and application downgrade, to achieve security assurance of external services after hybrid deployment of applications. By applying the present invention, the utilization efficiency of application servers can be improved, project costs can be reduced, and dynamic adjustment of application roles can be supported to reduce operation and maintenance pressure.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed architecture and automatic program design, and in particular to a multi-task allocation batch processing method and system based on pre-calculation. Background Art

[0002] With the development of information technology, the system development model has evolved from the original traditional monolithic architecture to a distributed architecture, and then to a SOA service-oriented architecture, a microservice architecture model, and even the latest grid service architecture model.

[0003] At present, applications in production environments are generally divided into two categories: those that provide online processing services and those that provide batch processing services. Application deployment is divided into online applications and batch applications for security isolation, so as to avoid mutual impact between batch and online operations.

[0004] If it is based on resource sharing or cost saving considerations, then ideally batch processing services need to be implemented on current online applications. This requires providing a mechanism to maximize resource sharing and save project costs while ensuring the safe provision of services.

[0005] In order to deal with this problem, the existing technology proposes to use timed task frameworks such as quartz and timer, as well as distributed task batch processing platforms such as XXL-job, elastic-job, and alibabaCloud Scheduler. By managing and batching timed tasks in a cluster and adopting distributed deployment, the high availability of the system is guaranteed, fault tolerance is improved, and it is ensured that the timed task is only executed on a certain node in the cluster, or how a task is split into multiple independent task items, which are executed separately by distributed machines.

[0006] However, the existing scheduled task framework and distributed task batch processing platform cannot realize the dynamic conversion of application roles as the task is executed, and can only complete simple task execution or task segmentation execution. Summary of the invention

[0007] In order to solve the deficiencies of the prior art, the present invention proposes a multi-task allocation batch processing method and system based on pre-calculation, which uses job resource pre-calculation to provide job pre-allocation strategy to improve job allocation efficiency; uses dynamic configuration to complete real-time monitoring of application jobs, role conversion, application online and offline, and application downgrade, and realizes the security guarantee of external services after application hybrid deployment. By applying the present invention, the utilization efficiency of application servers can be improved, project costs can be reduced, and dynamic adjustment of application roles can be supported to reduce operation and maintenance pressure.

[0008] To achieve the above objectives, the technical solutions adopted by the present invention include:

[0009] A multi-task allocation batch processing method based on pre-calculation, characterized by comprising:

[0010] S1. Receive a user request and determine whether the user request is an online request or a batch service request;

[0011] S2. When the user request is an online request, the user request is directly forwarded to the application for service processing, and the processing result is fed back;

[0012] S3. When the user request belongs to a batch service request, perform resource pre-calculation;

[0013] S4. Adjust the routable state of each application according to the result of resource pre-calculation;

[0014] S5. Send the user request to the corresponding application for service processing according to the routable status of each application, and feedback the processing result.

[0015] Furthermore, the step S2 comprises:

[0016] Get the list of application services;

[0017] Identify the applications marked in the normal state in the application service list;

[0018] Forward user requests to any application in normal state for service processing.

[0019] Furthermore, the step S2 further includes:

[0020] Change the mark of the app that receives the user request to In Use in the App Service List.

[0021] Furthermore, the step S3 comprises:

[0022] Identify batch transactions corresponding to user requests;

[0023] Calculate the average CPU usage, memory consumption, and total time consumption expected for the batch transaction to obtain the resource pre-calculation results.

[0024] Furthermore, the step S4 comprises:

[0025] The health status of each application in the application service list is determined according to the result of resource pre-calculation, and n applications with the best health status are selected to be adjusted to the routable state, where n is a preset positive integer.

[0026] Furthermore, the step S5 further includes:

[0027] When the application is performing service processing, the health status of the application is monitored using heartbeat information.

[0028] Furthermore, the step S5 further includes:

[0029] Setting a first degradation threshold and a second degradation threshold, wherein the set value of the first degradation threshold is greater than the second degradation threshold;

[0030] When the remaining resources of the application are lower than the first degradation threshold, the health status level of the application is reduced from green to yellow, and the service flow of the application is limited according to the preset policy;

[0031] When the remaining resources of the application are lower than the second degradation threshold, the health status level of the application is reduced from green or yellow to red, and the service of the application is stopped and the application is taken offline from the application service list.

[0032] The present invention also relates to a multi-task allocation batch processing system based on pre-calculation, characterized in that it comprises:

[0033] A user request determination module, used for receiving a user request and determining whether the user request is an online request or a batch service request;

[0034] A resource pre-calculation module, used for performing resource pre-calculation;

[0035] The gateway is used to send user requests to the corresponding application and receive processing feedback from the application;

[0036] Service registration module, used to manage the application service list;

[0037] The task processing module is used to provide applications to process user requests.

[0038] The present invention also relates to a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and the computer program implements the above method when executed by a processor.

[0039] The present invention also relates to an electronic device, characterized in that it comprises a processor and a memory;

[0040] The memory is used to store user requests and application service lists;

[0041] The processor is used to execute the above method by calling the user request and the application service list.

[0042] The present invention also relates to a computer program product, comprising a computer program and / or instructions, characterized in that the computer program and / or instructions implement the steps of the above method when executed by a processor.

[0043] The beneficial effects of the present invention are:

[0044] The multi-task allocation batch processing method and system based on pre-calculation described in the present invention is adopted to provide a job pre-allocation strategy by using job resource pre-calculation to improve the efficiency of job allocation; the real-time monitoring, role conversion, application online and offline, and application downgrade of application jobs are completed by using dynamic configuration to achieve the security guarantee of external services after mixed deployment of applications. By applying the present invention, the utilization efficiency of application servers can be improved, project costs can be reduced, and dynamic adjustment of application roles can be supported to reduce the pressure of operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The present invention is a flowchart of a multi-task allocation batch processing method based on pre-calculation.

[0046] Figure 2 The schematic diagram of the structure of the multi-task allocation batch processing system based on pre-calculation of the present invention is shown. DETAILED DESCRIPTION

[0047] In order to more clearly understand the content of the present invention, it will be described in detail with reference to the drawings and embodiments.

[0048] The first aspect of the present invention relates to a process of steps as follows Figure 1 The multi-task allocation batch processing method based on pre-computation shown includes:

[0049] S1. Receive a user request and determine whether the user request is an online request or a batch service request. The user's request can be a batch upload or download request for data. After the request is transmitted to the gateway, it is not directly assigned and processed, but the nature of the request is first determined.

[0050] Through the judgment method, the online request can be processed directly, reducing the subsequent pre-computation requirements.

[0051] S2. When the user request is an online request, the user request is directly forwarded to the application for service processing, and the processing result is fed back. During execution, for a single online request, it is still necessary to determine whether the corresponding application is in a normal state. Specifically, the application service list is obtained, the application marked in the application service list as being in a normal state is identified, and the user request is forwarded to any application in a normal state for service processing. For an application that is already occupied, its status is changed to in use in the application service list.

[0052] S3. When the user request belongs to a batch service request, perform resource pre-calculation. The resource pre-calculation is used to predict the response capabilities of each application to the batch service request before actual allocation. Specifically, the batch transaction corresponding to the user request is identified, and the average CPU occupancy, memory consumption and total time consumption expected to be used by the batch transaction are calculated to obtain the result of resource pre-calculation.

[0053] For example, the information {"jobname":"uploadTsk1","cpu":"25%","mem":"200m","cost":"100m"} means that the average cpu consumption of the current task uploadTsk1 is 25%, the memory consumption is 200 megabytes, and the total estimated time is 100 minutes.

[0054] The existing online services and batch services are provided by different applications respectively. The service gateway forwards user requests to the online service (microservice) application or the batch service (batchService) application respectively. This deployment method can clearly isolate the online service and the batch service. The fluctuation of the batch service business will not affect the online service, but the resource utilization rate is not high.

[0055] By pre-calculating resources, dynamic application allocation can be achieved, thereby realizing hybrid application deployment; job resource pre-calculation is used to provide job pre-allocation strategies to improve job allocation efficiency; dynamic configuration is used to complete real-time monitoring of application jobs, role conversion, application online and offline, and application downgrade, to achieve security guarantees for external services after hybrid application deployment.

[0056] S4. Adjust the routable state of each application according to the result of resource pre-calculation. In particular, determine the health state of each application in the application service list according to the result of resource pre-calculation, and select n applications with the best health state to adjust to the routable state, where n is a preset positive integer.

[0057] The configuration center dynamically adjusts the configuration of the service registration center based on the pre-calculated results. The service registration center determines the application address to which the current request can be routed based on the health status of each service, and sends the address to the gateway in real time. The gateway forwards the request to the corresponding application based on the mapping address of the service registration center. The application uses heartbeats to maintain communication with the service registration center during the execution of batch tasks.

[0058] S5. Send the user request to the corresponding application for service processing according to the routable status of each application, and feedback the processing result. During the execution process, it is preferred to continuously monitor the health status of the application, especially to monitor the health status of the application by means of heartbeat information containing application survival information and health information, and further dynamically adjust the application processing process according to the application resource occupancy (average CPU occupancy, memory consumption and total time consumption) obtained by monitoring. For the actual execution of the application, according to the system security guarantee, it is preferred to set a first degradation threshold and a second degradation threshold, and the set value of the first degradation threshold is greater than the second degradation threshold; when the remaining resources of the application are lower than the first degradation threshold, the health status level of the application is reduced from green to yellow, and the service flow of the application is limited according to the preset strategy; when the remaining resources of the application are lower than the second degradation threshold, the health status level of the application is reduced from green or yellow to red, and the service of the application is stopped and the application is offline from the application service list.

[0059] For example, the first downgrade threshold can be set to {"cpuIdle":20%, "memIdle":200m} (i.e., cpu is 20% idle, memory is 200 megabytes idle), and the second downgrade threshold can be set to {"cpuIdle":10%, "memIdle":100m} (i.e., cpu is 10% idle, memory is 100 megabytes idle). When the remaining resources of the current application are lower than the first downgrade threshold, the application is downgraded from green to yellow, and the application is subject to service flow control for a period of time (dynamically adjusting the flow control strategy of the configuration center), and the actual execution resource consumption of the batch task is updated to the configuration center in real time. When the remaining resources of the current application are lower than the second downgrade threshold, the application is further downgraded to red, and the application is offline from the service registration center for a period of time, and the actual execution resource consumption of the batch task is updated to the configuration center in real time. When the corresponding batch task is executed, check the application marked in red. If the application is still alive, mark it as green and put it back online in the application service list. If the red application survives after completing the batch task, the task can be converted to a normal application and put online in the service registration center.

[0060] Another aspect of the present invention also relates to a multi-task allocation batch processing system based on pre-calculation, the structure of which is as follows: Figure 2 As shown, including:

[0061] The user request judgment module is used to receive a user request and judge whether the user request is an online request or a batch service request.

[0062] The resource pre-calculation module is used to perform resource pre-calculation, especially including calculating the average CPU occupancy, memory consumption and total time consumption expected to be used by the batch transaction, and obtaining the resource pre-calculation result.

[0063] The gateway is used to send user requests to the corresponding application and receive processing feedback from the application. When the user request is an online request, it is directly sent to the available application. When the user request is a batch service request, the application is allocated based on the result of resource pre-calculation.

[0064] The service registration module is used to manage the application service list and update the application status in the application service list in real time based on the feedback from other modules.

[0065] The task processing module is used to provide applications with processing user requests, record the heartbeat information of the application, and dynamically adjust the application processing status.

[0066] By using this system, the above-mentioned operation processing method can be executed and the corresponding technical effects can be achieved.

[0067] An embodiment of the present invention also provides a computer-readable storage medium capable of implementing all the steps in the method in the above embodiment, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all the steps in the method in the above embodiment are implemented.

[0068] An embodiment of the present invention also provides an electronic device for executing the above method. As an implementation device of the method, the electronic device has at least a processor and a memory, and in particular, the memory stores data and related computer programs required for executing the method, such as user requests and application service lists, etc., and the processor calls the data and programs in the memory to execute all steps of the implementation method and obtains the corresponding technical effects.

[0069] Preferably, the electronic device may include a bus architecture, the bus may include any number of interconnected buses and bridges, and the bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are all well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same component, i.e., a transceiver, providing a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used to store data used by the processor when performing operations.

[0070] In addition, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operation control) used may include a microprocessor or other processor device and / or a logic device, which receives input and controls the operation of various components of the electronic device; the memory may be one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices, which may store the above-mentioned related data information, and may also store a program for executing related information, and the processor may execute the program stored in the memory to realize information storage or processing, etc.; the input unit is used to provide input to the processor, for example, it may be a key or a touch input device; the power supply is used to provide power to the electronic device; the display is used to display display objects such as images and text, for example, it may be an LCD display. The communication module is a transmitter / receiver that sends and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal. Based on different communication technologies, multiple communication modules may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless local area network module, etc. The communication module (transmitter / receiver) is also coupled to a speaker and a microphone via an audio processor to provide an audio output via the speaker and receive an audio input from the microphone, thereby realizing a common telecommunication function. The audio processor may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor is also coupled to a central processor, so that recording can be performed on the machine through the microphone, and the sound stored on the machine can be played through the speaker.

[0071] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0072] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0073] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in a box or multiple boxes. Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0075] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multi-task batch processing method based on pre-calculation, It is characterized in that include: S1. Receive a user request and determine whether the user request is an online request or a batch service request; S2. When the user request is an online request, the user request is directly forwarded to the application for service processing, and the processing result is fed back; S3. When the user request is a batch service request, perform resource pre-calculation, including: Identify batch transactions corresponding to user requests; Calculate the average CPU usage, memory consumption, and total time consumption of the batch transaction to obtain the resource pre-calculation results; S4. Adjusting the routable state of each application according to the result of resource pre-calculation, including: judging the health state of each application in the application service list according to the result of resource pre-calculation, selecting n applications with the best health state to be adjusted to the routable state, where n is a preset positive integer; S5. Send the user request to the corresponding application for service processing according to the routable status of each application, and feedback the processing result. When the application is performing service processing, the health status of the application is monitored using heartbeat information; Setting a first degradation threshold and a second degradation threshold, wherein the set value of the first degradation threshold is greater than the second degradation threshold; When the remaining resources of the application are lower than the first degradation threshold, the health status level of the application is reduced from green to yellow, and the service flow of the application is limited according to the preset policy; When the remaining resources of the application are lower than the second degradation threshold, the health status level of the application is reduced from green or yellow to red, and the service of the application is stopped and the application is taken offline from the application service list.

2. The method according to claim 1, It is characterized in that The step S2 comprises: Get the list of application services; Identify the applications marked in the normal state in the application service list; Forward user requests to any application in normal state for service processing.

3. The method according to claim 2, It is characterized in that The step S2 further comprises: Change the mark of the app that receives the user request to In Use in the App Service List.

4. A multi-task allocation batch processing system based on pre-calculation, It is characterized in that include: A user request determination module, used for receiving a user request and determining whether the user request is an online request or a batch service request; Resource pre-calculation module, used for pre-calculation of resources, including: Identify batch transactions corresponding to user requests; Calculate the average CPU usage, memory consumption, and total time consumption of the batch transaction to obtain the resource pre-calculation results; The gateway is used to send user requests to the corresponding application and receive processing feedback from the application; The service registration module is used to manage the application service list, including determining the health status of each application in the application service list according to the result of resource pre-calculation, and selecting n applications with the best health status to be adjusted to a routable state, where n is a preset positive integer; A task processing module, used to provide an application to process user requests, wherein when the application performs service processing, a heartbeat information method is used to monitor the health status of the application, including setting a first degradation threshold and a second degradation threshold, wherein the setting value of the first degradation threshold is greater than the second degradation threshold; When the remaining resources of the application are lower than the first degradation threshold, the health status level of the application is reduced from green to yellow, and the service flow of the application is limited according to the preset policy; When the remaining resources of the application are lower than the second degradation threshold, the health status level of the application is reduced from green or yellow to red, and the service of the application is stopped and the application is taken offline from the application service list.

5. A computer-readable storage medium, It is characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

6. An electronic device, It is characterized in that including a processor and a memory; The memory is used to store user requests and application service lists; The processor is used to execute the method according to any one of claims 1 to 3 by calling a user request and an application service list.

7. A computer program product comprising a computer program and / or instructions, It is characterized in that When the computer program and / or the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images