Task Processing Method, Device, Storage Medium and Program Product

Abstract

The present application provides a task processing method, device, storage medium and program product. By obtaining a new task instruction; then using a container management module to determine at least one task to be processed and a task execution instruction according to the new task instruction, the task execution instruction corresponding to the task to be processed; then using a preset virtual machine runtime module to determine one or more virtual machines and virtual machine containers according to the task execution instruction, the virtual machine container being a container running in the virtual machine; and finally using the virtual machine container to call mirror resources to process the task to be processed. It solves the technical problems of low security and high operation and maintenance costs caused by insufficient isolation in the application of existing distributed batch task processing technologies in the field of fintech. It achieves the technical effect of improving the complete isolation effect of batch task processing in the field of fintech, improving the security of batch task processing, and reducing the operation and maintenance costs at the same time.

Description

Technical Field

[0001] This application relates to the field of financial technology (Fintech), and in particular, to a task processing method, device, storage medium, and program product. Background Art

[0002] With the development of computer technology, more and more technologies are applied in the financial field. The traditional financial industry is gradually transforming into financial technology (Fintech), and the distributed batch task processing technology is no exception. However, due to the security and real-time requirements of the financial industry, higher requirements are also put forward for the technology.

[0003] Existing distributed batch task processing technologies need to reserve a large amount of computing resources upward to ensure the computing resources during the operation of batch programs. However, due to the security isolation requirements of each task when applied in the field of financial technology, it is difficult to share or reuse computing resources among computing resources, resulting in low utilization rate of computing resources and increasing huge server costs and labor costs for the operation and maintenance of financial enterprises.

[0004] Although the emergence of container technology has alleviated the problem of low resource utilization rate in distributed batch task processing to a certain extent, the existing container technology still cannot solve the contradiction among high isolation, high security, and cost management in the field of financial technology, that is, when the existing distributed batch task processing technology is applied in the field of financial technology, there are technical problems of low security and high operation and maintenance costs caused by insufficient isolation. Summary of the Invention

[0005] This application provides a task processing method, device, storage medium, and program product to solve the technical problems of low security and high operation and maintenance costs caused by insufficient isolation when the existing distributed batch task processing technology is applied in the field of financial technology.

[0006] In a first aspect, this application provides a task processing method, including:

[0007] Obtain a new task instruction;

[0008] Use a container management module to determine at least one task to be processed and a task execution instruction according to the new task instruction, and the task execution instruction corresponds to the task to be processed;

[0009] Use a preset virtual machine runtime module to determine one or more virtual machines and virtual machine containers according to the task execution instruction, where the virtual machine container is a container running in the virtual machine;

[0010] Use the virtual machine container to call image resources to process the task to be processed.

[0011] This application uses virtual machine containers to replace traditional containers. While inheriting the advantages of traditional container technology, it introduces virtual machines, overcoming the problem of sharing the host kernel among containers, which easily leads to kernel escape, resulting in insufficient isolation or only node-level isolation, causing deficiencies in the isolation security of batch tasks in the fintech field. Virtual machine containers further improve resource utilization, and through the high isolation of virtual machines, an additional layer of virtual machine isolation is added on top of the original container-level isolation, thus preventing the problem of insufficient isolation caused by sharing the kernel among containers, achieving a complete isolation effect for batch task processing in the fintech field, improving the security of batch task processing, and simultaneously reducing the technical effect of operation and maintenance costs.

[0012] In a second aspect, this application provides a task processing device, including:

[0013] An acquisition module, configured to acquire a new task instruction;

[0014] A processing module, configured to:

[0015] Utilize a container management module to determine at least one task to be processed and a task execution instruction according to the new task instruction, where the task execution instruction corresponds to the task to be processed;

[0016] Utilize a preset virtual machine runtime module to determine one or more virtual machines and virtual machine containers according to the task execution instruction, where the virtual machine container is a container running in the virtual machine;

[0017] Utilize the virtual machine container to call mirror resources to process the task to be processed.

[0018] In a third aspect, this application provides an electronic device, including:

[0019] A memory, configured to store program instructions;

[0020] A processor, configured to call and execute the program instructions in the memory to execute any possible method for determining item storage information provided in the first aspect.

[0021] In a fourth aspect, this application provides a storage medium, where a computer program is stored in the readable storage medium, and the computer program is used to execute any possible task processing method provided in the first aspect.

[0022] In a fifth aspect, this application further provides a computer program product, including a computer program, which when executed by a processor, implements any possible task processing method provided in the first aspect.

[0023] The present application provides a task processing method, device, storage medium, and program product. By obtaining a new task instruction; then using a container management module to determine at least one task to be processed and a task execution instruction according to the new task instruction, where the task execution instruction corresponds to the task to be processed; then using a preset virtual machine runtime module to determine one or more virtual machines and virtual machine containers according to the task execution instruction, and the virtual machine container is a container running in the virtual machine; finally, using the virtual machine container to call mirror resources to process the task to be processed. This solves the technical problems of low security and high operation and maintenance costs caused by insufficient isolation in the application of existing distributed batch task processing technologies in the field of fintech. It achieves the technical effects of improving the complete isolation effect of batch task processing in the field of fintech, improving the security of batch task processing, and reducing operation and maintenance costs at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0025] Figure 1 FIG. is a schematic diagram of an application scenario of a task processing method provided by the present application;

[0026] Figure 2 FIG. is a schematic flowchart of a task processing method provided by the present application;

[0027] Figure 3 FIG. is a schematic flowchart of another task processing method provided for the implementation of the present application;

[0028] Figure 4 FIG. is a schematic diagram of the interaction process of virtual machine containers provided for the embodiments of the present application

[0029] Figure 5 FIG. is a schematic structural diagram of a task processing device provided for the embodiments of the present application;

[0030] Figure 6 FIG. is a schematic structural diagram of an electronic device provided by the present application.

[0031] Through the above drawings, specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Apparently, the described embodiments are only a part rather than all of the embodiments of this application. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts, including but not limited to combinations of multiple embodiments, fall within the scope of protection of this application.

[0033] The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims, and above-mentioned accompanying drawings of this application are used to distinguish similar objects and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of this application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0034] The following explains the professional terms involved in this application:

[0035] Virtual Machine Manager (VMM): Used to establish and maintain a framework for managing virtual machines.

[0036] Kubernetes: A distributed container orchestration and scheduling system.

[0037] CRI (Kubernetes Runtime Interface): The container runtime interface of Kubernetes. Implementing this interface allows access to Kubernetes.

[0038] Open Container Initiative (OCI): Used to establish the methods for container image creation, authentication, deployment, and naming.

[0039] Node: A distributed computing node, the smallest physical unit that provides computing power.

[0040] Kubelet: The agent component of Kubernetes on each node.

[0041] Image: A form of file storage, a software package that contains all the elements required to run a preset task in any environment.

[0042] Container: An instance created through an image.

[0043] Secure container: Uses a lightweight virtual machine to run the container and implements the requirements of OCI itself.

[0044] Corn: A kind of timer.

[0045] CornJob: A computing task that runs according to the cron time rule, such as running once a day.

[0046] Job: A computing task that runs only once.

[0047] Quality of Service (QoS): Used to ensure that high-priority tasks can have sufficient resources to run. When the cluster resources are insufficient, the QoS mechanism will, according to the configured task levels, force low-priority tasks to release their computing resources to guarantee the execution of high-priority tasks.

[0048] First, the problems existing in the traditional batch task processing technology discovered by the inventors of this application in their long-term work practice will be introduced below:

[0049] The traditional batch task processing technology is also called the traditional machine solution. In this solution, it is necessary to calculate the required computing resources in advance when running a batch program. However, this kind of calculation can only reserve computing resources upward. Different machines need to be allocated for different batch processing tasks (i.e., batch task processing). However, the resources required by the batch system are very huge. When traditional batch tasks are working, the workloads of all machines are almost full; but during non-task time periods, the machines are often idle and difficult to be reused. For other tasks that require computing resources, due to the requirements of isolation or security specifications, they cannot run on these idle machines. As a result, the existing computing resources cannot be fully utilized, causing a huge waste of computing power or resources in the financial technology field, that is, the utilization rate of computing resources is low, increasing the huge server costs and labor costs for the operation and maintenance of financial enterprises.

[0050] Meanwhile, when new batch tasks need to be launched or taken offline, manual installation and deployment of machines or machine dismantling are required, which also results in a large waste of human resources. Although the emergence of container technology has alleviated some resource waste to a certain extent, the existing container technology still fails to meet the customized requirements of various batch tasks in the fintech field, such as loading different kernel modules and running at different times. For tasks that require special configuration, only node-level isolation can be used to complete them, which causes resources to be unable to be fully utilized, that is, container technology has not fundamentally solved the problem of low utilization rate of computing resources. Nor has it solved the problem that various batch tasks of financial enterprises require high isolation. Once a security problem occurs, it will cause a fatal blow.

[0051] Generally speaking, when the existing batch task processing technology is applied in the fintech field, the following problems exist:

[0052] 1. Low security: The existing container technology needs to share the host kernel, which leads to the easy occurrence of kernel escape and the triggering of security problems. So-called kernel escape means that the attacker realizes container escape through vulnerabilities in the host kernel. Specifically, the attacker first obtains the command execution ability under a certain privilege in the container by hijacking the containerized business logic or directly controlling (such as in scenarios where legitimate control of the container is obtained like CaaS (Communications as a Service)), etc.; then the attacker uses this command execution ability and further obtains the command execution ability under a certain privilege on the direct host where the container is located through some means, thus triggering a security problem.

[0053] 2. Unable to isolate the running environment: Variables required for different batch tasks during processing (i.e., a specific manifestation of computing resources, such as environment variables and kernel parameters, etc.) may have a competitive relationship, resulting in the failure of batch tasks to run.

[0054] 3. Resource waste: Due to the competition of running environment variables, more machines are needed to run tasks. In addition, the idle computing power cannot be utilized to provide computing for other tasks during the task dormancy period.

[0055] 4. Unable to complete task scheduling according to the personalized requirements of batch tasks (such as personalized customization requirements for environment variables, operating system versions, kernel modules, CPU (Central Processing Unit), memory, IO (Input / Output) ports, etc.).

[0056] In summary, when the existing distributed batch task processing technology is applied in the fintech field, there are technical problems of low security and high operation and maintenance costs caused by insufficient isolation.

[0057] To solve the above technical problems, the inventive concept of this application is as follows:

[0058] When processing batch tasks in the field of fintech, instead of directly running containers to call images to complete task processing, a secure virtual isolation environment is first created for the containers, enabling the containers to run in this virtual isolation environment. This application uses virtual machines to implement this virtual isolation environment, integrating computing resources through virtual machines to improve the reuse rate of computing resources. At the same time, each batch task runs in a different virtual isolation environment, fundamentally solving the container escape vulnerability caused by the shared host kernel in existing container technologies, that is, improving the isolation and security of the batch system.

[0059] The following uses specific embodiments to elaborate in detail on the technical solution of this application and how the technical solution of this application solves the above technical problems. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0060] Figure 1 It is a schematic diagram of the application scenario of a task processing method provided by this application. As Figure 1 shown, the operation and maintenance personnel 101 create batch tasks through the user terminal 102, and the batch system 103 assigns task types to each task, including Job and CornJob. Then, the execution instructions of the tasks are passed to kubernetes, that is, the container management module 104, and kubernetes forwards the execution instructions to VMM, that is, the virtual machine management module 105, to apply for creating a virtual isolation environment. The virtual machine management module constructs each virtual machine 106, and runs each virtual machine container 107 for processing specific tasks corresponding to the batch tasks in the virtual machine 106. The virtual container 107 calls various image packages in the batch system 103 to complete specific computing tasks.

[0061] It should be noted that during the development and maintenance of the batch system, developers can store various image packages in the database of the batch system 103 in advance for subsequent calls by virtual machine containers.

[0062] Figure 2 It is a schematic flowchart of a task processing method provided by an embodiment of this application. As Figure 2 shown, the specific steps of this task processing method include:

[0063] S201. Obtain a new task instruction.

[0064] In this step, the new task instruction includes the construction information of at least one task to be processed.

[0065] Specifically, the operation and maintenance personnel create a batch task through the user terminal. The batch task includes at least one task to be processed. The operation and maintenance personnel input the construction information of each task to be processed through the user terminal and upload it to the batch system.

[0066] It should be noted that the developer uploads the pre-made image package files for processing various types of tasks to the database of the batch system for storage in advance for subsequent virtual machine containers to call.

[0067] It should also be noted that the operation and maintenance personnel can manage batch tasks through the task interface of the user terminal, including: creating tasks, viewing task progress, viewing task results, and so on.

[0068] S202. Use the container management module to determine at least one task to be processed and the task execution instruction according to the new task instruction.

[0069] In this step, the task execution instruction corresponds to the task to be processed. The tasks to be processed include: one-time tasks and periodic tasks.

[0070] In this embodiment, the container management module includes a task management interface. Using the task management interface, at least one task to be processed is determined according to the task information in the new task instruction.

[0071] Specifically, after the batch system receives the new task instruction sent by the operation and maintenance personnel through the user terminal, it creates a batch task, that is, at least one task to be processed, through the corresponding interface APIs (Application Programming Interface) of Job (i.e., one-time task) and CronJob (i.e., periodic task) of kunernetes (i.e., the container management module), and generates the execution instructions corresponding to each task to be processed.

[0072] S203. Use the preset virtual machine runtime module to determine one or more virtual machines and virtual machine containers according to the task execution instruction.

[0073] In this step, the virtual machine container is a container running in the virtual machine.

[0074] In this embodiment, the batch system sends the task execution instruction to the virtual machine runtime module through the first container runtime interface of the container management module; through the virtual machine runtime module, it connects to the virtual machine management module according to the task execution instruction; using the virtual machine management module and the virtual machine runtime module, one or more virtual machines and virtual machine containers are created according to the task execution instruction.

[0075] It should be noted that the batch system in this embodiment supports the container management module to run virtual machines in the same way as running containers through the virtual machine runtime module, enabling virtual machines to have the behavioral characteristics of containers.

[0076] Specifically, the batch system forwards the instruction to create a container in the task execution instruction to WCS-RUNTIME (i.e., the virtual machine runtime module) through the CRI of Kubernetes. Multiple VMMs (i.e., virtual machine management modules) are selected by WCS-RUNTIME to create virtual machine containers. WCS-RUNTIME implements the OCI open container standard. Through this module, the batch system can effectively access Kubernetes, run virtual machines like containers, and accept the management and scheduling of Kubernetes.

[0077] It should be specifically noted that the OCI open container standard was established by the Linux Foundation in June 2015 to form an open industrial standard around container formats and runtimes. Currently, there are mainly two standard documents: the container runtime standard (runtime spec) and the container image standard (image spec).

[0078] Generally speaking, the purpose of formulating the container format standard is not to be bound by the upper-layer structure, such as specific clients, orchestration stacks, etc., nor to be bound by specific vendors or projects, that is, not limited to a specific operating system, hardware, CPU architecture, public cloud, etc.

[0079] These two protocols are connected together through the standard format of the OCI runtime filesytem bundle. OCI images can be converted into bundles through tools, and then the OCI container engine can recognize this bundle to run containers.

[0080] It should be noted that WCS-RUNTIME is a container management service platform independently developed based on Kubernetes in the embodiment. Since Kubernetes is a distributed system specifically for container orchestration and scheduling, it can only call containers and cannot directly call virtual machines. To implement the inventive concept of this application, running containers on virtual machines with high isolation and high resource reuse rate requires an intermediate medium to enable Kubernetes to call VMM to create and manage each virtual machine. And WCS-RUNTIME is this intermediate medium. That is, the virtual machine runtime module is used to connect the container management module and the virtual machine container.

[0081] It should also be noted that the virtual machine management module in this embodiment is used to implement the Open Container Initiative (OCI) standard, and lightweight virtual machines can be used to run containers. When the containers are running, there is no need to share the host kernel (because what is shared is the kernel of the virtual machine rather than the actual host kernel), achieving the purpose of completely isolating the containers, meeting the financial-level isolation requirements during the processing of each batch task, and improving the security of the batch system.

[0082] S204. Use the virtual machine container to call the mirror resource to process the task to be processed.

[0083] In this step, the virtual machine container calls the mirror package corresponding to the task to be processed, which is pre-stored in the database, to complete the processing process of the task to be processed and obtain the processing result.

[0084] In this embodiment, by docking with Kubernetes, the distributed batch processing scheduling process is completed. By using virtual machines to replace traditional containers, a financial-level security level is achieved. Therefore, it has the advantages of the container batch system at the same time, making full use of idle computing resources. After the batch task goes into hibernation or exits, the computing resources can be given to other tasks in need. Thus, computing resource reuse is achieved, reducing machine resource waste and labor costs. Moreover, by using the self-developed batch system and the self-developed WCS-RUNTIME to access Kubernetes, the advantages of Kubernetes in distributed system scheduling are fully utilized, while avoiding security vulnerabilities therein.

[0085] It should be noted that the task processing method provided in this embodiment is carried in the self-developed batch system. In a possible design, the batch system uses a mirror repository to store the batch program mirrors uploaded by developers. The batch system classifies these mirrors by means of tags and mirror directories, facilitating users to retrieve and use them. The batch system itself does not pay attention to the developers' batch programs and only checks whether the mirrors are standard OCI mirrors.

[0086] The operation and maintenance personnel can create batch tasks according to the existing mirrors and time rules. These batch tasks are all stored in the database and can be resumed at any time.

[0087] The batch system queries the progress of the batch tasks through the API of Kubernetes at any time, and defines the status of the batch tasks as: not started, running, failed, completed. The operation and maintenance personnel can view the progress of the batch tasks through the web page and operate on the failed tasks to retry, etc.

[0088] In a possible design, the batch system visualizes the APIs of Kubernetes' Job and CronJob, and through input on the web page, the task type can be selected: one-time task, periodic task. The periodic task can be represented by a cron expression, supports running multiple replicas simultaneously, and specifies the CPU, memory, and disk size required for each replica.

[0089] And event triggers can be created for tasks, triggering events based on the task status to recycle resources or notify the operation and maintenance personnel. All these greatly facilitate the operation and maintenance personnel to use the system, thus saving a large amount of manpower for installing and uninstalling batch programs.

[0090] The embodiment of the present application provides a task processing method, which includes obtaining a new task instruction; then using a container management module to determine at least one task to be processed and a task execution instruction according to the new task instruction, where the task execution instruction corresponds to the task to be processed; then using a preset virtual machine runtime module to determine one or more virtual machines and virtual machine containers according to the task execution instruction, and the virtual machine container is a container running in the virtual machine; finally, using the virtual machine container to call image resources to process the task to be processed. This solves the technical problems of low security and high operation and maintenance costs caused by insufficient isolation in the application of existing distributed batch task processing technologies in the field of fintech. It achieves the technical effects of improving the complete isolation effect of batch task processing in the field of fintech, improving the security of batch task processing, and reducing the operation and maintenance costs at the same time.

[0091] Figure 3 It is a schematic flowchart of another task processing method provided for the implementation of the present application. As Figure 3 shown, the specific steps of this task processing method include:

[0092] S301. Obtain a new task instruction.

[0093] S302. Use the container management module to determine at least one task to be processed and a task execution instruction according to the new task instruction.

[0094] In this step, the task execution instruction corresponds to the task to be processed, and the tasks to be processed include: one-time tasks and periodic tasks.

[0095] In this embodiment, the container management module includes a task management interface, and using the task management interface, at least one task to be processed is determined according to the task information in the new task instruction.

[0096] Specifically, after the batch system receives the new task instruction sent by the operation and maintenance personnel through the user terminal, it creates a batch task, that is, at least one task to be processed, through the corresponding interface APIs (Application Programming Interfaces) of Job (i.e., one-time task) and CronJob (i.e., periodic task) of kunernetes (i.e., container management module), and generates the execution instructions corresponding to each task to be processed.

[0097] S303. Send the task execution instruction to the virtual machine runtime module through the first container runtime interface of the container management module.

[0098] In this step, the container management module includes: node proxy components on each node, the first container runtime interface, and the container runtime management module. The virtual machine runtime module includes: the second container runtime interface;

[0099] This step specifically includes: using the node proxy component, through the first container runtime interface, sending the task execution instruction to the container runtime management module;

[0100] Using the container runtime management module to forward the task execution instruction to the second container runtime interface, so as to realize sending the task execution instruction to the virtual machine runtime module.

[0101] In this embodiment, the virtual machine runtime module includes: the second container runtime interface, and the second container runtime interface is an intermediate link medium between the container runtime management module and the virtual machine.

[0102] Specifically, Kubelet (i.e., node proxy component) sends an instruction to the container runtime management program containerd (i.e., container runtime management module) through the CRI interface (i.e., the first container runtime interface), and containerd will forward it to wcs-shim (i.e., the second container runtime interface) after receiving the instruction.

[0103] S304. Through the virtual machine runtime module, connect to the virtual machine management module according to the task execution instruction.

[0104] In this step, after the second container runtime interface of the virtual machine runtime module receives the task execution instruction, it parses the container creation information in the task execution instruction, establishes a connection with the virtual machine management module through the second container runtime interface, and passes the container creation information to the virtual machine management module. The virtual machine management module creates one or more corresponding virtual machines to create a corresponding virtual isolation running environment for the container.

[0105] S305. Using the virtual machine management module and the virtual machine runtime module, create one or more virtual machines and virtual machine containers according to the task execution instruction.

[0106] In this embodiment, the virtual machine runtime module further includes: a container process management module, a virtual machine kernel, and a file system image. The virtual machine kernel is the kernel module used when the virtual machine is running, and the file system image is the mini file system used when the virtual machine is running the virtual machine container.

[0107] In this step, use the virtual machine management module to create one or more virtual machines according to the task execution instruction, the virtual machine kernel, and the file system image.

[0108] Use the container process management module to send the task execution instruction to the minimum container deployment unit (i.e., pod) of the virtual machine through the Remote Procedure Call (RPC) protocol, so as to create one or more virtual machine containers and manage the working status of all virtual machine containers.

[0109] In a possible design, the virtual machine kernel includes a personalized customization tool, which is used to perform personalized customization on all sub-modules and various parameters in the virtual machine kernel.

[0110] The file system image is completely isolated from the host machine carrying the virtual machine and can be used independently.

[0111] The container process management module starts at the same time as the virtual machine and runs in the virtual machine as an initialization process. The container process management module is also used to expose the communication protocol file to the host machine carrying the virtual machine, so that the container process management module communicates with the second container runtime interface through the communication protocol file.

[0112] Specifically, the virtual machine runtime module in this embodiment is the self-developed WCS-RUNTIME, and its components are shown in Table 1:

[0113]

[0114]

[0115] Table 1

[0116] Figure 4 It is a schematic diagram of the virtual machine container interaction process provided by the embodiment of the present application. As Figure 4 shown:

[0117] Kubelet Node Agent Component 401: Send instructions to the containerd container runtime management module 402 through the CRI interface. After receiving the instructions, the containerd container runtime management module 402 will forward them to the wcs-shim second container runtime interface 403. According to the virtual machine situation, the wcs-shim second container runtime interface 403 sends the commands to the Pod through a private RPC process. After receiving the instructions, the wcs-agent container process management module 404 manages the container status according to the instruction content.

[0118] wcs-shim Second Container Runtime Interface 403: When receiving the instruction to create a container, WCS-RUNTIME will also create the wcs-shim second container runtime interface 403, which serves as an intermediate module for communication between the container runtime and the virtual machine. The wcs-shim second container runtime interface 403 is a bridge connecting the virtual machine and the container runtime management module.

[0119] wcs-agent Container Process Management Module 404: When the virtual machine starts, the wcs-agent container process management module 404 will be started and run as an init initialization process in the virtual machine. This component exposes the vsocks file (i.e., the communication protocol file) to the host, thereby completing communication with the wcs-shim second container runtime interface 403 through vsocks. The wcs-agent container process management module 404 has all the behaviors under the OCI standard, including: creating, starting, and shutting down the container 407, I / O stream communication, and all the behaviors of traditional containers.

[0120] Guest Kernel Virtual Machine Kernel 405: The kernel used when the virtual machine is actually running, rather than traditional containers sharing the host kernel. All modules and parameters of this kernel can be customized, and a complete set of tools is provided to customize the personalized kernel.

[0121] Guest Image File System Image 406: Also known as Guest OS, it is the operating system that actually runs the containers, is completely isolated from the host, and can use an independent file system.

[0122] Through the above components in the independently developed WCS-RUNTIME, the gap between containers and virtual machines is completely bridged, enabling the virtual machine to have the advantages of containers: fast, supporting one-time packaging and running anywhere, and also having the traditional advantages of virtual machines: secure, strong isolation, and strong customization.

[0123] S306: Pass the annotation tool to the container runtime management module through the first container runtime interface.

[0124] In the embodiment of this step, the container management module further includes an annotation tool, and the functions of the annotation tool include carrying customized parameters defined by users.

[0125] In order to provide a completely free customized configuration ability. In a possible design, the functions of the global configuration parameters include: redefining each global default parameter and the type of the container running in the virtual machine;

[0126] The functions of the container runtime configuration parameters include: specifying whether to create a new network namespace for the container runtime module, determining the network interfaces of the virtual machine and the virtual machine container, and determining whether to manage each custom process in the virtual machine runtime module only in the sandbox control group;

[0127] The functions of the container process management module configuration parameters include: specifying the size of the standard transmission pipeline created for the virtual machine container, and determining the list of kernel modules loaded into the virtual machine kernel and the parameters of each item in the kernel module list;

[0128] The functions of the virtual machine configuration parameters include: determining the driver used for the block device, determining the list of functions passed to the central processing unit of the virtual machine, determining the operating system when the virtual machine runs the virtual machine container, determining the system kernel module of the operating system, specifying the use of the communication protocol file during proxy communication, determining whether to enable the memory swap of the virtual machine, and specifying the addresses of each virtual machine management module. Among them, the block device is used to store information in fixed-size data blocks, and each data block has corresponding address information.

[0129] S307. Use the container runtime management module to convert the customized parameters in the annotation tool into standard annotation information according to the open container standard.

[0130] In this step, after receiving the customized parameters passed by the annotation tool annotation, the container runtime management module converts them into OCI standard annotation information according to the OCI standard.

[0131] Specifically, in this embodiment, four aspects of kubernetes annotations are formulated:

[0132] The global configuration parameters are shown in Table 2:

[0133] Keyword Value Type Comment io.wcs.config_path string Override the default wcs configuration io.wcs.pkg.oci.type string OCI container type

[0134] Table 2

[0135] The container runtime configuration parameters are shown in Table 3:

[0136]

[0137] Table 3

[0138] The configuration parameters of the container process management module are shown in Table 4 as follows:

[0139]

[0140] Table 4

[0141] The configuration parameters of the virtual machine are shown in Table 5 as follows:

[0142]

[0143]

[0144] Table 5

[0145] S308. Send the standard annotation information to the container process management module through the first container runtime interface and the remote procedure call protocol.

[0146] In this step, WCS-RUNTIME provides this information to wcs-agent through RPC.

[0147] S309. Use the container process management module to control the startup content of the virtual machine according to the standard annotation information.

[0148] In this step, the wcs-agent container process management module 404 completes the final action, that is, to control the startup of the virtual machine and the containers running in the virtual machine, namely the virtual machine containers. The startup content includes the custom creation of each virtual machine container and the setting of the startup status.

[0149] Specifically, for steps S306 - S309, when creating virtual machine containers, or rather when creating virtual machines, for each running task, customized parameters can be passed through the annotation (i.e., annotation tool) of kubernetes. Kubernetes passes the annotation to the container runtime management module and converts it into OCI annotations. Then WCS-RUNTIME provides this information to wcs-agent through RPC, and wcs-agent completes the final action.

[0150] For example, the parameters in Table 2 - 5 can be configured through the annotationss attribute of the kubectl yaml file, such as:

[0151]

[0152] In the above code example, a one-time batch task is directly defined through YAML. Customized io.wcs.config.agent.kernel_modules and io.wcs.config.hypervisor.enable_swap are specified through annotations. After the above file is submitted to Kubernetes through the kubectl command, after being scheduled and matched by Kubernetes, kubelet is selected to call the CRI interface. Through the CRI container creation command, the annotations are passed to CRI to implement containerd. And containerd, through the CRI plugin, calls wcs-runtime to create a virtual machine container, parses the annotations parameters and passes them to wcs-runtime, and wcs-runtime calls the VMM according to the parameters to generate the corresponding virtual machine container.

[0153] S310. Use the virtual machine container to call the image resource to process the task to be processed.

[0154] The task processing method provided in this embodiment is carried on a batch system. The batch system itself does not care about the developer's batch program and only checks whether the image is a standard OCI image.

[0155] The batch system uses an image repository to store the batch program images uploaded by developers. The batch system classifies these images by means of tags and image directories, so as to facilitate users to retrieve and use.

[0156] The operation and maintenance personnel can create batch tasks according to the existing images and time rules. These batch tasks are all stored in the database and can be restored and run at any time.

[0157] The batch system queries the progress of the batch task through the API of Kubernetes at any time, and defines the status of the batch task as: not started, running, failed, completed. The operation and maintenance personnel can view the progress of the batch task through the web page and operate on the retry of the failed task, etc.

[0158] The batch system visualizes the APIs of Kubernetes' Job and CronJob, and through web page input, the task type can be selected: one-time task, periodic task. The periodic task can be represented by a cron expression, supports running multiple replicas simultaneously, and specifies the required CPU, memory, and disk size for each replica.

[0159] Moreover, it can trigger events for task creation and trigger events based on task status to recycle resources or notify operation and maintenance personnel. All these greatly facilitate the operation and maintenance personnel to use the system, thus saving a large amount of manpower for installing and uninstalling batch programs.

[0160] Specifically, the number of running instances can be set for each task to be processed. When the entire cluster resources (i.e., all image resources or computing resources in the batch system) meet the instance number requirements, the batch tasks, that is, all tasks to be processed, can start execution. And the batch task instances are allocated to different nodes, and finally WCS-RUNTIME completes the startup of these batch tasks. However, in the actual production environment, the importance levels of batch tasks are also different. For tasks with a high importance level, the batch system uses the qos mechanism of kubernetes to give priority to ensuring tasks with a high importance level. By setting the minimum and maximum resources of the task to be the same, the batch system considers that the batch task has a high priority and will fully guarantee the resources required during its operation. When the minimum resource of the batch task is set to be less than the maximum resource, the batch allows it to use the resources within its resource setting range, but it cannot guarantee that it will definitely be able to use these resources. When the task does not set resource requirements, the batch system will try its best to meet the resource requirements during the operation of the batch task, but the priority of this batch task is the lowest. Therefore, when resources are insufficient, this type of task will be recycled first.

[0161] The batch system can label nodes according to the actual node characteristics, such as: network bandwidth size, disk performance, GPU, physical address identifier (computer room, rack), etc. By specifying labels in the batch task, the batch task can run on nodes with these labels to meet customized requirements.

[0162] In a possible design, the cluster scale of the batch system can also be expanded or reduced.

[0163] The processing scale of the distributed cluster is very important. In order to make full use of cheap servers for batch task computing, the functions of expanding and reducing the cluster are very necessary. The operation and maintenance personnel can view the workload of the entire system through the batch system and automatically include nodes in the cluster or kick them out of the cluster according to the load situation. The operation and maintenance personnel only need to make decisions on expansion and contraction according to the cluster load situation, without having to view the load situation of individual machines, leaving the resource scheduling ability to the batch system, and the operation and maintenance personnel only need to control the cluster scale. This fully reduces the mental burden of the operation and maintenance personnel.

[0164] An embodiment of the present application provides a task processing method, which includes obtaining a new task instruction; then using a container management module to determine at least one task to be processed and a task execution instruction according to the new task instruction, where the task execution instruction corresponds to the task to be processed; then using a preset virtual machine runtime module to determine one or more virtual machines and virtual machine containers according to the task execution instruction, where the virtual machine container is a container running in the virtual machine; and finally using the virtual machine container to call image resources to process the task to be processed. This solves the technical problems of low security and high operation and maintenance costs caused by insufficient isolation in the application of existing distributed batch task processing technologies in the field of fintech. It achieves the technical effects of improving the complete isolation effect of batch task processing in the field of fintech, improving the security of batch task processing, and reducing operation and maintenance costs at the same time.

[0165] Figure 5 It is a schematic structural diagram of a task processing device provided by an embodiment of the present application. The task processing device 500 can be implemented by software, hardware, or a combination of both.

[0166] As Figure 5 shown, the task processing device 500 includes:

[0167] An obtaining module 501, configured to obtain a new task instruction;

[0168] A processing module 502, configured to:

[0169] Use a container management module to determine at least one task to be processed and a task execution instruction according to the new task instruction, where the task execution instruction corresponds to the task to be processed;

[0170] Use a preset virtual machine runtime module to determine one or more virtual machines and virtual machine containers according to the task execution instruction, where the virtual machine container is a container running in the virtual machine;

[0171] Use the virtual machine container to call image resources to process the task to be processed.

[0172] In a possible design, the processing module 502 is configured to:

[0173] Send the task execution instruction to the virtual machine runtime module through a first container runtime interface of the container management module;

[0174] Connect to a virtual machine management module according to the task execution instruction through the virtual machine runtime module;

[0175] Use the virtual machine management module and the virtual machine runtime module to create one or more virtual machines and virtual machine containers according to the task execution instruction.

[0176] In a possible design, the container management module includes: node agent components on each node, a first container runtime interface, and a container runtime management module. The virtual machine runtime module includes: a second container runtime interface;

[0177] Correspondingly, the processing module 502 is configured to:

[0178] Use the node agent component to send a task execution instruction to the container runtime management module through the first container runtime interface;

[0179] Use the container runtime management module to forward the task execution instruction to the second container runtime interface.

[0180] In a possible design, the virtual machine runtime module includes: a second container runtime interface, a container process management module, a virtual machine kernel, and a file system image. The second container runtime interface is an intermediate link medium between the container runtime management module and the virtual machine. The virtual machine kernel is a kernel module used when the virtual machine is running. The file system image is a miniature file system used when the virtual machine is running a virtual machine container;

[0181] Correspondingly, the processing module 502 is configured to:

[0182] Use the virtual machine management module to create one or more virtual machines according to the task execution instruction, the virtual machine kernel, and the file system image;

[0183] Use the container process management module to send the task execution instruction to the minimum container deployment unit of the virtual machine through the remote procedure call protocol to create one or more virtual machine containers and manage the working status of all virtual machine containers.

[0184] In a possible design, the virtual machine kernel includes a personalized customization tool, which is used to perform personalized customization on all sub-modules and various parameters in the virtual machine kernel;

[0185] The file system image is completely isolated from the host machine carrying the virtual machine and can be used independently;

[0186] The container process management module starts at the same time as the virtual machine and runs as an initialization process in the virtual machine. The container process management module is also used to expose the communication protocol file to the host machine carrying the virtual machine, so that the container process management module communicates with the second container runtime interface through the communication protocol file.

[0187] In a possible design, the container management module includes an annotation tool, and the functions of the annotation tool include carrying user-defined customization parameters. Correspondingly, the processing module 502 is further configured to:

[0188] Pass the annotation tool to the container runtime management module through the first container runtime interface;

[0189] Use the container runtime management module to convert the customized parameters in the annotation tool into standard annotation information according to the open container standard, and send the standard annotation information to the container process management module through the first container runtime interface and the remote procedure call protocol;

[0190] Use the container process management module to control the startup content of the virtual machine according to the standard annotation information, and the startup content includes the custom creation of each virtual machine container and the setting of the startup status.

[0191] In a possible design, the customized parameters include: global configuration parameters, container runtime configuration parameters, container process management module configuration parameters, and virtual machine configuration parameters;

[0192] Among them, the functions of the global configuration parameters include: redefining various global default parameters and the types of containers running in the virtual machine;

[0193] The functions of the container runtime configuration parameters include: specifying whether to create a new network namespace for the container runtime module, determining the network interfaces of the virtual machine and the virtual machine containers, and determining whether to manage each custom process in the virtual machine runtime module only in the sandbox control group;

[0194] The functions of the container process management module configuration parameters include: specifying the size of the standard transmission pipeline created for the virtual machine container, and determining the list of kernel modules loaded into the virtual machine kernel and the parameters of each item in the kernel module list;

[0195] The functions of the virtual machine configuration parameters include: determining the driver for the block device, determining the list of functions of the central processing unit passed to the virtual machine, determining the operating system when the virtual machine runs the virtual machine container, determining the system kernel module of the operating system, specifying the use of the communication protocol file during proxy communication, determining whether to enable the memory swap of the virtual machine, and specifying the addresses of each virtual machine management module. Among them, the block device is used to store information in fixed-size data blocks, and each data block has corresponding address information.

[0196] In a possible design, the container management module includes a task management interface. Correspondingly, the processing module 502 is used to use the task management interface to determine at least one task to be processed according to the task information in the new task instruction. The tasks to be processed include: one-time tasks and periodic tasks.

[0197] It is worth noting that Figure 5The device provided by the illustrated embodiment can execute the functions on the consensus node side in the method provided in any of the above method embodiments. The specific implementation principles, technical features, explanations of professional terms, and technical effects are similar, and will not be elaborated here.

[0198] Figure 6 This is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 6 shown, the electronic device 600 may include: at least one processor 601 and a memory 602. Figure 6 The electronic device shown is taken as an example with one processor.

[0199] The memory 602 is used to store a program. Specifically, the program may include program code, and the program code includes computer operation instructions.

[0200] The memory 602 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0201] The processor 601 is used to execute the computer execution instructions stored in the memory 602 to implement the methods described in the above method embodiments.

[0202] Among them, the processor 601 may be a central processing unit (CPU for short), or an application specific integrated circuit (ASIC for short), or one or more integrated circuits configured to implement the embodiments of the present application.

[0203] Optionally, the memory 602 may be either independent or integrated with the processor 601. When the memory 602 is a device independent of the processor 601, the electronic device 600 may further include:

[0204] A bus 603, used to connect the processor 601 and the memory 602. The bus may be an industry standard architecture (ISA for short) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0205] Optionally, in a specific implementation, if the memory 602 and the processor 601 are integrated on a single chip, the memory 602 and the processor 601 can communicate through an internal interface.

[0206] The embodiments of the present application also provide a computer-readable storage medium, which may include: various media that can store program codes, such as USB flash drives, external hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs. Specifically, the computer-readable storage medium stores program instructions, and the program instructions are used for the methods in the foregoing method embodiments.

[0207] The embodiments of the present application also provide a computer program product, including a computer program, which implements the methods in the foregoing method embodiments when executed by a processor.

[0208] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include known common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and the embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the claims of the present application.

[0209] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A task processing method, characterized in that, Including: Obtain a new task instruction; Using a container management module, according to the new task instruction, determine at least one task to be processed and a task execution instruction, the task execution instruction corresponding to the task to be processed, the container management module including: node agent components on each node, a first container runtime interface, a container runtime management module, and an annotation tool, the function of the annotation tool including carrying user-defined customization parameters; Using a preset virtual machine runtime module, according to the task execution instruction, determine one or more virtual machines and virtual machine containers, the virtual machine containers being containers running in the virtual machines; the virtual machine runtime module including: a second container runtime interface, a container process management module, a virtual machine kernel, and a file system image, the second container runtime interface being an intermediate link medium between the container runtime management module and the virtual machine, the virtual machine kernel being a kernel module used by the virtual machine when running, and the file system image being a mini file system used by the virtual machine when running the virtual machine containers; Pass the annotation tool to the container runtime management module through the first container runtime interface; Using the container runtime management module, according to the open container standard, convert the customization parameters in the annotation tool into standard annotation information, and send the standard annotation information to the container process management module through the first container runtime interface and the remote procedure call protocol; Using the container process management module, according to the standard annotation information, control the startup content of the virtual machine, the startup content including the custom creation of each of the virtual machine containers and the startup status setting; Use the virtual machine containers to call mirror resources to process the tasks to be processed.

2. The task processing method according to claim 1, characterized in that, The using the preset virtual machine runtime module to determine one or more virtual machines and virtual machine containers according to the task execution instruction includes: Send the task execution instruction to the virtual machine runtime module through the first container runtime interface of the container management module; Connect to the virtual machine management module through the virtual machine runtime module according to the task execution instruction; Using the virtual machine management module and the virtual machine runtime module, create one or more virtual machines and virtual machine containers according to the task execution instruction.

3. The task processing method according to claim 2, wherein Sending the task execution instruction to the virtual machine runtime module through the first container runtime interface of the container management module includes: Using the node agent component, send the task execution instruction to the container runtime management module through the first container runtime interface; Use the container runtime management module to forward the task execution instruction to the second container runtime interface.

4. The task processing method according to claim 2 or 3, characterized in that Using the virtual machine management module and the virtual machine runtime module to create one or more virtual machines and virtual machine containers according to the task execution instruction includes: Using the virtual machine management module, create one or more of the virtual machines according to the task execution instruction, the virtual machine kernel, and the file system image; Using the container process management module, send the task execution instruction to the minimum container deployment unit of the virtual machine through the remote procedure call protocol to create one or more of the virtual machine containers and manage the working status of all the virtual machine containers.

5. The task processing method according to claim 4, characterized in that The virtual machine kernel includes a personalized customization tool for personalizing all sub-modules and various parameters in the virtual machine kernel; The file system image is completely isolated from the host machine carrying the virtual machine and can be used independently; The container process management module starts at the same time as the virtual machine and runs as an initialization process in the virtual machine. The container process management module is further configured to expose a communication protocol file to the host machine carrying the virtual machine, so that the container process management module communicates with the second container runtime interface through the communication protocol file.

6. The task processing method according to claim 1, characterized in that, The customization parameters include: global configuration parameters, container runtime configuration parameters, container process management module configuration parameters, and virtual machine configuration parameters; Among them, the functions of the global configuration parameters include: redefining various global default parameters and the types of containers running in the virtual machine; The functions of the container runtime configuration parameters include: specifying whether to create a new network namespace for the container runtime module, determining the network interfaces of the virtual machine and the virtual machine container, and determining whether to manage each custom process in the virtual machine runtime module only in the sandbox control group; The functions of the container process management module configuration parameters include: specifying the size of the standard transmission pipeline created for the virtual machine container, determining the list of kernel modules loaded into the virtual machine kernel and various parameters in the kernel module list; The functions of the virtual machine configuration parameters include: determining the driver for the block device, determining the list of functions passed to the central processing unit of the virtual machine, determining the operating system when the virtual machine runs the virtual machine container, determining the system kernel module of the operating system, specifying the use of the communication protocol file during proxy communication, determining whether to enable memory swapping of the virtual machine, and specifying the addresses of each virtual machine management module, where the block device is used to store information in fixed-size data blocks, and each data block has corresponding address information.

7. The task processing method according to claim 1, wherein The container management module includes a task management interface. Using the container management module, determine at least one task to be processed according to the new task instruction, including: Using the task management interface, determine at least one of the tasks to be processed according to the task information in the new task instruction. The tasks to be processed include: one-time tasks and periodic tasks.

8. An electronic device, characterized in that, Including: A processor; And, A memory for storing the computer program of the processor; Wherein, the processor is configured to execute the task processing method according to any one of claims 1 to 7 by executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the task processing method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the task processing method according to any one of claims 1 to 7.

Images