Container cache allocation method, system and equipment and storage medium

Abstract

The invention relates to the field of container cache allocation, and discloses a container cache allocation method, system and device and a storage medium, and the method comprises the following steps: when a plurality of mass data stream processing flows exist, establishing constraint conditions in the process of allocating physical resources to container slices and a resource allocation income calculation model; calculating a resource allocation income obtained in the process of allocating the physical resources to the container slices through a resource allocation income calculation model; and based on the constraint condition, with the maximum resource allocation income as the target, performing iterative operation through a CRO algorithm, and generating an optimal strategy for allocating the physical resources to the container slices. Limited physical cluster resources are allocated to the virtual container slices for caching through the CRO algorithm, so that the data has the advantage of container fault isolation during processing, and resources can be allocated most reasonably under the condition that physical resource clusters are limited.

Description

Technical Field

[0001] The present disclosure relates to the technical field of container cache allocation, and in particular to a container cache allocation method, system, device and storage medium. Background Art

[0002] In the field of network operating systems, processing large-scale basic data has become a key technical challenge. The basic data is usually unprocessed data, which is characterized by a large amount of data, inconsistent data length, significant data fluctuations, obvious data transmission discontinuity, and high concurrency scenarios often occur in practice. In view of the high requirements for real-time processing and high concurrency, the operating system needs to enable a large number of resources for multi-process / thread processing, while also ensuring data reliability, especially fault isolation. Fault isolation means that when a process has a problem and hangs up, it cannot affect other processes to ensure that data interfaces do not affect each other.

[0003] In order to solve the fault isolation problem, hardware resource virtualization is adopted and container technology is introduced. Since containers need to load mapping, start and other time-consuming operations, they cannot process data immediately. A container cache pool is proposed to put some started containers into the cache pool to improve data processing efficiency. It is divided into multiple container resources with different resource sizes, and different containers are used to connect APIs on different data collection sides. In order to ensure the real-time performance of data processing, the industry usually expands the container. When the data reaches a certain threshold, a new container is started and placed in the cache area. However, the introduction of containers will bring another problem, whether the physical resources allocated to the container match the data volume of the processing interface. When the processing capacity provided by the resource allocation of the container is higher than the data volume of the interface, it will lead to resource waste. When the resource allocation of the container is small and the amount of data connected to this container is large, its processing speed is unacceptable. Based on the above content, the current problem to be solved is: how to reasonably allocate physical resources in the cache container slices. How to allocate slices in the container cache pool for data processing to maximize resource utilization. Summary of the invention

[0004] The embodiments of the present disclosure provide a container cache allocation method, system, device, and storage medium to solve or alleviate one or more of the above technical problems in the prior art.

[0005] According to one aspect of the present disclosure, a container cache allocation method is provided, comprising:

[0006] When there are multiple massive data stream processing processes, establish the constraints for allocating physical resources to container slices and the resource allocation benefit calculation model;

[0007] The resource allocation benefit calculation model is used to calculate the resource allocation benefit obtained in the process of allocating physical resources to container slices;

[0008] Based on the constraints and with the goal of maximizing the resource allocation benefit, the CRO algorithm is iteratively calculated to generate the optimal strategy for allocating physical resources to container slices.

[0009] In one possible implementation, the constraints in allocating physical resources to container slices are expressed as:

[0010]

[0011] In the formula, x ij Indicates that the memory of the i-th physical node is allocated to the container slice node j; represents the amount of cache resources allocated from physical node i to container slice j; R i Represents a container resource.

[0012] In a possible implementation, the resource allocation benefit calculation model in the process of allocating physical resources to container slices is expressed as:

[0013]

[0014] In the formula, F represents the total value of all container slices per unit time; F buff represents the total cost, p1 represents the cost coefficient, including energy consumption cost and hardware equipment cost; p i represents the energy consumption and hardware cost per unit cache content; u ij Indicates the number of iteration updates of the container slice; x ij Indicates that the memory of the i-th physical node is allocated to the container slice node j; It represents the amount of cache resources allocated from physical node i to container slice j; T represents unit time.

[0015] In a possible implementation, based on the constraints and with the goal of maximizing the resource allocation benefit, the optimal strategy for allocating physical resources to container slices is generated through iterative calculations using the CRO algorithm, including:

[0016] Initializing various parameters, including the number of molecules in the container, the upper limit of kinetic energy loss in the invalid collision reaction of the molecules, and the initial value of the kinetic energy of the molecules;

[0017] Randomly select a container slice and allocate a random number of physical resources to form an indication matrix X, which is used to indicate the container slice that obtains the physical resources;

[0018] Generate a resource allocation matrix F based on the indication matrix X, which is used to indicate the amount of physical resources acquired by the container slice;

[0019] Select one of the mechanisms in the CRO algorithm, take the maximum resource allocation benefit as the goal, iterate the resource allocation matrix F, and generate the intermediate matrix F1;

[0020] Compare the intermediate matrix F1 with the constraints and output the target resource allocation matrix F* that meets the constraints.

[0021] According to one aspect of the present disclosure, a container cache allocation system is provided, comprising:

[0022] An establishment unit is used to establish constraints and resource allocation benefit calculation models in the process of allocating physical resources to container slices when there are multiple massive data stream processing processes;

[0023] Resource allocation benefit calculation model, used to calculate the resource allocation benefits obtained in the process of allocating physical resources to container slices;

[0024] The computing unit is used to generate the optimal strategy for allocating physical resources to container slices by performing iterative operations through the CRO algorithm based on constraints and with the goal of maximizing resource allocation benefits.

[0025] In one possible implementation, the constraints in allocating physical resources to container slices are expressed as:

[0026]

[0027] In the formula, x ij Indicates that the memory of the i-th physical node is allocated to the container slice node j; represents the amount of cache resources allocated from physical node i to container slice j; R i Represents a container resource.

[0028] In a possible implementation, the resource allocation benefit calculation model in the process of allocating physical resources to container slices is expressed as:

[0029]

[0030] In the formula, F represents the total value of all container slices per unit time; F buff represents the total cost, p1 represents the cost coefficient, including energy consumption cost and hardware equipment cost; p i represents the energy consumption and hardware cost per unit cache content; u ij Indicates the number of iteration updates of the container slice; x ij Indicates that the memory of the i-th physical node is allocated to the container slice node j; It represents the amount of cache resources allocated from physical node i to container slice j; T represents unit time.

[0031] In a possible implementation, the computing unit includes:

[0032] An initialization module, used to initialize various parameters, including the number of molecules in the container, the upper limit of kinetic energy loss in the invalid collision reaction of the molecules, and the initial value of the kinetic energy of the molecules;

[0033] A forming module, used to randomly select a container slice and allocate a random number of physical resources, forming an indication matrix X, used to indicate the container slice for obtaining the physical resources;

[0034] A generating module, used to generate a resource allocation matrix F based on the indication matrix X, used to indicate the amount of physical resources acquired by the container slice;

[0035] The operation module is used to select one of the mechanisms in the CRO algorithm, take the maximum resource allocation benefit as the goal, perform iterative operations on the resource allocation matrix F, and generate an intermediate matrix F1;

[0036] The output module is used to compare the intermediate matrix F1 with the constraint conditions and output the target resource allocation matrix F* that meets the constraint conditions.

[0037] According to one aspect of the present disclosure, there is provided a device, comprising:

[0038] Processor and memory;

[0039] The memory is used to store a computer program, and the processor calls the computer program stored in the memory to execute any one of the container cache allocation methods described above.

[0040] According to one aspect of the present disclosure, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the processor is enabled to execute any of the container cache allocation methods described above.

[0041] The present disclosure has the following beneficial effects: the present disclosure allocates limited physical cluster resources to virtual container slice cache through the CRO algorithm, so that when processing data, it has the advantages of container fault isolation and can allocate resources most reasonably when the physical resource cluster is limited.

[0042] The details of one or more embodiments of the present application are presented in the following drawings and descriptions. Other features and advantages of the present application will become apparent from the accompanying drawings. It should be understood that the above general description and the detailed descriptions below are only exemplary and explanatory and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0044] Figure 1 is a flow chart of a container cache allocation method of the present exemplary embodiment;

[0045] Figure 2 is a schematic diagram of the application of container cache of this exemplary embodiment in a high data concurrency scenario;

[0046] Figure 3 is a flow chart of container resource allocation based on the CRO algorithm of this exemplary embodiment;

[0047] Figure 4 is a schematic diagram of a target resource allocation matrix generation process of this exemplary embodiment;

[0048] Figure 5 is a block diagram of a container cache allocation system of the present exemplary embodiment;

[0049] Figure 6 It is a schematic structural diagram of a device of this exemplary embodiment. DETAILED DESCRIPTION

[0050] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0051] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware units or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0052] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0053] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example.

[0054] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or sub-modules is not necessarily limited to those steps or sub-modules explicitly listed, but may include other steps or sub-modules not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0055] Figure 1 is a flow chart of a container cache allocation method of this exemplary embodiment. Figure 1 As shown, an exemplary embodiment of the present disclosure provides a container cache allocation method, including:

[0056] When there are multiple massive data stream processing processes, establish the constraints for allocating physical resources to container slices and the resource allocation benefit calculation model;

[0057] The resource allocation benefit calculation model is used to calculate the resource allocation benefit obtained in the process of allocating physical resources to container slices;

[0058] Based on the constraints and with the goal of maximizing the resource allocation benefit, the CRO algorithm is iteratively calculated to generate the optimal strategy for allocating physical resources to container slices.

[0059] This embodiment specifically provides a container cache allocation method after physical resource virtualization to cope with the following problems faced by the network operating system: Figure 2In order to improve the allocation efficiency of physical resources in the high-concurrency data transmission scenario shown in the figure, the specific technical solution is as follows:

[0060] Through virtualization technology, physical resources are unevenly allocated to multiple containers. The uneven allocation of physical resources is to obtain the best resource allocation method in the corresponding scenario through various algorithms. The resource allocation method is described as follows: This embodiment optimizes resource allocation by using the heuristic algorithm CRO (Chemical Reaction Optimization), and trains the algorithm through high-concurrency data scenarios to obtain the best resource allocation method.

[0061] It is worth mentioning that the CRO algorithm is a chemical reaction optimization algorithm that simulates the interaction between molecules in a chemical reaction to seek the process of minimum system potential energy.

[0062] The application of the CRO algorithm in the container allocation process is as follows: fix the number of containers and the number of resources, and abstract the number of allocated resources of the container into a matrix. Randomly generate a set of data matrices and put them into the CRO algorithm process for iterative calculation. The matrix output by the CRO algorithm is the number of container resources allocated to each container in the cluster. The data is iterated through the four CRO algorithms either randomly or in a certain order.

[0063] The first of the four CRO algorithms is a single-molecule wireless collision reaction, which randomly generates a row or a column of new random data. The second algorithm is a decomposition reaction, in which random rows or columns of the matrix are removed from the original matrix, and the missing content of the two matrices is filled with new random numbers. The third algorithm is a single-molecule invalid collision reaction, in which the two matrices after the split each generate a row or a column of new random data. The fourth algorithm is a synthesis reaction, in which the random rows and complementary residual random rows of the two matrices after the split are merged, and the other data are discarded.

[0064] The above algorithm process needs to be iterated repeatedly, controlling and changing the number of container slices and the number of physical clusters to perform repeated operations. The optimal allocation matrix that meets the constraints and benefits is calculated, and the solution obtained by the optimal allocation matrix at this time is the maximum solution.

[0065] The constraint condition is to calculate the container resource allocation with the maximum utilization rate through the CRO algorithm in a specific scenario with a fixed number of physical resources and a fixed number of containers. After obtaining the resource allocation matrix, the container deployment is performed according to the number of containers, the number of clusters, and the allocation method of the optimal solution.

[0066] The container deployment is to distribute physical resources among several containers, and isolate the physical resources between containers. At this time, a failure of a container will not affect other containers and will not affect data processing.

[0067] The container resource allocation is achieved by mapping physical resources to a virtual platform, which can control the allocation of physical resources and allocate different resource sizes to each container. The allocation method is to allocate through an allocation matrix obtained by training the CRO algorithm.

[0068] The allocation matrix needs to be iteratively solved according to different data concurrent application scenarios, different numbers of physical resources, and different containers.

[0069] The container is started according to the resource allocation method obtained by iterating the allocation matrix, and containers of different sizes are stored in the container cache when the operating system is initialized.

[0070] The container cache area is an area for placing idle containers. Since it takes a long time to start a container, setting a container cache area can improve data processing speed.

[0071] Specifically, the constraints in allocating physical resources to container slices are expressed as:

[0072]

[0073] In the formula, x ij Indicates that the memory of the i-th physical node is allocated to the container slice node j; represents the amount of cache resources allocated from physical node i to container slice j; R i Represents a container resource.

[0074] This embodiment abstracts the problem of how to improve the resource utilization of container cache processing data into a maximum optimization problem of resource benefits, virtualizes physical resources, and defines the unit of each physical resource as the resource provided by the unit node. The CPU computing power, memory resources, etc. that each node can provide are quantified as a unit resource value unit U. Assume that when the utilization rate of each node is 100%, the hardware resources that the node can provide are 1 U of resources. At the same time, the container slice processing data process is introduced into the benefit value R. Assume that each unit of hardware resources can bring 2 Rs to the operator's network improvement at 100% utilization.

[0075] In order to solve the problem of how to reasonably allocate physical resources in cache container slices, it is assumed that the physical cluster provides resources for the container. The physical network is represented as a weighted vectorless graph g = (I, L), where I = {1, 2, 3, ..., i} is a physical node and L is a set of physical links. The container resources are represented as R i , container slice is represented by J, and the value of J is J = {1, 2, ..., j}. This embodiment introduces a variable x ij , where x ij It means that the memory of the ith physical node is allocated to the cache container slice node j. Otherwise, x ij= 0. Introduce another variable It represents the amount of cache resources allocated by node i to container j. The above statement has the formula:

[0076]

[0077] Specifically, the resource allocation benefit calculation model in the process of allocating physical resources to container slices is expressed as:

[0078]

[0079] In the formula, F represents the total value of all container slices per unit time; F buff represents the total cost, p1 represents the cost coefficient, including energy consumption cost and hardware equipment cost; p i represents the energy consumption and hardware cost per unit cache content; u ij Indicates the number of iteration updates of the container slice; x ij Indicates that the memory of the i-th physical node is allocated to the container slice node j; It represents the amount of cache resources allocated from physical node i to container slice j; T represents unit time.

[0080] Since each slice can process different amounts of data, different costs, and different economic benefits, the economic benefits it brings are also different. The economic benefits of container slices that can process massive amounts of data with high concurrency are high, while the economic benefits of containers that process less data are low. Container caching is a manifestation of increased costs, but container cache slices can process incoming data in a timely manner. As for how to reasonably allocate physical resources and container resources to improve the overall economic benefits, this model configures weights for each node. Assuming w i is the weight of node i, and the value of each capacity cache resource is F i , then F i =fw i Based on the above statement, the value of a container slice per unit time T is:

[0081]

[0082] Therefore, the total value of all container slices per unit time is:

[0083]

[0084] Using container slices will incur costs. This example also establishes a cost model:

[0085]

[0086] Among them, F buff is the total cost, and p1 is the cost coefficient, including energy consumption cost and hardware equipment cost.i is the energy consumption and hardware cost of the unit cache content. u is the number of iteration updates of the container slice.

[0087] Based on the above costs and values, this embodiment proposes that by optimizing the cache configuration, the optimization problem can be transformed into a total benefit maximum problem, and the total benefit is defined as E Total , this problem can be expressed as:

[0088]

[0089] Where I = {1,2,3,...,i}, J = {1,2,...,j} and the formula For restriction conditions.

[0090] This embodiment provides a method for solving the cache container resource allocation problem based on the CRO algorithm. The calculation formula for the total benefit is an indeterminate polynomial problem, which is difficult to solve. Heuristic algorithms are an important way to solve such problems. The design of the chemical reaction optimization algorithm is derived from the chemical reaction process. In the CRO algorithm, the main research object is molecules, and each molecule is mainly composed of molecular structure and molecular energy, which mainly includes molecular potential energy and molecular kinetic energy. The molecular potential energy represents the objective function value, and the molecular structure represents the solution to a given problem. The core of the CRO algorithm is to simulate the process of chemical molecules reaching a low-energy stable state after interacting with each other.

[0091] Specifically, based on the constraints and with the goal of maximizing the resource allocation benefit, the CRO algorithm is iterated to generate the optimal strategy for allocating physical resources to container slices, including:

[0092] Initializing various parameters, including the number of molecules in the container, the upper limit of kinetic energy loss in the invalid collision reaction of the molecules, and the initial value of the kinetic energy of the molecules;

[0093] Randomly select a container slice and allocate a random number of physical resources to form an indication matrix X, which is used to indicate the container slice that obtains the physical resources;

[0094] Generate a resource allocation matrix F based on the indication matrix X, which is used to indicate the amount of physical resources acquired by the container slice;

[0095] Select one of the mechanisms in the CRO algorithm, take the maximum resource allocation benefit as the goal, iterate the resource allocation matrix F, and generate the intermediate matrix F1;

[0096] Compare the intermediate matrix F1 with the constraints and output the target resource allocation matrix F* that meets the constraints.

[0097] The cache container resource allocation mechanism process of the CRO algorithm in this embodiment is described as follows: Figure 3As shown in the flowchart, when there are multiple massive data flow processing processes, first, a container is randomly selected to allocate cache resources, where the matrix X is an indicator matrix, indicating which container obtains resources, and the optimal resource allocation is obtained by finding the optimal solution F. The optimal solution matrix F indicates how much physical resources are allocated to each container slice. The flowchart is described in detail below:

[0098] Initialization process: Generate a set of random input parameters X, initialize various parameters, set the number of container molecules, the upper limit of kinetic energy loss in invalid collision reactions of molecules, the initial value of molecular kinetic energy and other parameters.

[0099] It is worth mentioning that the CRO algorithm includes four mechanisms, namely, the cache allocation mechanism for single-molecule invalid collision reactions, the cache resource allocation mechanism for decomposition reactions, the intermolecular invalid collision reaction mechanism, and the cache allocation mechanism for synthesis reactions. The four mechanisms are explained as follows:

[0100] Cache allocation mechanism for single-molecule invalid collision reaction: A slight collision between a molecule and the container wall will have a slight effect on the molecular structure. The algorithm will have a slight effect in this step. Under the constraints, the resource cache is randomly redistributed. Thus, a new cache allocation matrix is ​​obtained. The implementation principle of the single-molecule invalid collision reaction is as follows. Assume that the input matrix X is a 4th-order matrix, and its input is [a 11 ,a 12 ,a 13 ,a 14 ; a 21 ,a 22 ,a 23 ,a 24 ; a 31 ,a 32 ,a 33 ,a 34 ; a 41 ,a 42 ,a 43 ,a 44 ], then the random line is a random number δ, and the output is [a 11 ,a 12 ,a 13 ,a 14 ; δ 21 ,δ 22 ,δ 23 ,δ 24 ; a 31 ,a 32 ,a 33 ,a 34 ; a 41 ,a 42 ,a 43 ,a 44 ].

[0101] Cache resource allocation mechanism for decomposition reaction: When a molecule collides with the container wall, two new molecules will be generated. The algorithm is as follows: randomly select a row of the cache resource allocation matrix F and randomly assign it to the corresponding row in the two new matrices until each row of the original matrix is ​​assigned to the new matrix, and randomly fill the unassigned places with positive numbers. The implementation principle of the decomposition reaction is as follows: the input is [a 11 ,a 12 ,a 13 ,a 14 ; a 21 ,a 22 ,a 23 ,a 24 ; a 31 ,a 32 ,a 33 ,a 34 ; a 41 ,a 42 ,a 43 ,a 44 ], this matrix is ​​randomly split into two, one of which is [a 11 ,a 12 ,a 13 ,a 14 β 21 ,β 22 ,β 23 ,β 24 ; a 31 ,a 32 ,a 33 ,a 34 β 41 ,β 42 ,β 43 ,β 44 ], and the other is [β 11 ,β 12 ,β 13 ,β 14 ; a 21 ,a 22 ,a 23 ,a 24 β 31 ,β 32 ,β 33 ,β 34 ; a 41 ,a 42 ,a 43 ,a 44 ].

[0102] Invalid collision reaction mechanism between molecules: When two chemical molecules collide, they will slightly affect each other. The principle of this mechanism is: get two new matrices from the cache allocation matrix, and each matrix has a slight change from the original matrix, and randomly fill a random row of the two original matrices with random numbers to generate a new matrix. Assuming that the input matrix X is a fourth-order matrix, the algorithm of the invalid collision reaction mechanism between molecules is: input matrix [a 11 ,a 12 ,a 13 ,a 14 ; a 21 ,a 22 ,a 23 ,a 24 ; a 31 ,a 32 ,a 33 ,a 34 ; a 41 ,a 42 ,a 43 ,a 44 ]and

[0103] [b 11 ,b 12 ,b 13 ,b 14 ; b 21 ,b 22 ,b 23 ,b 24 ; b 31 ,b 32 ,b 33 ,b 34 ; b 41 ,b 42 ,b 43 ,b 44 ], then the matrix after the intermolecular invalid collision reaction algorithm is [a 11 ,a 12 ,a 13 ,a 14 ; a 21 ,a 22 ,a 23 ,a 24 ; a 31 ,a 32 ,a 33 ,a 34 β 41 ,β 42 ,β 43 ,β 44 ] and [b 11 ,b 12 ,b 13 ,b 14 β 21 ,β 22 ,β23 ,β 24 ; b 31 ,b 32 ,b 33 ,b 34 ; b 41 ,b 42 ,b 43 ,b 44 ].

[0104] Cache allocation mechanism for synthesis reactions: After two molecules collide, they undergo a drastic change and become a new molecule. Randomly select rows from the two original matrices and generate them into a new matrix. The rows of the new matrix all come from the old matrix, and no new values ​​are generated. The algorithm process is as follows: First, input the matrix [a 11 ,a 12 ,a 13 ,a 14 ; a 21 ,a 22 ,a 23 ,a 24 ; a 31 ,a 32 ,a 33 ,a 34 ; a 41 ,a 42 ,a 43 ,a 44 ] and [b 11 ,b 12 ,b 13 ,b 14 ; b 21 ,b 22 ,b 23 ,b 24 ; b 31 ,b 32 ,b 33 ,b 34 ; b 41 ,b 42 ,b 43 ,b 44 The output result is the composite value of the two matrices, and the matrix result is [b 11 ,b 12 ,b 13 ,b 14 ; a 21 ,a 22 ,a 23 ,a 24 ; b 31 ,b 32 ,b 33 ,b 34 ; a 41 ,a 42 ,a 43 ,a44 ].

[0105] Adjustment function: Compare the new solution with the constraints. If it meets the constraints, pass this solution. If it does not meet the constraints, stop generating this new solution and continue iterating with the original solution.

[0106] Stop basis: This experiment is set as a for loop iteration, and the number of iterations is limited to θ. When the iteration stops, the matrix F* is output, and the molecular potential energy and molecular structure are output, which are the target equation and the solution to the given problem.

[0107] like Figure 4 As shown in the figure, after obtaining the solution of the objective equation given by the scenario data, that is, the optimal resource allocation matrix, the containers are deployed according to the number of containers, the number of clusters, and the allocation method of the obtained optimal solution.

[0108] When deploying containers, first pull up the management container, which is responsible for pulling up, managing, and communicating with all business containers. Containers only divide physical resources, and the operating system image needs to be loaded in the container to provide services. Container resources are allocated according to the optimal resource allocation matrix, and all container images are pulled up, started, and placed in the container cache.

[0109] In actual applications, the container cache can process real-time data after it is pulled up. Since the amount of data to be processed is often inconsistent and there are high concurrency scenarios, the container cache will promptly dispatch a suitable container to process real-time data. When the container finishes processing the data, it is in an idle state and is then recycled back to the container cache.

[0110] In summary, the present disclosure belongs to a container orchestration method for processing high-concurrency data scenarios, using container technology for data processing fault isolation, and storing redundant containers in a container cache pool for efficient data processing. The present disclosure analyzes limited physical resources, and uses the heuristic algorithm CRO to solve NP-hard problems to rationally plan containers, so as to ensure efficient data processing while maximizing resource utilization, and solve the problem that there is no algorithm to solve the container allocation strategy for resource-limited scenarios in the prior art. The proposal of the present disclosure requires in-depth research on the current implementation mechanism and principle of high-concurrency data processing and the combination of equipment requirements for fault isolation scenarios; the CRO algorithm is used to allocate limited physical cluster resources to the virtual container slice cache, so that the data can have the advantages of container fault isolation when processing, and can allocate resources most reasonably when the physical resource cluster is limited.

[0111] Figure 5 is a block diagram of a container cache allocation system of the present exemplary embodiment, such as Figure 5 As shown, an exemplary embodiment of the present disclosure provides a container cache allocation system, including:

[0112] An establishing unit 10 is used to establish constraint conditions and a resource allocation benefit calculation model in the process of allocating physical resources to container slices when there are multiple massive data stream processing processes;

[0113] The resource allocation benefit calculation model 20 is used to calculate the resource allocation benefits obtained in the process of allocating physical resources to container slices;

[0114] The operation unit 30 is used to generate an optimal strategy for allocating physical resources to container slices by performing iterative operations through the CRO algorithm based on constraint conditions and with the goal of maximizing resource allocation benefits.

[0115] Specifically, the constraints in allocating physical resources to container slices are expressed as:

[0116]

[0117] In the formula, x ij Indicates that the memory of the i-th physical node is allocated to the container slice node j; y xij represents the amount of cache resources allocated from physical node i to container slice j; R i Represents a container resource.

[0118] Specifically, the resource allocation benefit calculation model in the process of allocating physical resources to container slices is expressed as:

[0119]

[0120] In the formula, F represents the total value of all container slices per unit time; F buff represents the total cost, p1 represents the cost coefficient, including energy consumption cost and hardware equipment cost; p i represents the energy consumption and hardware cost per unit cache content; u ij Indicates the number of iteration updates of the container slice; x ij Indicates that the memory of the i-th physical node is allocated to the container slice node j; It represents the amount of cache resources allocated from physical node i to container slice j; T represents unit time.

[0121] Specifically, the computing unit includes:

[0122] An initialization module, used to initialize various parameters, including the number of molecules in the container, the upper limit of kinetic energy loss in the invalid collision reaction of the molecules, and the initial value of the kinetic energy of the molecules;

[0123] A forming module, used to randomly select a container slice and allocate a random number of physical resources, forming an indication matrix X, used to indicate the container slice for obtaining the physical resources;

[0124] A generating module, used to generate a resource allocation matrix F based on the indication matrix X, used to indicate the amount of physical resources acquired by the container slice;

[0125] The operation module is used to select one of the mechanisms in the CRO algorithm, take the maximum resource allocation benefit as the goal, perform iterative operations on the resource allocation matrix F, and generate an intermediate matrix F1;

[0126] The output module is used to compare the intermediate matrix F1 with the constraint conditions and output the target resource allocation matrix F* that meets the constraint conditions.

[0127] Figure 6 Schematic diagram of the structure of a device of this exemplary embodiment. Figure 6 As shown, corresponding to the container cache allocation method provided above, the present disclosure also provides a device. Since the embodiment of the device is similar to the above method embodiment, the description is relatively simple. Please refer to the description of the above method embodiment for relevant details. The device described below is only schematic. The device may include: a processor 1, a memory 2, a communication bus (i.e., the above device bus) and a search engine, wherein the processor 1 and the memory 2 communicate with each other through the communication bus and communicate with the outside through the communication interface. The processor 1 can call the logic instructions in the memory 2 to execute the container cache allocation method.

[0128] In addition, the logic instructions in the above-mentioned memory 2 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: a storage chip, a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and other media that can store program codes.

[0129] On the other hand, an embodiment of the present disclosure further provides a processor-readable storage medium, on which a computer program 3 is stored. When the computer program 3 is executed by the processor 1, the container cache allocation method provided in the above embodiments is implemented.

[0130] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor 1, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.

[0131] The above are only preferred implementations of the present disclosure, and the protection scope of the present disclosure is not limited to the above embodiments. All technical solutions under the concept of the present disclosure belong to the protection scope of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present disclosure should be regarded as the protection scope of the present disclosure.

Claims

1. A container cache allocation method, characterized in that: include: When there are multiple massive data stream processing processes, establish the constraints for allocating physical resources to container slices and the resource allocation benefit calculation model; The resource allocation benefit calculation model is used to calculate the resource allocation benefit obtained in the process of allocating physical resources to container slices; Based on the constraints and with the goal of maximizing the resource allocation benefit, the CRO algorithm is iteratively calculated to generate the optimal strategy for allocating physical resources to container slices.

2. The container cache allocation method according to claim 1, characterized in that: The constraints in allocating physical resources to container slices are expressed as: In the formula, x ij Indicates that the memory of the i-th physical node is allocated to the container slice node j; represents the amount of cache resources allocated from physical node i to container slice j; R i Represents a container resource.

3. The container cache allocation method according to claim 1, characterized in that: The resource allocation benefit calculation model in the process of allocating physical resources to container slices is expressed as: In the formula, F represents the total value of all container slices per unit time; F buff represents the total cost, p1 represents the cost coefficient, including energy consumption cost and hardware equipment cost; p i represents the energy consumption and hardware cost per unit cache content; u ij Indicates the number of iterations and updates of the container slice; x ij Indicates that the memory of the i-th physical node is allocated to the container slice node j; It represents the amount of cache resources allocated from physical node i to container slice j; T represents unit time.

4. The container cache allocation method according to claim 1, characterized in that: Based on the constraints and with the goal of maximizing the resource allocation benefit, the CRO algorithm is iterated to generate the optimal strategy for allocating physical resources to container slices, including: Initializing various parameters, including the number of molecules in the container, the upper limit of kinetic energy loss in the invalid collision reaction of the molecules, and the initial value of the kinetic energy of the molecules; Randomly select a container slice and allocate a random number of physical resources to form an indication matrix X, which is used to indicate the container slice that obtains the physical resources; Generate a resource allocation matrix F based on the indication matrix X, which is used to indicate the amount of physical resources acquired by the container slice; Select one of the mechanisms in the CRO algorithm, take the maximum resource allocation benefit as the goal, iterate the resource allocation matrix F, and generate the intermediate matrix F1; Compare the intermediate matrix F1 with the constraints and output the target resource allocation matrix F* that meets the constraints.

5. A container cache allocation system, characterized in that: include: An establishment unit is used to establish constraints and resource allocation benefit calculation models in the process of allocating physical resources to container slices when there are multiple massive data stream processing processes; Resource allocation benefit calculation model, used to calculate the resource allocation benefits obtained in the process of allocating physical resources to container slices; The computing unit is used to generate the optimal strategy for allocating physical resources to container slices by performing iterative operations through the CRO algorithm based on constraints and with the goal of maximizing resource allocation benefits.

6. The container cache allocation system according to claim 5, characterized in that: The constraints in allocating physical resources to container slices are expressed as: In the formula, x ij Indicates that the memory of the i-th physical node is allocated to the container slice node j; represents the amount of cache resources allocated from physical node i to container slice j; R i Represents a container resource.

7. The container cache allocation system according to claim 5, characterized in that: The resource allocation benefit calculation model in the process of allocating physical resources to container slices is expressed as: In the formula, F represents the total value of all container slices per unit time; F buff represents the total cost, p1 represents the cost coefficient, including energy consumption cost and hardware equipment cost; p i represents the energy consumption and hardware cost per unit cache content; u ij Indicates the number of iterations and updates of the container slice; x ij Indicates that the memory of the i-th physical node is allocated to the container slice node j; It represents the amount of cache resources allocated from physical node i to container slice j; T represents unit time.

8. The container cache allocation system according to claim 5, characterized in that: The computing unit includes: An initialization module, used to initialize various parameters, including the number of molecules in the container, the upper limit of kinetic energy loss in the invalid collision reaction of the molecules, and the initial value of the kinetic energy of the molecules; A forming module, used to randomly select a container slice and allocate a random number of physical resources, forming an indication matrix X, used to indicate the container slice for obtaining the physical resources; A generating module, used to generate a resource allocation matrix F based on the indication matrix X, used to indicate the amount of physical resources acquired by the container slice; The operation module is used to select one of the mechanisms in the CRO algorithm, take the maximum resource allocation benefit as the goal, perform iterative operations on the resource allocation matrix F, and generate an intermediate matrix F1; The output module is used to compare the intermediate matrix F1 with the constraint conditions and output the target resource allocation matrix F* that meets the constraint conditions.

9. A device, characterized in that: include: Processor and memory; The memory is used to store a computer program, and the processor calls the computer program stored in the memory to execute the container cache allocation method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is enabled to execute the container cache allocation method according to any one of claims 1 to 4.

Images