Application deployment method, apparatus, system, device, and computer readable storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WEBANK (CHINA)
- Filing Date
- 2019-08-12
- Publication Date
- 2026-05-29
AI Technical Summary
The current deployment of financial business applications on cloud service machines involves cumbersome and complex procedures, and redeployment is particularly troublesome when data surges or drops sharply, making it impossible to deploy applications flexibly and conveniently.
The application deployment method based on Docker is adopted. The target data volume of the business platform is predicted through the batch data management platform, the number of clusters is calculated using the preset strategy model, and the target clusters are automatically applied for and deployed through the Docker container management platform.
It enables flexible application deployment, avoids deployment lag, reduces operational complexity, and improves deployment efficiency and resource utilization.
Smart Images
Figure CN110471671B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of financial technology (Fintech), and more particularly to application deployment methods, apparatus, systems, devices, and computer-readable storage media. Background Technology
[0002] With the rapid development of internet technology, especially internet financial technology (Fintech), more and more technologies (big data, distributed systems, blockchain, artificial intelligence, etc.) are being applied in the financial field, and financial business platforms are also increasing.
[0003] Currently, in order to effectively manage financial business, financial business applications are usually deployed on cloud service machines (cloud servers use minicomputers or virtual machines). The application deployment process first requires operations and maintenance personnel to apply for machines on the cloud service, configure relevant machine environment variables, install the application software corresponding to the financial business, adjust the associated application configuration information on the DCN (Data Center Node) area, and finally complete the application deployment. This application deployment method has a cumbersome and complex operation process and high risks. When there is a surge or drop in data, the application redeployment is also particularly troublesome. Summary of the Invention
[0004] The main objective of this invention is to propose an application deployment method, apparatus, system, device, and computer-readable storage medium based on Docker, aiming to solve the current technical problem that applications cannot be deployed flexibly and conveniently according to the actual business data volume of the application.
[0005] To achieve the above objectives, the present invention provides a Docker-based application deployment method, which includes the following steps:
[0006] Upon receiving an application deployment request from the business platform, obtain the application information corresponding to the application deployment request;
[0007] Query the application database of the business platform, and predict the target amount of data to be processed by the business platform based on the historical business data in the application database;
[0008] The target data volume is input into a preset strategy model to calculate the number of clusters.
[0009] The application information is sent to the Docker container management platform, and a target cluster corresponding to the number of clusters is requested from the Docker container management platform so that the application can be deployed in the target cluster.
[0010] Optionally, the step of querying the application database of the business platform and predicting the target data volume to be processed by the business platform based on historical business data in the application database includes:
[0011] Query the application database of the business platform to obtain historical business data within a preset time period:
[0012] The historical business data is input into a preset data analysis model, filtered by the data analysis model, and the filtered historical business data is associated according to the business process to generate a relational data table.
[0013] The data processing volume of each business process in the relational data table is counted, and the sum of the data processing volumes is taken as the target data volume to be processed by the business platform in a future preset time period.
[0014] Optionally, before the step of inputting the target data volume into a preset strategy model to calculate the number of clusters, the following steps are included:
[0015] Output a time setting prompt message to allow the user to set a time indicator for processing the target amount of data;
[0016] The step of inputting the target data volume into a preset strategy model to calculate the number of clusters includes:
[0017] The target data volume and the time metric are input into a preset strategy model, the preset strategy model is run, and the number of clusters is calculated.
[0018] Optionally, the step of inputting the target data volume and the time indicator into a preset strategy model, running the preset strategy model, and calculating the number of clusters includes:
[0019] The target data volume and the time index are input into a preset strategy model, and the target data volume and the time index are processed by the preset strategy model to obtain the number of clusters;
[0020] The algorithm in the preset strategy model includes:
[0021]
[0022] In the formula, n is the number of clusters; f(S) is the target data volume; and G... i Multi-process processing capability for each machine step; the T i The time metrics required to be processed for each machine; the P i The time for processing each data entry in a single step after adding a machine; f(D) represents the change in the amount of data for each month's date; the CPUi The CPU utilization rate for each machine running a batch, the CPU max The maximum CPU utilization for each machine running a batch; the I / O i The IO utilization rate for each machine running a batch, the IO max The MER is the upper limit of IO utilization for each machine running a batch; i The MER represents the memory usage for each machine running a batch. max The maximum memory usage per machine for batch processing; the GW i The gateway occupancy rate for each machine running batches, the GW max Set the upper limit for gateway occupancy for each machine running batches.
[0023] Optionally, after the step of sending the application information to the Docker container management platform and requesting a target cluster corresponding to the number of clusters from the Docker container management platform for application deployment in the target cluster, the method includes:
[0024] When the batch processing task is detected to have completed execution, a resource reclamation request is sent to the Docker container management platform to reclaim resources.
[0025] In addition, to achieve the above objectives, the present invention also provides a Docker-based application deployment system, which includes: a batch data management platform, a business platform, and a Docker container management platform connected by communication;
[0026] The Docker-based application deployment system implements the following steps:
[0027] When the batch data management platform receives an application deployment request sent by the business platform, the batch data management platform obtains the application information corresponding to the application deployment request.
[0028] The batch data management platform queries the application database of the business platform, and predicts the target amount of data to be processed by the business platform based on the historical business data in the application database.
[0029] The batch data management platform inputs the target data volume into a preset strategy model and calculates the number of clusters.
[0030] The batch data management platform sends the application information to the Docker container management platform and requests the target clusters corresponding to the number of clusters from the Docker container management platform;
[0031] The Docker container management platform sends the application information to the target cluster for application deployment within the target cluster.
[0032] Optionally, the Docker-based application deployment system further includes: a target cluster, which includes a master cluster and slave clusters connected by communication;
[0033] After the step of the Docker container management platform sending the application information to the target cluster for application deployment in the target cluster, the following steps are included:
[0034] When the main control cluster receives a batch processing task sent by the service platform, the main control cluster obtains the batch service data corresponding to the batch processing task.
[0035] The master control cluster counts the number of processes in each slave cluster to obtain the total number of execution processes. The master control cluster then divides the batch of business data according to the total number of execution processes to obtain sharded business data.
[0036] The master control cluster sends the fragmented service data to each slave cluster;
[0037] When a slave cluster receives the fragmented service data sent by the master cluster, the slave cluster processes the fragmented service data and feeds back the processing result of the fragmented service data to the master cluster.
[0038] The master cluster receives the processing results from each slave cluster, and the master cluster summarizes the processing results to obtain the total processing result of the batch processing task.
[0039] When the overall processing result is "processing complete", the main control cluster sends a notification message indicating that the batch processing task has been completed to the batch data management platform.
[0040] The batch data management platform sends the notification message to the Docker container management platform;
[0041] The Docker container management platform performs resource reclamation based on the aforementioned prompts.
[0042] Optionally, when the slave cluster receives the sharded service data sent by the master cluster, after the slave cluster processes the sharded service data, the process includes:
[0043] The subordinate cluster sends heartbeat data back to the master cluster;
[0044] The master control cluster receives heartbeat data from each of the slave clusters and sends the heartbeat data to the business platform.
[0045] The business platform receives the heartbeat data from each of the aforementioned heartbeat data, and then feeds back the heartbeat data to the batch data management platform.
[0046] The batch data management platform receives each heartbeat data sent by the business platform, and compares the device identifier in each heartbeat data with a preset identifier corresponding to the number of clusters;
[0047] If the device identifier in each of the heartbeat data is not exactly the same as the preset identifier corresponding to the number of clusters, the batch data management platform applies to the Docker container management platform for a new target cluster;
[0048] The Docker container management platform sends the application information to the new target cluster for application deployment in the new target cluster.
[0049] Furthermore, to achieve the above objectives, the present invention also provides a Docker-based application deployment device, the Docker-based application deployment device comprising:
[0050] The request receiving module is used to obtain the application information corresponding to the application deployment request when it receives the application deployment request sent by the business platform.
[0051] The query prediction module is used to query the application database of the business platform and predict the target amount of data to be processed by the business platform based on the historical business data in the application database.
[0052] The quantity determination module is used to input the target data volume into a preset strategy model and calculate the number of clusters.
[0053] The application deployment module is used to send the application information to the Docker container management platform, apply to the Docker container management platform for the target cluster corresponding to the number of clusters, and deploy the application in the target cluster.
[0054] In addition, to achieve the above objectives, the present invention also provides a Docker-based application deployment device, which includes: a memory, a processor, and a Docker-based application deployment program stored on the memory and executable on the processor. When the Docker-based application deployment program is executed by the processor, it implements the steps of the Docker-based application deployment method as described above.
[0055] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a Docker-based application deployment program, which, when executed by a processor, implements the steps of the Docker-based application deployment method as described above.
[0056] This invention provides a Docker-based application deployment method, apparatus, system, device, and computer-readable storage medium. In this embodiment, a batch data management platform predicts the target data volume to be generated by the business platform in the future based on historical business data. The batch data management platform inputs the target data volume into a preset strategy model to obtain the number of clusters required to process the target data volume. The platform then deploys the target clusters corresponding to the number of clusters to process the target data volume. This embodiment determines the number of clusters based on the target data volume for application deployment, achieving flexible application deployment and avoiding deployment delays. Furthermore, by using a Docker container management platform for application deployment, manual deployment by the user is eliminated, reducing the complexity of application deployment operations. Attached Figure Description
[0057] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention;
[0058] Figure 2 This is a flowchart illustrating the first embodiment of the Docker-based application deployment method of the present invention.
[0059] Figure 3 This is a flowchart illustrating the second embodiment of the Docker-based application deployment method of the present invention;
[0060] Figure 4 This is a schematic diagram of a specific scenario of the second embodiment of the Docker-based application deployment method of the present invention;
[0061] Figure 5 This is a schematic diagram of the functional modules of an embodiment of the Docker-based application deployment device of the present invention.
[0062] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0063] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0064] like Figure 1 As shown, Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention.
[0065] In this embodiment of the invention, the application deployment device based on Docker can be a PC or a server.
[0066] like Figure 1As shown, the Docker-based application deployment device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0067] Those skilled in the art will understand that Figure 1 The device structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0068] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a Docker-based application deployment program.
[0069] exist Figure 1 In the device shown, the network interface 1004 is mainly used to connect to the backend server and communicate with the backend server; the user interface 1003 is mainly used to connect to the client (user terminal) and communicate with the client; and the processor 1001 can be used to call the Docker-based application deployment program stored in the memory 1005 and execute the operations in the Docker-based application deployment method described below. The method implemented when the Docker-based application deployment program running on the processor is executed can be referred to the various embodiments of the Docker-based application deployment method of the present invention, which will not be repeated here.
[0070] Based on the above hardware structure, an embodiment of the application deployment method based on Docker of the present invention is proposed.
[0071] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the Docker-based application deployment method of the present invention. The method includes:
[0072] Step S10: Upon receiving an application deployment request sent by the business platform, obtain the application information corresponding to the application deployment request.
[0073] This embodiment of the application deployment method based on Docker is applied to a Docker-based application deployment device (also called GMP, GrobalManager Processor, batch data management platform). In this embodiment, the batch data management platform acts as an intermediary platform to connect the business platform and the Docker container management platform. In addition, the Docker container management platform communicates with multiple clusters that process business.
[0074] The batch data management platform receives application deployment requests sent by the business platform. The triggering method for these requests is not specifically limited; that is, the requests can be initiated by operations personnel or automatically by the business platform. In this embodiment, the number of business platforms is not specifically limited; that is, there can be one or more business platforms. This embodiment uses one business platform triggering one application deployment request as an example for explanation. Specifically:
[0075] When the batch data management platform receives an application deployment request from the business platform, it obtains application information related to the application deployment request. Application information refers to information related to application deployment, such as the configuration file for the execution of the business program.
[0076] That is, when the business platform detects an application deployment request, it associates the application deployment-related information with the application deployment request and sends it to the batch data management platform, so that the batch data management platform can deploy the application based on the application information. Specifically:
[0077] Step S20: Query the application database of the business platform, and predict the target amount of data to be processed by the business platform based on the historical business data in the application database.
[0078] The batch data management platform queries the application database of the business platform and obtains historical business data from the application database. Historical business data refers to data generated during application execution. Based on historical business data, the batch data management platform predicts the amount of target data to be processed that the business platform may generate in the future.
[0079] For example, if the business platform is an e-commerce transaction platform, historical business data includes: unit price of goods, customer information, sales time of goods, sales volume of goods, etc. The batch data management platform obtains one day's historical business data from the application database of the e-commerce transaction platform: 100 million historical business data were generated in the e-commerce transaction platform on November 11, 2018. The batch data management platform multiplies the historical business data by the year-on-year growth rate to obtain 120 million business data generated in the e-commerce transaction platform on November 11, 2019. The batch data management platform uses 120 million business data as the target data volume to be processed by the business platform.
[0080] It is understandable that the batch data management platform can predict the target data volume to be processed by the business platform based on historical business data in different ways. Specifically, in this embodiment, the batch data management platform's prediction of the target data volume to be processed by the business platform involves specific data analysis. If the application database of the business platform is a traditional relational database, such as Oracle or MySQL, the batch data management platform queries the application database to obtain historical business data and inputs the historical business data into a preset large-scale data management platform (the preset large-scale data management platform refers to a pre-set large-scale data management platform, such as Hadoop or Hive) for MapReduce distributed computing analysis to accurately and quickly predict the target data volume to be processed by the business platform.
[0081] Step S30: Input the target data volume into the preset strategy model to calculate the number of clusters.
[0082] After obtaining the target data volume, the batch data management platform determines the number of clusters to be used for data processing based on the target data volume. Specifically:
[0083] In this embodiment, a preset strategy model is a cluster quantity estimation model trained in advance based on historical business data. Specifically, the preset strategy model is derived by analyzing the relationships between the total data processing time of each machine in the cluster, the average processing time per data item per machine, the processing time per step per machine, the peak date variation for each month, holidays, marketing dates, the upper limit of CPU utilization, the upper limit of IO utilization, the upper limit of memory utilization, and the upper limit of gateway utilization for each machine in the cluster when the batch data management platform processes historical business data. The preset strategy model includes algorithms such as:
[0084]
[0085] In the formula, n is the number of clusters; f(S) is the target data volume; and G... i Multi-process processing capability for each machine step; the T i The time metrics required to be processed for each machine; the P i The time for processing each data entry in a single step after adding a machine; f(D) represents the change in the amount of data for each month's date; the CPU i The CPU utilization rate for each machine running a batch, the CPU maxThe maximum CPU utilization for each machine running a batch; the I / O i The IO utilization rate for each machine running a batch, the IO max The MER is the upper limit of IO utilization for each machine running a batch; i The MER represents the memory usage for each machine running a batch. max The maximum memory usage per machine for batch processing; the GW i The gateway occupancy rate for each machine running batches, the GW max Set the upper limit for gateway occupancy for each machine running batches.
[0086] Before determining the number of clusters using the preset strategy model in this embodiment, the batch data management platform needs to set time indicators. That is, before step S30, the batch data management platform outputs time setting prompt information so that the user can set the time indicator for processing the target amount of data.
[0087] After obtaining the time metrics for the target data volume, the batch data management platform inputs the target data volume and time metrics into a preset strategy model. The algorithm in the preset strategy model processes the target data volume and time metrics to obtain the number of clusters.
[0088] Specifically, in the preset strategy model algorithm, the dependent variable is the number of clusters, and the independent variables are the target data volume and time indicators. In addition, the preset strategy model algorithm also sets other influencing factors, such as the time for processing each data item in a single step after adding a machine; the change in the amount of data per month and date; the CPU utilization rate of each machine running the batch, and the upper limit of the CPU utilization rate of each machine running the batch; the IO utilization rate of each machine running the batch, and the upper limit of the IO utilization rate of each machine running the batch; the memory utilization rate of each machine running the batch, and the upper limit of the memory utilization rate of each machine running the batch; the gateway utilization rate of each machine running the batch, and the upper limit of the gateway utilization rate of each machine running the batch. Furthermore, those skilled in the art will understand that the influencing factors in the preset strategy model algorithm can be a fixed constant or a variable constant within a certain numerical range.
[0089] Step S40: Send the application information to the Docker container management platform and request a target cluster corresponding to the number of clusters from the Docker container management platform so as to deploy the application in the target cluster.
[0090] The batch data management platform sends application information to the Docker container management platform and requests the target cluster corresponding to the number of clusters from the Docker container management platform. In other words, the batch data management platform sends application information to the target cluster through the Docker container management platform so that the application can be deployed in the target cluster.
[0091] For example, before scaling up Docker, the production environment consisted of 8 machines. Due to rapid growth in data volume caused by business expansion, the processing time for batch applications increased significantly. After using dynamic scaling, a customized latest batch processing completion time was set. Based on the existing data volume, it was determined that the number of Docker containers needed to be increased by 100%. After scaling up from 8 machines to 16 Docker machines, the average processing time decreased to 52%. Table 1 below shows the comparison and optimization of Docker scaling time before (8 machines) and after (16 machines).
[0092]
[0093] Table 1
[0094] In this embodiment of the invention, the batch data management platform predicts the target data volume to be generated by the business platform in the future based on historical business data. The batch data management platform inputs the target data volume into a preset strategy model to obtain the number of clusters required to process the target data volume. The target clusters corresponding to the number of clusters are then deployed to process the target data volume. In this embodiment, the number of clusters is determined based on the target data volume for application deployment. On the one hand, this achieves flexible application deployment and avoids the lag in application deployment. On the other hand, application deployment is carried out through the Docker container management platform, which eliminates the need for users to manually deploy applications and reduces the complexity of application deployment operations.
[0095] Furthermore, referring to Figure 3 Based on the first embodiment of the Docker-based application deployment method of the present invention, a second embodiment of the Docker-based application deployment method of the present invention is proposed.
[0096] This embodiment is a refinement of step S20 in the first embodiment. The difference between this embodiment and the first embodiment is that:
[0097] Step S21: Query the application database of the business platform to obtain historical business data within a preset time period.
[0098] The batch data management platform queries the application database of the business platform and obtains historical business data within a preset time period. The preset time period can be flexibly set according to specific scenarios, such as one month, one week, or one day.
[0099] Step S22: Input the historical business data into a preset data analysis model, filter the historical business data through the data analysis model, and associate the filtered historical business data according to the business process to generate a relational data table.
[0100] The batch data management platform has a preset data analysis model. The data analysis model refers to the data processing code written by developers according to the business process. For example, the preset data analysis model is a Hive script.
[0101] The batch data management platform inputs historical business data into a preset data analysis model. The model filters the historical business data, removing redundant data. The platform then associates the filtered historical business data according to business processes, generating relational data tables. Figure 4 The batch data management platform extracts and analyzes historical business data from the DB (Database application database) to generate relational data tables.
[0102] Step S23: Calculate the data processing volume of each business process in the relational data table, and use the sum of the data processing volumes as the target data volume to be processed by the business platform in the future preset time period.
[0103] The batch data management platform statistically analyzes the data processing volume of each business process in a relational data table. The sum of these volumes is then used as the target data volume to be processed by the business platform within a predetermined timeframe. In this embodiment, the batch data management platform utilizes a pre-defined data analysis model to analyze business data, accurately analyzing the target data volume based on historical business data. This approach ensures accuracy while accelerating data analysis efficiency.
[0104] Furthermore, based on the above embodiments of the Docker-based application deployment method of the present invention, a third embodiment of the Docker-based application deployment method of the present invention is proposed.
[0105] This embodiment follows step S40 in the first embodiment. The difference between this embodiment and the first embodiment is that:
[0106] When the batch processing task is detected to have completed execution, a resource reclamation request is sent to the Docker container management platform to reclaim resources.
[0107] When the target cluster completes the batch processing task, it sends a completion message to the business platform. The business platform then forwards this message to the batch data management platform. Upon receiving this completion message, the batch data management platform detects that the batch processing task has finished and sends a resource reclamation request to the Docker container management platform. This request allows the Docker container management platform to register the target cluster's information for resource reclamation. Figure 4 .
[0108] In this embodiment, the batch data management platform intelligently identifies the completion status of batch operations and automatically recycles the processing flow, realizing dynamic expansion and dynamic recycling of resources, effectively avoiding the situation where resources cannot be dynamically recycled and thus causing resource waste.
[0109] Furthermore, this embodiment of the invention also provides an application deployment system based on Docker. In a first embodiment of the application deployment system based on Docker of this invention, the application deployment system based on Docker includes: a batch data management platform, a business platform, and a Docker container management platform that are connected by communication.
[0110] The Docker-based application deployment system implements the following steps:
[0111] When the batch data management platform receives an application deployment request sent by the business platform, the batch data management platform obtains the application information corresponding to the application deployment request.
[0112] The batch data management platform queries the application database of the business platform, and predicts the target amount of data to be processed by the business platform based on the historical business data in the application database.
[0113] The batch data management platform inputs the target data volume into a preset strategy model and calculates the number of clusters.
[0114] The batch data management platform sends the application information to the Docker container management platform and requests the target clusters corresponding to the number of clusters from the Docker container management platform;
[0115] The Docker container management platform sends the application information to the target cluster for application deployment within the target cluster.
[0116] The first embodiment of the Docker-based application deployment system of the present invention can be referred to in the various embodiments of the Docker-based application deployment method of the present invention, which will not be repeated here.
[0117] Furthermore, based on the first embodiment of the Docker-based application deployment system of the present invention, a second embodiment of the Docker-based application deployment system of the present invention is proposed.
[0118] The difference between this embodiment and the first embodiment is that the application deployment system based on Docker in this embodiment further includes: a target cluster, which includes a master cluster (also called a master machine) and a slave cluster (also called a slave machine) connected by communication.
[0119] After the step of the Docker container management platform sending the application information to the target cluster for application deployment in the target cluster, the following steps are included:
[0120] When the main control cluster receives a batch processing task sent by the service platform, the main control cluster obtains the batch service data corresponding to the batch processing task.
[0121] The master control cluster counts the number of processes in each slave cluster to obtain the total number of execution processes. The master control cluster then divides the batch of business data according to the total number of execution processes to obtain sharded business data.
[0122] The master control cluster sends the fragmented service data to each slave cluster;
[0123] When a slave cluster receives the fragmented service data sent by the master cluster, the slave cluster processes the fragmented service data and feeds back the processing result of the fragmented service data to the master cluster.
[0124] The master cluster receives the processing results from each slave cluster, and the master cluster summarizes the processing results to obtain the total processing result of the batch processing task.
[0125] When the overall processing result is "processing complete", the main control cluster sends a notification message indicating that the batch processing task has been completed to the batch data management platform.
[0126] The batch data management platform sends the notification message to the Docker container management platform;
[0127] The Docker container management platform performs resource reclamation based on the aforementioned prompts.
[0128] Specifically, after the application deployment is completed, the business platform sends batch business data to the target cluster and uses the target cluster to process the batch business data. That is, the business platform receives business operation requests and sends the batch processing tasks corresponding to the business operation requests to the master control cluster in the target cluster. When the master control cluster receives the batch processing tasks sent by the business platform, the master control cluster obtains the batch business data corresponding to the batch processing tasks.
[0129] The master control cluster counts the number of processes in each slave cluster to obtain the total number of executing processes. Based on the total number of executing processes, the master control cluster divides the batch business data into shards, resulting in shards of business data. For example, if there are 5 slave clusters and each slave cluster has 4 processes (which can be understood as one process corresponding to one CPU), then the total number of executing processes is 20. The master control cluster divides the batch business data into 20 parts, resulting in 20 shards of business data. The master control cluster then sends each shard of business data to its respective slave cluster.
[0130] When a slave cluster receives fragmented service data from the master cluster, the slave cluster processes the fragmented service data. After processing, the slave cluster sends the processing result back to the master cluster. The master cluster receives the processing results from each slave cluster and summarizes them to obtain the overall processing result of the batch processing task. When the overall processing result indicates completion, the master cluster sends a notification message indicating that the batch processing task has been completed to the batch data management platform.
[0131] The batch data management platform sends a notification to the Docker container management platform; the Docker container management platform then undoes the previous application deployment and reclaims resources based on the notification. In this embodiment, resource reclamation can be performed promptly upon completion of the batch processing task to prevent resource hoarding and effectively reduce the number of orphan machines.
[0132] Furthermore, based on the above embodiments of the Docker-based application deployment system of the present invention, a third embodiment of the Docker-based application deployment system of the present invention is proposed.
[0133] The difference between this embodiment and the above embodiment is that, when the slave cluster receives the sharded service data sent by the master cluster, after the slave cluster processes the sharded service data, it further includes:
[0134] The subordinate cluster sends heartbeat data back to the master cluster;
[0135] The master control cluster receives heartbeat data from each of the slave clusters and sends the heartbeat data to the business platform.
[0136] The business platform receives the heartbeat data from each of the aforementioned heartbeat data, and then feeds back the heartbeat data to the batch data management platform.
[0137] The batch data management platform receives each heartbeat data sent by the business platform, and compares the device identifier in each heartbeat data with a preset identifier corresponding to the number of clusters;
[0138] If the device identifier in each of the heartbeat data is not exactly the same as the preset identifier corresponding to the number of clusters, the batch data management platform applies to the Docker container management platform for a new target cluster;
[0139] The Docker container management platform sends the application information to the new target cluster for application deployment in the new target cluster.
[0140] Specifically, in this embodiment, when the slave clusters process sharded service data, the slave clusters will send heartbeat data back to the master cluster in real time. The master cluster receives the heartbeat data from each slave cluster and sends each heartbeat data to the service platform; the service platform receives each heartbeat data and sends the information from each heartbeat data to the batch data management platform; the batch data management platform receives each heartbeat data sent by the service platform and compares the device identifier in each heartbeat data with the preset identifier corresponding to the number of clusters; wherein, the preset identifier refers to the identifier information of the target cluster to be deployed that is set in advance.
[0141] If the device identifier in each heartbeat data is exactly the same as the preset identifier corresponding to the number of clusters, the batch data management platform determines that all target clusters have been successfully deployed. The batch data management platform then checks the task execution status of each target cluster in order to reclaim resources when the batch tasks are completed.
[0142] If the device identifier in each heartbeat data is not completely the same as the preset identifier corresponding to the number of clusters, the batch data management platform determines that there is a cluster deployment failure. The batch data management platform applies to the Docker container management platform for a new target cluster. The Docker container management platform sends the application information to the new target cluster for application deployment.
[0143] For example, if batch data management requires the deployment of 10 clusters, the batch data management system requests 10 clusters from the Docker container management platform. The preset identifiers corresponding to the 10 clusters are 1, 2, 3, 4 up to 10. The batch data management platform receives 9 heartbeat data points with identifiers 2, 3, 4, 5 up to 10. The batch data management platform determines that the deployment of the target cluster with device identifier 1 has failed, and then requests a new target cluster from the Docker container management platform. The Docker container management platform sends the application information to the new target cluster for application deployment.
[0144] In this embodiment, the Docker container management platform can flexibly apply for target clusters during business data processing, enabling dynamic and flexible deployment of applications.
[0145] This embodiment specifically illustrates a flexible deployment scenario when some target cluster applications fail to deploy. However, actual application deployment scenarios are not limited to deployment failures. If the target cluster corresponding to the number of clusters has been successfully deployed, but the business data processing time of the business platform does not meet the expected processing time settings, the batch data management platform can also apply for a new target cluster from the Docker container management platform. This will not be elaborated upon in this embodiment.
[0146] refer to Figure 5 The present invention also provides a Docker-based application deployment device, the Docker-based application deployment device comprising:
[0147] The request receiving module 10 is used to obtain application information corresponding to the application deployment request when it receives an application deployment request sent by the business platform.
[0148] The query prediction module 20 is used to query the application database of the business platform and predict the target amount of data to be processed by the business platform based on the historical business data in the application database.
[0149] The quantity determination module 30 is used to input the target data volume into a preset strategy model and calculate the number of clusters.
[0150] The application deployment module 40 is used to send the application information to the Docker container management platform, apply to the Docker container management platform for the target cluster corresponding to the number of clusters, and deploy the application in the target cluster.
[0151] In one embodiment, the query prediction module 20 includes:
[0152] The data query unit is used to query the application database of the business platform to obtain historical business data within a preset time period.
[0153] The table generation unit is used to input the historical business data into a preset data analysis model, filter the historical business data through the data analysis model, and associate the filtered historical business data according to the business process to generate a relational data table.
[0154] The statistical prediction unit is used to calculate the data processing volume of each business process in the relational data table, and to take the sum of the data processing volumes as the target data volume to be processed by the business platform in a future preset time period.
[0155] In one embodiment, the Docker-based application deployment apparatus includes:
[0156] The indicator setting module is used to output time setting prompts so that users can set time indicators for processing the target amount of data.
[0157] The quantity determination module 30 is used to input the target data volume and the time index into a preset strategy model, run the preset strategy model, and calculate the number of clusters.
[0158] In one embodiment, the quantity determination module 30 is further configured to: input the target data volume and the time indicator into a preset strategy model, process the target data volume and the time indicator through the preset strategy model, and obtain the cluster quantity;
[0159] The algorithm in the preset strategy model includes:
[0160]
[0161] In the formula, n is the number of clusters; f(S) is the target data volume; and G... i Multi-process processing capability for each machine step; the T i The time metrics required to be processed for each machine; the P i The time for processing each data entry in a single step after adding a machine; f(D) represents the change in the amount of data for each month's date; the CPU i The CPU utilization rate for each machine running a batch, the CPU max The maximum CPU utilization for each machine running a batch; the I / O i The IO utilization rate for each machine running a batch, the IO max The MER is the upper limit of IO utilization for each machine running a batch; i The MER represents the memory usage for each machine running a batch. max The maximum memory usage per machine for batch processing; the GW i The gateway occupancy rate for each machine running batches, the GW max Set the upper limit for gateway occupancy for each machine running batches.
[0162] In one embodiment, the Docker-based application deployment apparatus includes:
[0163] The execution recycling module is used to send a resource recycling request to the Docker container management platform to recycle resources when the batch processing task is detected to be completed.
[0164] The methods executed by the above-mentioned program modules can be referred to in the various embodiments of the application deployment method based on Docker of the present invention, and will not be repeated here.
[0165] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0166] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0167] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0168] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A method for deploying applications based on Docker, characterized in that, The Docker-based application deployment method includes the following steps: Upon receiving an application deployment request from the business platform, obtain the application information corresponding to the application deployment request; The application database of the business platform is queried, and the target amount of data to be processed by the business platform is predicted based on the historical business data in the application database; the historical business data refers to the data generated during the execution of the application. Output a time setting prompt message to allow the user to set a time indicator for processing the target amount of data; Input the target data volume and the time index into the preset strategy model, run the preset strategy model, and calculate the number of clusters; The application information is sent to the Docker container management platform, and a target cluster corresponding to the number of clusters is requested from the Docker container management platform so that the application can be deployed in the target cluster.
2. The application deployment method based on Docker as described in claim 1, characterized in that, The step of querying the application database of the business platform and predicting the target data volume to be processed by the business platform based on historical business data in the application database includes: Query the application database of the business platform to obtain historical business data within a preset time period. The historical business data is input into a preset data analysis model, filtered by the data analysis model, and the filtered historical business data is associated according to the business process to generate a relational data table. The data processing volume of each business process in the relational data table is counted, and the sum of the data processing volumes is taken as the target data volume to be processed by the business platform in a future preset time period.
3. The application deployment method based on Docker as described in claim 1, characterized in that, The step of inputting the target data volume and the time indicator into a preset strategy model, running the preset strategy model, and calculating the number of clusters includes: The target data volume and the time index are input into a preset strategy model, and the target data volume and the time index are processed by the preset strategy model to obtain the number of clusters; The algorithm in the preset strategy model includes: In the formula, n is the number of clusters; The target data volume; Multi-process processing capability for each machine step; the The time metrics required to be processed for each machine; The time for processing each piece of data in a single step after each additional machine is added; The amount of date data changes for each month; The CPU utilization rate for each machine running the batch, the The maximum CPU utilization for each machine running a batch; The IO utilization rate for each machine running the batch, the The maximum IO utilization rate for each machine running a batch; The memory usage for batch processing on each machine, the The maximum memory usage limit for each machine running a batch; The gateway occupancy rate for each machine running batches, the Set the upper limit for gateway occupancy for each machine running batches.
4. The application deployment method based on Docker as described in claim 1, characterized in that, After the steps of sending the application information to the Docker container management platform, requesting a target cluster corresponding to the number of clusters from the Docker container management platform, and deploying the application in the target cluster, the following steps are included: When the batch processing task is detected to have completed execution, a resource reclamation request is sent to the Docker container management platform to reclaim resources.
5. An application deployment system based on Docker, characterized in that, The Docker-based application deployment system includes: a batch data management platform, a business platform, and a Docker container management platform connected by communication; The Docker-based application deployment system implements the following steps: When the batch data management platform receives an application deployment request sent by the business platform, the batch data management platform obtains the application information corresponding to the application deployment request. The batch data management platform queries the application database of the business platform. Based on the historical business data in the application database, the batch data management platform predicts the target amount of data to be processed by the business platform. The historical business data refers to the data generated during application execution. The batch data management platform sets prompts for output time to allow users to set time indicators for processing the target amount of data. Input the target data volume and the time index into the preset strategy model, run the preset strategy model, and calculate the number of clusters; The batch data management platform sends the application information to the Docker container management platform and requests the target clusters corresponding to the number of clusters from the Docker container management platform; The Docker container management platform sends the application information to the target cluster for application deployment within the target cluster.
6. The Docker-based application deployment system as described in claim 5, characterized in that, The Docker-based application deployment system also includes: a target cluster, which comprises a master cluster and slave clusters connected by communication; After the step of the Docker container management platform sending the application information to the target cluster for application deployment in the target cluster, the following steps are included: When the main control cluster receives a batch processing task sent by the service platform, the main control cluster obtains the batch service data corresponding to the batch processing task. The master control cluster counts the number of processes in each slave cluster to obtain the total number of execution processes. The master control cluster then divides the batch of business data according to the total number of execution processes to obtain sharded business data. The master control cluster sends the fragmented service data to each slave cluster; When a slave cluster receives the fragmented service data sent by the master cluster, the slave cluster processes the fragmented service data and feeds back the processing result of the fragmented service data to the master cluster. The master cluster receives the processing results from each slave cluster, and the master cluster summarizes the processing results to obtain the total processing result of the batch processing task. When the overall processing result is "processing complete", the main control cluster sends a notification message indicating that the batch processing task has been completed to the batch data management platform. The batch data management platform sends the notification message to the Docker container management platform; The Docker container management platform performs resource reclamation based on the aforementioned prompts.
7. The Docker-based application deployment system as described in claim 6, characterized in that, When the slave cluster receives the sharded service data sent by the master cluster, after processing the sharded service data, the slave cluster includes: The subordinate cluster sends heartbeat data back to the master cluster; The master control cluster receives heartbeat data from each of the slave clusters and sends the heartbeat data to the business platform. The business platform receives the heartbeat data from each of the aforementioned heartbeat data, and then feeds back the heartbeat data to the batch data management platform. The batch data management platform receives each heartbeat data sent by the business platform, and compares the device identifier in each heartbeat data with a preset identifier corresponding to the number of clusters; If the device identifier in each of the heartbeat data is not exactly the same as the preset identifier corresponding to the number of clusters, the batch data management platform applies to the Docker container management platform for a new target cluster; The Docker container management platform sends the application information to the new target cluster for application deployment in the new target cluster.
8. An application deployment device based on Docker, characterized in that, The Docker-based application deployment device includes: The request receiving module is used to obtain the application information corresponding to the application deployment request when it receives the application deployment request sent by the business platform. The query prediction module is used to query the application database of the business platform and predict the target amount of data to be processed by the business platform based on the historical business data in the application database; the historical business data refers to the data generated during application execution. The quantity determination module is used to output time setting prompts so that the user can set the time index for processing the target amount of data; Input the target data volume and the time index into the preset strategy model, run the preset strategy model, and calculate the number of clusters; The application deployment module is used to send the application information to the Docker container management platform, apply to the Docker container management platform for the target cluster corresponding to the number of clusters, and deploy the application in the target cluster.
9. An application deployment device based on Docker, characterized in that, The Docker-based application deployment device includes: a memory, a processor, and a Docker-based application deployment program stored on the memory and capable of running on the processor. When the Docker-based application deployment program is executed by the processor, it implements the steps of the Docker-based application deployment method as described in any one of claims 1 to 4.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a Docker-based application deployment program, which, when executed by a processor, implements the steps of the Docker-based application deployment method as described in any one of claims 1 to 4.