Data processing method and device and storage medium

Abstract

The embodiment of the invention discloses a data processing method and device and a storage medium, which are used for application deployment, operation and maintenance and monitoring of multiple clusters and multiple environments. The method comprises the following steps: making a Helm-chart template file in response to an operation instruction of a developer; the Helm-chart template file is submitted to a code warehouse; and when assembly line construction is triggered, performing application deployment and release on different data center clusters based on a code warehouse and a parallel assembly line mode. According to the method, a Helm-Charts deployment scheme is adopted, the overall production efficiency of deployment is greatly improved, original k8s resource deployment is changed into batch application deployment one by one, development technicians can better understand, operate and maintain the k8s resource field effect, real devops is achieved, time and human resources can be saved, the parallel construction technology supports continuous integration and continuous delivery, and the development efficiency is improved. And inter-team cooperation and feedback are promoted, so that continuous improvement and innovation are realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly relates to a data processing method, device, and storage medium. Background Art

[0002] A cluster refers to connecting multiple computer nodes together through a network to work collaboratively to provide higher performance, availability, and scalability. Multiple clusters mean that there are multiple such cluster systems, and each cluster can operate independently or cooperate with each other or perform data interaction as needed.

[0003] Inside each cluster, there are usually multiple different operating environments to meet different business requirements and the requirements of stages such as development, testing, and operation and maintenance. Common environments include development environment, testing environment, production environment, etc.

[0004] In the process of deploying multi-data center applications that support the company's strategic-level services, due to the particularity of the company's business, the number of service deployment applications in a single data center can reach hundreds. To effectively manage and expand this complexity, the company urgently needs an efficient and reliable solution. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide a data processing method, device, and storage medium, aiming to optimize the underlying deployment process of application programs and improve the operation and maintenance efficiency for multi-data center clusters.

[0006] To achieve the above purpose, in a first aspect, the embodiments of the present invention provide a data processing method for application deployment, operation and maintenance, and monitoring in multi-cluster and multi-environment. The method includes:

[0007] Respond to the operation instruction of the developer to create a Helm-chart template file according to the operation instruction;

[0008] Submit the Helm-chart template file to the code repository for storage;

[0009] When the pipeline construction is triggered, perform application deployment on different data center clusters based on the code repository and the parallel pipeline method to obtain multiple different application deployment files;

[0010] Release multiple different application deployment files.

[0011] As a specific implementation manner of the present application, creating a Helm-chart template file according to the operation instruction is specifically:

[0012] Define all resource files required for the application deployment of the data center cluster according to the operation instruction;

[0013] Set the variable difference items to the form of reading the variables in the Values.yaml file.

[0014] As a specific implementation manner of the present application, multiple different application deployment files are obtained, specifically:

[0015] When the pipeline build is triggered, pull the chart package information from the code repository;

[0016] Adopt multiple parallel flow platform pipelines, and adjust the corresponding variables in the chart package information according to the application deployment requirements of different data center clusters to obtain multiple different application deployment files.

[0017] As a preferred implementation manner of the present application, after publishing multiple different application deployment files, the data processing method further includes:

[0018] Perform operation and maintenance management on different data center clusters that have completed application deployment, including real-time monitoring, configuration change comparison, log management, and resource control.

[0019] In a second aspect, an embodiment of the present application further provides a data processing device for application deployment, operation and maintenance, and monitoring in multiple clusters and multiple environments. The device includes:

[0020] A template making unit, configured to respond to an operation instruction of a developer to make a Helm-chart template file according to the operation instruction;

[0021] A code repository, configured to store the Helm-chart template file submitted by the template making unit;

[0022] An application deployment unit, configured to perform application deployment on different data center clusters based on the code repository and the parallel pipeline method when the pipeline build is triggered to obtain multiple different application deployment files;

[0023] A publishing unit, configured to publish multiple different application deployment files.

[0024] As a preferred implementation manner of the present application, the data processing device further includes an operation and maintenance unit for:

[0025] Perform operation and maintenance management on different data center clusters that have completed application deployment, including real-time monitoring, configuration change comparison, log management, and resource control.

[0026] In a third aspect, an embodiment of the present invention further provides a data processing device for application deployment, operation and maintenance, and monitoring in multiple clusters and multiple environments, including a processor, an input device, an output device, and a memory. The processor, the input device, the output device, and the memory are interconnected. Among them, the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method in the first aspect above.

[0027] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the method in the first aspect above.

[0028] Implementing the data processing method provided by the embodiment of the present invention, adopting the Helm-Charts deployment solution, greatly improves the overall production efficiency of deployment. It changes the original one-by-one deployment of k8s resources into batch application deployment, and also enables development technicians to better understand the role of k8s resource fields in operation and maintenance, realizing true devops. It can save time and human resources, and the parallel construction technology supports continuous integration and continuous delivery, promoting collaboration and feedback among teams, so as to achieve continuous improvement and innovation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art.

[0030] Figure 1 It is a flowchart of a data processing method provided by an embodiment of the present invention;

[0031] Figure 2 is Figure 1 Another flowchart of the data processing method shown;

[0032] Figure 3 It is an example diagram of the file structure of a helm-chart resource package;

[0033] Figure 4 It is an example diagram of a parallel pipeline;

[0034] Figure 5 It is a structural diagram of a data processing device provided by an embodiment of the present invention;

[0035] Figure 6 is Figure 5 Another structural diagram of the data processing device shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0037] It should be understood that when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0038] The explanations of related terms are as follows:

[0039] Helm-chart template: It is a collection of yaml file resource files, which contains application information introduction files and required resource files, and is a configuration template file for deploying resources.

[0040] Flow platform pipeline: The flow platform is an internal company cicd pipeline construction platform that provides operations such as program code compilation, packaging, and automatic release and deployment.

[0041] Data center application group: A collection of applications deployed under different k8s clusters.

[0042] The inventive concept of the present invention is:

[0043] Based on the problems mentioned in the background technology, the embodiments of the present invention explore and introduce Helm technology as part of the infrastructure to achieve standardized and modular deployment of application programs and simplify the operation and maintenance process. By leveraging the advantages of HelmCharts, this application can more flexibly define and manage large-scale application deployments, improve deployment efficiency, reduce risks, and optimize the company's operating costs and resource utilization to the greatest extent.

[0044] In addition to deploying application programs, effective operation and maintenance and monitoring are also key aspects to ensure system stability and performance. In this multi-data center environment, it is necessary to establish a comprehensive set of operation and maintenance tools on this basis to support operations such as real-time monitoring, log management, and resource adjustment of deployed applications.

[0045] Please refer to Figure 1 and Figure 2 , the data processing method provided by the embodiments of the present invention is used for application deployment, operation and maintenance, and monitoring in a multi-cluster and multi-environment, including the following steps:

[0046] S1. In response to the operation instruction of the developer, a Helm-chart template file is created according to the operation instruction.

[0047] Specifically, the developer develops and writes a basic helm-chart resource package, and the sample file structure of the resource package is as Figure 3 shown.

[0048] Chart.yaml Introduction of the chart package

[0049] templates contains all k8s resource files

[0050] charts is a collection of sub-chart files and is also an independent chart. The role of sub-Charts in Helm is to help organize and manage complex application structures, improving the flexibility and maintainability of deployment.

[0051] Values.yaml is used to replace the placeholder fields in the resource template to achieve the function of flexible configuration replacement.

[0052] That is, develop and write the basic version of helm-chart, define all the resources required for the application deployment of the data center service group, and set the variable differences as the form of reading the Values.yaml file variables to achieve scalability and flexibility.

[0053] S2. Submit the Helm-chart template file to the code repository for storage.

[0054] Specifically, push the completed basic helm-chart to the gitlab code repository for storage.

[0055] S3. When the pipeline build is triggered, based on the code repository and the parallel pipeline method, perform application deployment for different data center clusters to obtain multiple different application deployment files.

[0056] Specifically, when the pipeline build is triggered, pull the chart package information from the code repository; adopt multiple parallel flow platform pipelines, and adjust the corresponding variables in the chart package information according to the application deployment requirements of different data center clusters to obtain multiple different application deployment files.

[0057] That is, in this embodiment, parallel pipeline construction is used to control the version differences of different data centers, which can improve the build and release efficiency (as Figure 4 shown).

[0058] S4. Publish multiple different application deployment files.

[0059] S5. Perform operation and maintenance management on different data center clusters that have completed application deployment.

[0060] After release, provide essential operation and maintenance functions such as real-time monitoring, configuration change comparison, log management, and resource control to improve operation and maintenance efficiency.

[0061] That is, this embodiment provides a function for comparing changes in the Values.yaml version configuration, which facilitates understanding the differences between the version to be upgraded and the running version, and provides operation and maintenance functions such as event monitoring, log management, and resource adjustment.

[0062] From the above description, it can be seen that by adopting the Helm-Charts deployment solution for the data processing method provided by the embodiments of the present invention, the overall production efficiency of deployment is greatly improved. Instead of deploying k8s resources one by one originally, it becomes batch application deployment, which also enables development and technical personnel to better understand the role of k8s resource fields, realizes true devops, can save time and human resources, and the parallel construction technology supports continuous integration and continuous delivery, promotes collaboration and feedback among teams, and thus realizes continuous improvement and innovation.

[0063] Based on the same inventive concept, as Figure 5 shown, the embodiments of the present invention further provide a data processing device for application deployment, operation and maintenance, and monitoring in multiple clusters and multiple environments. The device includes:

[0064] A template making unit, which is used to respond to the operation instructions of developers and make a Helm-chart template file according to the operation instructions;

[0065] A code repository, which is used to store the Helm-chart template file submitted by the template making unit;

[0066] An application deployment unit, which is used to perform application deployment on different data center clusters based on the code repository and the parallel pipeline method when the pipeline construction is triggered, and obtain multiple different application deployment files;

[0067] A release unit, which is used to release multiple different application deployment files;

[0068] An operation and maintenance unit, which is used to perform operation and maintenance management on different data center clusters that have completed application deployment, including real-time monitoring, configuration change comparison, log management, and resource control.

[0069] Optionally, as Figure 6As shown in the figure, another embodiment of the present invention further provides a data processing device, which may include: one or more processors 101, one or more input devices 102, one or more output devices 103, and a memory 104. The above-mentioned processors 101, input devices 102, output devices 103, and memory 104 are interconnected through a bus 105. The memory 104 is used to store a computer program, and the computer program includes program instructions. The processor 101 is configured to call the program instructions to execute the methods in the method embodiment part described above.

[0070] It should be understood that in the embodiments of the present invention, the so-called processor 101 may be a central processing unit (CPU), and this processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0071] The input device 102 may include a keyboard, etc., and the output device 103 may include a display (such as an LCD), a speaker, etc.

[0072] The memory 104 may include a read-only memory and a random access memory, and provide instructions and data to the processor 101. A part of the memory 104 may also include a non-volatile random access memory. For example, the memory 104 may also store information about the device type.

[0073] In specific implementation, the processors 101, input devices 102, and output devices 103 described in the embodiments of the present invention may implement the implementation manners described in the embodiments of the data processing method provided by the embodiments of the present invention, which will not be elaborated herein.

[0074] It should be noted that for the specific working process of the data processing device, please refer to the method embodiment part described above, which will not be elaborated herein.

[0075] Correspondingly, an embodiment of the present invention provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, the following is implemented: the above-mentioned data processing method.

[0076] The computer-readable storage medium may be an internal storage unit of the system described in any of the foregoing embodiments, such as the hard disk or memory of the system. The computer-readable storage medium may also be an external storage device of the system, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the system. Further, the computer-readable storage medium may also include both the internal storage unit and the external storage device of the system. The computer-readable storage medium is used to store the computer program and other programs and data required by the system. The computer-readable storage medium may also be used to temporarily store the data that has been output or is to be output.

[0077] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0078] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Additionally, the displayed or discussed couplings or direct couplings or communication connections to each other can be indirect couplings or communication connections through some interfaces, devices, or units, and can also be in the form of electrical, mechanical, or other connections.

[0079] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiments of the present invention.

[0080] In addition, in each embodiment of the present invention, the functional units may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0081] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0082] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The data processing method is used for application deployment, operation, maintenance and monitoring of multiple clusters and multiple environments, and the method includes: Responding to the operation instruction of the developer to create a Helm-chart template file according to the operation instruction; Submit the Helm-chart template file to the code repository for storage; When the pipeline construction is triggered, the application is deployed to different data center clusters based on the code repository and the parallel pipeline method to obtain multiple different application deployment files; Publish multiple different application deployment files.

2. The data processing method according to claim 1, characterized in that: The Helm-chart template file includes an application information introduction file and resource files required for deployment.

3. The data processing method according to claim 1, characterized in that: Create a Helm-chart template file according to the operating instructions, specifically: Define all resource files required for data center cluster application deployment according to the operation instructions; Set the variable difference to read the Values.yaml file variables.

4. The data processing method according to claim 1, characterized in that: Get multiple different application deployment files, specifically: When the pipeline build is triggered, the chart package information is pulled from the code repository; By using multiple parallel flow platform pipelines, corresponding variables in the chart package information are adjusted according to the application deployment requirements of different data center clusters to obtain multiple different application deployment files.

5. The data processing method according to claim 4, characterized in that: After publishing a plurality of different application deployment files, the data processing method further includes: Perform operation and maintenance management on different data center clusters that have completed application deployment, including real-time monitoring, configuration change comparison, log management, and resource control.

6. A data processing device, characterized in that: The data processing device is used for application deployment, operation, maintenance and monitoring of multiple clusters and multiple environments, and the device includes: A template making unit, used to respond to an operation instruction of a developer to make a Helm-chart template file according to the operation instruction; A code repository, used to store the Helm-chart template file submitted by the template making unit; An application deployment unit, used to deploy applications to different data center clusters based on the code repository and parallel pipeline mode when pipeline construction is triggered, to obtain multiple different application deployment files; The publishing unit is used to publish multiple different application deployment files.

7. The data processing device according to claim 6, characterized in that The application deployment unit is specifically used for: When the pipeline build is triggered, the chart package information is pulled from the code repository; By using multiple parallel flow platform pipelines, corresponding variables in the chart package information are adjusted according to the application deployment requirements of different data center clusters, and multiple different application deployment files have been obtained.

8. The data processing device according to claim 6 or 7, characterized in that The data processing device further includes an operation and maintenance unit, which is used to: Perform operation and maintenance management on different data center clusters that have completed application deployment, including real-time monitoring, configuration change comparison, log management, and resource control.

9. A data processing device, characterized in that: The data processing device is used for application deployment, operation, maintenance and monitoring of multiple clusters and multiple environments. The data processing device includes a processor, an input device, an output device and a memory. The processor, input device, output device and memory are interconnected, wherein the memory is used to store a computer program, and the computer program includes program instructions. The processor is configured to call the program instructions to execute the method described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 5.

Images