Big data component management device and method based on cloud native technology
Through the big data component management method based on cloud-native technology, the configuration files are uniformly defined and distributed, and the configuration complexity and low resource utilization in component management are solved, and the consistency and dynamic scalability of components in different environments are achieved. , improves the stability and scalability of the system.
Patent Information
- Application Number
- CN202410535158.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-30
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-04-30
AI Technical Summary
The existing big data component management has challenges in terms of many configuration parameters, complex deployment processes, low resource utilization and poor scalability, resulting in system performance degradation, compatibility issues and difficulty in dynamically adjusting resources.
Using the big data component management device and method based on cloud native technology, the component operation template configuration module, the load description template configuration module and the configuration file distribution module are unifiedly defined component roles, dependencies and configuration parameters, and the configuration parameters are generated. files and distributed on multiple physical servers to ensure configuration consistency and dynamic scheduling of resources.
Reduces configuration errors, simplifies deployment processes, improves resource utilization and system scalability, ensures consistency and stability of components in different environments, and supports the growth of business scale and data volume.
Smart Images

Figure CN118312240B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data processing technology, and in particular to a big data component management device and method based on cloud native technology. Background Art
[0002] With the rapid development of information technology, the amount of global data has exploded, and traditional data processing methods have gradually exposed their limitations. In order to better cope with this challenge, big data technology has emerged and has become a powerful assistant for processing large-scale data. Big data technology covers a large number of components and tools, such as Hadoop, Spark, Kafka, etc., which together build an efficient and stable data processing ecosystem.
[0003] However, although these big data components perform well in data processing, there are some problems in component management. First, the configuration parameters of these components are numerous and prone to errors. For example, Hadoop's HDFS and MapReduce require the configuration of many parameters, such as the number of nodes, memory size, task scheduling strategy, etc. Once improperly configured, it may lead to system performance degradation or even crash.
[0004] Secondly, the deployment process is complex and prone to inconsistencies. Due to the complex dependencies between big data components, deployment needs to be performed in a certain order and steps, otherwise it may cause compatibility issues between components. In addition, the deployment configurations in different environments may also be different, further increasing the difficulty of deployment.
[0005] Finally, resource utilization is low and difficult to expand. Under the traditional component management method, resource allocation and scheduling often rely on static configuration and predefined policies, which are difficult to dynamically adjust according to actual needs. At the same time, with the expansion of business scale and the growth of data volume, the scalability of the system is also facing challenges. Summary of the invention
[0006] The purpose of this application is to propose a big data component management device and method based on cloud native technology to solve or alleviate technical problems existing in the prior art.
[0007] The technical solutions provided by the embodiments of this application are as follows:
[0008] A big data component management device based on cloud native technology, comprising:
[0009] A description file configuration module, used to configure a big data component description file, wherein the big data component description file at least defines cloud-native-based big data component roles, dependencies between big data components, and configuration parameters of big data components;
[0010] A component operation template configuration module, used to configure an operation configuration template file for running the big data component, wherein the operation configuration template file at least includes operation parameters and configuration information of the big data component, wherein the configuration information at least includes a port number, a storage path, and a log level;
[0011] A load description template configuration module, used to configure a load description template file when the big data component is working, wherein the load description template file at least includes the operation mode, resource requirements, and number of copies of the big data component in the cloud environment;
[0012] A component operation configuration module, used to generate a configuration file required for the operation of the big data component according to the big data component description file and the operation configuration template file, wherein the configuration file required for the operation of the big data component is used to guide the deployment of the big data component;
[0013] A configuration file distribution module is used to distribute the configuration files required for the operation of the big data component to multiple physical servers to be deployed, so as to generate a description file based on which the big data component is deployed on the multiple physical servers to be deployed, so that the big data components laid out on each of the physical servers to be deployed have configuration consistency characteristics.
[0014] A big data component management method based on cloud native technology, comprising:
[0015] Description file configuration step: configure a big data component description file, wherein the big data component description file at least defines cloud-native-based big data component roles, dependencies between big data components, and configuration parameters of big data components;
[0016] Component operation template configuration step: configuring an operation configuration template file for running the big data component, wherein the operation configuration template file at least includes operation parameters and configuration information of the big data component, and the configuration information at least includes a port number, a storage path, and a log level;
[0017] Load description template configuration step: configuring a load description template file when the big data component is working, wherein the load description template file at least includes the operation mode, resource requirements, and number of copies of the big data component in the cloud environment;
[0018] Component operation configuration step: generating a configuration file required for the operation of the big data component according to the big data component description file and the operation configuration template file, wherein the configuration file required for the operation of the big data component guides the deployment of the big data component;
[0019] Configuration file distribution step: distribute the configuration files required for the operation of the big data component to multiple physical servers to be deployed, so as to generate a description file based on which the big data component is deployed on multiple physical servers to be deployed, so that the big data components arranged on each of the physical servers to be deployed have configuration consistency characteristics.
[0020] The present application provides a big data component management method and device based on cloud native technology, the device comprising: a description file configuration module, used to configure a big data component description file, wherein the big data component description file at least defines a cloud-native big data component role, dependencies between big data components, and configuration parameters of the big data component; a component operation template configuration module, used to configure an operation configuration template file for running the big data component, wherein the operation configuration template file at least includes the operation parameters and configuration information of the big data component, wherein the configuration information at least includes a port number, a storage path, and a log level; a load description template configuration module, used to configure a load description template file when the big data component is working, wherein the load description template file The description template file at least includes the operation mode, resource requirements, and number of copies of the big data component in the cloud environment; the component operation configuration module is used to generate the configuration file required for the operation of the big data component according to the big data component description file and the operation configuration template file, and the configuration file required for the operation of the big data component is used to guide the deployment of the big data component; the configuration file distribution module is used to distribute the configuration file required for the operation of the big data component to multiple physical servers to be deployed, so as to generate a description file based on which the big data component is deployed on multiple physical servers to be deployed, so that the big data components laid out on each of the physical servers to be deployed have configuration consistency characteristics. It can be seen that the description file of the big data component is centrally configured through the description file configuration module, and the role, dependency, and configuration parameters of the component are uniformly defined, thereby reducing the possibility of configuration errors and ensuring the consistency and accuracy of the configuration. In addition, through clear definition of dependencies, the deployment process can be simplified and compatibility issues caused by dependency errors can be reduced. Furthermore, the running configuration template file is configured based on the component running template configuration module. This module can standardize the running parameters and configuration information of the big data component, including key parameters such as port number, storage path, log level, etc. The unified configuration of these parameters ensures the consistency of the components in different environments. At the same time, this also simplifies the deployment process and reduces the deployment difficulty caused by configuration inconsistency. Further, the load description template file is configured based on the load description template configuration module, which defines the operation mode, resource requirements and number of copies of the big data component in the cloud environment, realizes the dynamic allocation and scheduling of resources, and improves resource utilization. At the same time, by configuring the number of copies, the reliability and scalability of the system can be ensured to meet the needs of business scale and data volume growth. Moreover, based on the component running configuration module, the configuration files required for the big data component runtime are generated according to the description file and the running configuration template file. These configuration files have clear guidance, can simplify the deployment process, and reduce the complexity of operation. At the same time, due to the consistency of the configuration files, it can be ensured that the big data components deployed on each physical server to be deployed have configuration consistency characteristics.Finally, through the configuration file distribution module, the configuration files required for the generated big data component runtime are distributed to multiple physical servers to be deployed, ensuring that the big data components on each server have consistent configuration and deployment methods, thereby improving the stability and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Hereinafter, some specific embodiments of the present application will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:
[0022] Figure 1 This is a flow chart of a big data component management method based on cloud native technology according to an embodiment of the present application. DETAILED DESCRIPTION
[0023] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.
[0024] Figure 1 This is a flow chart of a big data component management method based on cloud native technology according to an embodiment of the present application. Figure 1 As shown, a big data component management device based on cloud native technology includes:
[0025] A description file configuration module, used to configure a big data component description file, wherein the big data component description file at least defines cloud-native-based big data component roles, dependencies between big data components, and configuration parameters of big data components;
[0026] A component operation template configuration module, used to configure an operation configuration template file for running the big data component, wherein the operation configuration template file at least includes operation parameters and configuration information of the big data component, wherein the configuration information at least includes a port number, a storage path, and a log level;
[0027] A load description template configuration module, used to configure a load description template file when the big data component is working, wherein the load description template file at least includes the operation mode, resource requirements, and number of copies of the big data component in the cloud environment;
[0028] A component operation configuration module, used to generate a configuration file required for the operation of the big data component according to the big data component description file and the operation configuration template file, wherein the configuration file required for the operation of the big data component is used to guide the deployment of the big data component;
[0029] A configuration file distribution module is used to distribute the configuration files required for the operation of the big data component to multiple physical servers to be deployed, so as to generate a description file based on which the big data component is deployed on the multiple physical servers to be deployed, so that the big data components laid out on each of the physical servers to be deployed have configuration consistency characteristics.
[0030] Optionally, the big data component management device further includes:
[0031] A cloud-native workload configuration module, used to generate a cloud-native workload description file according to the workload description template file, wherein the workload description file is used to at least describe the operation specifications, resource requirements and replica strategies of the big data component when it is running in a cloud environment built based on the physical server to be deployed;
[0032] A big data component running module is used to start running the big data component in a cloud environment built based on the physical server to be deployed according to the cloud-native workload description file.
[0033] To this end, in order to optimize the deployment and management of big data components, an automated method of generating workload description files is adopted. This method reduces the need for manual configuration, reduces the complexity and error rate of deployment. The workload description file provides a standardized way to define the operating specifications of big data components, ensuring consistency and predictability across different environments. The cloud-native workload configuration module can optimize resource allocation and replication strategies based on the data in the workload description file, thereby improving resource utilization and cost efficiency.
[0034] The scalability of cloud-native technology is one of its key advantages. When business needs grow, resources and the number of replicas can be easily increased to accommodate larger workloads. In addition, big data components in cloud environments can be quickly started or stopped as needed, providing flexibility for development and testing, and supporting rapid iteration and deployment of new versions.
[0035] In a cloud-native architecture, when a failure occurs, components can be quickly restarted or redeployed to other servers, reducing downtime. Because components are built and managed in a cloud environment, consistency between development, test, and production environments can be ensured, reducing problems caused by environmental differences. The modular design makes it easier to maintain and upgrade big data components, and individual components can be updated independently without affecting the stability of the entire system.
[0036] Through the load description template file, you can preset the load balancing strategy, so that the big data component can dynamically adjust the resources and number of copies according to the actual load. In addition, the integrated monitoring component can provide real-time data about the performance of the component to help with performance analysis and troubleshooting. These features together improve the deployment efficiency and operation stability of the big data component, and provide strong support for enterprise data processing and analysis.
[0037] This application also provides an exemplary code on a product:
[0038]
[0039]
[0040]
[0041] In the above code, the CloudNativeWorkloadConfigModule class is responsible for generating the workload description file, while the DataComponentRuntimeModule class is responsible for starting the big data component based on the description file.
[0042] Optionally, the description file configuration module is used to configure a big data component description file, wherein the big data component description file at least defines a cloud-native big data component role, dependencies between big data components, and configuration parameters of the big data component, and specifically includes:
[0043] Parsing the big data component role model to determine the big data component roles, the big data component roles at least including: data source, data processor, data storage, data analyzer;
[0044] Parsing the property files of each big data component based on the dependency interpreter, obtaining the dependency description between the big data components, and defining the dependency description in the form of key-value pairs to generate the dependency between the big data components;
[0045] Parsing the configuration files of each big data component based on the configuration parameter parser to determine the placeholders therein, querying the configuration parameters of the big data component in the configuration files of the big data component based on the placeholders, and converting the configuration parameters into key-value pairs;
[0046] Based on a structured language, using a set configuration engine, a container orchestration tool is called to containerize the big data component roles, dependencies between big data components, and configuration parameters of big data components to generate the big data component description file.
[0047] To this end, in order to achieve efficient, accurate and scalable big data component management, an automated parsing and configuration method is adopted. First, by parsing the big data component role model, the function of each component, such as data source, data processor, etc., is clearly defined, and the component usage and maintenance process is standardized. This not only improves the interoperability of components, but also reduces maintenance costs.
[0048] Secondly, it realizes automatic parsing and management of dependencies between components, thus reducing manual configuration errors. By automatically parsing the placeholders in the configuration file and converting them into key-value pairs, the configuration management process is simplified, the accuracy and maintainability of the configuration are improved, which not only reduces operational risks but also improves development efficiency.
[0049] In terms of containerization, configuration engines and container orchestration tools are used to automate containerized big data components. This not only improves the speed and consistency of deployment, but also brings better isolation and portability. Containerized big data components can be more easily expanded to cope with different workload requirements. At the same time, the use of container orchestration tools provides standardized interfaces, making the interaction between different components more standardized and simplifying the integration process.
[0050] In addition, structured languages and configuration engines are used to maintain and update big data components. This makes it easier to modify and upgrade components and supports rapid iteration and continuous integration. Through automated resource configuration, resources can be allocated more accurately, avoiding resource waste and improving resource utilization.
[0051] Automated configuration management also emphasizes security and portability. Automation reduces the possibility of human error, especially in the deployment and configuration of complex big data components. In addition, the automatically generated configuration files are also part of the documentation, which helps team members understand and learn the configuration of the system. Automated security best practices, such as the management of environment variables, are also implemented to reduce security vulnerabilities.
[0052] This application also provides an exemplary code on a product:
[0053] To demonstrate the implementation of the description file configuration module, the following is an exemplary pseudo code that describes how to parse the roles, dependencies, and configuration parameters of a big data component and containerize it using a container orchestration tool. Please note that this is just a conceptual example, and the actual code will depend on the specific technology stack and cloud platform API.
[0054]
[0055]
[0056] In the above code, classes are defined to handle role resolution, dependency resolution, configuration parameter resolution, and containerization respectively. The `DescriptionFileConfigModule` class combines these components to generate a description file for the big data component. The actual implementation will interact with specific container orchestration tools (such as Kubernetes) to handle file I / O, as well as network communication and security measures that may be involved. In addition, the real implementation also needs to consider factors such as error handling, logging, and user input validation.
[0057] Optionally, the structured language-based method uses a set configuration engine to call a container orchestration tool to containerize the big data component roles, dependencies between big data components, and configuration parameters of big data components to generate the big data component description file, specifically including:
[0058] Based on a structured language, using a set configuration engine, calling a container orchestration tool to define resource objects, a service discovery mechanism, and a configuration parameter chart, wherein the resource object is at least used to specify a container image, a number of replicas, and a port, the service discovery mechanism is used to encapsulate the dependencies between the big data components, and the configuration parameter chart is used to map the configuration parameters of the big data components;
[0059] Trigger the resource object to reference the service discovery mechanism and the configuration parameter chart for containerization, so as to encapsulate the code of the big data component through the container image, configure the number of copies of the big data component with dependencies through the number of copies, and define the access mechanism of the big data component based on the configuration parameters of the big data component through the port to generate the big data component description file.
[0060] To this end, the automated containerization process significantly reduces the need for manual configuration, significantly increases deployment speed, and reduces the risk of human error. This automation not only speeds up the development cycle, but also ensures greater configuration consistency, whether in development, testing, or production environments.
[0061] In addition, the use of structured languages further enhances the standardization and consistency of configuration, providing a solid foundation for cross-environment deployment. Containerized components are easier to maintain and upgrade due to their encapsulation and isolation characteristics, significantly reducing dependence on the host environment. This encapsulation also makes resource management more sophisticated, enabling precise control of the computing resources required by each big data component, such as CPU and memory, thereby optimizing resource usage.
[0062] The service discovery mechanism simplifies the communication problem between components and allows dynamic discovery and interaction. This mechanism not only improves the scalability of the system, but also reduces the complexity of manual management. By adjusting the number of replicas, it is easy to achieve horizontal expansion of big data components to cope with various load requirements.
[0063] At the same time, by setting a separate security policy for each big data component, the container provides an additional isolation layer, effectively reducing security risks. In addition, containerized big data components can be easily ported across different cloud platforms and operating systems, significantly improving the flexibility and adaptability of the system.
[0064] With the support of the service discovery mechanism, load balancing becomes easier to implement, which helps improve system availability and responsiveness. The integration of containerized components with monitoring and logging systems is also more convenient, providing the operation and maintenance team with more powerful system status tracking and analysis capabilities.
[0065] The fast start and stop feature of containers plays an important role in system maintenance and updates, significantly reducing downtime. Optimized resource usage and automated management strategies not only improve operational efficiency but also reduce operating costs. Finally, the seamless integration of container images with version control systems makes component version tracking simpler and more efficient.
[0066] This application also provides an exemplary code on a product:
[0067]
[0068]
[0069]
[0070]
[0071] The above sample code shows how to define a function to build a Kubernetes Deployment object, which describes a containerized deployment of a big data component. The code uses the Kubernetes Python client library to create and operate Kubernetes resources.
[0072] Optionally, the component operation template configuration module is used to configure an operation configuration template file for running the big data component, the operation configuration template file at least includes operation parameters and configuration information of the big data component, and the configuration information at least includes a port number, a storage path, and a log level, including:
[0073] Obtaining an operating parameter description document and a configuration item description document of a big data component, and parsing the operating parameter description document and the configuration item description document based on regular matching to obtain the operating parameters and configuration information of the big data component;
[0074] The operation parameters and configuration information of the big data component are encapsulated into structured format data, and the created formatting template file is called to synchronously replace the structured format data with the replaceable parameters defined in the formatting template file through the parameterized language in the formatting template file, so as to generate the operation configuration template file of the big data component.
[0075] To this end, by using template files, configuration consistency between different environments or different deployments is ensured, reducing differences and errors caused by manual configuration. Automatically generating configuration files reduces manual intervention in the deployment process, speeds up deployment, and improves efficiency. When a component configuration needs to be changed, only the template file needs to be updated and the configuration file needs to be regenerated, without manually modifying the configuration of each instance. The use of parameterized templates allows for flexible customization of configurations for different deployment environments without changing the structure of the template. Template files can be tracked by version control systems, making configuration change history and auditing easier. The use of templates reduces the repetitive work of manually writing configuration files for each big data component. Generating configuration files through templates ensures the safe injection of sensitive information (such as database passwords) instead of hard-coding them in configuration files. Using templates reduces common errors in manual configuration, such as spelling errors, format errors, etc. The template configuration module can dynamically adjust configuration parameters such as port numbers or storage paths according to the target environment to adapt to different operating environments. Developers and testers can easily generate configuration files for development or testing purposes without having to understand the underlying configuration details. The template file itself can be used as part of the documentation to help new team members quickly understand how the system is configured. Centrally managed template files simplify configuration maintenance, especially when the system expands or the configuration needs to be updated. When a problem occurs, the configuration parameters in the template file can be quickly located to facilitate troubleshooting and fixing the problem. The template configuration module can be easily integrated into the continuous integration / continuous deployment (CI / CD) process to automate the testing and deployment process. The design of the template configuration module allows more configuration parameters to be added as needed to adapt to future system development and changes. The template configuration module allows easy setting and management of log levels, which helps monitor system health and debug. Templates can include configuration of resource paths, which helps optimize data storage and access paths and improve performance. The use of templates promotes the standardization of configuration, allowing configuration templates to be shared between different teams and projects to improve collaboration efficiency.
[0076] This application also provides an exemplary code on a product:
[0077]
[0078]
[0079] In the above code, two string variables `run_params_doc` and
[0080] `config_items_doc` is used to simulate the run parameter description document and the configuration item description document. Then, regular expressions are used to parse these documents to extract key run parameters and configuration information. The Jinja2 library (a popular Python template engine) is used to load and render templates. The template content is designed to contain placeholders that correspond to the parameters we extracted from the document. Finally, the extracted structured data is used to render the template to generate the final run configuration template file
[0081] `run_config_template.yaml`.
[0082] Optionally, the load description template configuration module is used to configure a load description template file when the big data component is working, and the load description template file at least includes the operation mode, resource requirements, and number of copies of the big data component in the cloud environment, including:
[0083] Based on the Prometheus monitoring component, we obtain the throughput samples, processing speed samples, data volume samples, computing resource samples, storage resource samples, network bandwidth samples, and replication factor samples of the big data components when they are running in the test cloud environment, and perform regression analysis on these data to estimate the operation mode, resource requirements, and number of replicas of the big data components in the cloud environment.
[0084] Based on the resource allocation strategy, replica allocation logic and performance indicator threshold of the big data component at runtime, call the automated script to tune the operation mode, resource requirements and number of replicas of the data component in the cloud environment, and instantiate the optimized operation mode, resource requirements and number of replicas to generate a load description template file for the big data component when it is working.
[0085] To this end, regression analysis can be performed on the performance sample data collected by the Prometheus monitoring component to more accurately estimate resource requirements and the number of replicas and optimize performance. The use of automated scripts reduces the workload of manually adjusting resources and the number of replicas, improves efficiency, and reduces errors caused by human factors. Automated tuning based on resource allocation strategies and performance indicator thresholds helps to achieve reasonable resource allocation and avoid resource waste or shortage. The load description template file defines the operation mode of the big data component, which helps to standardize the deployment process and ensure consistency between different environments. The automated load description template generation process supports rapid iteration and can quickly respond to changes in requirements and updates to performance goals. By accurately estimating and tuning resource requirements, unnecessary resource usage can be reduced and the cost of cloud resources can be reduced. Templated configuration files are easy to understand and maintain, which helps new team members get started quickly. The load description template file can be customized according to different business requirements and performance goals, providing flexibility. The automated tuning process reduces the risk of system failures caused by improper configuration. Integration with monitoring tools such as Prometheus provides real-time performance feedback, which helps to continuously optimize the system. The template file itself can be used as a document to record the load configuration and tuning history of the big data component. Through template configuration, big data components can more easily adapt to different cloud environments and workload requirements. Automated load description template generation can quickly restore components to the optimal operating state and reduce system recovery time. Performance data collected in the test cloud environment can be used to verify the effectiveness of the configuration and ensure stability in the production environment. Through automated tuning, load balancing can be better achieved and the throughput and response speed of the system can be improved. The load description template configuration module can be integrated with the continuous monitoring system to adjust the configuration in real time to adapt to changes in workloads. The generated load description template file can be used directly for deployment, which simplifies the deployment process and speeds up deployment.
[0086] This application also provides an exemplary code on a product:
[0087]
[0088]
[0089] In this example, we first simulate the performance sample data obtained from Prometheus and convert it into a Pandas DataFrame for easy processing. Then, we use a linear regression model to perform regression analysis on the replica factor. Next, we define the resource allocation strategy and performance indicator thresholds, and call an automated script `tune_resources` to tune the resources and number of replicas. This script should contain specific tuning logic to adjust resource allocation according to the actual strategy and thresholds. Finally, we instantiate the tuned operation mode, resource requirements, and number of replicas and write them to the load description template file.
[0090] in `load_description_template.json`.
[0091] Optionally, the component operation configuration module is used to generate a configuration file required for the big data component to run according to the big data component description file and the operation configuration template file, and the configuration file required for the big data component to run is used to guide the deployment of the big data component, including:
[0092] Generate an operation configuration decision tree model according to the big data component description file and the operation configuration template file;
[0093] Based on the operation configuration decision tree model, a configuration file required for the big data component to run is generated.
[0094] To this end, configuration files are automatically generated based on component descriptions and running templates, reducing manual configuration errors and workload. Through the decision tree model, the logic and decision process of configuration generation are clearer and more traceable, making it easier to understand and maintain. The decision tree model can be flexibly adjusted according to different component descriptions and running templates, making it easy to expand new scenarios and requirements. The deployment configuration process of big data components is standardized, improving the consistency and repeatability of deployment. Through intelligent decision-making, resources can be allocated more reasonably, resource utilization can be optimized, and resource waste can be avoided. The decision tree model can predict the optimal configuration parameters based on the performance indicators of the component and improve the operating efficiency of the component. Automated configuration generation can quickly respond to environmental changes or changes in business needs, accelerate the deployment and adjustment process, reduce human participation in the configuration process, and thus reduce errors caused by human factors. The decision tree model can be easily integrated into existing automated deployment processes, such as CI / CD pipelines. Accurate configuration file generation helps improve the stability and reliability of big data component operation. The decision tree model can handle complex decision logic containing multiple conditions and parameters. When the configuration needs to be updated or modified, only the decision tree model needs to be adjusted, without manually modifying each configuration file. The decision tree model can generate adaptive configurations according to different operating environments (such as development, test, and production environments). Automatically generated configuration files can be easily combined with version control systems to achieve version control and historical tracking of configurations. Automated configuration generation can make it easier to implement security best practices, such as avoiding hard-coding sensitive information in configuration files. When problems occur in the system, the decision path of the decision tree model can be used as a reference for fault diagnosis. The decision tree model can consider the needs of load balancing and generate configuration files that can support load balancing. Through the decision tree model, a cost-effective trade-off can be made between resource allocation and performance to achieve the optimal operating cost. Automatically generated configuration files can be easily used in test environments to accelerate the testing process. The decision tree model and the generated configuration files can be used as documents to help team members understand the configuration logic of the system.
[0095] This application also provides an exemplary code on a product:
[0096]
[0097]
[0098]
[0099] In the above example, a `DecisionTreeNode` class is first defined, which represents a node in a decision tree, containing a condition, a true branch, and a false branch. Each branch can be another decision node or a leaf node, and the leaf node returns the configuration file content. The `DecisionTree` class is responsible for generating the configuration file. It has a `generate_config` method that recursively evaluates the decision tree until a suitable leaf node is found. The `build_decision_tree` function is used to build a decision tree, which defines the decision logic and leaf nodes, and then creates and connects the nodes of the decision tree. A component description file and run configuration template file for an example are created, and the configuration file content is generated using the built decision tree, and then the content is written to a file.
[0100] Optionally, generating an operation configuration decision tree model according to the big data component description file and the operation configuration template file includes:
[0101] Performing operation logic analysis on the big data component description file and the operation configuration template file, and building a component operation navigation path;
[0102] Generate an operation configuration decision tree model according to the component operation navigation path;
[0103] The generating, based on the operation configuration decision tree model, a configuration file required for the big data component to run, comprises:
[0104] The operation configuration decision tree model is matched with the configuration file template constructed when the big data component is tested at runtime to generate the configuration file required for the big data component to run.
[0105] To this end, by running logic analysis, complex configuration logic can be structured to make it easier to understand and maintain. The component run navigation path provides a clear way to determine how to generate configuration based on component descriptions and run templates. The automated configuration generation process reduces the need for manual configuration, improves efficiency and reduces human errors. The decision tree model can be flexibly adjusted according to different run logics and navigation paths to adapt to different big data components and deployment environments. The decision tree model can be easily integrated into existing deployment processes, including automated deployment and continuous integration / continuous deployment (CI / CD) systems. Through precise run logic analysis, the optimal configuration can be generated for big data components to improve the performance of the components. The decision tree model can help determine resource allocation, such as CPU, memory, and storage, to optimize resource usage and reduce costs. The decision tree provides a way to track the configuration generation process, which helps diagnose configuration errors or system failures. By matching the configuration file template at test runtime, existing templates can be reused to reduce duplication of work. The automatically generated configuration file can be easily combined with the version control system to achieve version control and history tracking of the configuration. Automated configuration generation makes it easier to implement security best practices, such as avoiding hard coding sensitive information in configuration files. The decision tree model can generate adaptive configurations based on different deployment environments (such as development, testing, and production). The decision tree model can consider the needs of load balancing and generate configuration files that can support load balancing. The automatically generated configuration files can be easily used in the test environment to accelerate the testing process. The decision tree model and the generated configuration files can be used as documents to help team members understand the configuration logic of the system. The decision tree model helps ensure that there are no conflicts in the configurations of different components. During the development and testing phases, different configuration options can be quickly generated and tested. The decision tree model can be used to simulate the configuration requirements of different environments to verify the behavior of the components. The decision-making process of the decision tree model is transparent and helps build the team's trust in the automated configuration process.
[0106] This application also provides an exemplary code on a product:
[0107]
[0108]
[0109] In the above example, a DecisionTree class is first defined, which can recursively traverse the decision tree nodes and generate configurations based on the component description and run template. 'condition_is_data_source' is an example condition function that determines whether it is a data source component based on the component type. A simple decision tree node 'root_node' is constructed, which contains the condition function and the corresponding true / false branches. Each branch is either a configuration object or contains another condition function and the corresponding branch. Then, a decision tree instance is created and the root node is passed in. The 'generate_config' method is called with the component description and run template to generate the configuration file content. Finally, the generated configuration file content is written to a file and printed out.
[0110] Optionally, the configuration file distribution module is used to distribute the configuration files required for the big data component to be run to multiple physical servers to be deployed, so as to generate a description file for deploying the big data component on the multiple physical servers to be deployed, so that the big data components arranged on each of the physical servers to be deployed have configuration consistency characteristics, including:
[0111] Trigger the SCP file distribution component, execute the written parsing script to obtain the physical server list, and parse the physical server list to obtain the IP address of the physical server; keep the SCP file distribution component in a triggered state to execute the written verification script, so as to synchronously distribute the configuration files required for the operation of the big data component to multiple physical servers to be deployed through the scheduled task cron job, so as to generate a description file based on which the big data component is deployed on the multiple physical servers to be deployed, so that the big data components laid out on each of the physical servers to be deployed have a configuration consistency feature.
[0112] To this end, ensure that the big data components on all physical servers to be deployed have the same configuration to avoid problems caused by inconsistent manual configuration. Automatically distribute configuration files to multiple servers, reducing the need for manual copying and pasting, and improving deployment efficiency. Automatic distribution reduces the possibility of human configuration errors and improves the stability and reliability of the system. When more servers need to be added, the configuration files can be easily distributed to the new servers without manual intervention. Automated distribution saves time in manually configuring each server, allowing operation and maintenance personnel to focus on other important tasks. Centralized configuration file management simplifies configuration updates and maintenance. Through automated distribution, it can be ensured that all servers use the latest security configuration, reducing the risk of security vulnerabilities. Tools such as SCP (Secure Copy Protocol) can be used for secure transmission, supporting multiple servers and operating systems. Configuration files can be distributed to multiple servers at the same time, improving the ability of batch deployment. The automated distribution process can be monitored and recorded, which helps to track the distribution status and history of configuration files. Different configuration files can be used for different deployment environments (such as development, test, and production) without changing the distribution process. When a problem occurs, it can be quickly determined whether it is caused by inconsistent configuration because all servers use the same configuration file. By ensuring that all servers have the same configuration, the availability and fault tolerance of the system can be improved. Automated distribution reduces the manpower and time costs of long-term configuration maintenance. Automatically distributed configuration files can be used in automated testing environments to ensure consistency in test results. Using scheduled tasks such as cron jobs, you can flexibly schedule the distribution time of configuration files to accommodate different deployment windows. The automated distribution process can be combined with a version control system to implement version control and rollback of configuration files. Automated distribution supports more frequent deployments and helps achieve continuous integration and continuous deployment (CI / CD). Fast configuration file distribution can reduce downtime during system updates and maintenance. Unified configuration file distribution ensures that all team members use the same configuration, improving team collaboration efficiency.
[0113] This application also provides an exemplary code on a product:
[0114]
[0115]
[0116] In the above example, a `get_servers_ip_list` function is first defined, which reads and parses the IP addresses from the server list file. Then, a `scp_file_to_servers` function is defined, which uses the `paramiko` library (a Python implementation of the SSHv2 protocol library) to distribute the configuration file to the specified server through the SCP protocol. Next, the server IP address list is traversed and the `scp_file_to_servers` function is called to distribute the configuration file to each server. In order to verify whether the file has been successfully distributed, a `verify_file_distribution` function is defined, which uses the `subprocess` library to connect to each server through SSH and check whether the configuration file exists. Finally, a scheduled task is set up using `crontab` to execute the verification script regularly.
[0117] Optionally, the cloud-native workload configuration module is used to generate a cloud-native workload description file according to the load description template file, wherein the workload description file is used to at least describe the operation specifications, resource requirements and replica strategies of the big data component when it is running in the cloud environment built based on the physical server to be deployed, including:
[0118] Parsing the load description template file, determining the parameter placeholder defined based on the parameterized syntax therein and obtaining the operation specifications, resource requirements and replica strategy when running in the cloud environment specified by the parameter placeholder;
[0119] When the load of the physical server application to be deployed reaches a set intensity threshold, based on the set load balancing consistency, the operating specifications, resource requirements and replica strategies during runtime in the cloud environment are optimized, so as to mirror the operating specifications, resource requirements and replica strategies during runtime in the optimized cloud environment to generate a cloud-native workload description file.
[0120] To this end, the automatic generation of workload description files reduces manual configuration errors and standardizes the deployment process of big data components in the cloud environment. By monitoring the load of physical servers in real time and dynamically adjusting resource allocation and replica strategies according to preset thresholds, cloud resources can be used more effectively and the performance and stability of the system can be improved. The optimized load balancing consistency ensures that the workload is evenly distributed among the components in the cloud environment, avoiding the situation where some components are overloaded and other components waste resources. The module can dynamically adjust the number of replicas according to the actual workload to achieve elastic scaling to adapt to different business needs. Through reasonable replica strategies and resource allocation, the availability and fault tolerance of big data components are improved, ensuring the continuous operation of key services. Optimizing resource use can reduce unnecessary resource waste, reduce the cost of cloud computing resources, and maximize cost-effectiveness. The module can quickly respond to load changes and adjust the configuration in time to adapt to the rapid changes in business needs. Centrally managed and automatically generated workload description files simplify the management and maintenance of cloud resources. Modules can be adapted according to different cloud environments and physical server characteristics, improving the adaptability of the system. Automated configuration generation can more easily integrate security best practices and improve the security of the entire system. Automatically generated configuration files can be easily version controlled and audited, which helps with compliance and problem tracking. Modules can be integrated into CI / CD processes to automate the testing and deployment of new versions of big data components. Load description templates can be reused in different deployment scenarios, reducing the workload of configuration development. Reasonable copy strategies help to quickly restore services in the event of failures and reduce business interruption time. Modules can be integrated with performance monitoring tools to provide real-time performance feedback and help continuously optimize system configuration.
[0121] This application also provides an exemplary code on a product:
[0122]
[0123]
[0124] In this example, a `parse_template_file` function is first defined, which extracts parameter placeholders from the template file. The `get_server_load` function simulates getting the current load of the physical server. The `should_optimize_load` function determines whether the workload description needs to be optimized based on the current load. The `optimize_workload` function adjusts the workload description based on the optimization parameters. Finally, the `generate_workload_description` function connects the above logic in series to generate and write the final workload description file.
[0125] Optionally, the big data component running module is used to start running the big data component in the cloud environment built based on the physical server to be deployed according to the cloud-native workload description file, including:
[0126] Register the physical server to be deployed as a node of the big data component running module, so as to call the containerized image of each physical server to be deployed configured mapped thereon based on the big data component running module, and start running the big data component in the cloud environment built based on the physical server to be deployed according to the cloud-native workload description file.
[0127] To this end, the module can automatically start the big data component in the cloud environment according to the workload description file, reducing the need for manual deployment and speeding up deployment. Cloud native environments usually support automatic scaling, which means that resources can be dynamically adjusted according to the workload, improving resource utilization and optimizing costs. By running multiple copies of big data components on multiple physical servers (nodes), the fault tolerance and availability of the system can be improved. Cloud platforms usually provide load balancing services to ensure that the workload is evenly distributed to each node and prevent any single point of overload. The use of containerized images simplifies the management and maintenance of big data components because containers can be quickly started, stopped and replaced. Containerization ensures consistency between development, test and production environments, reducing problems caused by "environment differences". Cloud native environments support rapid iteration, allowing new versions of big data components to be quickly deployed and tested. Cloud platforms usually provide integrated monitoring and logging tools to facilitate tracking and analysis of the performance and status of big data components. The security features of cloud platforms, such as network isolation and role-based access control (RBAC), can improve the operational security of big data components. Cloud service providers usually provide disaster recovery solutions to ensure that services can be quickly restored in the event of a failure. Containerized images can run on different cloud platforms and local environments, improving the compatibility and flexibility of big data components. The integration of cloud-native environments with CI / CD processes makes automated testing and deployment of new versions easier.
[0128] This application also provides an exemplary code on a product:
[0129]
[0130]
[0131]
[0132] In this example, the Kubernetes configuration file is first loaded, which usually points to a kubeconfig file, which contains the authentication information required to interact with the Kubernetes cluster. Then, a `V1Deployment` object is created, which defines the containerized image, number of replicas, ports, and environment variables for the big data component. The number of replicas can be set based on the information in the workload description template file. Next, a `V1Service` object is created, which allows external traffic to access the Pods in the Deployment through the defined port. Finally, the `AppsV1Api` and `CoreV1Api` instances are used to create the Deployment and Service respectively in the Kubernetes cluster.
[0133] The present application also provides a big data component management method based on cloud native technology, which includes:
[0134] Description file configuration step: configure a big data component description file, wherein the big data component description file at least defines cloud-native-based big data component roles, dependencies between big data components, and configuration parameters of big data components;
[0135] Component operation template configuration step: configuring an operation configuration template file for running the big data component, wherein the operation configuration template file at least includes operation parameters and configuration information of the big data component, and the configuration information at least includes a port number, a storage path, and a log level;
[0136] Load description template configuration step: configuring a load description template file when the big data component is working, wherein the load description template file at least includes the operation mode, resource requirements, and number of copies of the big data component in the cloud environment;
[0137] Component operation configuration step: generating a configuration file required for the operation of the big data component according to the big data component description file and the operation configuration template file, wherein the configuration file required for the operation of the big data component guides the deployment of the big data component;
[0138] Configuration file distribution step: distribute the configuration files required for the operation of the big data component to multiple physical servers to be deployed, so as to generate a description file based on which the big data component is deployed on multiple physical servers to be deployed, so that the big data components arranged on each of the physical servers to be deployed have configuration consistency characteristics.
[0139] For exemplary descriptions of the various steps in the above method embodiment, reference may be made to the above device embodiment.
[0140] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.
Claims
1. A big data component management device based on cloud native technology, characterized in that: include: A description file configuration module, used to configure a big data component description file, wherein the big data component description file at least defines cloud-native-based big data component roles, dependencies between big data components, and configuration parameters of big data components; A component operation template configuration module, used to configure an operation configuration template file for running the big data component, wherein the operation configuration template file at least includes operation parameters and configuration information of the big data component, wherein the configuration information at least includes a port number, a storage path, and a log level; A load description template configuration module, used to configure a load description template file when the big data component is working, wherein the load description template file at least includes the operation mode, resource requirements, and number of copies of the big data component in the cloud environment; A component operation configuration module, used to generate a configuration file required for the operation of the big data component according to the big data component description file and the operation configuration template file, wherein the configuration file required for the operation of the big data component is used to guide the deployment of the big data component; A configuration file distribution module, used to distribute the configuration files required for the big data component to be run to multiple physical servers to be deployed, so as to generate a description file for deploying the big data component on the multiple physical servers to be deployed, so that the big data components deployed on each of the physical servers to be deployed have a configuration consistency feature; The configuration file distribution module is used to distribute the configuration files required for the big data component to be run to multiple physical servers to be deployed, so as to generate a description file for deploying the big data component on the multiple physical servers to be deployed, so that the big data components arranged on each of the physical servers to be deployed have configuration consistency characteristics, including: Trigger the SCP file distribution component, execute the written parsing script to obtain the physical server list, and parse the physical server list to obtain the IP address of the physical server; keep the SCP file distribution component in a triggered state to execute the written verification script, so as to synchronously distribute the configuration files required for the operation of the big data component to multiple physical servers to be deployed through the scheduled task cron job, so as to generate a description file based on which the big data component is deployed on the multiple physical servers to be deployed, so that the big data components laid out on each of the physical servers to be deployed have a configuration consistency feature.
2. According to the big data component management device based on cloud native technology according to claim 1, it is characterized in that: The big data component management device further includes: A cloud-native workload configuration module, used to generate a cloud-native workload description file according to the workload description template file, wherein the workload description file is used to at least describe the operation specifications, resource requirements and replica strategies of the big data component when it is running in a cloud environment built based on the physical server to be deployed; A big data component running module is used to start running the big data component in a cloud environment built based on the physical server to be deployed according to the cloud-native workload description file.
3. According to the big data component management device based on cloud native technology according to claim 1, it is characterized in that: The description file configuration module is used to configure a big data component description file, wherein the big data component description file at least defines a cloud-native big data component role, dependencies between big data components, and configuration parameters of the big data component, specifically including: Parsing the big data component role model to determine the big data component roles, the big data component roles at least including: data source, data processor, data storage, data analyzer; Parsing the property files of each big data component based on the dependency interpreter, obtaining the dependency description between the big data components, and defining the dependency description in the form of key-value pairs to generate the dependency between the big data components; Parsing the configuration files of each big data component based on the configuration parameter parser to determine the placeholders therein, querying the configuration parameters of the big data component in the configuration files of the big data component based on the placeholders, and converting the configuration parameters into key-value pairs; Based on a structured language, using a set configuration engine, a container orchestration tool is called to containerize the big data component roles, dependencies between big data components, and configuration parameters of big data components to generate the big data component description file.
4. The big data component management device based on cloud native technology according to claim 1, characterized in that: Based on the structured language, using the set configuration engine, calling the container orchestration tool to containerize the big data component roles, the dependencies between the big data components, and the configuration parameters of the big data components to generate the big data component description file specifically includes: Based on a structured language, using a set configuration engine, calling a container orchestration tool to define resource objects, a service discovery mechanism, and a configuration parameter chart, wherein the resource object is at least used to specify a container image, a number of replicas, and a port, the service discovery mechanism is used to encapsulate the dependencies between the big data components, and the configuration parameter chart is used to map the configuration parameters of the big data components; Trigger the resource object to reference the service discovery mechanism and the configuration parameter chart for containerization, so as to encapsulate the code of the big data component through the container image, configure the number of copies of the big data component with dependencies through the number of copies, and define the access mechanism of the big data component based on the configuration parameters of the big data component through the port to generate the big data component description file.
5. The big data component management device based on cloud native technology according to claim 1, characterized in that: The component operation template configuration module is used to configure the operation configuration template file of the big data component, the operation configuration template file at least includes the operation parameters and configuration information of the big data component, and the configuration information at least includes the port number, storage path, and log level, including: Obtaining an operating parameter description document and a configuration item description document of a big data component, and parsing the operating parameter description document and the configuration item description document based on regular matching to obtain the operating parameters and configuration information of the big data component; The operation parameters and configuration information of the big data component are encapsulated into structured format data, and the created formatting template file is called to synchronously replace the structured format data with the replaceable parameters defined in the formatting template file through the parameterized language in the formatting template file, so as to generate the operation configuration template file of the big data component.
6. The big data component management device based on cloud native technology according to claim 1, characterized in that: The load description template configuration module is used to configure the load description template file when the big data component is working. The load description template file at least includes the operation mode, resource requirements, and number of copies of the big data component in the cloud environment, including: Based on the Prometheus monitoring component, we obtain the throughput samples, processing speed samples, data volume samples, computing resource samples, storage resource samples, network bandwidth samples, and replication factor samples of the big data components when they are running in the test cloud environment, and perform regression analysis on these data to estimate the operation mode, resource requirements, and number of replicas of the big data components in the cloud environment. Based on the resource allocation strategy, replica allocation logic and performance indicator threshold of the big data component at runtime, call the automated script to tune the operation mode, resource requirements and number of replicas of the data component in the cloud environment, and instantiate the optimized operation mode, resource requirements and number of replicas to generate a load description template file for the big data component when it is working.
7. The big data component management device based on cloud native technology according to claim 1, characterized in that: The component operation configuration module is used to generate a configuration file required for the big data component to run according to the big data component description file and the operation configuration template file. The configuration file required for the big data component to run is used to guide the deployment of the big data component, including: Generate an operation configuration decision tree model according to the big data component description file and the operation configuration template file; Based on the operation configuration decision tree model, a configuration file required for the big data component to run is generated.
8. The big data component management device based on cloud native technology according to claim 7, characterized in that: The step of generating an operation configuration decision tree model according to the big data component description file and the operation configuration template file includes: Performing operation logic analysis on the big data component description file and the operation configuration template file, and building a component operation navigation path; Generate an operation configuration decision tree model according to the component operation navigation path; The generating, based on the operation configuration decision tree model, a configuration file required for the big data component to run, comprises: The operation configuration decision tree model is matched with the configuration file template constructed when the big data component is tested at runtime to generate the configuration file required for the big data component to run.
