Heterogeneous big data resource mining and scheduling method and system based on hybrid cloud native architecture

Abstract

The application relates to the field of new-generation information technology and discloses a heterogeneous big data resource mining scheduling method and system based on a hybrid cloud native architecture, which comprises the following steps: S1: collecting various node data in the architecture through a data collection agent deployed in the hybrid cloud native architecture to obtain an original heterogeneous data source set, performing a metadata labeling operation on the original heterogeneous data source set to obtain a resource attribution identification data set; S2: performing a data hierarchical label marking operation on the resource attribution identification data set to obtain a data set carrying hierarchical labels, performing a format normalization operation on the data set carrying the hierarchical labels to standardize the heterogeneous data set; S3: constructing a resource topology graph based on the standardized heterogeneous data set to obtain a resource topology graph, and performing a graph node load pre-detection operation on the cross-domain resource topology graph to obtain a load topology graph.

Description

Technical Field

[0001] This invention relates to the field of next-generation information technology, and in particular to a method and system for mining and scheduling heterogeneous big data resources based on a hybrid cloud-native architecture. Background Technology

[0002] With the continuous iteration and implementation of cloud computing technology, hybrid cloud-native architecture has gradually become the mainstream carrier for enterprises' heterogeneous business deployments. The architecture encompasses IaaS layer hardware resources, PaaS layer microservice components, and various upper-layer business systems, with massive amounts of multi-source heterogeneous operational data continuously generated along with business operations. In the early days, resource scheduling relied on manual configuration of fixed parameters, and operations and maintenance personnel relied on historical experience to set resource allocation values, scaling thresholds, and read / write traffic splitting rules, which could only adapt to simple scenarios with stable business volume and minimal fluctuations in resource load.

[0003] In existing technologies, traditional topology graph construction relies solely on resource ownership information to build nodes and edges, without supplementing node labels based on real-time resource load status. As a result, the graph cannot intuitively reflect the differences in load levels among different resource entities. Subsequent edge attribute assignments rely only on basic operational indicators, lacking load dimension data support. Consequently, the graph depicts a one-sided resource operational status, and the scheduling reference is not comprehensive enough. Second, conventional DQN optimization models use fixed-size state and action spaces, which cannot adapt to the addition or removal of hybrid cloud resource topologies. When new or offline resource nodes are added to the architecture, the model input dimensions cannot match the actual map data, which can easily lead to model calculation errors and optimization results failure. Third, the existing DQN reward function only builds optimization logic around the single indicator of resource usage cost. The model iteration process only pursues the minimization of resource cost, which is very easy to over-compress hardware resource quotas in order to reduce costs. This can lead to the transmission latency of the business link exceeding the upper limit agreed by the SLA protocol and the business throughput failing to meet the standard. It is impossible to simultaneously take into account both cost control and business performance constraints. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art and to achieve the above objectives, the present invention proposes the following technical solution: A heterogeneous big data resource mining and scheduling method based on a hybrid cloud-native architecture includes: S1: Collect data from various nodes within the architecture through a data collection agent deployed on a hybrid cloud-native architecture to obtain a set of raw heterogeneous data sources. Perform metadata annotation operations on the raw heterogeneous data source set to obtain a resource ownership identifier dataset. S2: Perform data hierarchical labeling on the resource ownership identifier dataset to obtain a dataset with hierarchical labels. Perform format normalization on the dataset with hierarchical labels to standardize the heterogeneous data set. S3: Construct a resource topology graph based on a standardized heterogeneous data set to obtain a resource topology graph. Perform a pre-detection operation on the graph node load of the cross-domain resource topology graph to obtain a load topology graph. Combine the standardized heterogeneous data set to perform edge attribute assignment operation on the load topology graph to obtain a cross-domain resource topology graph with attributes. S4: Perform format conversion on the cross-domain resource topology map with attributes to obtain the model state input. Configure model constraints according to the hybrid cloud native architecture operation rules to obtain the model constraint set. Combine the model state input and the model constraint set to run the DQN reinforcement learning cost optimization model to perform calculations and obtain the mining scheduling strategy. S5: Deploy the resource mining and scheduling strategy to the hybrid cloud-native architecture control plane to execute the configuration and implement resource mining and scheduling.

[0005] The process of obtaining the original heterogeneous data source set includes: Start the data collection agent inside the hybrid cloud native architecture, and poll the IaaS layer resource nodes in the architecture according to the preset period to obtain the raw IaaS layer data; Start the data collection agent inside the hybrid cloud native architecture, and poll the PaaS layer service nodes in the architecture according to the preset period to obtain the raw data of the PaaS layer; Start the data collection agent within the hybrid cloud-native architecture, and poll the business system nodes within the architecture according to the preset period to obtain the raw data of the business system; The original data from the IaaS layer, PaaS layer, and business system are merged and summarized to obtain the original heterogeneous data source set.

[0006] The process of obtaining the resource ownership identifier dataset includes: Read each data source unit in the original heterogeneous data source set, identify the architecture level corresponding to the data source unit, and obtain the data source level identification result; Based on the data source hierarchy identification result, a preset hierarchy code is matched, and the corresponding hierarchy code is bound to each data source unit to obtain the bound code data source unit; The bound encoding data source units are arranged in an orderly manner according to the architecture hierarchy to obtain the resource ownership identifier dataset.

[0007] The process of obtaining a dataset with hierarchical labels includes: Based on each data unit in the resource ownership identifier dataset, the business priority determination result is obtained by judging the business importance corresponding to the data unit through historical data. Based on each data unit in the resource ownership identifier dataset, the confidentiality control level corresponding to the data unit is determined by historical data to obtain the security level determination result; Based on the business priority determination result and the security level determination result, a corresponding hierarchical label is attached to each data unit to obtain the data unit with the attached label. All the data units with the additional labels are classified and integrated according to the hierarchical labels to obtain a dataset with hierarchical labels.

[0008] The process of obtaining a resource topology map includes: Resource ownership identifiers are extracted from each data unit of a standardized heterogeneous dataset to obtain a complete sequence of resource identifiers; The unified control plane of the hybrid cloud-native architecture is invoked to map the resource identifier sequence to the corresponding physical resource entity, thereby obtaining a set of resource entity vertices; By analyzing the runtime dependencies between different resource entity vertices in the resource entity vertex set using a unified control plane, a set of entity dependencies is obtained. A graph structure is constructed based on the set of resource entity vertices and the set of entity dependencies to obtain the resource topology graph.

[0009] The process of performing a load pre-detection operation on the cross-domain resource topology graph to obtain the load topology graph includes: Traverse all resource entity vertices within the resource topology graph, collect the real-time running load value corresponding to each vertex, and obtain the vertex load value set. The values ​​in the vertex load value set are compared with the preset load threshold to classify the load level of each vertex and obtain the vertex load level result. Based on the vertex load level results, a corresponding load level label is attached to each resource entity vertex to obtain a graph node with the attached load label; The original graph structure is reconstructed based on the graph nodes with the additional load labels to obtain the load topology graph.

[0010] The process of obtaining a cross-domain resource topology map with attributes includes: Extract the unified logical model identifier carried by the data unit from the standardized heterogeneous data set, and count the access frequency, throughput and latency indicators corresponding to each logical model identifier to obtain the set of operation indicators. Traverse all connecting edges inside the load topology graph, bind the corresponding indicators in the running indicator set to the connecting edges, and obtain the connecting edges after attribute assignment. By integrating the connected edges with all assigned attributes with the original graph vertices, a cross-domain resource topology graph with attributes is obtained.

[0011] The process of obtaining the model constraint set includes: Retrieve the preset resource consumption cost limit parameters of the hybrid cloud-native architecture to obtain the cost constraints; The performance constraints are obtained by retrieving the default service level agreement standard parameters of the hybrid cloud-native architecture. The cost constraint and the performance constraint are combined to obtain the model constraint set.

[0012] The process of obtaining the mining scheduling strategy includes: The model state input is loaded into the input interface of the DQN reinforcement learning cost optimization model to initialize the model's state space and action space, thus obtaining the model's initial running state. Input the set of model constraints into the DQN reinforcement learning cost optimization model, set the objective function boundary, performance index threshold and cost upper limit of the model, and obtain the model running constraint configuration. Based on the initial running state of the model and the configuration of the model running constraints, with the optimization objective of minimizing resource consumption cost, the iterative training process of the DQN reinforcement learning cost optimization model is started to obtain the optimal action sequence; The optimal action sequence is segmented into three independent action codes by using a fixed field partitioning technique according to a preset coding interval. The segmentation results in a resource allocation coding segment, an elastic scaling coding segment, and a read-write separation coding segment. Retrieve the pre-stored encoding resource mapping lookup table of the hybrid cloud-native architecture, match the resource allocation encoding segments within the three independent action codes after splitting, and calculate the configuration parameters of each resource entity based on the corresponding mapping relationship between the encoding and resource quota to obtain the resource allocation scheme; Retrieve the same coding resource mapping lookup table, match the elastic scaling coding segments within the three types of independent action codes after splitting, calculate the expansion and contraction threshold parameters of each resource node, and obtain the elastic scaling rules. Retrieve the same encoding resource mapping table, match the read-write separation encoding segments within the three types of independent action encodings after splitting, calculate the read-write splitting ratio parameters of the business link, and obtain the read-write separation strategy; Resource allocation schemes, elastic scaling rules, and read / write separation strategies are encapsulated and integrated according to a preset format recognizable by the hybrid cloud-native architecture control plane to obtain the mining scheduling strategy.

[0013] A heterogeneous big data resource mining and scheduling system based on a hybrid cloud-native architecture is characterized by: Resource acquisition module: Collects data from various nodes within the architecture through a data acquisition agent deployed on a hybrid cloud-native architecture to obtain a set of raw heterogeneous data sources. Performs metadata annotation operations on the raw heterogeneous data source set to obtain a resource ownership identifier dataset. Heterogeneous processing module: Performs data hierarchical labeling on the resource ownership identifier dataset to obtain a dataset with hierarchical labels, and performs format normalization on the dataset with hierarchical labels to standardize the heterogeneous data set; The graph construction module constructs a resource topology graph based on a standardized heterogeneous data set, obtains a resource topology graph, performs a pre-detection operation on the graph node load of the cross-domain resource topology graph to obtain a load topology graph, and performs edge attribute assignment operation on the load topology graph in combination with the standardized heterogeneous data set to obtain a cross-domain resource topology graph with attributes. Strategy optimization module: The cross-domain resource topology map with attributes is format converted to obtain the model state input. The model constraints are configured according to the hybrid cloud native architecture operation rules to obtain the model constraint set. The DQN reinforcement learning cost optimization model is run in combination with the model state input and the model constraint set to obtain the mining scheduling strategy. Resource mining and scheduling module: Distributes resource mining and scheduling strategies to the hybrid cloud-native architecture control plane to execute configurations and implement resource mining and scheduling.

[0014] The present invention has the following beneficial effects: 1. By adding a node load pre-detection step after the topology graph is built, load level labels are added to the graph nodes based on real-time load values. Load dimension information is added on the basis of the original resource association relationship. Subsequent edge attribute assignment is carried out in combination with both operation indicators and node load data to improve the data integrity of the graph. This allows the resource topology to fully restore the real operating status of the architecture and improve the data reference accuracy of subsequent scheduling parameter calculation.

[0015] 2. By dynamically determining the state space dimension and action space value range of the DQN model based on the total number of vertices and the range of edge attribute values ​​in the topological graph, the model network structure can adapt to changes in the scale of hybrid cloud resources, avoiding the problem of model dimension incompatibility after resource increases or decreases, improving the model's environmental adaptability in heterogeneous hybrid cloud scenarios, and broadening the model's applicable scope.

[0016] 3. By introducing cost constraints and SLA performance constraints simultaneously during the model configuration phase, cost and latency indicators are coupled together into the model reward function. During model iterative optimization, the upper limit of resource usage cost and the lower limit of business performance are simultaneously constrained. This achieves resource-intensive cost reduction while ensuring that various indicators of the business link meet the service agreement requirements, balancing the two needs of cost optimization and stable business operation, and overcoming the business performance control defects caused by single cost optimization.

[0017] 4. By segmenting the optimal action sequence output by the model using a fixed delimiter code, and relying on a pre-compiled encoding resource mapping table, the encoding and scheduling parameters are standardized and mapped. The segmentation rules and mapping rules are fixed in advance, eliminating the need for temporary custom parsing scripts. This enables the accurate conversion of continuous encoding into resource allocation schemes, elastic scaling rules, and read-write separation strategies, improving the efficiency and accuracy of converting model optimization results into actual scheduling instructions. Attached Figure Description

[0018] Figure 1 This diagram illustrates the method steps of the heterogeneous big data resource mining and scheduling method and system based on a hybrid cloud-native architecture proposed in this invention.

[0019] Figure 2 This is a system block diagram of the heterogeneous big data resource mining and scheduling method and system based on a hybrid cloud-native architecture proposed in this invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1 like Figure 1 As shown, the heterogeneous big data resource mining and scheduling method based on a hybrid cloud-native architecture proposed in this invention includes: S1: Collect data from various nodes within the architecture through a data collection agent deployed on a hybrid cloud-native architecture to obtain a set of raw heterogeneous data sources. Perform metadata annotation operations on the raw heterogeneous data source set to obtain a resource ownership identifier dataset. Specifically, the process of obtaining the original heterogeneous data source set is as follows: Start all data collection agents deployed within the hybrid cloud-native architecture, poll IaaS layer resource nodes according to the architecture's preset fixed collection cycle, capture virtual machine and storage hardware operation logs and resource usage data to obtain raw IaaS layer data; reuse the same batch of data collection agents, poll PaaS layer service nodes according to the same preset cycle, capture middleware and container cluster call log data to obtain raw PaaS layer data; Continue to reuse the data collection agent to poll the front-end business system nodes, capture business interactions and database read / write logs, and obtain the raw data of the business system; The existing data merging and summarizing tool is invoked to perform field alignment and data concatenation on the original data from the IaaS layer, PaaS layer, and business system, and the resulting set of original heterogeneous data sources is obtained after merging and summarizing. Specifically, the process of obtaining the resource ownership identifier dataset is as follows: Traverse the original heterogeneous data source collection All data source units within the system call the existing field keyword matching algorithm to identify the data's IaaS layer, PaaS layer, or business system through the data's built-in path field, thus obtaining the data source layer identification result; Retrieve the pre-stored hierarchical coding lookup table of the architecture, match the corresponding numerical codes according to the data source hierarchical identification results, and add coding identifiers to the data source units one by one to obtain the bound coding data source units; Based on the sorting rule of coded values ​​from smallest to largest, the existing data sorting component is invoked to perform an ordered rearrangement of all bound coded data source units, ultimately generating a resource ownership identifier dataset. ; in, This represents the resource ownership identifier corresponding to the i-th data source.

[0022] S2: Perform data hierarchical labeling on the resource ownership identifier dataset to obtain a dataset with hierarchical labels. Perform format normalization on the dataset with hierarchical labels to standardize the heterogeneous data set. Specifically, the process of obtaining a dataset with hierarchical labels is as follows: Read each data unit in the resource ownership identifier dataset, retrieve the business access frequency records in the architecture historical business operation database, use the frequency interval division algorithm to determine the importance of the business corresponding to the data, and obtain the business priority determination result; Continue reading the same batch of data units, retrieve the security level configuration parameters in the architecture security management ledger, determine the confidentiality management level based on the existing data security level classification standard, and obtain the security level determination result; Based on the preset priority-security level association mapping table, and combined with the business priority determination result and the security level determination result, a unique hierarchical label is attached to each data unit to obtain the data unit with the attached label; By calling existing classification and aggregation tools, all data units with additional labels are grouped and integrated according to the identification content of the hierarchical labels to obtain a dataset with hierarchical labels; Then, the existing heterogeneous data format normalization tool is called to unify the storage format and encoding rules of the dataset with hierarchical labels, and noise filtering for missing and outlier removal is performed simultaneously to obtain a standardized heterogeneous data set.

[0023] S3: Construct a resource topology graph based on a standardized heterogeneous data set to obtain a resource topology graph. Perform a pre-detection operation on the graph node load of the cross-domain resource topology graph to obtain a load topology graph. Combine the standardized heterogeneous data set to perform edge attribute assignment operation on the load topology graph to obtain a cross-domain resource topology graph with attributes. Specifically, the process of obtaining the resource topology map is as follows: From each data unit of the standardized heterogeneous dataset, the pre-annotated resource ownership identifiers are extracted through field truncation operations. All identifier characters are then summarized to obtain the complete resource identifier sequence. The field truncation uses existing SQL string truncation functions. Call the resource mapping interface of the built-in unified control plane of the hybrid cloud native architecture, such as the existing K8s resource query interface, to map all resource identifier sequences one by one to the real physical resource entities inside the architecture, and summarize the mapping results to obtain the set of resource entity vertices; Based on the resource call chain log built into the unified control plane, a mining algorithm is used to analyze the deployment dependency and call dependency relationships between different resources within the resource entity vertex set, and all relationships are summarized to obtain the entity dependency relationship set; The graph construction interface of the existing graph database Neo4j is called to build a graph structure with resource entity vertices as nodes and entity dependencies as edges, and generate a resource topology graph. Specifically, the process of obtaining the load topology map includes: The process iterates through all resource entity vertices within the resource topology graph, invokes the Prometheus operations and maintenance monitoring component (a performance monitoring component), and uses the component's built-in node probes to connect to the corresponding IaaS layer physical host and PaaS layer container instance for each vertex. It periodically captures two types of real-time raw data for each hardware unit: instantaneous CPU utilization and used memory capacity. For multiple time-series data collected from the same vertex, an arithmetic average algorithm is used for aggregation. The aggregated values ​​for all vertices are then summarized to obtain the vertex load value set. This process is represented as follows:

[0024] in, Represents the set of vertex load values. This represents the load value after aggregating the j-th resource vertex; This represents the original load sample value of the j-th vertex at time t, where T represents the total number of samples in a single sampling period. The vertex set represents the complete set of vertices in the resource topology graph, and k represents the total number of resource vertices in the graph; Retrieve the pre-defined load grading threshold table from the hybrid cloud-native architecture configuration library. The threshold table is divided into three critical values: high load threshold, medium load threshold, and low load threshold, based on the upper limit of the architecture's service capacity. Then, retrieve each item from the set of peak load values. Compare the values ​​with the three thresholds one by one, and classify the load levels into high, medium and low based on the range of the values. Determine the load level of each resource vertex to obtain the vertex load level result. Based on graph database field extension technology, the load information of each vertex in the vertex load level result is used as the label content. A load label field is added to the original vertex attribute field of the resource topology graph. The corresponding level label is written for each graph vertex one by one. After the label binding operation is completed, the graph node with the attached load label is obtained. Based on the existing node replacement and graph reconstruction interfaces provided by Neo4j graph database, graph nodes with added load labels are used to replace the original nodes in the resource topology graph that originally had no load information. The graph storage structure is reconstructed without changing the original vertex dependency edge association relationship, and finally a load topology graph is generated. Specifically, the process of obtaining a cross-domain resource topology map with attributes includes: The unified logical model identifier carried by each data unit is extracted from the standardized heterogeneous data set. The existing big data SQL aggregation statistics technology is used to group the data according to the unified logical model identifier. The three types of operation indicators for each group of identifiers are statistically analyzed: the access frequency per unit time, the data throughput per second, and the average transmission latency of the request. The abnormal indicator data after grouping is processed by the existing median noise reduction. All grouped indicator data are summarized to obtain the set of operation indicators. Traverse all connecting edges within the load topology graph. Based on the association mapping relationship of resource ownership identifiers, match the unified logical model identifiers corresponding to the vertices at both ends of the current edge. Retrieve the three indicator parameters bound to the identifier from the set of running indicators: time access frequency, data throughput per second, and average request transmission latency. Call the graph attribute writing existing interface and write the three retrieved indicators as edge extended attributes into the attribute fields of the corresponding connecting edge. Complete the indicator binding of all edges one by one to obtain the connecting edges after attribute assignment. While maintaining the original graph node structure with added load labels, the connecting edges with all attribute values ​​are structurally reorganized with the original graph vertices. The graph edge attribute fields are then completed to integrate the entire graph data and finally generate a cross-domain resource topology graph with attributes.

[0025] S4: Perform format conversion processing on the cross-domain resource topology map with attributes to obtain the model state input. Configure model constraints according to the hybrid cloud native architecture operation rules to obtain the model constraint set. Combine the model state input and the model constraint set to run the optimization model to perform calculations and obtain the mining scheduling strategy. Specifically, the process of obtaining the model state input is as follows: Read the pre-entered raw parameters of various resource unit prices and maximum resource purchase quotas from the hybrid cloud-native architecture operation and maintenance configuration database, calculate the unit usage cost of all resource categories based on the pricing rules of each resource category, summarize them, and construct the upper limit constraint relationship of the total resource cost to obtain the cost constraint condition, expressed as:

[0026] in, Represents the total actual cost of all resources actually used by the architecture. Represents the unit usage cost of resource type j. This represents the actual quantity of resource j allocated. This represents the maximum cost limit for all resources pre-defined by the architecture; Specifically, the process of obtaining the model constraint set is as follows: Retrieve the Service Level Agreement (SLA) document of the hybrid cloud-native architecture archive storage, extract two types of hard performance parameters from the document's legal terms: maximum allowable latency and minimum guaranteed throughput of the business link, organize them to form the full-link performance constraint boundary, and obtain the performance constraint conditions; The parameters are all legally mandated standard values ​​that were signed and implemented before the business went live. A JSON structured encapsulation tool is used to package and integrate cost constraints and performance constraints in key-value pair format, and encapsulate them into a structured parameter group that the model can directly read. After integration, a model constraint group is generated. Specifically, the process of obtaining the mining scheduling strategy is as follows: The model constraint set is imported into the standardized data input interface of the DQN reinforcement learning cost optimization model. The built-in parameter initialization maturity function of the DQN reinforcement learning cost optimization model is called. The Xavier initialization rule is adopted to extract the total number of elements of the vertex set of the cross-domain resource topology graph from the model state input. The dimension of the model state space is determined by the number of vertices. Extract the range of edge attribute values ​​from the set of attributed edges in the cross-domain resource topology graph, and combine it with the upper and lower limits of the quota for hybrid cloud resources to define the legal value ranges of various resource allocation values, scaling thresholds, and read / write ratios in the action space, thus completing the initial configuration of the action space. The state space dimension and action space are used as configuration parameters for the model framework. The original vertex, edge, and metric data carried by the model state input are bound to these parameters. After being encapsulated by the model's built-in initialization function, the initial running state of the model is generated, represented as follows:

[0027] in, Represents the initial running state of the model. Represents the model state input. Represents the state space dimension. Represents the action space; Initialize the model with a function; Based on the initial running state of the model, Using the model environment input and model operation constraints as the computational boundary, and minimizing the total resource cost as the global optimization objective, the built-in DQN reinforcement learning cost optimization model is activated. Iteratively train the existing algorithm and construct a single-step instant reward function for iterative updates of model parameters. The reward function formula is as follows:

[0028] in, This represents the instantaneous reward value obtained by the model in the t-th iteration. Let t be the actual total cost of resources in round t. The weighting coefficient of the representative cost item, This represents the penalty coefficient for exceeding the latency limit. For real-time link latency, The maximum allowable delay is agreed upon for the SLA. This represents a penalty that is only activated when the actual latency exceeds the limit; otherwise, the value of this item is 0. The model continuously iterates and updates network parameters through environmental interaction and reward feedback until the model loss function converges to a preset minimum threshold. After the iteration terminates, the discrete coding sequence is extracted from the model output layer to obtain the optimal action sequence. The training process of the DQN reinforcement learning cost optimization model includes: The training input consists of the model's initial running state and a set of model constraints. First, the dimensions of the model's state space and action space are determined based on the initial running state, and then the training network parameters are initialized. With target network parameters Meanwhile, a blank experience replay pool is constructed, and fixed hyperparameters are set to obtain the initial configuration for model training; The training process employs a multi-round iterative interaction approach, with the model adjusting based on the current environmental state. The greedy strategy selects the currently scheduled action in the action space, applies the action to the hybrid cloud resource environment, obtains the immediate reward and the updated environment state, forms an experience sample and stores it in the experience replay pool. When the number of samples in the experience replay pool meets the sampling conditions, a batch of samples is randomly drawn to calculate the target Q value. The error between the predicted Q value and the target Q value of the trained network is calculated using the loss function, and the parameters of the trained network are updated in reverse using the gradient descent algorithm. The model synchronizes the trained network parameters to the target network parameters every fixed number of iterations. To ensure training stability; The above interaction, sampling, and parameter update process is iterated repeatedly to continuously maximize the cumulative reward and minimize the resource deployment cost. When the model loss fluctuation is less than the preset threshold or the maximum number of iterations is reached, the model is determined to have converged and training is stopped. Finally, the current system state is greedily solved based on the converged network parameters to output the optimal action sequence. The optimal action sequence is segmented using a preset delimiter code, resulting in three independent coded segments: resource allocation, elastic scaling, and read / write separation. These three independent action codes are represented as follows:

[0029] in, This represents the complete set of codes for the optimal action sequence. This represents the resource allocation code segment after splitting. This represents the elastically scalable encoded segment after splitting. This represents a split, read-write separated encoded segment; Retrieve the pre-stored encoding resource mapping table of the architecture and match the resource allocation encoding segment. The resource allocation scheme is obtained by converting the number of resource instances allocated and the storage quota parameters. Multiplexing lookup table matching flexible scaling code segment The expansion and contraction trigger thresholds for each node are calculated to obtain the elastic scaling rules, and then the read-write separation coding segments are matched again by looking up the table. The read / write separation strategy is obtained by converting the read / write request splitting ratio of each business link with the target node number. According to the preset configuration file format for the control plane identification of hybrid cloud native architecture, the existing configuration encapsulation tool is called to integrate and package the three contents of resource allocation scheme, elastic scaling rules, and read-write separation strategy into a configuration message that can be directly parsed by the interface, and finally obtain the mining scheduling strategy.

[0030] S5: Distribute the resource mining and scheduling strategy to the hybrid cloud-native architecture control plane to execute the configuration and realize resource mining and scheduling; The system reads and extracts three types of structured configuration content encapsulated within the scheduling strategy: resource allocation scheme, elastic scaling rules, and read / write separation strategy. It then uses an existing JSON parsing tool to split the internal parameter fields of the strategy, resulting in three independent configuration parameters. Based on the parameters of the split resource allocation scheme, the control plane interfaces with the IaaS layer resource management interface and adjusts the allocation quota of virtual machine computing power and physical storage capacity one by one according to the resource quota values ​​in the parameters to complete the dynamic quota configuration of IaaS layer hardware resources. Based on the scaling threshold and shrinking threshold parameters included in the elastic scaling rules, modify the container scaling trigger configuration of the PaaS layer microservice cluster, and automatically trigger instance addition or instance destruction actions when the cluster load touches the corresponding threshold. Based on the business traffic splitting ratio and target node number carried by the read / write separation strategy, modify the business system database read / write routing configuration to split business read requests and write requests to the corresponding storage nodes according to the preset ratio.

[0031] Example 2 like Figure 2 As shown, the heterogeneous big data resource mining and scheduling system based on a hybrid cloud-native architecture includes: Resource acquisition module: Collects data from various nodes within the architecture through a data acquisition agent deployed on a hybrid cloud-native architecture to obtain a set of raw heterogeneous data sources. Performs metadata annotation operations on the raw heterogeneous data source set to obtain a resource ownership identifier dataset. Heterogeneous processing module: Performs data hierarchical labeling on the resource ownership identifier dataset to obtain a dataset with hierarchical labels, and performs format normalization on the dataset with hierarchical labels to standardize the heterogeneous data set; The graph construction module constructs a resource topology graph based on a standardized heterogeneous data set, obtains a resource topology graph, performs a pre-detection operation on the graph node load of the cross-domain resource topology graph to obtain a load topology graph, and performs edge attribute assignment operation on the load topology graph in combination with the standardized heterogeneous data set to obtain a cross-domain resource topology graph with attributes. Strategy optimization module: The cross-domain resource topology map with attributes is format converted to obtain the model state input. The model constraints are configured according to the hybrid cloud native architecture operation rules to obtain the model constraint set. The DQN reinforcement learning cost optimization model is run in combination with the model state input and the model constraint set to obtain the mining scheduling strategy. Resource mining and scheduling module: Distributes resource mining and scheduling strategies to the hybrid cloud-native architecture control plane to execute configurations and implement resource mining and scheduling.

[0032] In the application, several formulas are calculated by removing dimensions and taking their numerical values. The formulas are established by collecting a large amount of data and simulating the most recent real situation. Some coefficients or weights in the formulas are set by those skilled in the art according to the actual situation, so they will not be elaborated here.

[0033] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0034] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A heterogeneous big data resource mining and scheduling method based on a hybrid cloud-native architecture, characterized in that, include: S1: Collect data from various nodes within the architecture through a data collection agent deployed on a hybrid cloud-native architecture to obtain a set of raw heterogeneous data sources. Perform metadata annotation operations on the raw heterogeneous data source set to obtain a resource ownership identifier dataset. S2: Perform data hierarchical labeling on the resource ownership identifier dataset to obtain a dataset with hierarchical labels. Perform format normalization on the dataset with hierarchical labels to standardize the heterogeneous data set. S3: Construct a resource topology graph based on a standardized heterogeneous data set to obtain a resource topology graph. Perform a pre-detection operation on the graph node load of the cross-domain resource topology graph to obtain a load topology graph. Combine the standardized heterogeneous data set to perform edge attribute assignment operation on the load topology graph to obtain a cross-domain resource topology graph with attributes. S4: Perform format conversion on the cross-domain resource topology map with attributes to obtain the model state input. Configure model constraints according to the hybrid cloud native architecture operation rules to obtain the model constraint set. Combine the model state input and the model constraint set to run the DQN reinforcement learning cost optimization model to perform calculations and obtain the mining scheduling strategy. S5: Deploy the resource mining and scheduling strategy to the hybrid cloud-native architecture control plane to execute the configuration and implement resource mining and scheduling.

2. The heterogeneous big data resource mining and scheduling method based on a hybrid cloud-native architecture according to claim 1, characterized in that, The process of obtaining the original heterogeneous data source set includes: Start the data collection agent inside the hybrid cloud native architecture, and poll the IaaS layer resource nodes in the architecture according to the preset period to obtain the raw IaaS layer data; Start the data collection agent inside the hybrid cloud native architecture, and poll the PaaS layer service nodes in the architecture according to the preset period to obtain the raw data of the PaaS layer; Start the data collection agent within the hybrid cloud-native architecture, and poll the business system nodes within the architecture according to the preset period to obtain the raw data of the business system; The original data from the IaaS layer, PaaS layer, and business system are merged and summarized to obtain the original heterogeneous data source set.

3. The heterogeneous big data resource mining and scheduling method based on a hybrid cloud-native architecture according to claim 2, characterized in that, The process of obtaining the resource ownership identifier dataset includes: Read each data source unit in the original heterogeneous data source set, identify the architecture level corresponding to the data source unit, and obtain the data source level identification result; Based on the data source hierarchy identification result, a preset hierarchy code is matched, and the corresponding hierarchy code is bound to each data source unit to obtain the bound code data source unit; The bound encoding data source units are arranged in an orderly manner according to the architecture hierarchy to obtain the resource ownership identifier dataset.

4. The heterogeneous big data resource mining and scheduling method based on a hybrid cloud-native architecture according to claim 3, characterized in that, The process of obtaining a dataset with hierarchical labels includes: Based on each data unit in the resource ownership identifier dataset, the business priority determination result is obtained by judging the business importance corresponding to the data unit through historical data. Based on each data unit in the resource ownership identifier dataset, the confidentiality control level corresponding to the data unit is determined by historical data to obtain the security level determination result; Based on the business priority determination result and the security level determination result, a corresponding hierarchical label is attached to each data unit to obtain the data unit with the attached label. All the data units with the additional labels are classified and integrated according to the hierarchical labels to obtain a dataset with hierarchical labels.

5. The heterogeneous big data resource mining and scheduling method based on a hybrid cloud-native architecture according to claim 1, characterized in that, The process of obtaining a resource topology map includes: Resource ownership identifiers are extracted from each data unit of a standardized heterogeneous dataset to obtain a complete sequence of resource identifiers; The unified control plane of the hybrid cloud-native architecture is invoked to map the resource identifier sequence to the corresponding physical resource entity, thereby obtaining a set of resource entity vertices; By analyzing the runtime dependencies between different resource entity vertices in the resource entity vertex set using a unified control plane, a set of entity dependencies is obtained. A graph structure is constructed based on the set of resource entity vertices and the set of entity dependencies to obtain the resource topology graph.

6. The heterogeneous big data resource mining and scheduling method based on a hybrid cloud-native architecture according to claim 5, characterized in that, The process of performing a load pre-detection operation on the cross-domain resource topology graph to obtain the load topology graph includes: Traverse all resource entity vertices within the resource topology graph, collect the real-time running load value corresponding to each vertex, and obtain the vertex load value set. The values ​​in the vertex load value set are compared with the preset load threshold to classify the load level of each vertex and obtain the vertex load level result. Based on the vertex load level results, a corresponding load level label is attached to each resource entity vertex to obtain a graph node with the attached load label; The original graph structure is reconstructed based on the graph nodes with the additional load labels to obtain the load topology graph.

7. The heterogeneous big data resource mining and scheduling method and system based on a hybrid cloud-native architecture as described in claim 5, characterized in that, The process of obtaining a cross-domain resource topology map with attributes includes: Extract the unified logical model identifier carried by the data unit from the standardized heterogeneous data set, and count the access frequency, throughput and latency indicators corresponding to each logical model identifier to obtain the set of operation indicators. Traverse all connecting edges inside the load topology graph, bind the corresponding indicators in the running indicator set to the connecting edges, and obtain the connecting edges after attribute assignment. By integrating the connected edges with all assigned attributes with the original graph vertices, a cross-domain resource topology graph with attributes is obtained.

8. The heterogeneous big data resource mining and scheduling method based on a hybrid cloud-native architecture according to claim 7, characterized in that, The process of obtaining the model constraint set includes: Retrieve the preset resource consumption cost limit parameters of the hybrid cloud-native architecture to obtain the cost constraints; The performance constraints are obtained by retrieving the default service level agreement standard parameters of the hybrid cloud-native architecture. The cost constraint and the performance constraint are combined to obtain the model constraint set.

9. The heterogeneous big data resource mining and scheduling method based on a hybrid cloud-native architecture according to claim 8, characterized in that, The process of obtaining the mining scheduling strategy includes: The model state input is loaded into the input interface of the DQN reinforcement learning cost optimization model to initialize the model's state space and action space, thus obtaining the model's initial running state. Input the set of model constraints into the DQN reinforcement learning cost optimization model, set the objective function boundary, performance index threshold and cost upper limit of the model, and obtain the model running constraint configuration. Based on the initial running state of the model and the configuration of the model running constraints, with the optimization objective of minimizing resource consumption cost, the iterative training process of the DQN reinforcement learning cost optimization model is started to obtain the optimal action sequence; The optimal action sequence is segmented into three independent action codes by using a fixed field partitioning technique according to a preset coding interval. The segmentation results in resource allocation coding segment, elastic scaling coding segment, and read-write separation coding segment. Retrieve the pre-stored encoding resource mapping lookup table of the hybrid cloud-native architecture, match the resource allocation encoding segments within the three independent action codes after splitting, and calculate the configuration parameters of each resource entity based on the corresponding mapping relationship between the encoding and resource quota to obtain the resource allocation scheme; Retrieve the same coding resource mapping lookup table, match the elastic scaling coding segments within the three types of independent action codes after splitting, calculate the expansion and contraction threshold parameters of each resource node, and obtain the elastic scaling rules. Retrieve the same encoding resource mapping table, match the read-write separation encoding segments within the three types of independent action encodings after splitting, calculate the read-write splitting ratio parameters of the business link, and obtain the read-write separation strategy; Resource allocation schemes, elastic scaling rules, and read / write separation strategies are encapsulated and integrated according to a preset format recognizable by the hybrid cloud-native architecture control plane to obtain the mining scheduling strategy.

10. A heterogeneous big data resource mining and scheduling system based on a hybrid cloud-native architecture, implemented according to any one of claims 1-9, characterized in that: Resource acquisition module: Collects data from various nodes within the architecture through a data acquisition agent deployed on a hybrid cloud-native architecture to obtain a set of raw heterogeneous data sources. Performs metadata annotation operations on the raw heterogeneous data source set to obtain a resource ownership identifier dataset. Heterogeneous processing module: Performs data hierarchical labeling on the resource ownership identifier dataset to obtain a dataset with hierarchical labels, and performs format normalization on the dataset with hierarchical labels to standardize the heterogeneous data set; The graph construction module constructs a resource topology graph based on a standardized heterogeneous data set, obtains a resource topology graph, performs a pre-detection operation on the graph node load of the cross-domain resource topology graph to obtain a load topology graph, and performs edge attribute assignment operation on the load topology graph in combination with the standardized heterogeneous data set to obtain a cross-domain resource topology graph with attributes. Strategy optimization module: The cross-domain resource topology map with attributes is format converted to obtain the model state input. The model constraints are configured according to the hybrid cloud native architecture operation rules to obtain the model constraint set. The DQN reinforcement learning cost optimization model is run in combination with the model state input and the model constraint set to obtain the mining scheduling strategy. Resource mining and scheduling module: Distributes resource mining and scheduling strategies to the hybrid cloud-native architecture control plane to execute configurations and implement resource mining and scheduling.

Images