A big data demand analysis method and system serving a smart cloud service platform
By performing semantic parsing and pattern extraction on the service request data of the smart cloud service platform, a comprehensive demand feature map is generated. Combined with an improved hierarchical clustering algorithm and a resource demand derivation model, the problem of mismatch between resource pre-allocation schemes and dynamic demands in existing technologies is solved, and high-precision resource scheduling is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIANGSU ROBIN TECH SERVICE CO LTD
- Filing Date
- 2026-04-30
- Publication Date
- 2026-06-12
Smart Images
Figure CN122196348A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of smart cloud big data analysis technology, specifically a big data demand analysis method and system for smart cloud service platforms. Background Technology
[0002] The current demand analysis of the smart cloud service platform relies heavily on semantic parsing of service request texts, combined with statistical calculations on static resource request data, and classifies service requests using a single feature similarity. The platform only performs semantic extraction on task description texts, only performs total statistics on resource demand data, does not conduct structured correlation analysis of task-related events, and represents demand features using only a single dimension.
[0003] Existing solutions fail to perform independent pattern mining for time-series resource demands, thus failing to reflect the dynamic changes in resource requirements. Task relationships are not structured into a network, and the three types of features—service intent, resource data, and event information—are independent and not integrated. Traditional hierarchical clustering relies solely on static feature similarity to group requests, ignoring the impact of evolving resource demand trends. Clustering grouping has a low degree of matching with actual needs, leading to biases in the derivation of aggregated resource demands, and making resource pre-allocation difficult to adapt to the platform's dynamic operational requirements.
[0004] It is necessary to combine service intent characteristics, resource demand evolution trajectories, and event correlation networks to construct a comprehensive demand representation method. The clustering criteria need to be optimized, and service requests should be accurately grouped by simultaneously combining feature similarity and evolutionary trends. Summary of the Invention
[0005] This invention aims to solve at least one of the technical problems existing in the prior art; Therefore, this invention proposes a big data demand analysis method for a smart cloud service platform, comprising: Collect raw service request data from the smart cloud service platform, including task description text, resource request history, and execution time logs; The original service request data is subjected to normalization and cleaning to generate a standardized service request record set, which includes a service topic summary, a resource requirement time sequence, and task-related event records. Semantic parsing is performed on the service topic summary to obtain service intent features; pattern extraction is performed on the resource demand time sequence to obtain the resource demand evolution trajectory; and correlation analysis is performed on the task-related event records to obtain the event correlation network. The service intent features, resource demand evolution trajectory, and event correlation network are input into the demand analysis model for feature fusion to generate a comprehensive demand feature map. Based on the comprehensive demand feature map, an improved hierarchical clustering algorithm is used to automatically group service requests, forming a grouping result containing multiple service request clusters. The improved hierarchical clustering algorithm performs clustering analysis based on feature similarity and evolution trend. For each service request cluster, the aggregated resource requirement specification of the corresponding cluster is calculated through the resource requirement derivation model to form a clustered requirement list; Based on the clustering requirements list, an optimized resource pre-allocation scheme is generated for the resource scheduling center of the smart cloud service platform.
[0006] Further, semantic parsing is performed on the service topic summary to obtain service intent features, including: The service topic summary is input into a pre-trained business intent understanding model to extract a preliminary set of intent keywords; The initial intent keyword set is subjected to contextual semantic disambiguation to eliminate word ambiguity and generate a clear set of business concepts; Knowledge graph embedding technology is used to map each concept in the explicit set of business concepts to a continuous vector space; The multiple concept vectors obtained from the mapping are weighted and synthesized to finally generate a dense vector representing the overall service intent, which serves as the service intent feature.
[0007] Furthermore, the execution pattern of the resource demand time series is extracted to obtain the resource demand evolution trajectory, including: The resource demand time series is subjected to outlier detection and smoothing to form a smoothed resource demand series; The smoothed resource demand sequence is decomposed into long-term trend components, periodic components, and random fluctuation components. Curve fitting is performed on the long-term trend component, periodic component, and random fluctuation component respectively to obtain the corresponding trend curve equation, periodic function model, and fluctuation envelope. The trend curve equation, periodic function model, and fluctuation envelope are recombined to construct a parameterized curve that describes the evolution of resource demand in three dimensions: trend, period, and randomness. This parameterized curve serves as the trajectory of the resource demand evolution.
[0008] Furthermore, the service intent features, resource demand evolution trajectory, and event correlation network are input into the demand analysis model for feature fusion to generate a comprehensive demand feature map, including: The requirements analysis model includes a feature encoding layer, an interaction fusion layer, and a graph construction layer; The feature encoding layer encodes the input service intent features into intent nodes, the resource demand evolution trajectory into time sequence nodes, and the event association network into a set of relationship edges. In the interactive fusion layer, an information transmission path is established between intent nodes, time sequence nodes and relation edges, so that the information of the intent node is transmitted to the relevant time sequence node along the relation edge, and the information of the time sequence node also updates the state of the intent node in reverse. After multiple rounds of information transmission and state updates, the state of the nodes in the interactive fusion layer reaches stability. The graph construction layer uses stable intent nodes and temporal nodes as vertices of the feature graph and the updated set of relational edges as edges of the feature graph to construct a weighted directed graph, which is the comprehensive requirement feature graph.
[0009] Furthermore, encoding the event association network into a set of relation edges includes: Analyze the event association network to identify all event nodes in the network and the event association edges connecting the event nodes; For each event-related edge, a correlation strength weight is calculated. The correlation strength weight is calculated based on the temporal proximity of the two event nodes, the overlap in resource access, and the relevance in service topics. All event-related edges carrying association strength weights are aggregated to form the relationship edge set, where each edge records the starting event node, the ending event node, and the association strength weight.
[0010] Furthermore, the improved hierarchical clustering algorithm performs clustering analysis based on feature similarity and evolutionary trends. The working process of the improved hierarchical clustering algorithm includes: The comprehensive demand feature map of each service request is used as the initial cluster node; Calculate the distance between any two cluster nodes, which is a weighted composite of static feature distance and dynamic trend distance, wherein the static feature distance is calculated based on the cosine similarity of service intent features, and the dynamic trend distance is calculated based on the morphological difference of the resource demand evolution trajectory. Based on the calculated distance matrix, the two cluster nodes with the smallest distance are iteratively merged to form a new cluster node; After each merge, the service intent features and resource demand evolution trajectories of the new cluster nodes are recalculated. The service intent features are obtained by calculating the centroids of the sub-cluster features, and the resource demand evolution trajectories are obtained by weighted averaging of the trajectories of the sub-clusters. The process of calculating distances and merging is repeated until a preset clustering termination condition is met. The clustering termination condition is that the number of clusters reaches a preset threshold or the minimum inter-cluster distance exceeds a preset threshold, and finally the grouping result is formed.
[0011] Furthermore, the dynamic trend distance calculation based on the morphological difference of the resource demand evolution trajectory includes: The two resource demand evolution trajectories to be compared are aligned on time axis. Extract the local extreme point sequence and global trend slope of each aligned trajectory separately; Compare the differences in location and amplitude of the local extreme point sequences of the two trajectories, and calculate the extreme point matching difference degree; Compare the global trend slopes of the two trajectories in terms of the direction and intensity of change, and calculate the trend consistency difference. The extreme point matching difference degree and the trend consistency difference degree are linearly combined to generate the final shape difference degree, which is used as the value of the dynamic trend distance.
[0012] Furthermore, the step of calculating the aggregated resource requirement specification for each service request cluster using a resource requirement derivation model includes: Extract the resource demand evolution trajectory of all service requests belonging to the service request cluster from the comprehensive demand feature map; The evolution trajectories of all extracted resource demands are superimposed at the same point in time to generate a baseline for the total cluster demand. Analyze the changes in the total demand baseline of the cluster over time to identify peak, trough, and stable demand periods; For each time period, calculate the peak demand, average demand, and demand fluctuation range of the cluster for computing resources, memory resources, and storage resources within the corresponding time period; The peak demand during peak periods, the average demand during trough periods, and the average demand during stable periods, along with their fluctuation ranges, are integrated to form a multi-dimensional resource demand profile for the corresponding service request cluster under different time periods, which serves as the aggregated resource demand specification for the corresponding cluster.
[0013] The process of overlaying all extracted resource demand evolution trajectories at the same point in time to generate a baseline for total cluster demand includes: Establish a unified time coordinate system and map the evolution trajectory of all resource demands onto the unified time coordinate system; On the unified time coordinate system, a set of discrete time sampling points are defined; At each time sampling point, the values of all resource demand evolution trajectories mapped to the current point are summed to obtain the total cluster demand value for the corresponding sampling point; Connect the total cluster demand corresponding to all time sampling points to form a continuous total demand curve, and use the total demand curve as the baseline of the total cluster demand.
[0014] Furthermore, based on the aforementioned clustering requirement list, an optimized resource pre-allocation scheme is generated for the resource scheduling center of the intelligent cloud service platform, including: Obtain the current status of the idle resource pool and the load status of each physical server in the resource scheduling center of the smart cloud service platform; The aggregated resource requirements in the clustered requirements list are matched with the status of the idle resource pool to assess the feasibility of direct allocation. When direct allocation is not feasible, calculate the resources that can be released after migrating or adjusting the existing low-priority tasks, based on the load status of each physical server and the aggregated resource requirements. By combining directly allocable resources, releasable resources, and the time characteristics of demand, a phased and batch-executed resource allocation schedule and physical deployment mapping table are generated, which together constitute the optimized resource pre-allocation scheme.
[0015] Furthermore, the present invention also includes a big data demand analysis system for a smart cloud service platform, the system including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, it implements the steps of the big data demand analysis method for a smart cloud service platform as described above.
[0016] Compared with the prior art, the beneficial effects of the present invention are: The service intent features are obtained through semantic parsing of service topic summaries. Pattern extraction is performed on the time-series of resource demands to form a resource demand evolution trajectory. Correlation analysis is conducted on task-related event records to construct an event association network. The service intent features, resource demand evolution trajectory, and event association network are input into the demand analysis model to achieve multi-dimensional feature fusion and generate a comprehensive demand feature map. Multi-dimensional feature fusion integrates three heterogeneous information types: semantic intent, time-series changes, and event associations. This avoids the limitations of single-feature representation, fully covers the multi-dimensional attributes of service requests, refines the granularity of demand feature representation, strengthens the intrinsic relationships between different features, accurately restores the constituent elements and dynamic change logic of demands, and improves the completeness and accuracy of demand feature representation.
[0017] The improved hierarchical clustering algorithm simultaneously adopts feature similarity and resource demand evolution trends as clustering criteria to achieve automatic grouping of service requests. The clustering process considers both static feature matching and dynamic temporal changes, optimizing the clustering criteria, improving the consistency of service request features within the same cluster, and reducing feature overlap between clusters. Based on the accurate grouping results, a resource demand derivation model can stably calculate the aggregated resource demand specifications for corresponding clusters, forming a clustered demand list that fits the platform's actual operating status. This reduces the discrepancy between pre-allocated resources and actual needs, optimizes the adaptation accuracy of the resource pre-allocation scheme, and ensures a high degree of alignment between resource scheduling configuration and service request requirements. Attached Figure Description
[0018] Figure 1 This is a flowchart illustrating the steps of a big data demand analysis method for a smart cloud service platform as described in this invention. Figure 2 A flowchart illustrating the process of performing semantic parsing on a service topic summary to obtain service intent features; Figure 3 This is a flowchart for extracting the resource demand evolution trajectory from the execution pattern of the resource demand time series. Detailed Implementation
[0019] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] See Figure 1 This invention provides a big data demand analysis method for a smart cloud service platform, the specific method including: Raw service request data, including task description text, resource request history, and execution time logs, is collected in real-time or periodically from the intelligent cloud service platform. A data cleaning rule base is used to perform field completion, noise filtering, and format standardization on the raw service request data, generating a standardized set of service request records. Each record in this set includes a service topic summary, a resource requirement time series, and task-related event records. Semantic parsing is performed on the service topic summary to transform it into machine-understandable service intent features. Pattern extraction is performed on the resource requirement time series to identify its inherent patterns and generate a resource requirement evolution trajectory. Correlation analysis is performed on the task-related event records to construct an event association network reflecting the dependencies between events. The service intent features, resource requirement evolution trajectory, and event association network are input into the requirement analysis model. Feature fusion is achieved through the model's internal feature encoding, interaction, and reconstruction mechanisms, outputting a comprehensive requirement feature map. Based on the comprehensive requirement feature map, an improved hierarchical clustering algorithm is used to automatically group all service requests. This algorithm comprehensively considers static attribute similarity and dynamic behavioral evolution trends when calculating sample spacing, ultimately forming several highly cohesive service request clusters. For each service request cluster, the resource demand derivation model is invoked to aggregate and calculate the historical resource usage of its members, resulting in an aggregated resource demand specification representing the needs of the entire group, which is then compiled into a clustered demand list. The system reads the clustered demand list and, combined with the platform's current resource availability and server load status, formulates a resource allocation plan that balances efficiency and cost through simulation scheduling, thus generating an optimized resource pre-allocation scheme for the intelligent cloud service platform's resource scheduling center.
[0021] In one embodiment of the invention. See also Figure 2 The standardized service topic summary is input into a business intent understanding model pre-trained on a large-scale industry corpus. This model, based on a deep neural network architecture, can extract a preliminary set of intent keywords from natural language text. Contextual semantic disambiguation is performed on these preliminary keywords, using dictionaries and domain ontology to eliminate polysemy and determine the precise meaning of each word in the current context, thus generating an unambiguous set of explicit business concepts. By constructing a complete cloud service knowledge graph and its embedding technology, each independent concept in the explicit business concept set is mapped to a coordinate point in a high-dimensional vector space, obtaining the corresponding concept vector representation. Based on the concept's hierarchical depth in the knowledge graph and its frequency of appearance in the service topic summary, different importance weights are assigned to each concept vector. These scattered concept vectors are then merged into a unified, dense real-valued vector through a weighted summation operation. This vector fully embodies the core objectives and functional orientation of the service, serving as the final service intent feature output.
[0022] In practical implementation, the raw service request data collected from the smart cloud service platform, after standardization, includes a service topic summary field containing natural language text describing the task objectives and service types. For example, a service topic summary might read, "Financial risk control model training task, requiring GPU-accelerated computation and access to the customer transaction database," summarizing the background and purpose of a data analysis task. This service topic summary is then input into a pre-trained business intent understanding model. This model, based on a deep learning architecture and fine-tuned on large-scale general text and cloud computing corpora, can identify entities and action phrases in the text. After processing the example text, the model outputs a preliminary set of intent keywords containing "finance," "risk control," "model training," "GPU," and "database access," which constitute a preliminary abstraction of the service objective.
[0023] In some embodiments, a contextual semantic disambiguation operation is performed on the initial intent keyword set. The disambiguation process relies on a pre-built domain ontology and a thesaurus. Taking the word "model" in the initial intent keyword set as an example, it may refer to a mathematical model or a software architecture model without context. However, in the complete context of the service topic summary, "model training" and "financial risk control" are strongly linked. Therefore, by querying the domain ontology, it is confirmed that "model" here specifically refers to a machine learning model, rather than a concept with the same name in other fields. After traversal processing, the ambiguous words in the initial intent keyword set are replaced one by one with standardized concepts with unique definitions, forming a clear set of business concepts. The elements in the set become unambiguous terms such as "financial risk control," "ML model training," "GPU computing," and "relational database access."
[0024] In practical implementation, knowledge graph embedding technology is used to map a set of explicit business concepts to a vector space. Each business concept in the cloud service knowledge graph is represented as a high-dimensional vector. For example, the concept of "ML model training" may be located in the region close to the concepts of "data processing" and "high-performance computing" in the vector space, while the concept of "relational database access" is close to the region of "data persistence" and "transaction processing". The mapping process converts each string concept in the set of explicit business concepts into a corresponding real number vector, generating a set of concept vectors with the same dimension.
[0025] In some embodiments, concept vectors are weighted and synthesized to generate service intent features. The weight of each concept vector is determined based on its frequency of occurrence in the service topic summary and its hierarchical depth in the knowledge graph. For example, the concept "financial risk control" appears frequently and is located in a deep classification node of the knowledge graph, so it is given a higher weight, while the auxiliary concept "temporary storage" has a lower weight. The weighted synthesis is achieved by multiplying each concept vector by its normalized weight and then adding them together, ultimately outputting a dense vector that comprehensively represents the overall service intent.
[0026] In practical implementation, the calculation formula for weighted synthesis is as follows: in: This represents the final generated service intent feature vector. This represents the total number of concepts in a clearly defined set of business concepts. Indicates the first The concept vector corresponding to each business concept Indicates the first The normalized weight coefficients of each business concept are calculated by combining the word frequency of the concept in the service topic summary with the depth information in the knowledge graph, and are processed by the Softmax function to ensure that the sum of all weights is 1.
[0027] In one embodiment of the present invention, see [reference] Figure 3The resource demand time series is analyzed using a sliding window and statistical testing methods to detect outliers and perform interpolation corrections. A filtering algorithm is then used to smooth short-term fluctuations, resulting in a smoothed resource demand series. Signal decomposition techniques are employed to decompose this smoothed series into three independent components: a long-term trend component reflecting macroeconomic increases and decreases, a periodic component embodying fixed cyclical patterns, and a random fluctuation component capturing occasional changes. Mathematical models are fitted to each of these components. The least squares method is used to fit the long-term trend component to obtain the trend curve equation. Fourier transform is used to fit the periodic component to construct a periodic function model, and the fluctuation range of the random fluctuation component is estimated to generate a fluctuation envelope. The obtained trend curve equation, periodic function model, and fluctuation envelope are mathematically superimposed and combined to construct a feature curve containing multiple parameters. This parameterized curve can characterize the evolution path of resource demand over time at different granularities, marking it as the resource demand evolution trajectory.
[0028] In practical implementation, the resource demand time series is a set of CPU utilization or memory usage data points extracted from a standardized service request record set and sorted by timestamp; for example, a 24-hour resource demand time series contains hourly sampled CPU utilization percentage values [45,48,52,50,80,85,88,86,55,53,49,47,46,44,42,41,40,38,36,35,33,32,30,29], of which 80%, 85% and 88% are identified as potential outliers because they deviate significantly from adjacent values. Outlier detection based on Z-score is performed on the resource demand time series. Values with an absolute Z-score greater than 3 are identified as outliers and replaced with the linear interpolation result of the two preceding and following normal values. Then, a moving average filtering algorithm with a window size of 5 is used to smooth short-term fluctuations, forming a smoothed resource demand series [46.0,47.2,51.6,54.8,58.4,62.0,65.6,68.8,59.2,56.8,52.4,48.8,45.6,43.2,41.6,39.2,37.6,36.0,34.4,33.6,32.0,31.2,30.4,29.6].
[0029] In some embodiments, the smoothed resource demand sequence is decomposed using a seasonal trend decomposition method at multiple scales to separate a long-term trend component, a periodic component, and a random fluctuation component. The long-term trend component reflects the overall growth or decline direction of resource demand, the periodic component captures recurring daily or weekly fluctuations, and the random fluctuation component includes residual changes not included in the former two. During the decomposition process, the long-term trend component is obtained through multinomial regression fitting, the periodic component is modeled using Fourier series, and the random fluctuation component is obtained through residual calculation. The three components satisfy an additive relationship numerically.
[0030] In practice, curve fitting is performed on the long-term trend component, the periodic component, and the random fluctuation component respectively. A second-order polynomial function is used to fit the long-term trend component to obtain the trend curve equation, which describes the linear or nonlinear growth law of resource demand over time. A periodic function model is constructed by fitting the periodic component with a combination of trigonometric functions containing the fundamental frequency and harmonics. This model quantifies the amplitude and phase of the repeated fluctuations in demand. Gaussian process regression is used to fit the random fluctuation component to generate the fluctuation envelope, which defines the upper and lower boundaries of the random fluctuation.
[0031] In some embodiments, the trend curve equation, periodic function model, and fluctuation envelope are recombined to construct a parametric curve, the mathematical expression of which is: in: Indicates at a point in time Resource demand forecasts at the location, The trend curve equation represents the time point The output value, This indicates the periodic function model at time points. The output value, This indicates the fluctuation envelope at time point The parameterized curve encodes information from long-term trend components, periodic components, and random fluctuation components into a unified mathematical form, outputting the trajectory of resource demand evolution.
[0032] It is understandable that parametric curves can flexibly adjust the time granularity, displaying fine-grained demand fluctuations by hour or aggregated by day or week to show macro trends. Optionally, when constructing parametric curves, additive or multiplicative combinations can be selected according to the actual application scenario to adapt to the characteristics of different types of resource demand sequences.
[0033] In one embodiment of the invention, the feature encoding layer of the demand analysis model receives three types of input: service intent features are directly mapped to an intent node with rich node attributes; the resource demand evolution trajectory is converted into a set of vectors arranged by time steps and encapsulated as a temporal node; and the event association network is sent to the encoder for processing. The encoder parses the topology of the event association network, locates the event nodes in the network and the directed edges connecting these nodes, calculates the association strength weight of each edge based on the time interval of event occurrence, the overlap of shared resource IDs, and the angle between the word vectors of the service topics, and gathers all weighted edges into a set of relation edges. A message passing mechanism is built in the interaction fusion layer, allowing intent nodes to send feature information to their associated temporal nodes through relation edges, and also allowing temporal nodes to feed back their own state changes to upstream intent nodes. This bidirectional flow, after multiple iterations, makes the system energy tend to stabilize. The graph construction layer captures the steady-state feature values of all nodes in the network and the weight values of the edges. Using the intention node and the time sequence node as vertices and the updated set of relational edges as edges, a weighted directed graph representing the degree of association is constructed. This graph is the comprehensive demand feature graph.
[0034] In practical implementation, the feature encoding layer of the demand analysis model receives three types of input data: the service intent feature is a 128-dimensional dense vector; the resource demand evolution trajectory is a parameterized curve containing 24 time steps, each time step corresponding to a resource demand value; and the event association network is a directed graph structure composed of several event nodes and connecting edges. The feature encoding layer directly maps the service intent feature to intent nodes, and the attribute vector of the intent node is the service intent feature itself; it divides the resource demand evolution trajectory into 24 vectors according to time steps, concatenates them, and encapsulates them into time-series nodes, the attribute vector of the time-series node containing the shape parameters and numerical sequence of the trajectory; the event association network is fed into the encoder for parsing, identifying the event nodes in the network and the directed edges connecting the event nodes.
[0035] In some embodiments, the encoding process of the event association network is as follows: The event association network includes event nodes "Start Container Instance", "Load Dataset", "Perform Preprocessing", and "Invoke Inference Service", as well as directed edges connecting these event nodes; an association strength weight is calculated for each event association edge, based on three indicators: temporal proximity, resource access overlap, and service topic relevance. Temporal proximity is measured by the inverse of the time difference between the events, resource overlap is measured by the proportion of two events accessing the same resource ID, and topic relevance is measured by the cosine similarity of the word vectors of the service topic summaries corresponding to the two events; the three indicators are normalized and weighted to obtain the comprehensive association strength weight, and all weights are summed to form the set of relationship edges. Refer to Table 1 to show the weight calculation of the event association edges.
[0036] Table 1: Calculation Table of Event-Related Edge Weights In practical implementation, the interaction fusion layer establishes information transmission paths between intent nodes, temporal nodes, and the set of relational edges. Information transmission follows a message passing mechanism: intent nodes transmit feature information to associated temporal nodes through relational edges, and temporal nodes update their own states upon receiving the information. Simultaneously, temporal nodes transmit their updated states back to intent nodes, which then integrate feedback information from multiple temporal nodes to adjust their feature representations. The information transmission process occurs in multiple iterations, with node states gradually converging to a stable state. In some embodiments, the information update formula for the interaction fusion layer is: in: Represents a node In the The state vector during each iteration. Represents a node In the The state vector during each iteration. This represents the state vector of node u in the k-th iteration. Represents the ReLU activation function. The weight matrix represents the state of a node itself. Represents the set of relation edges. Indicates from node Pointing to node The directed edge, Representing an edge The correlation strength weight, This represents a weight matrix indicating the information of neighboring nodes. The formula ensures that node state updates retain both their own historical state and the weighted information from neighboring nodes.
[0037] It is understandable that multiple rounds of information transmission allow node states to gradually absorb global relational information, reaching a stable state where they no longer change significantly. The graph construction layer uses stable intent nodes and temporal nodes as vertices of the feature graph, and the updated set of relational edges as edges, constructing a weighted directed graph. The weight carried by each edge in the graph reflects the strength of the association between nodes, and the vertex attributes store the fused feature information. The generated weighted directed graph is the comprehensive requirement feature graph. Optionally, when constructing the weighted directed graph, a threshold filter can be set for the edge weights, retaining only strongly correlated edges with weights higher than a specified value to simplify the graph structure.
[0038] In one embodiment of the present invention, the improved hierarchical clustering algorithm treats the comprehensive demand feature map of each service request as an independent cluster node during initialization. During the distance calculation phase, for any two cluster nodes, not only is the cosine distance between their service intent feature vectors calculated as the static feature distance, but the morphological differences in their resource demand evolution trajectories are also specifically evaluated. The calculation of trajectory morphological differences requires first aligning the two trajectories along the time dimension, then extracting their respective local maxima and minima sequences and the slope changes of the overall trend; the extreme point matching difference is calculated by comparing the distribution offset and magnitude difference of the extreme points on the time axis, and the trend consistency difference is calculated by comparing the direction and magnitude of the overall slope. The two are then linearly weighted to obtain the dynamic trend distance. The static feature distance and the dynamic trend distance are weighted to synthesize the final distance metric. Based on this distance matrix, the algorithm continuously merges the two closest cluster nodes, and after each merge, recalculates the centroid features and weighted average trajectory of the new cluster. This process is repeated until the number of clusters reaches a set upper limit or the minimum inter-cluster distance exceeds a threshold, thereby completing the grouping of service requests.
[0039] In its implementation, the improved hierarchical clustering algorithm initializes each service request's comprehensive demand feature map as an independent cluster node. For example, if there are three service requests, they are denoted as cluster node A, cluster node B, and cluster node C. Each cluster node contains two parts: a service intent feature vector and a resource demand evolution trajectory. When calculating the distance between cluster nodes, the distance metric is a weighted composite of static feature distance and dynamic trend distance. The static feature distance is calculated based on the cosine similarity of the service intent feature vectors of two cluster nodes, while the dynamic trend distance is calculated based on the morphological differences of the resource demand evolution trajectories of two cluster nodes. The static feature distance is calculated by normalizing the service intent feature vectors, calculating the cosine similarity, and then converting the similarity into a distance value. The dynamic trend distance requires aligning the two resource demand evolution trajectories along their time axes and extracting the local extreme point sequence and the global trend slope for comparison. Refer to Table 2, which shows the static feature distance, dynamic trend distance, and final composite distance between each pair of the three cluster nodes. Table 2: Static feature distance, dynamic trend distance, and final composite distance between each pair of three cluster nodes. In practical implementation, the specific calculation process of dynamic trend distance is as follows: For two resource demand evolution trajectories to be compared, their sampling time intervals are first aligned to ensure consistent time axis lengths and overlapping starting points; then, the aligned trajectory data points are traversed to identify all local maxima and minima, forming a sequence of local extrema; simultaneously, the global trend slope is obtained by linearly fitting the entire trajectory. The local extrema matching difference is calculated by comparing the positional deviation and amplitude difference of the extrema points of the two trajectories on the time axis, and the trend consistency difference is calculated by comparing the sign and magnitude difference of the global trend slopes of the two trajectories; the weighted combination of the two difference values yields the morphological difference, which serves as the value of the dynamic trend distance. The formula for calculating the dynamic trend distance is: in: This represents the dynamic trend distance between cluster node X and cluster node Y. It is a balancing factor that adjusts the relative importance of differences between local extreme points and differences in global slope. Indicates the degree of difference in matching local extrema. This indicates the degree of difference in the consistency of trends.
[0040] In some embodiments, the improved hierarchical clustering algorithm iteratively merges cluster nodes with the smallest distance based on the calculated distance matrix; for example, cluster node A and cluster node B in the table above have the smallest composite distance, so they are preferentially merged into a new cluster node AB. After each merge, the feature representation of the new cluster node is recalculated: the service intent feature is obtained by calculating the centroid of all service intent feature vectors of the sub-cluster, and the resource demand evolution trajectory is obtained by weighted averaging of the parameters of all trajectories of the sub-cluster; the weighting weights can be adjusted according to the confidence level of the trajectory or the data quality. The merging process continues until the number of clusters reaches a preset upper limit threshold or the minimum inter-cluster distance exceeds a preset separation threshold, ultimately forming the grouping result of service requests.
[0041] In one embodiment of the present invention, all resource demand evolution trajectories belonging to the target service request cluster are selected from the comprehensive demand feature map. A unified time coordinate system covering the time range of all trajectories is established, and the value of each trajectory is mapped to this coordinate system. A series of discrete time sampling points are uniformly selected on this coordinate system. At each sampling point, the trajectory values that have passed through that point are summed to obtain the total cluster demand value at that moment. The total value connecting all sampling points generates the total cluster demand baseline. The fluctuation pattern of this baseline is analyzed to divide the peak period of high demand, the trough period of low demand, and the intermediate period of stable demand. The usage of computing resources, memory resources, and storage resources are statistically analyzed in each period, and the peak readings, arithmetic mean, and standard deviation representing the fluctuation range are recorded. These data, organized by resource and time dimensions, are packaged into the aggregated resource demand specification for the cluster. The resource scheduling module obtains this specification list and compares it with the current available resource slots and physical machine load information of the scheduling center. If there are insufficient idle resources, it plans to free up resources by migrating low-priority tasks. Based on this, a resource pre-allocation scheme is prepared, specifying the allocation timing, quantity, and target physical nodes.
[0042] In practical implementation, from a service request cluster containing five service requests, the resource demand evolution trajectory of all service requests belonging to the cluster is extracted from the comprehensive demand feature map. Each of the five resource demand evolution trajectories corresponds to five different task execution instances, and each trajectory describes the hourly memory usage change over a 24-hour period. A unified time coordinate system is established, with the time axis ranging from 0 to 23 hours and the time scale in hours. All resource demand evolution trajectories are mapped onto this unified time coordinate system, ensuring that the time points of each trajectory are aligned with the hourly scale of the coordinate system. On this unified time coordinate system, a set of discrete time sampling points is defined, with each sampling point representing an hour (0, 1, 2, ..., 23). At each sampling point, the memory usage values of all resource demand evolution trajectories mapped to that point are summed to obtain the total cluster memory demand for that sampling point. Connecting the total cluster memory demand values corresponding to all sampling points forms a continuous total demand curve. This total demand curve serves as the baseline for the total cluster demand, which shows a significant increase in memory demand between 10:00 and 14:00.
[0043] In some embodiments, the changes in the total cluster demand baseline over time are analyzed to identify peak, trough, and stable demand periods. Based on the total cluster demand baseline, periods where memory demand is consistently 20% higher than the overall average are classified as peak periods, periods where memory demand is consistently 20% lower than the overall average are classified as trough periods, and the remaining periods are classified as stable periods. In the example, the peak period is defined as 10:00 to 14:00, the trough period as 02:00 to 05:00, and the remaining time is considered stable. For each period, the peak, average, and fluctuation ranges of the cluster's demand for computing, memory, and storage resources are calculated, with the fluctuation range measured by standard deviation. The peak demand of the peak period, the average demand of the trough period, and the average demand and fluctuation range of the stable period are integrated to form a multi-dimensional resource demand profile of the corresponding service request cluster under different time periods, serving as the aggregated resource demand specification for the corresponding cluster.
[0044] In implementation, an optimized resource pre-allocation plan is generated for the smart cloud service platform resource scheduling center based on the clustered demand list. The current status of the idle resource pool and the load status of each physical server in the smart cloud service platform resource scheduling center are obtained. The idle resource pool shows 512GB of available memory, while the peak memory demand during peak periods in the clustered demand list is 600GB, indicating that direct allocation is not feasible. Based on the load status of each physical server and the aggregated resource demand specifications, the resources that can be released after migrating or adjusting existing low-priority tasks are calculated. Migrating low-priority tasks on three physical servers can release 160GB of memory. Combining directly allocable resources, releaseable resources, and the time characteristics of the demand, a phased and batch-executed resource allocation schedule and physical deployment mapping table are generated, together forming the optimized resource pre-allocation plan. The resource allocation schedule stipulates that 512GB of idle memory is allocated before 09:30, low-priority task migration is completed before 10:00 and an additional 160GB of memory is allocated to meet peak demand, and after 14:00, excess resources are gradually reclaimed for other tasks.
[0045] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.
Claims
1. A big data demand analysis method for a smart cloud service platform, characterized in that, The method includes: Collect raw service request data from the smart cloud service platform, including task description text, resource request history, and execution time logs; The original service request data is subjected to normalization and cleaning to generate a standardized service request record set, which includes a service topic summary, a resource requirement time sequence, and task-related event records. Semantic parsing is performed on the service topic summary to obtain service intent features; pattern extraction is performed on the resource demand time sequence to obtain the resource demand evolution trajectory; and correlation analysis is performed on the task-related event records to obtain the event correlation network. The service intent features, resource demand evolution trajectory, and event correlation network are input into the demand analysis model for feature fusion to generate a comprehensive demand feature map. Based on the comprehensive demand feature map, an improved hierarchical clustering algorithm is used to automatically group service requests, forming a grouping result containing multiple service request clusters. The improved hierarchical clustering algorithm performs clustering analysis based on feature similarity and evolution trend. For each service request cluster, the aggregated resource requirement specification of the corresponding cluster is calculated through the resource requirement derivation model to form a clustered requirement list; Based on the clustering requirements list, an optimized resource pre-allocation scheme is generated for the resource scheduling center of the smart cloud service platform.
2. The big data demand analysis method for a smart cloud service platform according to claim 1, characterized in that, Semantic parsing is performed on the service topic summary to obtain service intent features, including: The service topic summary is input into a pre-trained business intent understanding model to extract a preliminary set of intent keywords; The initial intent keyword set is subjected to contextual semantic disambiguation to eliminate word ambiguity and generate a clear set of business concepts; Knowledge graph embedding technology is used to map each concept in the explicit set of business concepts to a continuous vector space; The multiple concept vectors obtained from the mapping are weighted and synthesized to finally generate a dense vector representing the overall service intent, which serves as the service intent feature.
3. The big data demand analysis method for a smart cloud service platform according to claim 1, characterized in that, The resource demand time series is subjected to pattern extraction to obtain the resource demand evolution trajectory, including: The resource demand time series is subjected to outlier detection and smoothing to form a smoothed resource demand series; The smoothed resource demand sequence is decomposed into long-term trend components, periodic components, and random fluctuation components. Curve fitting is performed on the long-term trend component, periodic component, and random fluctuation component respectively to obtain the corresponding trend curve equation, periodic function model, and fluctuation envelope. The trend curve equation, periodic function model, and fluctuation envelope are recombined to construct a parameterized curve that describes the evolution of resource demand in three dimensions: trend, period, and randomness. This parameterized curve serves as the trajectory of the resource demand evolution.
4. The big data demand analysis method for a smart cloud service platform according to claim 1, characterized in that, The service intent features, resource demand evolution trajectory, and event correlation network are input into the demand analysis model for feature fusion to generate a comprehensive demand feature map, including: The requirements analysis model includes a feature encoding layer, an interaction fusion layer, and a graph construction layer; The feature encoding layer encodes the input service intent features into intent nodes, the resource demand evolution trajectory into time sequence nodes, and the event association network into a set of relationship edges. In the interactive fusion layer, an information transmission path is established between intent nodes, time sequence nodes and relation edges, so that the information of the intent node is transmitted to the relevant time sequence node along the relation edge, and the information of the time sequence node also updates the state of the intent node in reverse. After multiple rounds of information transmission and state updates, the state of the nodes in the interaction fusion layer reaches stability. The graph construction layer uses stable intent nodes and temporal nodes as vertices of the feature graph and the updated set of relational edges as edges of the feature graph to construct a weighted directed graph, which is the comprehensive requirement feature graph.
5. The big data demand analysis method for a smart cloud service platform according to claim 4, characterized in that, The process of encoding the event association network into a set of relation edges includes: Analyze the event association network to identify all event nodes in the network and the event association edges connecting the event nodes; For each event-related edge, a correlation strength weight is calculated. The correlation strength weight is calculated based on the temporal proximity of the two event nodes, the overlap in resource access, and the relevance in service topics. All event-related edges carrying association strength weights are aggregated to form the relationship edge set, where each edge records the starting event node, the ending event node, and the association strength weight.
6. The big data demand analysis method for a smart cloud service platform according to claim 1, characterized in that, The improved hierarchical clustering algorithm performs clustering analysis based on feature similarity and evolutionary trends. The working process of the improved hierarchical clustering algorithm includes: The comprehensive demand feature map of each service request is used as the initial cluster node; Calculate the distance between any two cluster nodes, which is a weighted composite of static feature distance and dynamic trend distance, wherein the static feature distance is calculated based on the cosine similarity of service intent features, and the dynamic trend distance is calculated based on the morphological difference of the resource demand evolution trajectory. Based on the calculated distance matrix, the two cluster nodes with the smallest distance are iteratively merged to form a new cluster node; After each merge, the service intent features and resource demand evolution trajectories of the new cluster nodes are recalculated. The service intent features are obtained by calculating the centroids of the sub-cluster features, and the resource demand evolution trajectories are obtained by weighted averaging of the trajectories of the sub-clusters. The process of calculating distances and merging is repeated until a preset clustering termination condition is met. The clustering termination condition is that the number of clusters reaches a preset threshold or the minimum inter-cluster distance exceeds a preset threshold, and finally the grouping result is formed.
7. The big data demand analysis method for a smart cloud service platform according to claim 6, characterized in that, The dynamic trend distance is calculated based on the morphological differences in the trajectory of resource demand evolution, including: The two resource demand evolution trajectories to be compared are aligned on time axis. Extract the local extreme point sequence and global trend slope of each aligned trajectory separately; Compare the differences in location and amplitude of the local extreme point sequences of the two trajectories, and calculate the extreme point matching difference degree; Compare the global trend slopes of the two trajectories in terms of the direction and intensity of change, and calculate the trend consistency difference. The extreme point matching difference degree and the trend consistency difference degree are linearly combined to generate the final shape difference degree, which is used as the value of the dynamic trend distance.
8. The big data demand analysis method for a smart cloud service platform according to claim 1, characterized in that, The step of calculating the aggregated resource requirement specification for each service request cluster using a resource requirement derivation model includes: Extract the resource demand evolution trajectory of all service requests belonging to the service request cluster from the comprehensive demand feature map; The evolution trajectories of all extracted resource demands are superimposed at the same point in time to generate a baseline for the total cluster demand. Analyze the changes in the total demand baseline of the cluster over time to identify peak, trough, and stable demand periods; For each time period, calculate the peak demand, average demand, and demand fluctuation range of the cluster for computing resources, memory resources, and storage resources within the corresponding time period; The peak demand during peak periods, the average demand during trough periods, and the average demand during stable periods, along with their fluctuation ranges, are integrated to form a multi-dimensional resource demand profile of the corresponding service request cluster under different time periods, which serves as the aggregated resource demand specification for the corresponding cluster. The process of overlaying all extracted resource demand evolution trajectories at the same point in time to generate a baseline for total cluster demand includes: Establish a unified time coordinate system and map the evolution trajectory of all resource demands onto the unified time coordinate system; On the unified time coordinate system, a set of discrete time sampling points are defined; At each time sampling point, the values of all resource demand evolution trajectories mapped to the current point are summed to obtain the total cluster demand value for the corresponding sampling point; Connect the total cluster demand corresponding to all time sampling points to form a continuous total demand curve, and use the total demand curve as the baseline of the total cluster demand.
9. A big data demand analysis method for a smart cloud service platform according to claim 1, characterized in that, Based on the clustering requirements list, an optimized resource pre-allocation scheme is generated for the resource scheduling center of the smart cloud service platform, including: Obtain the current status of the idle resource pool and the load status of each physical server in the resource scheduling center of the smart cloud service platform; The aggregated resource requirements in the clustered requirements list are matched with the status of the idle resource pool to assess the feasibility of direct allocation. When direct allocation is not feasible, calculate the resources that can be released after migrating or adjusting the existing low-priority tasks, based on the load status of each physical server and the aggregated resource requirements. By combining directly allocable resources, releasable resources, and the time characteristics of demand, a phased and batch-executed resource allocation schedule and physical deployment mapping table are generated, which together constitute the optimized resource pre-allocation scheme.
10. A big data demand analysis system serving a smart cloud service platform, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the big data demand analysis method for a smart cloud service platform as described in any one of claims 1 to 9.