Cloud platform monitoring method and device, computer device, readable storage medium and program product

Abstract

The application relates to a cloud platform monitoring method and device, computer equipment, a readable storage medium and a program product. The method comprises the following steps: performing data processing on historical data of a plurality of cloud platforms to obtain a first sample data set, wherein the first sample data set comprises a plurality of sample data of different data types; obtaining a positive sample pair and a negative sample pair according to sample data of a text type and sample data of a numerical value type in the first sample data set, training an encoder in a to-be-trained model according to the positive sample pair and the negative sample pair, and obtaining a target encoder; training a decoder in the to-be-trained model according to the sample data of the numerical value type and sample data of an alarm type in the first sample data set, and obtaining a target decoder; obtaining a target monitoring model according to the target encoder and the target decoder; and monitoring the plurality of cloud platforms based on the target monitoring model to obtain monitoring results of the cloud platforms. The method can realize the monitoring of the plurality of cloud platforms.

Description

Technical Field

[0001] This application relates to the field of cloud platform monitoring technology, and in particular to a cloud platform monitoring method, apparatus, computer equipment, readable storage medium, and program product. Background Technology

[0002] The use of multiple cloud platforms has become an important strategy for enterprises. Using multiple cloud service providers makes it easier to leverage the advantages of different cloud platforms, optimize costs, enhance business flexibility, and back up data and applications across multiple cloud platforms, thereby enhancing system redundancy and disaster recovery capabilities.

[0003] However, this multi-cloud environment also brings complex management challenges. Therefore, how to monitor multiple cloud platforms has become an urgent problem to be solved. Summary of the Invention

[0004] Therefore, it is necessary to provide a cloud platform monitoring method, device, computer equipment, readable storage medium, and program product that can monitor multiple cloud platforms, addressing the aforementioned technical problems.

[0005] Firstly, this application provides a cloud platform monitoring method, which includes: processing historical data from multiple cloud platforms to obtain a first sample dataset, the first sample dataset including sample data of multiple different data types; obtaining positive sample pairs and negative sample pairs based on text-type sample data and numerical-type sample data in the first sample dataset, and training the encoder in the training model based on the positive sample pairs and negative sample pairs to obtain a target encoder; training the decoder in the training model based on the numerical-type sample data and alarm-type sample data in the first sample dataset to obtain a target decoder; obtaining a target monitoring model based on the target encoder and target decoder; and monitoring multiple cloud platforms based on the target monitoring model to obtain monitoring results for each cloud platform.

[0006] In one embodiment, historical data includes target data, log data, and security data. Target data includes resource usage data and / or application performance data. Data processing is performed on historical data from multiple cloud platforms to obtain a first sample dataset, including: overlaying different types of target data corresponding to the same timestamp to obtain numerical sample data for each timestamp; vectorizing log data for each timestamp to obtain text sample data for each timestamp; vectorizing security data for each timestamp to obtain alarm sample data for each timestamp; and obtaining the first sample dataset based on the numerical sample data, text sample data, and alarm sample data for each timestamp.

[0007] In one embodiment, the decoder includes a first decoder, which trains the decoder in the model to be trained based on numerical sample data and alarm type sample data in the first sample dataset to obtain a target decoder. The training process includes: inputting numerical sample data into a target encoder to obtain a first output result from the target encoder, and inputting the first output result into the first decoder to obtain alarm data output by the first decoder; obtaining an alarm loss value based on the alarm data and alarm type sample data, training the first decoder based on the alarm loss value to obtain a first target decoder; and obtaining the target decoder based on the first target decoder.

[0008] In one embodiment, the decoder further includes a second decoder. The first sample dataset includes multiple timestamps corresponding to numerical sample data. Obtaining a target decoder based on the first target decoder includes: splitting the multiple numerical sample data into first sample data and second sample data; inputting the first sample data into a target encoder to obtain a second output result from the target encoder, and inputting the second output result into the second decoder to obtain target prediction data output by the second decoder. The target prediction data includes resource demand prediction data and / or application performance prediction data; obtaining a target prediction loss value based on the target prediction data and the second sample data; training the second decoder based on the target prediction loss value to obtain a second target decoder; and obtaining the target decoder based on the first target decoder and the second target decoder.

[0009] In one embodiment, obtaining a target monitoring model based on a target encoder and a target decoder includes: obtaining a monitoring model based on the target encoder, the target decoder, and an output layer; replacing the output layer in the monitoring model with a linear layer to obtain a target reward model; initializing the monitoring model and inputting the numerical sample data into the initialized monitoring model to obtain the output result of the initialized monitoring model; inputting the numerical sample data and the output result of the initialized monitoring model into the target reward model to obtain the reward value output by the target reward model; and adjusting the parameters of the initialized monitoring model based on the reward value output by the target reward model to obtain the target monitoring model.

[0010] In one embodiment, replacing the output layer of the monitoring model with a linear layer to obtain the target reward model includes: replacing the output layer of the monitoring model with a linear layer to obtain a reward model; inputting numerical sample data into the monitoring model K times to obtain 2K output results from the monitoring model, wherein the 2K output results include K alarm data and K target prediction data; obtaining a second sample dataset based on the numerical sample data input into the monitoring model and the 2K output results; and training the reward model based on the second sample dataset to obtain the target reward model.

[0011] Secondly, this application also provides a cloud platform monitoring device, which includes:

[0012] The processing module is used to process historical data from multiple cloud platforms to obtain a first sample dataset, which includes sample data of multiple different data types.

[0013] The first training module is used to obtain positive and negative sample pairs based on the text and numerical sample data in the first sample dataset, and to train the encoder in the model to be trained based on the positive and negative sample pairs to obtain the target encoder.

[0014] The second training module is used to train the decoder in the model to be trained based on the numerical sample data and the alarm type sample data in the first sample dataset, so as to obtain the target decoder.

[0015] The determination module is used to obtain the target monitoring model based on the target encoder and target decoder;

[0016] The monitoring module is used to monitor multiple cloud platforms based on the target monitoring model and obtain the monitoring results of each cloud platform.

[0017] In one embodiment, historical data includes target data, log data, and security data. Target data includes resource usage data and / or application performance data. The processing module is specifically used to overlay different types of target data corresponding to the same timestamp to obtain sample data of numerical type corresponding to each timestamp; to vectorize the log data of each timestamp to obtain sample data of text type corresponding to each timestamp; to vectorize the security data of each timestamp to obtain sample data of alarm type corresponding to each timestamp; and to obtain a first sample dataset based on the sample data of numerical type, text type, and alarm type corresponding to each timestamp.

[0018] In one embodiment, the decoder includes a first decoder and a second training module, specifically configured to input numerical sample data into the target encoder to obtain a first output result from the target encoder, and input the first output result into the first decoder to obtain alarm data output by the first decoder; obtain an alarm loss value based on the alarm data and sample data of alarm types; train the first decoder based on the alarm loss value to obtain a first target decoder; and obtain a target decoder based on the first target decoder.

[0019] In one embodiment, the decoder further includes a second decoder. The first sample dataset includes multiple timestamps corresponding to numerical sample data. The second training module is specifically used to split the multiple numerical sample data into first sample data and second sample data; input the first sample data into the target encoder to obtain the second output result of the target encoder, and input the second output result into the second decoder to obtain the target prediction data output by the second decoder. The target prediction data includes resource demand prediction data and / or application performance prediction data; obtain the target prediction loss value based on the target prediction data and the second sample data; train the second decoder based on the target prediction loss value to obtain the second target decoder; and obtain the target decoder based on the first target decoder and the second target decoder.

[0020] In one embodiment, the determining module is specifically configured to: obtain a monitoring model based on the target encoder, the target decoder, and the output layer; replace the output layer in the monitoring model with a linear layer to obtain a target reward model; initialize the monitoring model and input the numerical sample data into the initialized monitoring model to obtain the output result of the initialized monitoring model; input the numerical sample data and the output result of the initialized monitoring model into the target reward model to obtain the reward value output by the target reward model; and adjust the parameters of the initialized monitoring model based on the reward value output by the target reward model to obtain the target monitoring model.

[0021] In one embodiment, the determining module is specifically used to replace the output layer in the monitoring model with a linear layer to obtain a reward model; input numerical sample data into the monitoring model K times to obtain 2K output results from the monitoring model, including K alarm data and K target prediction data; based on the numerical sample data input into the monitoring model and the 2K output results, a second sample dataset is obtained; and the reward model is trained based on the second sample dataset to obtain a target reward model.

[0022] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any one of the first aspects above.

[0023] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0024] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0025] The aforementioned cloud platform monitoring method, device, computer equipment, readable storage medium, and program product process historical data from multiple cloud platforms to obtain sample data including multiple different data types. Then, based on text and numerical sample data in the first sample dataset, positive and negative sample pairs are obtained. These pairs are then used to train the encoder in the training model, resulting in a target encoder. Similarly, based on the numerical sample data and alarm sample data from the first sample dataset, the decoder is trained, resulting in a target decoder. Finally, a target monitoring model is obtained based on the target encoder and decoder. This target monitoring model is then used to monitor multiple cloud platforms, yielding monitoring results for each platform. This method, using a target monitoring model trained on historical data from multiple cloud platforms, not only achieves multi-cloud platform monitoring but also improves the management efficiency and reliability of multi-cloud environments compared to manual analysis of cloud platform data. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is an application environment diagram of a cloud platform monitoring method in one embodiment;

[0028] Figure 2 This is a flowchart illustrating a cloud platform monitoring method in one embodiment;

[0029] Figure 3 This is a schematic diagram of the encoder training process in one embodiment;

[0030] Figure 4 This is a schematic diagram of the decoder training process in one embodiment;

[0031] Figure 5 This is a schematic diagram illustrating the principle of the monitoring model reinforcement training process in one embodiment;

[0032] Figure 6 This is a structural block diagram of a cloud platform monitoring device in one embodiment;

[0033] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0035] Before introducing the cloud platform monitoring method provided in this application, some technical terms involved in this application will be explained:

[0036] PPO (Proximal Policy Optimization) is an optimization algorithm used in reinforcement learning. It is mainly used to train agents to make decisions in the environment with the goal of maximizing long-term rewards. Its core idea is to update the policy by optimizing the objective function, while controlling the update magnitude to ensure the stability of training.

[0037] RLHF (Reinforcement Learning with Human Feedback) is a machine learning approach that combines reinforcement learning with human feedback. This method allows the model to adjust its policy not only based on reward signals from the environment but also by leveraging human judgment. This is particularly useful in tasks such as natural language processing and complex decision-making systems, where human intuition and values ​​can help the model better understand the task and optimize its performance.

[0038] Transformer is a deep learning model architecture based on attention mechanisms. It uses self-attention to perform correlation computation on various positions in an input sequence. It is widely used in natural language processing and sequence tasks. Its core innovation is the introduction of self-attention, enabling the model to focus on information at different positions when processing the input sequence, thereby capturing contextual relationships. A Transformer typically consists of an encoder and a decoder. The encoder converts the input sequence into a feature representation, and the decoder uses this representation to generate the target output. Multi-head attention mechanisms allow the model to process multiple self-attention mechanisms in parallel, enhancing information extraction capabilities. To address the problem of missing positional information in sequence data, Transformer introduces positional encoding to ensure the model understands the order of elements in the sequence. Furthermore, layer normalization and residual connections make the training of deep networks more stable. Compared to traditional recurrent neural networks, Transformer has stronger parallel processing capabilities, enabling efficient training and handling of long-range dependencies. Due to these advantages, the Transformer architecture has achieved significant results in various natural language processing tasks and has also been extended to other fields such as image processing and time-series data prediction, becoming a significant milestone in the field of deep learning.

[0039] The use of multi-cloud platforms has become an important strategy for enterprises, as using multiple cloud service providers allows them to leverage the advantages of different cloud platforms, optimize costs, and enhance business flexibility. However, this multi-cloud environment also brings complex management challenges, especially in resource monitoring, security management, and performance optimization. Multi-cloud platforms refer to enterprises or organizations simultaneously using multiple different cloud service providers to deploy and manage their computing resources and services. This strategy is typically used to: reduce dependence on a single vendor by distributing resources across different cloud platforms; select the most cost-effective solution based on the pricing and service characteristics of different cloud service providers; and enhance system redundancy and disaster recovery capabilities by backing up data and applications across multiple cloud platforms.

[0040] Existing traditional monitoring technologies typically monitor resources on a single cloud platform. However, as enterprises advance their multi-cloud strategies, these traditional monitoring tools and methods face challenges in multi-cloud environments because they often lack cross-platform integration capabilities and struggle to uniformly manage data and metrics across different cloud platforms. Therefore, how to achieve monitoring across multiple cloud platforms has become an urgent problem to solve.

[0041] Therefore, it is necessary to propose effective technical means to solve the above problems. The technical solution of this application and how it solves the above technical problems will be described in detail below with specific embodiments.

[0042] The cloud platform monitoring method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 12 communicates with server 14 via a network. A data storage system can store the data that server 14 needs to process. The data storage system can be integrated onto server 14 or located in the cloud or on other network servers. Terminal 12 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 14 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0043] In one exemplary embodiment, such as Figure 2 As shown, a cloud platform monitoring method is provided, which can be applied to... Figure 1 Taking the server in the example, the explanation includes the following steps 101 to 105. Wherein:

[0044] Step 101: Process the historical data from multiple cloud platforms to obtain the first sample dataset, which includes sample data of multiple different data types.

[0045] Historical data includes target data, log data, and security data. Target data includes resource usage data and / or application performance data.

[0046] Resource usage data includes at least one of the following: CPU (Central Processing Unit) utilization, memory usage, storage usage, and network traffic. CPU utilization refers to the CPU usage of different instances or virtual machines; memory usage refers to the amount of memory used and the idle state of memory; storage usage refers to disk usage, including read and write operations and space usage; and network traffic refers to inbound network traffic, outbound network traffic, network latency data, and packet loss rate.

[0047] Application performance data includes at least one of response time, throughput, and error rate. Response time refers to the time between request and response in an application; throughput refers to the number of requests or transactions processed per second; and error rate refers to the rate of error and exception occurrence in an application or service.

[0048] Log data includes at least one of the following: transaction logs, operating system logs, application logs, and security logs. Transaction logs refer to detailed transaction and operation logs recorded by the application; operating system logs refer to system event and error logs generated by the operating system; application logs refer to runtime logs generated by the application, including debugging information and error reports; and security logs refer to logs recording security events, user access records, and abnormal activity.

[0049] Security data includes at least one of security incidents and intrusion detection data. Security incidents refer to records of attack attempts, unauthorized access, and other security threats; intrusion detection data includes alerts and reports from intrusion detection systems.

[0050] Historical data from multiple cloud platforms can be obtained periodically through APIs (Application Programming Interfaces) provided by different cloud service providers.

[0051] The target data includes resource usage data and application performance data, both of which are numerical data; log data is text data; and security data is alarm data. Data processing is performed on historical data from multiple cloud platforms, including at least one of the following: normalization, data smoothing, and encoding for numerical data; vectorization for text data; and vectorization for alarm data.

[0052] Step 102: Based on the text-type sample data and numerical-type sample data in the first sample dataset, obtain positive sample pairs and negative sample pairs, and train the encoder in the model to be trained based on the positive sample pairs and negative sample pairs to obtain the target encoder.

[0053] In the first sample dataset, multiple sample data of different data types are corresponding according to timestamps. Therefore, based on the text type sample data and the numeric type sample data in the first sample dataset, positive sample pairs and negative sample pairs are obtained, including: taking the text type sample data and the numeric type sample data corresponding to the same timestamp as positive sample pairs; and taking the text type sample data and the numeric type sample data corresponding to different timestamps as negative sample pairs.

[0054] It is understandable that historical data is obtained from multiple cloud platforms. Therefore, in the process of obtaining positive and negative sample pairs, the consistency of cloud platforms must also be considered. That is, the text type sample data and the numerical type sample data in the positive sample pair belong to the same cloud platform; the text type sample data and the numerical type sample data in the negative sample pair may or may not belong to the same cloud platform.

[0055] The encoder in the model to be trained includes a first encoder and a second encoder, both of which can be transformer encoders. The target encoder is trained based on positive and negative sample pairs, including: inputting numerical sample data from positive sample pairs into the first encoder to obtain first numerical features; inputting text sample data from positive sample pairs into the second encoder to obtain first text features; performing cosine similarity processing on the first numerical and first text features to obtain the cosine similarity of the positive sample pairs; inputting numerical sample data from negative sample pairs into the first encoder to obtain second numerical features; inputting text sample data from negative sample pairs into the second encoder to obtain second text features; performing cosine similarity processing on the second numerical and second text features to obtain the cosine similarity of the negative sample pairs; and training the first and second encoders based on the cosine similarity of the positive and negative sample pairs and constraints to obtain the target encoder. The constraints are that the cosine similarity of positive samples is greater than or equal to a first threshold, and the cosine similarity of negative samples is less than or equal to a second threshold; the first threshold is close to 1, and the second threshold is close to 0.

[0056] The first encoder and the second encoder are trained based on the cosine similarity of positive sample pairs, the cosine similarity of negative sample pairs, and constraints. This includes obtaining the cosine similarity loss value of positive sample pairs based on the cosine similarity of positive sample pairs and the cosine similarity loss function; obtaining the cosine similarity loss value of negative sample pairs based on the cosine similarity of negative sample pairs and the cosine similarity loss function; and training the first encoder and the second encoder based on the cosine similarity loss value of positive sample pairs, the cosine similarity loss value of negative sample pairs, and constraints to obtain the first target encoder and the second target encoder.

[0057] After obtaining the first target encoder and the second target encoder, the first target encoder can be used as the target encoder.

[0058] The cosine similarity loss function is as follows:

[0059]

[0060]

[0061] in, denoted by ; E represents the expectation; i represents the axis, i=0 represents the row axis, i=1 represents the column axis; L0 and L1 represent the losses on different axes, i.e., the loss values ​​of text features and numerical features, respectively; j represents the sample pair, j=0 represents the positive sample pair, j=1 represents the negative sample pair; n and w are the numerical features and text features after L2 normalization, respectively; Sim is the cosine similarity; both the first encoder and the second encoder include multiple projection layers, and D is the same projection layer of the first encoder and the second encoder.

[0062] To facilitate understanding of the encoder training process in the above-mentioned model to be trained, the following is provided: Figure 3 The diagram shown illustrates the encoder training process.

[0063] It is understandable that, given that the text part already has mature Transformer feature extraction parameters, in order to reduce the complexity of parameter updates, the model parameters in the second encoder can be kept fixed during training, or after several rounds of training, the model parameters of the two encoder parts can be stopped from being updated, and only the projection layer with fewer parameters can be updated to accelerate the training process.

[0064] Step 103: Train the decoder in the model to be trained based on the numerical sample data and the alarm type sample data in the first sample dataset to obtain the target decoder.

[0065] The decoder in the model to be trained may include a first decoder, and the corresponding target decoder includes a first target decoder. Alternatively, the decoder in the model to be trained may include a first decoder and a second decoder, and the corresponding target decoder may include a first target decoder and a second target decoder.

[0066] Training the first decoder can be done by taking numerical sample data as input and alarm sample data as labels, thus obtaining the first target decoder. It can be understood that since the first target decoder is trained based on alarm sample data, it can output alarm information.

[0067] Training the second decoder can be done by using a portion of the numerical sample data from the first sample dataset as input and another portion of the numerical sample data as labels, thus obtaining the second target decoder. It can be understood that if the second target decoder is trained based on resource usage data, it can output resource demand prediction information; if it is trained based on application performance data, it can output application performance prediction information.

[0068] Step 104: Obtain the target monitoring model based on the target encoder and target decoder.

[0069] Based on the target encoder, target decoder, and output layer, a monitoring model is obtained. After obtaining the monitoring model, it can be directly used as the target monitoring model, or the monitoring model can be further optimized to obtain the target monitoring model.

[0070] Step 105: Monitor multiple cloud platforms based on the target monitoring model to obtain the monitoring results for each cloud platform.

[0071] The monitoring results include at least one of the following: alarm information, resource demand prediction information, and application performance prediction information.

[0072] In one example, after obtaining the target monitoring model, resource usage data and / or application performance data are collected from each cloud platform, and the resource usage data and / or application performance data are input into the target monitoring model to obtain the monitoring results of the cloud platform output by the target monitoring model.

[0073] In the aforementioned cloud platform monitoring method, historical data from multiple cloud platforms is processed to obtain sample data of various data types. Then, based on text and numerical sample data from the first sample dataset, positive and negative sample pairs are obtained. These pairs are then used to train the encoder in the training model, resulting in the target encoder. Similarly, based on the numerical sample data and alarm sample data from the first sample dataset, the decoder is trained, resulting in the target decoder. Finally, the target monitoring model is derived from the target encoder and decoder. This target monitoring model is then used to monitor multiple cloud platforms, yielding monitoring results for each platform. This method, using a target monitoring model trained on historical data from multiple cloud platforms, not only achieves multi-cloud platform monitoring but also improves the management efficiency and reliability of multi-cloud environments compared to manual data analysis.

[0074] In an exemplary embodiment, historical data includes target data, log data, and security data. The target data includes resource usage data and / or application performance data. Data processing is performed on historical data from multiple cloud platforms to obtain a first sample dataset, including the following steps 1011 to 1014, wherein:

[0075] Step 1011: Overlay different types of target data corresponding to the same timestamp to obtain sample data of numerical type corresponding to each timestamp.

[0076] Optionally, multiple target data points in the historical data are arranged by timestamp, with the data range being the timestamp range of two adjacent timestamp log data points. Each target data point includes at least two of the following: CPU utilization, memory usage data, storage usage data, network traffic, response time, throughput, and error rate.

[0077] Normalization, data smoothing, and encoding are performed on each type of data in multiple target datasets.

[0078] Normalization is performed on the data of each type from multiple target datasets to obtain normalized data of each type. The normalization formula is as follows:

[0079]

[0080] Taking CPU utilization as an example, This represents the normalized CPU utilization rate at time point t. This represents the CPU utilization rate at time point t. This represents the average value; It represents the standard deviation.

[0081] Data smoothing is then applied to the normalized data of each type to obtain the smoothed data. The formula for data smoothing is as follows:

[0082]

[0083] Taking CPU utilization as an example, This represents the CPU utilization rate after smoothing the data at time point t. This represents the standard deviation, which is understandable. The standard deviation of the Gaussian function determines the degree of smoothing; N is the latest time point.

[0084] After data smoothing, one-hot encoding is performed on each type of smoothed data to obtain one-hot encoded data of each type. The one-hot encoded data of each type corresponding to the same timestamp are then superimposed to obtain sample data of the numerical type corresponding to each timestamp. The sample data of the numerical type corresponding to multiple timestamps constitute a time series vector, which can be denoted as... .

[0085] For example, the target data includes CPU utilization and throughput. The CPU utilization after one-hot encoding corresponding to timestamp t is 1,000,000, and the throughput after one-hot encoding corresponding to timestamp t is 0100,000. By superimposing the data of each type after one-hot encoding corresponding to timestamp t, the sample data corresponding to time t is 1,100,000.

[0086] Step 1012: Vectorize the log data for each timestamp to obtain sample data of text type corresponding to each timestamp.

[0087] The log data includes at least one log type, such as transaction logs, operating system logs, application logs, and security logs.

[0088] For log data of each log type, word segmentation and vectorization are performed on the log data at each timestamp of that log type to obtain sample text data corresponding to each timestamp of that log type. The sample text data corresponding to multiple timestamps of that log type can constitute a word vector sequence of the log data of that log type, which can be denoted as... Among these, word segmentation and vectorization can be performed using NLP (Natural Language Processing).

[0089] Step 1013: Vectorize the security data for each timestamp to obtain sample data of the alarm type corresponding to each timestamp.

[0090] Security data, similar to log data, is text-based. Therefore, the security data for each timestamp undergoes word segmentation and vectorization to obtain sample data for each alarm type. This alarm type sample data serves as labels for training the model. Thus, the sample data for alarm types corresponding to multiple timestamps constitute a sequence of word vectors for the labels, which can be denoted as... .

[0091] Step 1014: Obtain the first sample dataset based on the sample data of numerical type, text type, and alarm type corresponding to each timestamp.

[0092] Optionally, the sample data of numerical type, sample data of text type, and sample data of alarm type are mapped according to the timestamp to construct the first sample dataset.

[0093] In an exemplary embodiment, the decoder includes a first decoder, which trains the decoder in the model to be trained based on numerical sample data and alarm type sample data in the first sample dataset to obtain a target decoder, including steps 1031 to 1033:

[0094] Step 1031: Input the numerical sample data into the target encoder to obtain the first output result of the target encoder, and input the first output result into the first decoder to obtain the alarm data output by the first decoder.

[0095] It is understandable that when the target data includes resource usage data, the numerical sample data obtained after processing the target data is the resource usage sample data, and thus the first output result is the resource numerical feature.

[0096] When the target data includes application performance data, the numerical sample data obtained after processing the target data is the application performance sample data, and the first output result is the performance numerical feature.

[0097] When the target data includes application performance data and resource usage data, the numerical sample data obtained after processing the target data is the fused sample data with resource usage and application performance information, and the first output result is the fused numerical feature.

[0098] Step 1032: Based on the alarm data and sample data of alarm types, obtain the alarm loss value, and train the first decoder based on the alarm loss value to obtain the first target decoder;

[0099] In this context, the sample data for alarm types are equivalent to the true labels. Based on the alarm data and the sample data for alarm types, the alarm loss value is obtained, including: calculating the cross-entropy loss of the alarm data and the sample data for alarm types to obtain the alarm loss value.

[0100] The formula for calculating alarm loss value is as follows:

[0101]

[0102] in, E represents the alarm loss value; c represents the expected value; and w represents the alarm data and sample data for the alarm type.

[0103] It is understandable that during the training of the first decoder, the timestamps of the numerical sample data input to the target encoder each time are consistent with the timestamps of the alarm type sample data used as labels.

[0104] Step 1033: Obtain the target decoder based on the first target decoder.

[0105] After obtaining the first target decoder, it can be used directly as the target decoder; or the target decoder can be obtained based on the first target decoder and the second target decoder.

[0106] In one example, the decoder further includes a second decoder, and the first sample dataset includes sample data of numerical types corresponding to multiple timestamps; step 1033 includes steps 10331 to 10334, wherein:

[0107] Step 10331: Split the sample data of multiple numerical types into first sample data and second sample data.

[0108] Multiple numerical data samples within a preset time period are split into first sample data and second sample data; the timestamp corresponding to the first sample data is less than the timestamp corresponding to the second sample data.

[0109] It is understandable that when the target data includes resource usage data, the numerical sample data obtained after processing the target data is the resource usage sample data. Thus, the first sample data and the second sample data are the resource usage sample data corresponding to different timestamps.

[0110] When the target data includes application performance data, the numerical sample data obtained after processing the target data is the application performance sample data. Thus, the first sample data and the second sample data are the application performance sample data corresponding to different timestamps.

[0111] Step 10332: Input the first sample data into the target encoder to obtain the second output result of the target encoder, and input the second output result into the second decoder to obtain the target prediction data output by the second decoder. The target prediction data includes resource demand prediction data and / or application performance prediction data.

[0112] It's understandable that if the target data includes resource usage data, the second output is the resource numerical characteristic. If the target data includes application performance data, the second output is the performance numerical characteristic. If the target data includes both application performance data and resource usage data, the second output is the fused numerical characteristic.

[0113] Step 10333: Based on the target prediction data and the second sample data, obtain the target prediction loss value, and train the second decoder based on the target prediction loss value to obtain the second target decoder.

[0114] The second sample data is equivalent to the true label. Based on the target prediction data and the second sample data, the target prediction loss value is obtained, including: calculating the cross-entropy loss between the target prediction data and the second sample data to obtain the target prediction loss value.

[0115] The formula for calculating the target prediction loss value is as follows:

[0116]

[0117] in, represents the alarm loss value; E represents the expected value; c represents the target prediction data; and w represents the second sample data.

[0118] Step 10334: Obtain the target decoder based on the first target decoder and the second target decoder.

[0119] To facilitate understanding the training process of the decoder in the above-mentioned model to be trained, the following is provided: Figure 4 The diagram shown illustrates the principle of the decoder training process.

[0120] In this embodiment, since the first target decoder is trained based on sample data of alarm types, it can output alarm information. If the second target decoder is trained based on resource usage data, it can output resource demand prediction information; if it is trained based on application performance data, it can output application performance prediction information. Therefore, the target decoder, composed of the first and second target decoders, can both generate alarms and predict future resource demands and application performance based on the cloud platform's resource usage data and application performance data.

[0121] In an exemplary embodiment, a target monitoring model is obtained based on the target encoder and the target decoder, including steps 1041 to 1045, wherein:

[0122] Step 1041: Obtain the monitoring model based on the target encoder, target decoder, and output layer.

[0123] The output layer converts the outputs of the first target decoder and the second target decoder into vector representations of alarm data and target prediction data.

[0124] It is understandable that the architecture of the monitoring model consists of a target encoder, a target decoder, and an output layer.

[0125] Step 1042: Replace the output layer in the monitoring model with a linear layer to obtain the target reward model.

[0126] Optionally, the output layer in the monitoring model is replaced with a linear layer to obtain a reward model; numerical sample data is input into the monitoring model K times to obtain 2K output results of the monitoring model; based on the numerical sample data input into the monitoring model and the 2K output results, a second sample dataset is obtained; the reward model is trained based on the second sample dataset to obtain the target reward model.

[0127] The linear layer is an initialized linear layer used to generate a scalar output (output score), which represents the output quality; a higher score indicates higher quality.

[0128] In one example, the target decoder includes a first target decoder and a second target decoder. Therefore, inputting numerical sample data into the monitoring model K times yields 2K output results, including K alarm data and K target prediction data. It can be understood that, where the numerical sample data includes resource usage sample data, the target prediction data can be resource demand prediction data.

[0129] Based on the numerical sample data input to the monitoring model and 2K output results, the second sample dataset is formatted as follows: one numerical sample data corresponds to K different alarm data and K different target prediction data. The 2K output results are then sorted, meaning one numerical sample data corresponds to two output sorts.

[0130] Inputting a numerical sample data point and its corresponding alarm data into the reward model yields the alarm score. Similarly, inputting a sample data point of a different data type and its corresponding target prediction data into the reward model yields the target prediction score, such as a resource demand prediction score. It can be understood that for a numerical sample data point, the reward model needs to be input 2K times to obtain 2K output results with corresponding scores, ensuring that these scores satisfy the sorting order of the second sample dataset.

[0131] Specifically, the numerical sample data is represented as x, and the corresponding K alarm data and K target prediction data are represented as y={y1,…,y K}, where the rating order is y1>y2>…>y K The region with high scores is represented as y. h ={y1,…,y w}, represent the region with the lowest score as y d ={y w+1 ,…,y K K represents the number of outputs corresponding to each input. Since alarm data and target prediction data each have K outputs, both types of outputs will have... Second comparison. The reward value is obtained by inputting numerical sample data and high-scoring outputs (alarm data or target prediction data) into the reward model. The reward value is obtained by inputting numerical sample data and low-scoring outputs (alarm data or target prediction data) into the reward model. The goal of training the reward model is to maximize the difference between high and low scores. Therefore, the loss function of the reward model is calculated using the following formula during training:

[0132]

[0133] in, This represents the expected value of the cross-entropy calculated from the numerical sample data x selected from the second sample dataset, and the alarm data / target prediction data. The function represents the sigmoid function.

[0134] The trained target reward model can identify the quality of the output generated by the monitoring model. That is, based on the sample data of the input data type, the target reward model can judge the accuracy of the alarm data and target prediction data output by the monitoring model.

[0135] Step 1043: Initialize the monitoring model and input the numerical sample data into the initialized monitoring model to obtain the output results of the initialized monitoring model.

[0136] Step 1044: Input the numerical sample data of the initialized monitoring model and the output of the initialized monitoring model into the target reward model to obtain the reward value output by the target reward model;

[0137] Step 1045: Adjust the parameters of the initialized monitoring model based on the reward value output by the target reward model to obtain the target monitoring model.

[0138] Steps 1043 to 1045 above involve further enhancing the monitoring model based on the pre-trained target reward model to obtain the target monitoring model. The specific enhancement process is as follows:

[0139] The monitoring model is initialized to obtain the initialized monitoring model. Numerical sample data x is input into the initialized monitoring model to obtain the output results, which include alarm data y and target prediction data y. The numerical sample data x and alarm data y can then be input into the target reward model, or the numerical sample data x and target prediction data y can be input into the reward value output by the target reward model. Use reward value The parameters of the monitoring model are adjusted to obtain the target monitoring model.

[0140] Using reward values During the parameter tuning process of the initialized monitoring model, KL divergence is used to constrain the monitoring model, preventing it from deviating significantly from the original monitoring model in pursuit of higher scores. The specific expression is:

[0141]

[0142] in, This represents the initialized monitoring model, RLM. This represents the monitoring model MPMM; , For adjustment coefficients; This represents the numerical sample data of the initialized monitoring model RLM and the output results of the initialized monitoring model.

[0143] To facilitate understanding of the above-described reinforcement training process for the monitoring model, the following is provided: Figure 5 The diagram shown illustrates the principle of the reinforcement training process for the monitoring model. This reinforcement training process employs the PPO algorithm and RLHF.

[0144] In this embodiment, the monitoring model is reinforced based on the target reward model, resulting in a more reliable target monitoring model.

[0145] Based on the aforementioned cloud platform monitoring method, the resulting target monitoring model can not only output alarms when monitoring the cloud platform, but also predict future resource demands and application performance trends. This allows users to receive timely prompts to expand resources, reducing issues such as service endpoints, storage overflow, and traffic congestion. Furthermore, this application uses different types of data from multiple cloud platforms for training, covering most types of risky operations and attacks, enabling more accurate identification of risk causes and faster risk handling, significantly improving the reliability and stability of monitoring.

[0146] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0147] Based on the same inventive concept, this application also provides a cloud platform monitoring device for implementing the cloud platform monitoring method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more cloud platform monitoring device embodiments provided below can be found in the limitations of the cloud platform monitoring method described above, and will not be repeated here.

[0148] In one exemplary embodiment, such as Figure 6As shown, a cloud platform monitoring device 600 is provided, comprising: a processing module 601, a first training module 602, a second training module 603, a determination module 604, and a monitoring module 605, wherein:

[0149] The processing module 601 is used to process historical data from multiple cloud platforms to obtain a first sample dataset, which includes sample data of multiple different data types.

[0150] The first training module 602 is used to obtain positive sample pairs and negative sample pairs based on the text type sample data and numerical type sample data in the first sample dataset, and to train the encoder in the model to be trained based on the positive sample pairs and negative sample pairs to obtain the target encoder.

[0151] The second training module 603 is used to train the decoder in the model to be trained based on the numerical sample data and the alarm type sample data in the first sample dataset, so as to obtain the target decoder.

[0152] The determination module 604 is used to obtain the target monitoring model based on the target encoder and the target decoder;

[0153] The monitoring module 605 is used to monitor multiple cloud platforms based on the target monitoring model and obtain the monitoring results of each cloud platform.

[0154] In one embodiment, historical data includes target data, log data, and security data. Target data includes resource usage data and / or application performance data. Processing module 601 is specifically used to overlay different types of target data corresponding to the same timestamp to obtain sample data of numerical type corresponding to each timestamp; to vectorize the log data of each timestamp to obtain sample data of text type corresponding to each timestamp; to vectorize the security data of each timestamp to obtain sample data of alarm type corresponding to each timestamp; and to obtain a first sample dataset based on the sample data of numerical type, text type, and alarm type corresponding to each timestamp.

[0155] In one embodiment, the decoder includes a first decoder and a second training module 603, specifically configured to input numerical sample data into the target encoder to obtain a first output result from the target encoder, and input the first output result into the first decoder to obtain alarm data output by the first decoder; obtain an alarm loss value based on the alarm data and sample data of alarm types; train the first decoder based on the alarm loss value to obtain a first target decoder; and obtain a target decoder based on the first target decoder.

[0156] In one embodiment, the decoder further includes a second decoder. The first sample dataset includes multiple timestamps corresponding to numerical sample data. The second training module 603 is specifically used to split the multiple numerical sample data into first sample data and second sample data; input the first sample data into the target encoder to obtain the second output result of the target encoder, and input the second output result into the second decoder to obtain the target prediction data output by the second decoder. The target prediction data includes resource demand prediction data and / or application performance prediction data; obtain the target prediction loss value based on the target prediction data and the second sample data; train the second decoder based on the target prediction loss value to obtain the second target decoder; and obtain the target decoder based on the first target decoder and the second target decoder.

[0157] In one embodiment, the determining module is specifically configured to: obtain a monitoring model based on the target encoder, the target decoder, and the output layer; replace the output layer in the monitoring model with a linear layer to obtain a target reward model; initialize the monitoring model and input the numerical sample data into the initialized monitoring model to obtain the output result of the initialized monitoring model; input the numerical sample data and the output result of the initialized monitoring model into the target reward model to obtain the reward value output by the target reward model; and adjust the parameters of the initialized monitoring model based on the reward value output by the target reward model to obtain the target monitoring model.

[0158] In one embodiment, the determining module 604 is specifically used to replace the output layer in the monitoring model with a linear layer to obtain a reward model; input numerical sample data into the monitoring model K times to obtain 2K output results from the monitoring model, including K alarm data and K target prediction data; based on the numerical sample data input into the monitoring model and the 2K output results, a second sample dataset is obtained; and the reward model is trained based on the second sample dataset to obtain a target reward model.

[0159] Each module in the aforementioned cloud platform monitoring device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0160] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as a first sample dataset, a second sample dataset, and monitoring results. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a cloud platform monitoring method.

[0161] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0162] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any one of the above method embodiments.

[0163] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the above method embodiments.

[0164] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in any one of the above method embodiments.

[0165] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0166] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0167] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0168] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A cloud platform monitoring method, characterized by, The method comprises: data processing is performed on historical data of a plurality of cloud platforms to obtain a first sample data set, the first sample data set comprising a plurality of sample data of different data types; obtaining positive sample pairs and negative sample pairs according to sample data of a text type and sample data of a numerical value type in the first sample data set, and training an encoder in a to-be-trained model according to the positive sample pairs and the negative sample pairs to obtain a target encoder; the decoder in the to-be-trained model comprises a first decoder, the sample data of the numerical value type is input into the target encoder to obtain a first output result output by the target encoder, and the first output result is input into the first decoder to obtain alarm data output by the first decoder; an alarm loss value is obtained according to the alarm data and sample data of an alarm type in the first sample data set, the first decoder is trained based on the alarm loss value to obtain a first target decoder; and a target decoder is obtained according to the first target decoder; a target monitoring model is obtained according to the target encoder and the target decoder; and a plurality of cloud platforms are monitored based on the target monitoring model to obtain monitoring results of the cloud platforms.

2. The method of claim 1, wherein, The historical data comprises target data, log data and security data, the target data comprises resource usage data and / or application performance data; the data processing on the historical data of the plurality of cloud platforms to obtain the first sample data set comprises: superimposing different types of target data corresponding to a same timestamp to obtain sample data of the numerical value type corresponding to each timestamp; vectorizing log data of each timestamp to obtain sample data of the text type corresponding to each timestamp; vectorizing security data of each timestamp to obtain sample data of the alarm type corresponding to each timestamp; the first sample data set is obtained according to the sample data of the numerical value type, the sample data of the text type and the sample data of the alarm type corresponding to each timestamp.

3. The method of claim 1, wherein, The encoder in the to-be-trained model comprises a first encoder and a second encoder; the training of the encoder in the to-be-trained model according to the positive sample pairs and the negative sample pairs to obtain the target encoder comprises: inputting sample data of a numerical value type in the positive sample pairs into the first encoder to obtain first numerical value features; and inputting sample data of a text type in the positive sample pairs into the second encoder to obtain first text features; performing cosine similarity processing on the first numerical value features and the first text features to obtain a cosine similarity of the positive sample pairs; inputting sample data of a numerical value type in the negative sample pairs into the first encoder to obtain second numerical value features; and inputting sample data of a text type in the negative sample pairs into the second encoder to obtain second text features; performing cosine similarity processing on the second numerical value features and the second text features to obtain a cosine similarity of the negative sample pairs; and The first encoder and the second encoder are trained based on the cosine similarity of the positive sample pair, the cosine similarity of the negative sample pair, and a constraint condition, to obtain the target encoder.

4. The method of claim 1, wherein, The decoder in the to-be-trained model further includes a second decoder, the first sample data set includes sample data of a plurality of timestamp corresponding numerical types, and the target decoder is obtained according to the first target decoder, including: The plurality of sample data of the numerical type is split into first sample data and second sample data; The first sample data is input into the target encoder to obtain a second output result output by the target encoder, and the second output result is input into the second decoder to obtain target prediction data output by the second decoder, the target prediction data including resource demand prediction data and / or application performance prediction data; According to the target prediction data and the second sample data, a target prediction loss value is obtained, and the second decoder is trained based on the target prediction loss value to obtain a second target decoder; The target decoder is obtained according to the first target decoder and the second target decoder.

5. The method of claim 1, wherein, The target monitoring model is obtained according to the target encoder and the target decoder, including: The monitoring model is obtained according to the target encoder, the target decoder, and an output layer; The output layer in the monitoring model is replaced by a linear layer to obtain a target reward model; The monitoring model is initialized, and the sample data of the numerical type is input into the initialized monitoring model to obtain an output result output by the initialized monitoring model; The sample data of the numerical type input into the initialized monitoring model and the output result output by the initialized monitoring model are input into the target reward model to obtain a reward value output by the target reward model; The initialized monitoring model is parameter adjusted based on the reward value output by the target reward model to obtain the target monitoring model.

6. The method of claim 5, wherein, The output layer in the monitoring model is replaced by a linear layer to obtain a target reward model, including: The output layer in the monitoring model is replaced by a linear layer to obtain a reward model; The sample data of the numerical type is input into the monitoring model K times to obtain 2K output results output by the monitoring model, the 2K output results including K alarm data and K target prediction data; Based on the sample data of the numerical type input into the monitoring model and the 2K output results, a second sample data set is obtained; The reward model is trained based on the second sample data set to obtain the target reward model.

7. A cloud platform monitoring apparatus characterized by comprising: The device includes: The processing module is configured to perform data processing on historical data of a plurality of cloud platforms to obtain a first sample data set, the first sample data set including a plurality of sample data of different data types; The first training module is configured to obtain positive sample pairs and negative sample pairs according to text type sample data and numerical value type sample data in the first sample data set, and train an encoder in a to-be-trained model according to the positive sample pairs and the negative sample pairs to obtain a target encoder. The decoder in the to-be-trained model comprises a first decoder. The second training module is configured to input the numerical value type sample data into the target encoder to obtain a first output result output by the target encoder, input the first output result into the first decoder to obtain alarm data output by the first decoder, obtain an alarm loss value according to the alarm data and alarm type sample data in the first sample data set, train the first decoder based on the alarm loss value to obtain a first target decoder, and obtain a target decoder according to the first target decoder. The determining module is configured to obtain a target monitoring model according to the target encoder and the target decoder. The monitoring module is configured to monitor a plurality of cloud platforms based on the target monitoring model to obtain monitoring results of the cloud platforms.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

Images