A method, system, device, and medium for genealogy determination based on Bayesian optimization.

Abstract

This invention relates to a data lineage identification method, system, device, and medium based on Bayesian optimization. By using a Bayesian process to optimize the pre-trained BERT model in the lineage identification task, the number of evaluations required to find the optimal solution is minimized. Bayesian optimization automatically finds the optimal BERT embedding strategy to improve the log anomaly classification effect based on KNN, thereby achieving efficient identification of whether logs are recorded in the system's lineage.

Description

Technical Field

[0001] This invention belongs to the field of intelligent operation and maintenance data processing technology, and relates to a data lineage identification method, system, device and medium based on Bayesian optimization. Background Technology

[0002] Intelligent Operations and Maintenance (AIOps) uses AI to provide end-to-end data to operations and maintenance personnel, assisting them in efficiently monitoring, diagnosing, and debugging systems. This process requires collecting, aggregating, and analyzing continuous performance data streams generated by system applications, as well as the hardware and network running those applications. After data aggregation, integration, and correlation analysis, a holistic view of the applications, infrastructure, and network is provided to facilitate the detection and prediction of anomalies. Data hierarchy provides a data organization solution for intelligent operations and maintenance.

[0003] Traditional data lineage construction requires system tracking and data collection, or data acquisition via data interfaces. This approach relies on secondary development, which is difficult to implement in systems with incomplete interfaces, numerous functional modules, and high coupling. Automated data lineage construction based on log systems can avoid these problems. The first step in automated data lineage construction based on log systems is lineage identification. Since log information is often rich in content, massive in volume, and continuously grows with system operation, it is necessary to extract key information that needs to be included in the lineage, filtering out duplicate, auxiliary, and mostly normal operating log records, and only marking key information such as system error messages, login information, and key data flows. This process is lineage identification. However, current methods for determining whether logs should be included in the system lineage still suffer from low efficiency. Summary of the Invention

[0004] To address the problems existing in the above-mentioned traditional methods, this invention proposes a data lineage determination method based on Bayesian optimization, a data lineage determination system based on Bayesian optimization, a computer device, and a computer-readable storage medium, which can achieve efficient determination of whether logs have been recorded in the system's lineage.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0006] On the one hand, a data lineage determination method based on Bayesian optimization is provided, including the following steps:

[0007] Obtain the log data to be judged;

[0008] After loading the pre-trained BERT model and word segmenter, the log data to be judged is preprocessed to obtain the embedding vector corresponding to the log data to be judged; the BERT model is obtained by pre-training based on log data with recorded lineage labels and using Bayesian optimization.

[0009] A machine learning model trained on a lineage identification task is used to identify the lineage of the embedded vectors and determine the lineage labels of the log data to be identified.

[0010] In one embodiment, the pre-training process of the BERT model and the training process of the machine learning model include:

[0011] Input log data with lineage tags;

[0012] After loading the BERT model and word segmenter, define the hyperparameters and search space; the hyperparameters and search space include the extraction level of the BERT model, the pooling method, and the input truncation length;

[0013] Log data is divided into training and validation sets, and an objective function is constructed. The objective function is to extract BERT embeddings from training and validation logs for a given combination of parameters.

[0014] The machine learning model is trained and predicted based on the training set and the validation set, and the weighted F1 score of the validation set is used as the optimization metric.

[0015] The F1 score for lineage determination is optimized based on the acquisition function defined in Bayesian optimization; the acquisition function is used to measure the expected growth of the objective function relative to the current best observation;

[0016] Determine whether the difference between the current iteration's collection function value and the previous iteration's value is less than a set threshold; Bayesian optimization iteratively explores within the search space;

[0017] If so, output a set of optimal hyperparameters to obtain the trained BERT model and machine learning model; otherwise, enter the next loop to continue training the machine learning model until the optimal hyperparameters are output.

[0018] In one embodiment, when defining hyperparameters and search space, it also includes defining whether to use PCA dimensionality reduction and the dimension of PCA dimensionality reduction.

[0019] In one embodiment, the machine learning model is the K-nearest neighbor algorithm or a decision tree.

[0020] On the other hand, a data lineage determination system based on Bayesian optimization is also provided, including:

[0021] The log acquisition module is used to acquire log data to be judged;

[0022] The preprocessing module is used to preprocess the log data to be judged after loading the pre-trained BERT model and word segmenter to obtain the embedding vector corresponding to the log data to be judged; the BERT model is obtained by pre-training based on log data with recorded lineage labels and using Bayesian optimization.

[0023] The pedigree determination module is used to perform pedigree determination on the embedded vectors using a machine learning model trained on the pedigree determination task, and to determine the pedigree labels of the log data to be determined.

[0024] On another front, a computer device is also provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the aforementioned Bayesian-optimized data lineage determination method.

[0025] Furthermore, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the aforementioned Bayesian optimization-based genealogy determination method.

[0026] One of the above technical solutions has the following advantages and beneficial effects:

[0027] The aforementioned data lineage identification method, system, device, and medium based on Bayesian optimization use Bayesian processes to optimize the pre-trained BERT model in the lineage identification task, minimizing the number of evaluations required to find the optimal solution. Bayesian optimization automatically finds the optimal BERT embedding strategy to improve the log anomaly classification effect based on KNN, thus achieving efficient identification of whether logs are recorded in the system's lineage. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating a data lineage determination method based on Bayesian optimization in one embodiment;

[0030] Figure 2 This is a schematic diagram of a Bayesian optimization process based on BERT+KNN in one embodiment;

[0031] Figure 3 This is a schematic diagram of the module framework of a data lineage determination system based on Bayesian optimization in one embodiment. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0033] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.

[0034] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0035] In one embodiment, such as Figure 1 As shown, a data lineage determination method based on Bayesian optimization is provided, which may include the following steps S12 to S16:

[0036] S12, obtain the log data to be judged;

[0037] S14, after loading the pre-trained BERT model and word segmenter, preprocess the log data to be judged to obtain the embedding vector corresponding to the log data to be judged; the BERT model is obtained by pre-training based on log data with recorded lineage labels and using Bayesian optimization;

[0038] S16. The machine learning model trained by the lineage identification task is used to identify the lineage of the embedded vector and determine the lineage label of the log data to be identified.

[0039] It is understood that this embodiment will optimize lineage determination through a Bayesian process. In the lineage determination process, the pre-trained text model BERT (Bidirectional Encoder Representations from Transformers) can be used to preprocess the raw, unstructured log information. The pre-processed embedding vectors are then trained using machine learning models such as, but not limited to, KNN (K-Nearest Neighbors) or decision trees for the determination task. The BERT model used in this process is a pre-trained model with a large number of parameters, making training difficult. Therefore, a Bayesian process is used for model optimization.

[0040] definition For pedigree determination, the F1 score is assumed to be... The value is sampled from a Gaussian distribution, i.e. .in, To measure two different data points x and The kernel function for the distance between them, in the absence of sufficient prior knowledge, assumes the mean of the Gaussian distribution is 0, and estimates the distance based on continuous sampling of the Gaussian distribution. The values ​​of the parameters. In order to find... To find the optimal value of the objective function, in this Gaussian process, a collection function *a* is defined to measure the expected growth of the objective function relative to the current best observation. The collection function *a* is as follows:

[0041] .

[0042] in, For data, , This represents the variance of the data noise. Let Variance be the variance of the Gaussian process. The mean of the Gaussian process is given. ( ) is the cumulative distribution function (CDF) of the standard normal distribution. For the hyperparameters of the Gaussian process, The probability density function (PDF) of the standard normal distribution is... The value at that location, For the nth sample The corresponding lineage labels, where N is the number of data entries (total data). Potential for standardization improvement. It can be written as:

[0043] .

[0044] in, This represents the optimal value for x at the current time. During each sampling, a new x is obtained through the optimal sampling function, and the corresponding... The data is added to the dataset to update the Gaussian process model. The steps of updating the acquisition function, selecting new data points, and updating the Gaussian model are repeated continuously. With each iteration, the Gaussian model becomes increasingly accurate, and the search gradually approaches the optimum. The optimization process continues until a convergence threshold is met. This structured approach enables Bayesian optimization to effectively explore complex environments by intelligently balancing the exploration of unknown regions with the utilization of potential regions, minimizing the number of evaluations required to find the optimal solution.

[0045] Bayesian optimization typically uses a Gaussian process (GP) to model the "black box objective function": using observed samples... After learning, the posterior mean μ(x) (reflecting the predicted value of the objective function) and posterior variance σ(x) (reflecting the uncertainty of the prediction) are output for a new point x. The role of the sampling function is to "quantify the value of sampling at point x", that is, to consider both the potential for the predicted value to be better than the current best value and the exploration value of high uncertainty regions.

[0046] The aforementioned data lineage determination method based on Bayesian optimization uses a Bayesian process to optimize the pre-trained BERT model in the lineage determination task, minimizing the number of evaluations required to find the optimal solution. Bayesian optimization automatically finds the optimal BERT embedding strategy to improve the log anomaly classification effect based on KNN, thus achieving efficient determination of whether logs are recorded in the system's lineage.

[0047] In one embodiment, such as Figure 2 As shown, the pre-training of the BERT model and the training process of the machine learning model can specifically include the following training steps:

[0048] Input log data with lineage tags;

[0049] After loading the BERT model and word segmenter, define the hyperparameters and search space; the hyperparameters and search space include the extraction level of the BERT model, the pooling method, and the input truncation length;

[0050] Log data is divided into training and validation sets, and an objective function is constructed. The objective function is to extract BERT embeddings from training and validation logs for a given combination of parameters.

[0051] The machine learning model is trained and predicted based on the training set and the validation set, and the weighted F1 score of the validation set is used as the optimization metric.

[0052] The F1 score for lineage determination is optimized based on the acquisition function defined in Bayesian optimization; the acquisition function is used to measure the expected growth of the objective function relative to the current best observation;

[0053] Determine whether the difference between the current iteration's collection function value and the previous iteration's value is less than a set threshold; Bayesian optimization iteratively explores within the search space;

[0054] If so, output a set of optimal hyperparameters to obtain the trained BERT model and machine learning model; otherwise, enter the next loop to continue training the machine learning model until the optimal hyperparameters are output.

[0055] It is understood that in this embodiment, Bayesian optimization can be used to automatically find the optimal BERT embedding strategy to improve the log anomaly classification performance based on KNN. In the lineage determination task, the Bayesian optimization process based on BERT+KNN is as follows: Figure 2 As shown.

[0056] Specifically, the process begins by loading a pre-trained BERT model and a tokenizer, defining a hyperparameter search space that includes the layer index extracted by the BERT model, pooling methods (such as CLS vectors, average pooling, or max pooling), and the input truncation length (max_len). Next, the log data is divided into training and validation sets, and an objective function is constructed: for a given combination of parameters, extract BERT embeddings from both training and validation logs. Then, KNN is used for training and prediction, with the weighted F1 score of the validation set used as the optimization metric. Bayesian optimization iteratively explores this search space, ultimately outputting a set of optimal hyperparameters. Based on these optimal hyperparameters, embeddings are re-extracted, the KNN model is trained, and a final evaluation is performed, thereby automatically and efficiently determining whether logs are included in the phylogenetic tree.

[0057] In one embodiment, when defining hyperparameters and the search space, it is also included to define whether to use PCA dimensionality reduction and the dimension of PCA dimensionality reduction.

[0058] It is understandable that when defining hyperparameters and search space, one can also define whether to use PCA (Principal Component Analysis) for dimensionality reduction and its dimension (i.e., PCAdim), so that PCA dimensionality reduction can be performed when necessary, and then KNN can be used for training and prediction. This allows high-dimensional log data to be mapped to a low-dimensional space while preserving as much key log information as possible, simplifying the data structure and improving processing efficiency.

[0059] It should be understood that, although Figure 1The steps are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed; they can be performed in other orders. Figure 1 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0060] In one embodiment, such as Figure 3 As shown, a data lineage determination system 100 based on Bayesian optimization is also provided, including a log acquisition module 11, a preprocessing module 13, and a lineage determination module 15. The log acquisition module 11 is used to acquire log data to be determined. The preprocessing module 13 is used to preprocess the log data to be determined after loading a pre-trained BERT model and a word segmenter, obtaining the embedding vectors corresponding to the log data to be determined; the BERT model is obtained by pre-training based on log data with entered lineage labels using Bayesian optimization. The lineage determination module 15 is used to perform lineage determination on the embedding vectors using a machine learning model trained on the lineage determination task, determining the lineage labels of the log data to be determined.

[0061] The aforementioned Bayesian optimization-based data lineage identification system 100 uses a Bayesian process to optimize the pre-trained model BERT in the lineage identification task, minimizing the number of evaluations required to find the optimal solution. Bayesian optimization automatically finds the optimal BERT embedding strategy to improve the log anomaly classification effect based on KNN, thus achieving efficient identification of whether logs are recorded in the system's lineage.

[0062] In one embodiment, the pre-training of the BERT model and the training of the machine learning model may include the following steps:

[0063] Input log data with lineage tags;

[0064] After loading the BERT model and word segmenter, define the hyperparameters and search space; the hyperparameters and search space include the extraction level of the BERT model, the pooling method, and the input truncation length;

[0065] Log data is divided into training and validation sets, and an objective function is constructed. The objective function is to extract BERT embeddings from training and validation logs for a given combination of parameters.

[0066] The machine learning model is trained and predicted based on the training set and the validation set, and the weighted F1 score of the validation set is used as the optimization metric.

[0067] The F1 score for lineage determination is optimized based on the acquisition function defined in Bayesian optimization; the acquisition function is used to measure the expected growth of the objective function relative to the current best observation;

[0068] Determine whether the difference between the current iteration's collection function value and the previous iteration's value is less than a set threshold; Bayesian optimization iteratively explores within the search space;

[0069] If so, output a set of optimal hyperparameters to obtain the trained BERT model and machine learning model; otherwise, enter the next loop to continue training the machine learning model until the optimal hyperparameters are output.

[0070] In one embodiment, when defining hyperparameters and the search space, it is also included to define whether to use PCA dimensionality reduction and the dimension of PCA dimensionality reduction.

[0071] In one embodiment, the machine learning model is either the K-nearest neighbor algorithm or a decision tree.

[0072] It is understood that the explanations of the features in the aforementioned Bayesian-optimized data phylogenetic identification system 100 can be understood by referring to the corresponding explanations in the embodiments of the aforementioned Bayesian-optimized data phylogenetic identification method. Each module in the aforementioned Bayesian-optimized data phylogenetic identification system 100 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of a device with data processing capabilities, or stored in software in the memory of the aforementioned device, so that the processor can call and execute the operations corresponding to each module. The aforementioned device can be, but is not limited to, various types of intelligent operation and maintenance system devices already existing in the art.

[0073] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following processing steps: acquiring log data to be discriminated; loading a pre-trained BERT model and a word segmenter, and then preprocessing the log data to be discriminated to obtain an embedding vector corresponding to the log data to be discriminated; obtaining the BERT model based on log data with entered lineage labels and using Bayesian optimization for pre-training; and using a machine learning model trained on the lineage discrimination task to perform lineage discrimination on the embedding vector to determine the lineage label of the log data to be discriminated.

[0074] In one embodiment, when the processor executes the computer program, it can also implement the steps or sub-steps added in the various embodiments of the data lineage determination method based on Bayesian optimization.

[0075] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following processing steps: acquiring log data to be discriminated; loading a pre-trained BERT model and a word segmenter, and then preprocessing the log data to be discriminated to obtain the embedding vector corresponding to the log data to be discriminated; the BERT model is obtained by pre-training based on log data with recorded lineage labels and using Bayesian optimization; and the machine learning model trained on the lineage discrimination task is used to perform lineage discrimination on the embedding vector to determine the lineage label of the log data to be discriminated.

[0076] In one embodiment, when the computer program is executed by a processor, it can also implement the steps or sub-steps added in the various embodiments of the data lineage determination method based on Bayesian optimization.

[0077] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus DRAM (RDRAM), and interface DRAM (DRDRAM), etc.

[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of protection of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and all such modifications and improvements fall within the scope of protection of the present invention.

Claims

1. A data provenance discrimination method based on Bayesian optimization, characterized in that, The method comprises the steps of: obtaining log data to be discriminated; loading a pre-trained BERT model and a word segmenter to pre-process the log data to be discriminated, and obtaining an embedding vector corresponding to the log data to be discriminated; the BERT model is pre-trained based on log data with an entry lineage label and by using Bayesian optimization; using a machine learning model trained for a lineage discrimination task to discriminate the embedding vector, and determining a lineage label of the log data to be discriminated; the lineage discrimination is to extract key information in the lineage information that needs to be entered into the lineage; the pre-training of the BERT model and the training process of the machine learning model comprise: inputting log data with an entry lineage label; loading the BERT model and the word segmenter, defining hyperparameters and a search space; the hyperparameters and the search space include a level extracted by the BERT model, a pooling mode, and an input truncation length; dividing the log data into a training set and a validation set, and constructing an objective function; the objective function is to extract BERT embeddings of the training and validation logs for a given parameter combination; training and predicting using the machine learning model based on the training set and the validation set, and using a weighted F1 score of the validation set as an optimization index; optimizing the F1 score of the lineage discrimination based on a sampling function defined in the Bayesian optimization; the sampling function is used to measure the expected growth of the objective function relative to the current best observation; determining whether a difference between a value of the sampling function in the current cycle and a value of the sampling function in the last cycle is less than a set threshold; the Bayesian optimization iteratively explores within the search space; if yes, outputting a set of optimal hyperparameters, obtaining a trained BERT model and a machine learning model; otherwise, continuing to train the machine learning model in the next cycle until the optimal hyperparameters are outputted. 2.The method of claim 1, wherein, When the hyperparameters and the search space are defined, whether to use PCA dimension reduction and the dimension of PCA dimension reduction are also defined.

3. The data provenance discrimination method based on Bayesian optimization according to claim 1 or 2, characterized in that, The machine learning model is a K-nearest neighbor algorithm or a decision tree.

4. A data provenance discrimination system based on Bayesian optimization, characterized in that, The method comprises: a log acquisition module configured to acquire log data to be discriminated; a preprocessing module configured to load a pre-trained BERT model and a word segmenter to pre-process the log data to be discriminated, and obtain an embedding vector corresponding to the log data to be discriminated; the BERT model is pre-trained based on log data with an entry lineage label and by using Bayesian optimization; a lineage determination module configured to use a machine learning model trained for a lineage discrimination task to discriminate the embedding vector, and determine a lineage label of the log data to be discriminated; the lineage discrimination is to extract key information in the lineage information that needs to be entered into the lineage; the pre-training of the BERT model and the training process of the machine learning model comprise: inputting log data with an entry lineage label; loading the BERT model and the word segmenter, defining hyperparameters and a search space; the hyperparameters and the search space include a level extracted by the BERT model, a pooling mode, and an input truncation length; dividing the log data into a training set and a validation set, and constructing an objective function; the objective function is to extract BERT embeddings of the training and validation logs for a given parameter combination; training and predicting using the machine learning model based on the training set and the validation set, and using a weighted F1 score of the validation set as an optimization index; Optimize the F1 score of lineage discrimination based on the acquisition function defined in Bayesian optimization; the acquisition function is used to measure the expected increase of the objective function relative to the current best observation; Determine whether the difference between the value of the acquisition function in this cycle and the last cycle is less than a set threshold; Bayesian optimization iteratively explores within the search space; If yes, output a set of optimal hyperparameters, obtain a trained BERT model and machine learning model; otherwise, continue training the machine learning model until the optimal hyperparameters are output.

5. The Bayesian optimization-based data provenance discrimination system of claim 4, wherein, When defining the hyperparameters and the search space, it also includes defining whether to use PCA dimension reduction and the dimension of PCA dimension reduction.

6. The Bayesian-optimization-based data provenance discrimination system according to claim 4 or 5, characterized in that, The machine learning model is a K-nearest neighbor algorithm or a decision tree. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The processor executes the computer program to realize the steps of the data lineage discrimination method based on Bayesian optimization in any one of claims 1 to 3.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the data lineage discrimination method based on Bayesian optimization in any one of claims 1 to 3.

Images