Log data generation method and device based on support vector machine, equipment and medium

Abstract

The invention discloses a log data generation method and device based on a support vector machine, equipment and a medium. The method comprises the steps of performing data segmentation on a target historical log data set corresponding to a current analysis demand according to a preset data segmentation strategy to obtain a target training set and a target test set, performing model training on a basic support vector machine model corresponding to the current analysis demand based on the target training set to obtain a to-be-selected support vector machine model, performing model optimization on the to-be-selected support vector machine model based on the target test set to obtain a target support vector machine model meeting a preset optimization standard; based on a preset scene demand and a preset scene generation strategy, determining a target definition scene corresponding to the current analysis demand; and determining target log data corresponding to the current analysis demand based on the target definition scene and the target support vector machine model. Through the technical scheme of the invention, the log behavior in a real scene can be accurately simulated, and the authenticity and reliability of the log data are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and medium for generating log data based on a support vector machine. Background Art

[0002] With the development of computer technology, log data from complex distributed systems has been widely used to monitor and troubleshoot complex distributed systems by inputting it into log analysis systems for data analysis. However, simulating real log data to test and verify the capabilities of log analysis systems has always been a challenge.

[0003] Existing technologies typically rely on simple rules to generate log data or manually generate log data. However, methods that rely on simple rules to generate log data or manually generate log data are usually based on static rules, making it difficult to simulate dynamic and complex system behaviors, which reduces the reliability of log data. In addition, methods in existing technologies often lack support for user-defined scenarios, which reduces the authenticity of log data. In addition, they require a lot of manual configuration and operation, which increases labor costs and the possibility of errors. Therefore, how to accurately simulate log behavior in real scenarios and improve the authenticity and reliability of log data is a problem that needs to be solved urgently. Summary of the Invention

[0004] The present invention provides a log data generation method, apparatus, device and medium based on a support vector machine, which can solve the problem of low authenticity and reliability of log data used for testing and verification of a log analysis system.

[0005] According to one aspect of the present invention, a method for generating log data based on a support vector machine is provided, comprising:

[0006] Obtain the target historical log data set corresponding to the current analysis requirements;

[0007] Performing data segmentation on the target historical log data set according to a preset data segmentation strategy to obtain a target training set and a target test set corresponding to the current analysis requirements, performing model training on a basic support vector machine model corresponding to the current analysis requirements based on the target training set to obtain a candidate support vector machine model, and performing model optimization on the candidate support vector machine model based on the target test set to obtain a target support vector machine model that meets the preset optimization criteria;

[0008] Based on the preset scenario requirements and the preset scenario generation strategy, determine the target definition scenario corresponding to the current analysis requirements;

[0009] Based on the target definition scenario and the target support vector machine model, target log data corresponding to the current analysis requirement is determined.

[0010] According to another aspect of the present invention, a log data generating device based on a support vector machine is provided, comprising:

[0011] The data collection module is used to obtain the target historical log data set corresponding to the current analysis requirements;

[0012] A model training module is used to perform data segmentation on the target historical log data set according to a preset data segmentation strategy to obtain a target training set and a target test set corresponding to the current analysis requirements, perform model training on the basic support vector machine model corresponding to the current analysis requirements based on the target training set to obtain a candidate support vector machine model, and perform model optimization on the candidate support vector machine model based on the target test set to obtain a target support vector machine model that meets the preset optimization criteria;

[0013] The scenario simulation module is used to determine the target definition scenario corresponding to the current analysis requirements based on the preset scenario requirements and the preset scenario generation strategy;

[0014] The log generation module is used to determine the target log data corresponding to the current analysis requirement based on the target definition scenario and the target support vector machine model.

[0015] According to another aspect of the present invention, an electronic device is provided, comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log data generation method based on the support vector machine described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log data generation method based on a support vector machine according to any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method for generating log data based on a support vector machine according to any embodiment of the present invention is implemented.

[0021] The technical solution of the embodiment of the present invention is to obtain a target training set and a target test set by performing data segmentation on the target historical log data set corresponding to the current analysis requirement according to a preset data segmentation strategy. Then, the basic support vector machine model corresponding to the current analysis requirement is trained based on the target training set to obtain a candidate support vector machine model, and the candidate support vector machine model is optimized based on the target test set to obtain a target support vector machine model that meets the preset optimization criteria. Afterwards, based on the preset scenario requirements and the preset scenario generation strategy, the target definition scenario corresponding to the current analysis requirement is determined. Finally, based on the target definition scenario and the target support vector machine model, the target log data corresponding to the current analysis requirement is determined. Since the support vector machine model is used to generate log data, and it adapts to a variety of usage scenarios according to the preset scenario requirements, the problem of low authenticity and reliability of log data used for testing and verification of log analysis systems in the prior art is solved, and the log behavior in real scenarios can be accurately simulated, thereby improving the authenticity and reliability of log data.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 Flowchart of a method for generating log data based on a support vector machine according to the first embodiment of the present invention;

[0025] Figure 2 is a flow chart of a log data generation method based on a support vector machine according to the second embodiment of the present invention;

[0026] Figure 3 1 is a schematic structural diagram of a log data generating device based on a support vector machine according to a third embodiment of the present invention;

[0027] Figure 4 The figure is a schematic structural diagram of an electronic device for implementing the log data generation method based on a support vector machine according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "objective", "basis" and the like in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or that are inherent to these processes, methods, products or apparatus.

[0030] It should be noted that the information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0031] Example 1

[0032] Figure 1 This is a flowchart of a log data generation method based on a support vector machine provided in the first embodiment of the present invention. This embodiment is applicable to the case of generating log data for testing and verifying a log analysis system. The method can be executed by a log data generation device based on a support vector machine. The log data generation device based on a support vector machine can be implemented in the form of hardware and / or software. The log data generation device based on a support vector machine can be configured in an electronic device. Figure 1 As shown, the method includes:

[0033] S110: Obtain a target historical log data set corresponding to the current analysis requirement.

[0034] Among them, the current analysis requirement may refer to the log data type requirement used for data analysis at the current moment. Exemplarily, the current analysis requirement may be determined based on the data type of log data used for testing and verifying the log analysis system at the current moment. Specifically, if the data type of log data used for testing and verifying the log analysis system at the current moment is load log data, the current analysis requirement may be determined as load log data. It is worth noting that, in an embodiment of the present invention, the current analysis requirement may be one analysis requirement or a set of multiple analysis requirements, and the embodiment of the present invention does not specifically limit this.

[0035] The historical log data may refer to log data generated during a historical period. The target historical log data may refer to historical log data that meets the preset format requirements corresponding to the current analysis needs. For example, the target historical log data may refer to historical log data that meets the preset field requirements. The target historical log data set may refer to a data set consisting of each target historical log data.

[0036] S120. Perform data segmentation on the target historical log data set according to a preset data segmentation strategy to obtain a target training set and a target test set corresponding to the current analysis requirements, perform model training on the basic support vector machine model corresponding to the current analysis requirements based on the target training set to obtain a candidate support vector machine model, and perform model optimization on the candidate support vector machine model based on the target test set to obtain a target support vector machine model that meets the preset optimization criteria.

[0037] Among them, the preset data segmentation strategy may refer to a pre-set segmentation scheme for the target historical log data set. Exemplarily, the preset data segmentation strategy may be that the training set accounts for 70% of the target historical log data set, and the test set accounts for 30% of the target historical log data set. It may also be that the training set accounts for 80% of the target historical log data set, and the test set accounts for 20% of the target historical log data set. Usually, the category ratio in the training set and the test set is the same as that in the target historical log data set, and the proportion of the training set is generally greater than or equal to 70%. The data in the training set and the test set are different from each other, and the training set and the test set are combined to form a complete target historical log data set. The target training set may refer to the training set obtained after data segmentation of the target historical log data set. The target test set may refer to the test set obtained after data segmentation of the target historical log data set.

[0038] Among them, support vectors (SV) can refer to data points on the boundary and closest to the hyperplane. These data points are very important for determining the position of the hyperplane because they directly affect the boundary of the classifier. The support vector machine (SVM) model can refer to a supervised learning model for classification and regression analysis. Generally, SVM separates data points of different categories by finding an optimal hyperplane, thereby achieving the purpose of classification. In classification problems, the goal of SVM is to find a hyperplane that can maximize the interval between categories so that the classifier has better generalization ability for unknown data. In the embodiment of the present invention, the support vector machine model can be used to analyze and learn the features and patterns in historical log data, and then generate a model that meets the expected log data according to these patterns during the simulation process.

[0039] The basic support vector machine model may refer to an untrained initial support vector machine model. The selected support vector machine model may refer to a trained support vector machine model obtained after model training based on a target training set. The preset optimization criterion may refer to a pre-set standard rule for evaluating the validation process. For example, the preset optimization criterion may include a model indicator parameter meeting a preset threshold. Generally, the preset optimization criterion can provide an end condition for the model optimization process. The target support vector machine model may refer to an optimized support vector machine model obtained after model optimization based on a target test set.

[0040] S130: Determine a target definition scenario corresponding to the current analysis requirement based on the preset scenario requirement and the preset scenario generation strategy.

[0041] The preset scenario requirement may refer to a pre-set scenario setting requirement. For example, the preset scenario requirement may include a scenario setting type and a scenario setting method. The scenario setting type may refer to the category attribute of the setting scenario. Examples include high-load scenarios, network anomaly scenarios, and hardware failure scenarios. The scenario setting method may refer to the method in which the user desires to generate the setting scenario. For example, selecting a scenario using a field in an existing template, or customizing the generated scenario.

[0042] The preset scenario generation strategy may refer to a pre-defined scenario generation rule. Typically, a preset scenario generation strategy can be constructed based on pre-defined scenario setting types and scenario setting methods. For example, the preset scenario generation strategy may be a strategy for generating scenario fields corresponding to the scenario setting type using the scenario setting method. The target-defined scenario may refer to the scenario fields generated based on the preset scenario requirements and the preset scenario generation strategy.

[0043] S140: Determine target log data corresponding to current analysis requirements based on the target definition scenario and the target support vector machine model.

[0044] The target log data may refer to log data corresponding to a target definition scenario generated based on a target support vector machine model.

[0045] In an optional embodiment, after determining the target log data corresponding to the current analysis requirement based on the target definition scenario and the target support vector machine model, it may also include: obtaining a preset output method corresponding to the current analysis requirement; and transmitting data to the target log data based on the preset output method. The preset output method may refer to a pre-set log data output strategy. Exemplarily, the preset output method may be storage to a local disk, storage to a database, or direct pushing to a log analysis system via an application programming interface (API). Typically, if the preset output method is storage to a database, a specified database is first created and connected to, and then the target log data is written to a database table in chronological order, with each log record including fields such as a timestamp, log level, event description, and associated system components. If the preset output method is to push to a log analysis system, the output result may be fed back, thereby facilitating user confirmation that the target log data has been successfully stored or injected into the specified log analysis system. Specifically, after determining the target log data corresponding to the current analysis needs based on the target definition scenario and the target support vector machine model, the system can obtain the preset output method pre-selected by the user. Then, based on the preset output method, the generated target log data can be output to a designated storage medium or directly injected into the log analysis system for further analysis and verification. This solves the problem that log data generated by traditional methods has a single format and is difficult to directly apply to different log analysis systems.

[0046] It is worth noting that before the target log data is transmitted based on the preset output method, the target log data can also be formatted into a unified output format, such as JavaScript Object Notation (JSON) or character-separated values ​​(CSV) format.

[0047] The technical solution of the embodiment of the present invention is to obtain a target training set and a target test set by performing data segmentation on the target historical log data set corresponding to the current analysis requirement according to a preset data segmentation strategy. Then, the basic support vector machine model corresponding to the current analysis requirement is trained based on the target training set to obtain a candidate support vector machine model, and the candidate support vector machine model is optimized based on the target test set to obtain a target support vector machine model that meets the preset optimization criteria. Afterwards, based on the preset scenario requirements and the preset scenario generation strategy, the target definition scenario corresponding to the current analysis requirement is determined. Finally, based on the target definition scenario and the target support vector machine model, the target log data corresponding to the current analysis requirement is determined. Since the support vector machine model is used to generate log data, and it adapts to a variety of usage scenarios according to the preset scenario requirements, the problem of low authenticity and reliability of log data used for testing and verification of log analysis systems in the prior art is solved, and the log behavior in real scenarios can be accurately simulated, thereby improving the authenticity and reliability of log data.

[0048] Example 2

[0049] Figure 2 A flowchart of a log data generation method based on a support vector machine is provided for the second embodiment of the present invention. This embodiment is refined based on the above embodiment. In this embodiment, the operation of obtaining a target historical log data set corresponding to the current analysis requirement is specifically refined, which may include: determining a distributed system set corresponding to the current analysis requirement based on the requirement type of the current analysis requirement; wherein the distributed system set includes each distributed system; obtaining a basic historical log data set generated by the target distributed system in a preset historical period; wherein the basic historical log data set includes each basic historical log data; performing feature extraction on the basic historical log data based on a preset dictionary library to obtain intermediate historical log data containing the extracted features; normalizing the intermediate historical log data to obtain candidate historical log data, and combining and processing each candidate historical log data corresponding to the same target distributed system to generate a candidate historical log data set corresponding to the target distributed system; combining and processing the candidate historical log data sets corresponding to each target distributed system to obtain the target historical log data set corresponding to the current analysis requirement. Figure 2 As shown, the method includes:

[0050] S210: Determine a distributed system set corresponding to the current analysis requirement based on the requirement type of the current analysis requirement; wherein the distributed system set includes various distributed systems.

[0051] The requirement type can refer to the type of log data corresponding to the current analysis requirement. The distributed system can refer to the computer system composed of business nodes that generate the log data to be studied. The distributed system set can refer to the set of distributed systems corresponding to the same current analysis requirement.

[0052] S220: Obtain a basic historical log data set generated by the target distributed system during a preset historical period; wherein the basic historical log data set includes various basic historical log data.

[0053] The target distributed system may refer to a distributed system selected from a set of distributed systems for data collection. The preset historical period may refer to a pre-set historical time range. For example, the preset historical period may be the previous three months. The basic historical log data may refer to unprocessed historical log data generated by the target distributed system during the preset historical period.

[0054] Specifically, after determining the corresponding distributed system set based on the current analysis requirement type, a connection can be established to the log storage node of the target distributed system in the distributed system set, for example, by creating a soft link using the target distributed system's address. Subsequently, a specific API interface or log parsing tool is used to collect data from the basic historical log data set generated by the target distributed system during a preset historical period, providing an effective foundation for subsequent operations.

[0055] It is worth noting that, in the embodiment of the present invention, the preset historical time periods corresponding to the various target distributed systems may be different, and the embodiment of the present invention does not specifically limit this.

[0056] S230 : Extract features from the basic historical log data based on a preset dictionary library to obtain intermediate historical log data containing the extracted features.

[0057] The preset dictionary library may refer to a pre-set dictionary library containing key fields. The preset dictionary library may be used to determine the key fields contained in the log data. Intermediate historical log data may refer to log data that only contains extracted features after feature extraction. Extracted features may refer to important features in the basic historical log data. Exemplarily, the extracted features may include timestamps, log levels, event descriptions, system components, and user identities (IDs).

[0058] Specifically, after collecting data from the basic historical log data set, a pre-set dictionary library can be used to identify the important features contained in the basic historical log data set. The identified features are then extracted, and useless features are removed, retaining only the important features, reducing the amount of data processing in subsequent processes.

[0059] S240 , normalizing the intermediate historical log data to obtain candidate historical log data, and combining and processing the candidate historical log data corresponding to the same target distributed system to generate a candidate historical log data set corresponding to the target distributed system.

[0060] Normalization refers to scaling the values ​​of different features to similar intervals to eliminate dimensional differences between variables and ensure that different features or attributes have similar numerical ranges. Normalization maps data to a specific range, such as [0, 1] or [-1, 1]. Specifically, if the intermediate historical log data is "2024-08-2010:15:45,INFO,WebServer,Request received,UserID:123,Memory:50%", "2024-08-2010:15:46,ERROR,Database,Connection timeout,UserID:123,Memory:80%", and "2024-08-2010:16:01,WARN,WebServer,High memory usage,Memory:85%", then the candidate historical log data obtained through normalization processing can be "[1,0,0,1,123,0.5]", "[0,1,1,0,123,0.8]", and "[0,0,1,1,123,0.85]". This ensures that the intermediate historical log data from multiple preset historical time periods and multiple target distributed systems have similar numerical ranges, which facilitates the training of the support vector machine model.

[0061] The historical log data to be selected may refer to the historical log data obtained after normalization processing, and the historical log data set to be selected may refer to a data set consisting of various historical log data to be selected corresponding to the same target distributed system.

[0062] It is worth noting that in order to ensure the smooth progress of the subsequent model verification process, in an embodiment of the present invention, a data label representing a scene mode can be added to the corresponding historical log data to be selected based on the log level and event description of the intermediate historical log data. For example, following the above example, if the log level of the intermediate historical log data is INFO, then after adding a data label representing a scene mode to the corresponding historical log data to be selected, the result is: [1,0,0,1,123,0.5]->Normal; if the log level of the intermediate historical log data is ERROR and the event description is Connection timeout, then after adding a data label representing a scene mode to the corresponding historical log data to be selected, the result is: [0,1,1,0,123,0.8]->Timeout; if the log level of the intermediate historical log data is WARN, then after adding a data label representing a scene mode to the corresponding historical log data to be selected, the result is: [0,0,1,1,123,0.85]->Warning.

[0063] S250 , combining and processing the candidate historical log data sets corresponding to the target distributed systems to obtain the target historical log data set corresponding to the current analysis requirement.

[0064] Specifically, after combining and processing the various candidate historical log data corresponding to the same target distributed system to obtain various candidate historical log data sets, the various candidate historical log data sets corresponding to the current analysis requirements can be combined and processed to obtain the target historical log data set corresponding to the current analysis requirements.

[0065] S260 , performing data segmentation on the target historical log data set according to a preset data segmentation strategy to obtain a target training set and a target test set corresponding to the current analysis requirements.

[0066] Specifically, after generating a candidate historical log data set corresponding to the target distributed system, the target historical log data set may be segmented according to a preset data segmentation strategy to obtain a target training set and a target test set that satisfy the preset data segmentation strategy.

[0067] S270. Obtain a basic support vector machine model corresponding to the current analysis requirement.

[0068] Specifically, before training the support vector machine model, the parameters in the support vector machine model, such as weights, can be initialized according to preset empirical values ​​to obtain a basic support vector machine model, and then the basic support vector machine model can be trained in subsequent processes.

[0069] S280: Input the target training set into the basic support vector machine model, perform model training on the basic support vector machine model according to the data features of each training data in the target training set, and obtain a trained candidate support vector machine model.

[0070] Specifically, during the model training process, the basic support vector machine model can identify and mark different scenario patterns based on a combination of feature values ​​such as timestamp, log level, user ID, and memory usage in the target training set. For example, the basic support vector machine model can identify that the scenario pattern corresponding to the training data with data features of [0, 0, 1, 1, 123, 0.85] is a high memory usage scenario; it can also identify that the scenario pattern corresponding to the training data with data features of [0, 1, 1, 0, 123, 0.8] is a database timeout scenario. Thus, by using all the training data in the target training set to train the basic support vector machine model, a trained candidate support vector machine model that can learn scenario patterns and data features can be obtained.

[0071] S290: Input the target test set into the selected support vector machine model to obtain the optimization index value corresponding to the target test set.

[0072] Among them, the optimization index may refer to an index used to evaluate the model optimization results. Exemplarily, the optimization index may be accuracy, recall rate, F1 score, etc. Accuracy may refer to the proportion of samples correctly predicted by the model to the total number of samples. Recall rate may refer to the proportion of all positive samples predicted by the model as positive examples, which is used to reflect the model's ability to capture positive examples. F1 score may refer to the harmonic mean of accuracy and recall rate, which is used to comprehensively reflect the accuracy and recall ability of the model. The optimization index value may refer to the specific value corresponding to each optimization index. Generally, the calculation formula of F1 score may be: F1 score = 2 / [(1 / accuracy) + (1 / recall)]. The closer the F1 score is to 1, the better the model performs in predicting positive examples.

[0073] Specifically, after inputting the target test set into the candidate support vector machine model, the candidate support vector machine model's prediction results for the target test set are obtained. Since the target test dataset includes log patterns and features with known results, namely, actual labels, the prediction results can be matched with the actual labels, and the number of samples correctly and incorrectly predicted by the candidate support vector machine model can be counted. For example, if the total number of samples in the target test set is 100, the number of correctly predicted samples is 85, and the number of incorrectly predicted samples is 15, then the accuracy is: Accuracy = 85 / 100 = 0.85. Furthermore, samples labeled as positive in the target test set, such as fault or exception logs, can be identified by the field names of the data fields. The number of positive examples predicted by the candidate support vector machine model is then compared with the actual number of positive examples in the target test set. For example, if the actual number of positive examples in the target test set is 40, the candidate support vector machine model predicts 38 positive examples, of which the actual number of positive examples is 30, then the recall is: Recall = 30 / 40 = 0.75. Furthermore, based on the above precision and recall, we can get the F1 score: F1 score = 2 / [(1 / 0.85)+(1 / 0.75)]≈0.796.

[0074] S2100 , optimizing the candidate support vector machine model based on a preset optimization standard and the optimization index value to obtain a target support vector machine model that meets the preset optimization standard.

[0075] Specifically, after obtaining the optimization index value corresponding to the target test set, the optimization index value can be evaluated according to the preset optimization standard. If the optimization index value meets the preset optimization standard, the model optimization process is ended and the model at this time is used as the target support vector machine model; conversely, if the optimization index value does not meet the preset optimization standard, the model is re-optimized and adjusted.

[0076] Specifically, the optimization and adjustment process can include: first, expanding and processing the dataset. Generally, if the recall rate is low, it indicates that the candidate support vector machine model may have missed positive examples. Therefore, the target training set can be expanded, particularly by increasing the number of positive examples, to enhance the model's ability to learn from positive examples. Data augmentation techniques can also be employed, such as introducing slight noise into the existing target training set, adjusting log timestamps or the order of log events, to generate more diverse training data. Next, hyperparameters of the candidate support vector machine model, such as the kernel function, regularization parameter, and error tolerance, can be adjusted. For example, the regularization parameter can be increased to reduce misclassifications. Grid search or random search methods can also be used to automatically adjust hyperparameters to find the optimal parameter combination for improved model performance. Furthermore, the candidate support vector machine model, after hyperparameter adjustment, can be more thoroughly evaluated using cross-validation to ensure good generalization across different datasets. For example, the target historical log dataset can be divided into k parts, with one part used as the target validation set and the remaining parts used as the target training set. This process can be repeated k times and the results averaged. Among them, k can refer to a numerical value for evaluating the amount of data segmentation, which can usually be adjusted according to actual applications, and the embodiments of the present invention do not specifically limit this. Further, the important features currently used, such as timestamps, event types, and log contents, are analyzed. If it is found that certain features contribute little to the model or introduce noise, these features can be removed to achieve feature selection. In addition, techniques such as principal component analysis or linear discriminant analysis can be used to reduce the dimensionality of important features to reduce feature redundancy and improve the computational efficiency of the model. Finally, the candidate support vector machine model is retrained and evaluated using a new target test set to calculate new accuracy, recall, and F1 scores until the optimization index values ​​meet the preset optimization criteria. The model optimization process is terminated, and the model at this time is used as the target support vector machine model.

[0077] S2110. Based on preset scenario requirements and preset scenario generation strategies, determine the target definition scenario corresponding to the current analysis requirements.

[0078] In an optional embodiment, the determination of the target definition scenario corresponding to the current analysis requirement based on the preset scenario requirement and the preset scenario generation strategy may include: if the preset scenario requirement is a scene field selection requirement, then based on the preset scenario generation strategy, determining the basic scene template corresponding to the scene field selection requirement in the preset scene template library as the target definition scenario corresponding to the current analysis requirement; if the preset scenario requirement is a scene template definition requirement, then based on the preset scenario generation strategy and the scene template definition requirement, generating a corresponding target scene template as the target definition scenario corresponding to the current analysis requirement.

[0079] Among them, the scenario field selection requirement may refer to the requirement to actively select a field containing a scenario pattern. Exemplarily, a user may select a display field containing a scenario pattern by clicking the mouse to realize the scenario field selection requirement. The preset scenario template may refer to a predefined scenario template. Exemplarily, the preset scenario template may include a high-load scenario template, a network anomaly scenario template, a hardware failure scenario template, etc. Usually, each scenario template contains a set of typical log features. For example, a high-load scenario template may include: high central processing unit (CPU) usage: CPU>90%, high memory usage: Memory>85% features. The preset scenario template library may refer to a database containing multiple scenario templates. The basic scenario template may refer to a scenario template in the preset scenario template library that corresponds to the scenario field selection requirement. Exemplarily, if the scenario field selection requirement is a high-load scenario, the high-load scenario template in the preset scenario template library may be determined as the basic scenario template.

[0080] Among them, the scenario template definition requirement may refer to the requirement for customized generation of the scenario template. For example, the user can set a specific log event type, generation frequency, time distribution, trigger conditions and logical relationship between events through a configuration file or a graphical interface to realize the scenario template definition requirement. The target scenario template may refer to a scenario template that is regenerated according to the scenario template definition requirement. For example, the user defines a scenario template definition requirement of "database timeout causes user login failure", and its target scenario template may be: log event type: ERROR; log generation frequency: generate one every 30 seconds; trigger condition: when the database connection times out, generate a user login failure log; event logical relationship: database timeout -> user login failure.

[0081] Specifically, if the preset scenario requirement is a scenario field selection requirement, then based on the scenario field in the scenario field selection requirement, a basic scenario template consistent with the scenario field is determined in the preset scenario template library and used as the target definition scenario corresponding to the current analysis requirement. If the preset scenario requirement is a scenario template definition requirement, the configuration conditions in the scenario template definition requirement are converted into rules based on the preset scenario generation strategy to generate a corresponding target scenario template as the target definition scenario corresponding to the current analysis requirement. Thus, by providing a preset scenario template library and supporting user-defined scenarios, it is possible to adapt to a variety of usage scenarios and requirements and achieve flexible scenario definition.

[0082] S2120: Input the target definition scenario into a target support vector machine model.

[0083] S2130. Determine the scenario feature sequence corresponding to the target definition scenario based on the target support vector machine model, and randomly process the scenario feature sequence based on the field type of each scenario feature in the scenario feature sequence to generate target log data corresponding to the current analysis requirements.

[0084] Among them, the scenario feature may refer to each important feature corresponding to the target definition scenario. Exemplarily, the scenario feature may be a timestamp, log level, event description, system component, and user ID, etc. The scenario feature sequence may refer to a numerical sequence composed of various scenario features. Exemplarily, the sequence order in the scenario feature sequence may be: [timestamp, log level, event description, system component, user ID]. The field type may refer to the feature type of the scenario feature. Generally, different scenario features correspond to different field types, and different field types correspond to different value ranges.

[0085] Specifically, after determining the target definition scenario corresponding to the current analysis requirement, the target definition scenario can be input into the target support vector machine model. Through the association relationship between the patterns and features learned by the target support vector machine model, the corresponding scenario feature sequence is determined for the target definition scenario. Furthermore, the field type of each scenario feature is determined according to the sequence order in the scenario feature sequence, and the scenario feature sequence is randomly processed based on the field type of each scenario feature to generate the target log data corresponding to the current analysis requirement. For example, if the target definition scenario is a high-load scenario, the corresponding target log data can be: INFO: CPU usage at 92%; WARN: Memory usage at 88%; ERROR: System response time exceeded threshold; INFO: CPU usage at 95%; WARN: Disk I / O utilization high, etc.

[0086] The technical solution of an embodiment of the present invention determines the distributed system set corresponding to the current analysis requirement based on the requirement type of the current analysis requirement, obtains a basic historical log data set generated by the target distributed system in the distributed system set during a preset historical period, extracts features from the basic historical log data based on a preset dictionary library to obtain intermediate historical log data containing the extracted features, normalizes the intermediate historical log data to obtain candidate historical log data, and combines and processes each candidate historical log data corresponding to the same target distributed system to generate a candidate historical log data set corresponding to the target distributed system. Furthermore, the candidate historical log data sets corresponding to each target distributed system are combined and processed to obtain a target historical log data set corresponding to the current analysis requirement. Furthermore, the target historical log data set is segmented according to a preset data segmentation strategy to obtain a target training set and a target test set corresponding to the current analysis requirement. A basic support vector machine model corresponding to the current analysis requirement is obtained, the target training set is input into the basic support vector machine model, and the basic support vector machine model is trained based on the data features of each training data in the target training set to obtain a trained candidate support vector machine model. Furthermore, the target test set is input into the candidate support vector machine model to obtain the optimization index value corresponding to the target test set. And based on the preset optimization criteria and optimization index values, the selected support vector machine model is optimized to obtain a target support vector machine model that meets the preset optimization criteria. Finally, based on the preset scenario requirements and the preset scenario generation strategy, the target definition scenario corresponding to the current analysis requirement is determined. The target definition scenario is input into the target support vector machine model, and the scenario feature sequence corresponding to the target definition scenario is determined based on the target support vector machine model. The scenario feature sequence is randomly processed based on the field type of each scenario feature in the scenario feature sequence to generate the target log data corresponding to the current analysis requirement. Since the support vector machine model is used to generate log data, and it adapts to a variety of usage scenarios through preset scenario requirements, it solves the problem of low authenticity and reliability of log data used for testing and verification of log analysis systems in the prior art, can accurately simulate log behavior in real scenarios, and improve the authenticity and reliability of log data.

[0087] Based on the above embodiment, the time interval, event sequence, and type of log generation can be dynamically adjusted according to different scenarios. Historical log data can be obtained according to the dynamically adjusted rules, and the support vector machine model can be retrained to more closely resemble log behavior in a real environment. This embodiment of the present invention is not specifically limited to this.

[0088] Example 3

[0089] Figure 3 This is a structural diagram of a log data generation device based on a support vector machine provided in the third embodiment of the present invention. Figure 3As shown, the device includes: a data acquisition module 310, a model training module 320, a scene simulation module 330 and a log generation module 340;

[0090] The data collection module 310 is used to obtain the target historical log data set corresponding to the current analysis requirements;

[0091] The model training module 320 is used to perform data segmentation on the target historical log data set according to a preset data segmentation strategy to obtain a target training set and a target test set corresponding to the current analysis requirements, perform model training on the basic support vector machine model corresponding to the current analysis requirements based on the target training set to obtain a candidate support vector machine model, and perform model optimization on the candidate support vector machine model based on the target test set to obtain a target support vector machine model that meets the preset optimization criteria;

[0092] A scenario simulation module 330 is used to determine a target definition scenario corresponding to the current analysis requirement based on the preset scenario requirement and the preset scenario generation strategy;

[0093] The log generation module 340 is used to determine the target log data corresponding to the current analysis requirement based on the target definition scenario and the target support vector machine model.

[0094] The technical solution of the embodiment of the present invention is to obtain a target training set and a target test set by performing data segmentation on the target historical log data set corresponding to the current analysis requirement according to a preset data segmentation strategy. Then, the basic support vector machine model corresponding to the current analysis requirement is trained based on the target training set to obtain a candidate support vector machine model, and the candidate support vector machine model is optimized based on the target test set to obtain a target support vector machine model that meets the preset optimization criteria. Afterwards, based on the preset scenario requirements and the preset scenario generation strategy, the target definition scenario corresponding to the current analysis requirement is determined. Finally, based on the target definition scenario and the target support vector machine model, the target log data corresponding to the current analysis requirement is determined. Since the support vector machine model is used to generate log data, and it adapts to a variety of usage scenarios according to the preset scenario requirements, the problem of low authenticity and reliability of log data used for testing and verification of log analysis systems in the prior art is solved, and the log behavior in real scenarios can be accurately simulated, thereby improving the authenticity and reliability of log data.

[0095] Optionally, the data acquisition module 310 may be used to:

[0096] Determine a distributed system set corresponding to the current analysis requirement based on the requirement type of the current analysis requirement; wherein the distributed system set includes each distributed system;

[0097] Obtaining a basic historical log data set generated by the target distributed system during a preset historical period; wherein the basic historical log data set includes various basic historical log data;

[0098] Performing feature extraction on the basic historical log data based on a preset dictionary library to obtain intermediate historical log data containing the extracted features;

[0099] Normalizing the intermediate historical log data to obtain candidate historical log data, and combining and processing the candidate historical log data corresponding to the same target distributed system to generate a candidate historical log data set corresponding to the target distributed system;

[0100] Combine and process the candidate historical log data sets corresponding to each target distributed system to obtain the target historical log data set corresponding to the current analysis requirements.

[0101] Optionally, the model training module 320 may be used to:

[0102] Obtain the basic support vector machine model corresponding to the current analysis requirements;

[0103] The target training set is input into the basic support vector machine model, and the basic support vector machine model is trained according to the data features of each training data in the target training set to obtain a trained support vector machine model to be selected.

[0104] Optionally, the model training module 320 may be used to:

[0105] Inputting the target test set into the selected support vector machine model to obtain an optimization index value corresponding to the target test set;

[0106] The selected support vector machine model is optimized based on the preset optimization criteria and the optimization index value to obtain a target support vector machine model that meets the preset optimization criteria.

[0107] Optionally, the scenario simulation module 330 may be used to:

[0108] If the preset scenario requirement is a scenario field selection requirement, determining a basic scenario template corresponding to the scenario field selection requirement in a preset scenario template library based on a preset scenario generation strategy as a target definition scenario corresponding to the current analysis requirement;

[0109] If the preset scenario requirement is a scenario template definition requirement, a corresponding target scenario template is generated based on the preset scenario generation strategy and the scenario template definition requirement as the target definition scenario corresponding to the current analysis requirement.

[0110] Optionally, the log generation module 340 may be used to:

[0111] Inputting the target definition scenario into a target support vector machine model;

[0112] The scenario feature sequence corresponding to the target definition scenario is determined based on the target support vector machine model, and the scenario feature sequence is randomly processed based on the field type of each scenario feature in the scenario feature sequence to generate target log data corresponding to the current analysis requirements.

[0113] Optionally, the log data generation device based on a support vector machine may further include: a data output module, which is used to obtain a preset output method corresponding to the current analysis requirement after determining the target log data corresponding to the current analysis requirement based on the target definition scenario and the target support vector machine model; and to transmit data of the target log data based on the preset output method.

[0114] The log data generation device based on the support vector machine provided in the embodiment of the present invention can execute the log data generation method based on the support vector machine provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0115] Example 4

[0116] Figure 4 A schematic diagram of the structure of an electronic device 410 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0117] like Figure 4As shown, the electronic device 410 includes at least one processor 420, and a memory connected to the at least one processor 420 in communication, such as a read-only memory (ROM) 430, a random access memory (RAM) 440, etc., wherein the memory stores a computer program that can be executed by the at least one processor, and the processor 420 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 430 or the computer program loaded from the storage unit 490 to the random access memory (RAM) 440. Various programs and data required for the operation of the electronic device 410 can also be stored in the RAM 440. The processor 420, ROM 430 and RAM 440 are connected to each other via a bus 450. An input / output (I / O) interface 460 is also connected to the bus 450.

[0118] Multiple components in the electronic device 410 are connected to the I / O interface 460, including an input unit 470, such as a keyboard, a mouse, etc.; an output unit 480, such as various types of displays, speakers, etc.; a storage unit 490, such as a magnetic disk, an optical disk, etc.; and a communication unit 4100, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 4100 allows the electronic device 410 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0119] The processor 420 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the processor 420 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 420 executes the various methods and processes described above, such as the log data generation method based on a support vector machine.

[0120] The method includes:

[0121] Obtain the target historical log data set corresponding to the current analysis requirements;

[0122] Performing data segmentation on the target historical log data set according to a preset data segmentation strategy to obtain a target training set and a target test set corresponding to the current analysis requirements, performing model training on a basic support vector machine model corresponding to the current analysis requirements based on the target training set to obtain a candidate support vector machine model, and performing model optimization on the candidate support vector machine model based on the target test set to obtain a target support vector machine model that meets the preset optimization criteria;

[0123] Based on the preset scenario requirements and the preset scenario generation strategy, determine the target definition scenario corresponding to the current analysis requirements;

[0124] Based on the target definition scenario and the target support vector machine model, target log data corresponding to the current analysis requirement is determined.

[0125] In some embodiments, the log data generation method based on the support vector machine can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 490. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 410 via the ROM 430 and / or the communication unit 4100. When the computer program is loaded into the RAM 440 and executed by the processor 420, one or more steps of the log data generation method based on the support vector machine described above can be performed. Alternatively, in other embodiments, the processor 420 can be configured to execute the log data generation method based on the support vector machine by any other appropriate means (for example, by means of firmware).

[0126] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0131] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0132] The present application also discloses a computer program product, comprising a computer program that, when executed by a processor, implements the support vector machine-based log data generation method provided in any of the embodiments of the present application. This program product shares the same inventive concept as the support vector machine-based log data generation method disclosed in each embodiment of the present application, and therefore is not further described here.

[0133] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0134] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A log data generation method based on support vector machine, characterized in that: include: Get the target historical log data set corresponding to the current analysis requirements; The target historical log data set is segmented according to a preset data segmentation strategy to obtain a target training set and a target test set corresponding to the current analysis requirement, a basic support vector machine model corresponding to the current analysis requirement is trained based on the target training set to obtain a candidate support vector machine model, and the candidate support vector machine model is optimized based on the target test set to obtain a target support vector machine model that meets the preset optimization criteria; Based on the preset scenario requirements and the preset scenario generation strategy, determine the target definition scenario corresponding to the current analysis requirements; Based on the target definition scenario and the target support vector machine model, target log data corresponding to the current analysis requirement is determined.

2. The method according to claim 1, characterized in that: The step of obtaining a target historical log data set corresponding to the current analysis requirement includes: Determine a distributed system set corresponding to the current analysis requirement based on the requirement type of the current analysis requirement; wherein the distributed system set includes each distributed system; Obtain a basic historical log data set generated by the target distributed system in a preset historical period; wherein the basic historical log data set includes various basic historical log data; Performing feature extraction on the basic historical log data based on a preset dictionary library to obtain intermediate historical log data containing the extracted features; Normalizing the intermediate historical log data to obtain the historical log data to be selected, and combining and processing the various historical log data to be selected corresponding to the same target distributed system to generate a historical log data set to be selected corresponding to the target distributed system; The candidate historical log data sets corresponding to each target distributed system are processed in combination to obtain the target historical log data set corresponding to the current analysis requirements.

3. The method according to claim 1, characterized in that The step of training the basic support vector machine model corresponding to the current analysis requirement based on the target training set to obtain a support vector machine model to be selected includes: Obtain the basic support vector machine model corresponding to the current analysis requirements; The target training set is input into the basic support vector machine model, and the basic support vector machine model is trained according to the data features of each training data in the target training set to obtain a trained support vector machine model to be selected.

4. The method according to claim 1, characterized in that The step of optimizing the candidate support vector machine model based on the target test set to obtain a target support vector machine model that meets a preset optimization standard includes: Inputting the target test set into the selected support vector machine model to obtain the optimization index value corresponding to the target test set; The selected support vector machine model is optimized based on the preset optimization standard and the optimization index value to obtain a target support vector machine model that meets the preset optimization standard.

5. The method according to claim 1, characterized in that The step of determining the target definition scenario corresponding to the current analysis requirement based on the preset scenario requirement and the preset scenario generation strategy includes: If the preset scenario requirement is a scenario field selection requirement, a basic scenario template corresponding to the scenario field selection requirement is determined in a preset scenario template library based on a preset scenario generation strategy as a target definition scenario corresponding to the current analysis requirement; If the preset scenario requirement is a scenario template definition requirement, a corresponding target scenario template is generated based on the preset scenario generation strategy and the scenario template definition requirement as a target definition scenario corresponding to the current analysis requirement.

6. The method according to claim 1, characterized in that The determining target log data corresponding to the current analysis requirement based on the target definition scenario and the target support vector machine model includes: Inputting the target definition scenario into a target support vector machine model; The scenario feature sequence corresponding to the target definition scenario is determined based on the target support vector machine model, and the scenario feature sequence is randomly processed based on the field type of each scenario feature in the scenario feature sequence to generate target log data corresponding to the current analysis requirement.

7. The method according to claim 1, characterized in that After determining the target log data corresponding to the current analysis requirement based on the target definition scenario and the target support vector machine model, the method further includes: Get the preset output method corresponding to the current analysis requirements; The target log data is data transmitted based on the preset output mode.

8. A log data generating device based on support vector machine, characterized in that: include: The data collection module is used to obtain the target historical log data set corresponding to the current analysis requirements; A model training module is used to perform data segmentation on the target historical log data set according to a preset data segmentation strategy to obtain a target training set and a target test set corresponding to the current analysis requirements, perform model training on a basic support vector machine model corresponding to the current analysis requirements based on the target training set to obtain a candidate support vector machine model, and perform model optimization on the candidate support vector machine model based on the target test set to obtain a target support vector machine model that meets the preset optimization criteria; A scenario simulation module is used to determine the target definition scenario corresponding to the current analysis requirement based on the preset scenario requirements and the preset scenario generation strategy; The log generation module is used to determine the target log data corresponding to the current analysis requirements based on the target definition scenario and the target support vector machine model.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log data generation method based on a support vector machine according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log data generation method based on a support vector machine according to any one of claims 1 to 7 when executed.

Images