Log sample screening method and device, electronic device, and storage medium
By automatically filtering log samples using MapReduce and semi-supervised K-means clustering, the problem of slow log filtering speed is solved, and the effect of efficiently filtering valuable logs is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-28
- Publication Date
- 2026-03-27
AI Technical Summary
Existing technologies are slow and inefficient in filtering logs, especially when processing massive amounts of log data, making it difficult to effectively filter out meaningful logs.
By employing the MapReduce filtering algorithm combined with the semi-supervised K-means clustering method, and by obtaining the sample labels of log samples and calculating the cosine distance, invalid, pending confirmation, and valid logs are identified, and valuable log samples are automatically filtered out.
It significantly improves the speed and efficiency of log filtering, requires no manual intervention, and can efficiently filter out invalid logs, providing valuable log samples for subsequent analysis.
Smart Images
Figure CN116881216B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of big data or other related fields, in particular, to a log sample screening method and device, an electronic device and a storage medium. BACKGROUND
[0002] Log screening is to screen out meaningful logs / valid logs from massive logs and eliminate meaningless logs / invalid logs (i.e. logs that cannot provide ideas for analyzing problems). Eliminating these invalid logs not only saves a large amount of storage space, but also highlights the effective logs with practical value, providing help for the next log terminal or analyst to analyze and classify logs.
[0003] In the related art, when performing log screening, the commonly used technical solution is to manually screen logs according to the observation results by manually observing the content of logs, or to use regular expressions to match the keywords in logs, thereby achieving the effect of filtering and screening.
[0004] However, the above-mentioned log screening by manual or regular expression has obvious shortcomings. The first one is that the manual screening of specific logs requires a large amount of manual time and is slow and inefficient. The second one is that the regular expression screening of logs is generally only suitable for filtering and matching screening on a single machine and is not suitable for massive log data. Screening massive data is extremely slow.
[0005] At present, no effective solution has been proposed to solve the above-mentioned problems. SUMMARY
[0006] The embodiments of the present application provide a log sample screening method and device, an electronic device and a storage medium to at least solve the technical problem of slow screening speed and low efficiency in related art when performing log screening.
[0007] According to an aspect of the embodiments of the present application, a method for screening log samples is provided, comprising: obtaining each log sample and a corresponding sample label in a target log set, wherein the sample label is one of the following: an invalid label, a to-be-confirmed label, and a valid label; inputting all log samples corresponding to the to-be-confirmed label and the corresponding sample labels into a data integration model, and outputting a new centroid set, wherein the new centroid set contains K new centroids, and K is a positive integer; calculating a cosine distance value between each to-be-confirmed log sample and each new centroid in the new centroid set, and selecting a minimum cosine distance and a new centroid corresponding to the to-be-confirmed log sample; determining a sample label corresponding to the to-be-confirmed log sample based on the minimum cosine distance and the new centroid, wherein the sample label is one of the following: an invalid label and a valid label; and outputting all log samples containing the valid label.
[0008] Optionally, the step of obtaining each log sample and a corresponding sample label in a target log set comprises: calculating a cosine distance between each log sample in the target log set and each initial centroid in a preset centroid set, to obtain a cosine distance set, wherein the preset centroid set contains K initial centroids after sample clustering; selecting a minimum cosine distance and a corresponding initial centroid in the cosine distance set; and determining a sample label corresponding to the log sample based on the minimum cosine distance and the initial centroid.
[0009] Optionally, before calculating a cosine distance between each log sample in a target log set and each centroid in a preset centroid set, the method further comprises: receiving a to-be-screened log set input by an external terminal, and selecting a preset part of log samples in the to-be-screened log set, wherein the selected preset part of log samples are used as the target log set; performing sample clustering processing on the target log set, to obtain K sample classes; and extracting a sample center of each sample class, wherein the sample center is used as the initial centroid, and the preset centroid set is obtained.
[0010] Optionally, after extracting a sample center of each sample class, the sample center is used as the initial centroid, and the preset centroid set is obtained, the method further comprises: reading the preset centroid set, and placing the preset centroid set in a preset distributed cache.
[0011] Optionally, the step of determining the sample label corresponding to the log sample based on the minimum cosine distance corresponding to the log sample and the initial centroid comprises: in a case where the minimum cosine distance is less than a first preset threshold, confirming that the sample label of the log sample is an invalid label; in a case where the minimum cosine distance is greater than or equal to the first preset threshold and less than a second preset threshold, confirming that the sample label of the log sample is a to-be-confirmed label, wherein the first preset threshold is less than the second preset threshold; and in a case where the minimum cosine distance is greater than the second preset threshold, confirming that the sample label of the log sample is a valid label.
[0012] Optionally, the step of inputting all the log samples corresponding to the to-be-confirmed labels and the corresponding sample labels into the data integration model and outputting the new centroid set comprises: obtaining all the log samples corresponding to the invalid labels in the target log set to obtain an invalid log sample set; calculating a clustering center of the invalid log sample set; and inputting the clustering center of the invalid log sample set, all the log samples corresponding to the to-be-confirmed labels and the corresponding sample labels into the data integration model to output the new centroid set.
[0013] Optionally, before the step of calculating the cosine distance value between each new centroid in the new centroid set and the to-be-confirmed log sample corresponding to the to-be-confirmed label, the method further comprises: reading all the log samples corresponding to the to-be-confirmed label and selecting part of the log samples in the all log samples, taking the selected part of the log samples as a to-be-confirmed log sample set; and taking each row of log sample in the to-be-confirmed log sample set as the to-be-confirmed log sample corresponding to the to-be-confirmed label.
[0014] Optionally, the step of determining the sample label corresponding to the to-be-confirmed log sample based on the minimum cosine distance corresponding to the to-be-confirmed log sample and the new centroid comprises: in a case where the minimum cosine distance is less than a third preset threshold, determining that the sample label of the to-be-confirmed log sample is an invalid label; and in a case where the minimum cosine distance is greater than or equal to the third preset threshold, determining that the sample label of the to-be-confirmed log sample is a valid label.
[0015] According to another aspect of the embodiments of the present application, a log sample screening device is also provided, comprising: an acquisition unit configured to acquire each log sample in a target log set and a corresponding sample label, wherein the sample label is one of: an invalid label, a to-be-confirmed label, and a valid label; an input unit configured to input all log samples corresponding to the to-be-confirmed label and the corresponding sample labels to a data integration model, and output a new centroid set, wherein the new centroid set contains K new centroids, and K is a positive integer; a calculation unit configured to calculate a cosine distance value between each to-be-confirmed log sample and each new centroid in the new centroid set, and select a minimum cosine distance and a new centroid corresponding to the to-be-confirmed log sample; a determination unit configured to determine a sample label corresponding to the to-be-confirmed log sample based on the minimum cosine distance and the new centroid corresponding to the to-be-confirmed log sample, wherein the sample label is one of: an invalid label and a valid label; and an output unit configured to output all log samples containing the valid label.
[0016] Optionally, the acquisition unit comprises: a first calculation module configured to calculate a cosine distance between each log sample in the target log set and each initial centroid in a preset centroid set, to obtain a cosine distance set, wherein the preset centroid set contains K initial centroids after sample clustering; a first selection module configured to select a minimum cosine distance and a corresponding initial centroid in the cosine distance set; and a first determination module configured to determine a sample label corresponding to the log sample based on the minimum cosine distance and the initial centroid corresponding to the log sample.
[0017] Optionally, the log sample screening device further comprises: a first receiving module configured to receive a to-be-screened log set input by an external terminal before calculating a cosine distance between each log sample in a target log set and each centroid in a preset centroid set, and select a preset part of log samples in the to-be-screened log set, wherein the selected preset part of log samples are taken as the target log set; a clustering module configured to perform sample clustering processing on the target log set to obtain K sample classes; and a first extraction module configured to extract a sample center of each sample class, and take the sample center as the initial centroid to obtain the preset centroid set.
[0018] Optionally, the log sample screening device further comprises: a putting module configured to read the preset centroid set after extracting a sample center of each sample class, taking the sample center as the initial centroid, and obtaining the preset centroid set, and put the preset centroid set into a preset distributed cache.
[0019] Optionally, the determining unit comprises: a second determining module, configured to determine that the sample label of the log sample is an invalid label when the minimum cosine distance is less than a first preset threshold; a third determining module, configured to determine that the sample label of the log sample is a to-be-confirmed label when the minimum cosine distance is greater than or equal to the first preset threshold and less than a second preset threshold, wherein the first preset threshold is less than the second preset threshold; and a fourth determining module, configured to determine that the sample label of the log sample is a valid label when the minimum cosine distance is greater than the second preset threshold.
[0020] Optionally, the input unit comprises: a first obtaining module, configured to obtain all log samples corresponding to invalid labels in the target log set to obtain an invalid log sample set; a second calculating module, configured to calculate a clustering center of the invalid log sample set; and an input module, configured to input the clustering center of the invalid log sample set and the log samples corresponding to all the to-be-confirmed labels and the corresponding sample labels into a data integration model to output the new center set.
[0021] Optionally, the log sample screening device further comprises: a first reading module, configured to read all log samples corresponding to the to-be-confirmed label and select part of the log samples in the all log samples as a to-be-confirmed log sample set before calculating the cosine distance value between the to-be-confirmed log sample corresponding to the to-be-confirmed label and each new center in the new center set; and a fifth determining module, configured to take each row of log sample in the to-be-confirmed log sample set as the to-be-confirmed log sample corresponding to the to-be-confirmed label.
[0022] Optionally, the fifth determining module comprises: a first determining submodule, configured to determine that the sample label of the to-be-confirmed log sample is an invalid label when the minimum cosine distance is less than a third preset threshold; and a second determining submodule, configured to determine that the sample label of the to-be-confirmed log sample is a valid label when the minimum cosine distance is greater than or equal to the third preset threshold.
[0023] According to another aspect of the embodiment of the present application, a computer readable storage medium is provided, which comprises a stored computer program, wherein the computer program controls the device where the computer readable storage medium is located to perform the log sample screening method when the computer program is running.
[0024] According to a further aspect of the embodiments of the present application, there is also provided an electronic device comprising one or more processors and a memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the log sample screening method according to any one of the preceding embodiments.
[0025] In the present disclosure, after receiving a log set to be screened, each log sample in the target log set and the corresponding sample label are obtained, and then all log samples corresponding to the to-be-confirmed label and the corresponding sample label are input into a data integration model, a new centroid set is output, the cosine distance value between the to-be-confirmed log sample corresponding to the to-be-confirmed label and each new centroid in the new centroid set is calculated, the minimum cosine distance corresponding to the to-be-confirmed log sample and the new centroid are selected, the sample label corresponding to the to-be-confirmed log sample is determined based on the minimum cosine distance corresponding to the to-be-confirmed log sample and the new centroid, and finally all log samples containing valid labels are output.
[0026] In the present disclosure, for a large amount of log sample data, after initial labeling, the to-be-confirmed samples are further subjected to label reconfirmation, thereby combining the MapReduce filtering algorithm to filter out all invalid logs. This screening method does not require human intervention, and the MapReduce filtering algorithm and the model clustering method can be used to screen all log samples containing valid labels, thereby significantly improving the sample screening speed and efficiency, and further solving the technical problems of slow screening speed and low efficiency in related technologies. BRIEF DESCRIPTION OF DRAWINGS
[0027] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:
[0028] Figure 1 is a flowchart of an optional log sample screening method according to an embodiment of the present application;
[0029] Figure 2 is a schematic diagram of an optional log sample screening device according to an embodiment of the present application;
[0030] Figure 3 is a hardware structure block diagram of an electronic device (or mobile device) for a log sample screening method according to an embodiment of the present application. DETAILED DESCRIPTION
[0031] In order to better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should belong to the scope of protection of the present application.
[0032] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.
[0033] In order for those skilled in the art to better understand the technical scheme of the present application, the following will explain some terms or nouns involved in the embodiments of the present application:
[0034] MapReduce, a programming model and computing framework for processing large-scale data sets, divides data processing tasks into two main stages: Map phase and Reduce phase.
[0035] K-means, a clustering algorithm, works by assigning each data point in the dataset to one of the k cluster centers closest to it, and then recalculating the positions of the cluster centers based on the assignments. This process is iterated until the positions of the cluster centers no longer change or a predetermined number of iterations is reached.
[0036] It should be noted that the screening method and device for log samples in the present disclosure can be used in the field of big data technology for screening log samples based on big data, and can also be used in any field other than the field of big data technology for screening log samples based on big data. The application field of the screening method and device for log samples in the present disclosure is not limited.
[0037] It should be noted that the related information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of countries and regions, and provide corresponding operation portal for user to choose authorization or refusal. For example, an interface is provided between the system and the related user or agency, and before obtaining the related information, the interface needs to send an acquisition request to the aforementioned user or agency, and after receiving the consent information fed back by the aforementioned user or agency, the related information is acquired.
[0038] The following embodiments of the present application can be applied to various systems / applications / devices for filtering log samples. In order to reduce the accumulation of platform logs and provide valuable logs for further problem analysis in the background, a log sample filtering method is proposed for the massive logs generated under the big data platform. The filtering method can filter log samples based on MapReduce, and combine the semi-supervised K-means specific log filtering method to eliminate invalid logs from the massive logs, and provide help for log analysis and classification by analysts in the next step.
[0039] The present application will be described in detail below in conjunction with various embodiments.
[0040] Embodiment one
[0041] According to the embodiments of the present application, an embodiment of a log sample filtering method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.
[0042] Figure 1 is a flowchart of an optional log sample filtering method according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:
[0043] Step S101, obtaining each row of log sample in the target log set and the corresponding sample label, wherein the sample label is one of the following: invalid label, to-be-confirmed label, valid label;
[0044] Step S102, inputting the log sample corresponding to all to-be-confirmed labels and the corresponding sample label into a data integration model, and outputting a new centroid set, wherein the new centroid set contains K new centroids, and K is a positive integer;
[0045] Step S103, calculate the cosine distance value between the to-be-confirmed log sample corresponding to the to-be-confirmed label and each new centroid in the new centroid set, and select the minimum cosine distance and the new centroid corresponding to the to-be-confirmed log sample;
[0046] Step S104, determine the sample label corresponding to the to-be-confirmed log sample based on the minimum cosine distance and the new centroid corresponding to the to-be-confirmed log sample, wherein the sample label is one of the following: invalid label, valid label;
[0047] Step S105, output all log samples containing valid labels.
[0048] Through the above steps, after receiving the to-be-screened log set, the target log set can be obtained, each row of log sample and the corresponding sample label in the target log set are obtained, and then all log samples corresponding to the to-be-confirmed label and the corresponding sample label are input into the data integration model, the new centroid set is output, the cosine distance value between the to-be-confirmed log sample corresponding to the to-be-confirmed label and each new centroid in the new centroid set is calculated, and the minimum cosine distance and the new centroid corresponding to the to-be-confirmed log sample are selected. The sample label corresponding to the to-be-confirmed log sample is determined based on the minimum cosine distance and the new centroid corresponding to the to-be-confirmed log sample, and finally all log samples containing valid labels are output. In this embodiment, for a large amount of log sample data, after initial labeling, the to-be-confirmed samples can be further labeled and confirmed, thereby combining the MapReduce filtering algorithm to filter out all invalid logs. This screening method does not require human intervention, and the MapReduce filtering algorithm and the model clustering method can be used to screen all log samples containing valid labels, thereby significantly improving the sample screening speed and sample screening efficiency, and further solving the technical problems of slow screening speed and low efficiency in related technologies.
[0049] The embodiments of the present application will be described in detail below in combination with the above steps.
[0050] In this embodiment, the massive logs received by the big data system (may include massive historical logs and real-time current logs, and after using the scheme provided by the embodiments of the present application to input a large amount of log data, less valid logs are obtained) are pre-screened to obtain a target log set.
[0051] Step S101, obtain each row of log sample and the corresponding sample label in the target log set, wherein the sample label is one of the following: invalid label, to-be-confirmed label, valid label.
[0052] Optionally, before calculating the cosine distance between each log sample in the target log set and each centroid in the preset centroid set, the method further comprises: receiving a to-be-screened log set input by an external terminal (types include but are not limited to: mobile terminal, PC terminal, server where big data system is located, etc.), and selecting a preset part of log samples in the to-be-screened log set, taking the selected preset part of log samples as the target log set; performing sample clustering processing on the target log set to obtain K sample classes; extracting a sample center of each sample class, taking the sample center as an initial centroid to obtain the preset centroid set.
[0053] The embodiment introduces the K-means algorithm, performs initial clustering on the selected log samples to obtain K sample classes and K sample centers, wherein in the process of extracting a sample center of each sample class and taking the sample center as an initial centroid, a sample with the minimum total distance to other samples in each sample class can be selected as the sample center by calculating the distance values between samples in each sample class. The obtained preset centroid set is defined as Cens in the embodiment.
[0054] Optionally, after extracting a sample center of each sample class, taking the sample center as an initial centroid, and obtaining the preset centroid set, the method further comprises: reading the preset centroid set and putting the preset centroid set into a preset distributed cache.
[0055] The embodiment introduces the MapReduce idea, and therefore, in order to facilitate subsequent calculation of the cosine distance between the centroid and the sample, the preset centroid set needs to be put into a preset distributed cache (the type of the cache is selected by the user, for example, a MapReduce distributed cache determined by the MapReduce architecture).
[0056] After putting the preset centroid set into the cache, the first execution of the Map program and the Reduce program is started. For the Map program, each Map program task reads the target log set to be classified, and outputs each sample and its corresponding label.
[0057] Optionally, step S101 comprises: calculating the cosine distance between each log sample in the target log set and each initial centroid in the preset centroid set to obtain a cosine distance set, wherein the preset centroid set contains K initial centroids after sample clustering; selecting the minimum cosine distance in the cosine distance set and the corresponding initial centroid; determining the sample label corresponding to the log sample based on the minimum cosine distance and the initial centroid corresponding to the log sample.
[0058] By reading each row sample in the target log set row by row, the cosine distance between the sample and each centroid in the preset centroid set Cens in the distributed cache is calculated, and then the minimum cosine distance and the corresponding initial centroid are found. In the embodiment, the minimum cosine distance is defined as dist_min, and the initial centroid obtained at this time is defined as Cen_m.
[0059] Optionally, based on the minimum cosine distance corresponding to the log sample and the initial centroid, the step of determining the sample label corresponding to the log sample comprises: in the case that the minimum cosine distance is less than a first preset threshold, confirming that the sample label of the log sample is an invalid label; in the case that the minimum cosine distance is greater than or equal to the first preset threshold and less than a second preset threshold, confirming that the sample label of the log sample is a to-be-confirmed label, wherein the first preset threshold is less than the second preset threshold; in the case that the minimum cosine distance is greater than the second preset threshold, confirming that the sample label of the log sample is a valid label.
[0060] In the determination of the sample label corresponding to each log sample, the pre-distance is compared with the distance threshold, and each sample is labeled according to the comparison result. If the cosine distance dist_min is less than the first preset threshold (defined as T1 in the embodiment), the log sample is labeled, and the log sample is determined as an invalid label. If the minimum cosine distance dist_min is between the first preset threshold T1 and the second preset threshold T2 (T2>T1), the sample is labeled as a to-be-determined label (defined as todo in the embodiment), and specific verification will be performed later. If the minimum cosine distance dist_min is greater than the threshold T2, the log sample is determined as a valid label (defined as use in the embodiment).
[0061] For the Reduce program, the input is the output result of all Map programs, that is, the sample and the corresponding label. Samples with the same label are summarized in a Reduce task, and the center of all samples under the same to-be-determined label (excluding the to-be-determined label todo and the valid label use) is calculated, and finally a new set of K centroids is obtained. That is, the valid label can be directly outputted, and the invalid label can be abandoned.
[0062] And the label reconfirmation mode for the to-be-determined label can be illustrated by the following steps.
[0063] In step S102, the log samples corresponding to all to-be-confirmed labels and the corresponding sample labels are inputted into a data integration model, and a new centroid set is outputted, wherein the new centroid set contains K new centroids, and K is a positive integer.
[0064] Optionally, the step S102 comprises: obtaining all log samples corresponding to the invalid label in the target log set to obtain an invalid log sample set; calculating a clustering center of the invalid log sample set; inputting the clustering center of the invalid log sample set, the log samples corresponding to all to-be-confirmed labels and the corresponding sample labels into the data integration model to output a new centroid set.
[0065] The embodiment defines the new centroid set as new_Cens, and at the same time, the embodiment also reads the new centroid set new_Cens in the preset distributed cache. Each Map program task reads a part of the log set of the to-be-determined label todo and outputs each sample and the corresponding label.
[0066] In step S103, the cosine distance values between the to-be-confirmed log sample corresponding to the to-be-confirmed label and each new centroid in the new centroid set are calculated, and the minimum cosine distance and the new centroid corresponding to the to-be-confirmed log sample are selected.
[0067] Optionally, before the cosine distance values between the to-be-confirmed log sample corresponding to the to-be-confirmed label and each new centroid in the new centroid set are calculated, the following steps are further included: reading all log samples corresponding to the to-be-confirmed label, and selecting part of the log samples in all the log samples as a to-be-confirmed log sample set; and taking each row of log sample in the to-be-confirmed log sample set as a to-be-confirmed log sample corresponding to the to-be-confirmed label.
[0068] For the Map program, each row sample of a part of the log set corresponding to the to-be-determined label todo is read row by row, the cosine distance between the sample and each centroid in the new centroid set new_Cens in the distributed cache is calculated, and then the minimum cosine distance dist_min and the corresponding centroid Cen_m are found out.
[0069] In step S104, based on the minimum cosine distance corresponding to the to-be-confirmed log sample and the new centroid, the sample label corresponding to the to-be-confirmed log sample is determined, wherein the sample label is one of the following: an invalid label, a valid label.
[0070] Optionally, the step of determining the sample label corresponding to the to-be-confirmed log sample based on the minimum cosine distance corresponding to the to-be-confirmed log sample and the new centroid comprises: in the case that the minimum cosine distance is less than a third preset threshold, determining that the sample label of the to-be-confirmed log sample is the invalid label; and in the case that the minimum cosine distance is greater than or equal to the third preset threshold, determining that the sample label of the to-be-confirmed log sample is the valid label.
[0071] If the minimum cosine distance dist_min is less than a third preset threshold value (defined as T3 in this embodiment), the sample is determined as an invalid log. If the minimum cosine distance dist_min is greater than or equal to the threshold value T3, the sample is labeled as valid.
[0072] It should be noted that the log sample labeled as invalid in this embodiment can be a repeated, erroneous, discarded, redundant, obsolete or insignificant log sample.
[0073] In step S105, all log samples containing the valid label are output.
[0074] Through the above embodiment, the massive logs generated under the big data platform can be screened, the invalid logs are removed from the massive logs, and all log samples containing the valid label are output.
[0075] The present application will be described below in conjunction with a specific embodiment.
[0076] For the log set to be classified, the logs in the log set log_set are divided into K classes (K is the actual number of classes) in advance. The sample centers of the K classes are taken respectively to obtain K centroids, and the centroid set is denoted as Cens.
[0077] The centroid set Cens is read and placed in the distributed cache of MapReduce, and each Map program task reads a part of the log set to be classified log_big (corresponding to the target log set described above) and outputs each sample and its corresponding label.
[0078] The specific task of the Map program is as follows:
[0079] Each row of sample in the part of log_big is read line by line, the cosine distance between the sample and each centroid in the centroid set Cens in the distributed cache is calculated, the minimum cosine distance dist_min and its corresponding centroid Cen_m are found. If the minimum cosine distance dist_min is less than a first preset threshold value T1, the sample is labeled with the centroid Cen_m (i.e., the sample is labeled as invalid), the class is divided, and the sample is regarded as an invalid log. If the minimum cosine distance dist_min is between (including) the threshold value T1 and a second preset T2 (T2>T1), the sample is labeled as todo, which will be verified later. If the minimum cosine distance dist_min is greater than the threshold value T2, the sample is regarded as a valid sample, and the sample is labeled with a valid label use.
[0080] The input of the specific code of the Map program is the log set to be classified log_set, and the output is: sample, label.
[0081] The input to the Reduce program is the output of all the Map programs, i.e., samples and their corresponding labels. Samples with the same label are aggregated into a Reduce task, which calculates the centroids of all samples under the same label (excluding the undetermined label "todo" and the valid label "use"), ultimately yielding a new set of K centroids.
[0082] The specific tasks of the Reduce program are:
[0083] For the input sample set, read the label of each sample, determine the label type, and if it is the undetermined label "todo", ignore the set and wait for the subsequent confirmation stage. If it is the valid label "use", directly output the sample and its corresponding label. If it is neither the undetermined label "todo" nor the valid label "use" (i.e., for invalid labels), calculate the centroid of the sample set under that label, and finally output the centroid of the sample under that label. Based on the centroids of all samples corresponding to the invalid label, and combining the above, a new set of K centroids is finally obtained, which can form a new centroid set new_Cens.
[0084] At this point, a portion of the valid logs have been filtered out, and the remaining portion needs to be further confirmed from the samples with the undetermined "todo" tag.
[0085] Reconfirmation phase:
[0086] The centroids of all samples corresponding to the invalid label, and the resulting set of K new centroids, are recorded as the latest centroid set, new_Cens. The latest centroid set, new_Cens, is stored in the distributed cache of MapReduce. Each Map task reads a portion of the log set containing the labels to be determined (todo), and outputs each sample and its corresponding label.
[0087] The specific tasks performed by the Map program:
[0088] Read each line of a subset of log samples corresponding to the label "todo" to be determined, calculate the cosine distance between the sample and each centroid in the centroid set "new_Cens" in the distributed cache, and find the smallest cosine distance "dist_min" and its corresponding centroid "Cen_m". If the cosine distance "dist_min" is less than the set third threshold "T3", then label the sample with centroid "Cen_m", classify it, and consider the sample as an invalid log. If the smallest cosine distance "dist_min" is greater than or equal to the threshold "T3", then consider the sample as a valid sample and label it "use".
[0089] The input of the Reduce program is the output of all Map programs, that is, samples and corresponding labels, and samples with the same label are aggregated into one reduce task, and the output contains all samples with the valid label use.
[0090] The specific task of the Reduce program is:
[0091] For the input sample set, the label of the sample set is read, the label type is judged, if it is a valid label use, the sample and the corresponding label are directly output, if it is not a valid label use, it is ignored. At this time, all valid logs are filtered out.
[0092] In the embodiment of the application, for the massive logs generated under the big data platform, in order to reduce the accumulation of platform logs and provide valuable logs for further problem analysis, a semi-supervised kmeans specific log filtering method based on MapReduce filtering is proposed, which removes invalid logs from massive logs. After using the log sample filtering method provided by the application to input a large amount of log data, a smaller amount of valid logs is obtained, which provides help for subsequent operation and maintenance personnel in analyzing problems and classifying logs.
[0093] The following will be described in detail in conjunction with another embodiment.
[0094] Embodiment two
[0095] The log sample filtering device provided in the embodiment includes a plurality of implementation units, each implementation unit corresponding to each implementation step in the above embodiment one.
[0096] Figure 2 It is a schematic diagram of an optional log sample filtering device according to an embodiment of the application, as shown in Figure 2 The real-name authentication device can include: an acquisition unit 20, an input unit 21, a calculation unit 22, a determination unit 23, and an output unit 24, wherein,
[0097] The acquisition unit 20 is configured to acquire each log sample and the corresponding sample label in the target log set, wherein the sample label is one of: an invalid label, a to-be-confirmed label, and a valid label.
[0098] The input unit 21 is configured to input the log samples corresponding to the to-be-confirmed labels and the corresponding sample labels to a data integration model, and output a new centroid set, wherein the new centroid set includes K new centroids, and K is a positive integer.
[0099] The calculation unit 22 is configured to calculate the cosine distance value between the to-be-confirmed log sample corresponding to the to-be-confirmed label and each new centroid in the new centroid set, and select the minimum cosine distance and the new centroid corresponding to the to-be-confirmed log sample.
[0100] The determination unit 23 is configured to determine a sample label corresponding to the to-be-confirmed log sample based on the minimum cosine distance corresponding to the to-be-confirmed log sample and the new centroid, wherein the sample label is one of an invalid label and a valid label.
[0101] The output unit 24 is configured to output all log samples containing the valid label.
[0102] The log sample screening device can obtain, by the acquisition unit 20, each log sample and the corresponding sample label in the target log set after receiving a to-be-screened log set, input, by the input unit 21, all log samples corresponding to the to-be-confirmed label and the corresponding sample label into the data integration model, output a new centroid set, calculate, by the calculation unit 22, the cosine distance value between the to-be-confirmed log sample corresponding to the to-be-confirmed label and each new centroid in the new centroid set, select the minimum cosine distance corresponding to the to-be-confirmed log sample and the new centroid, determine, by the determination unit 23, the sample label corresponding to the to-be-confirmed log sample based on the minimum cosine distance corresponding to the to-be-confirmed log sample and the new centroid, and finally output, by the output unit 24, all log samples containing the valid label. In this embodiment, for a large amount of log sample data, the to-be-confirmed samples can be further labeled and confirmed after initial labeling, thereby combining the MapReduce filtering algorithm to filter out all invalid logs. This screening method does not require human intervention, and the MapReduce filtering algorithm and the model clustering method can screen all log samples containing the valid label, significantly improve the sample screening speed and efficiency, and solve the technical problems of slow screening speed and low efficiency in related technologies.
[0103] Optionally, the acquisition unit includes a first calculation module configured to calculate the cosine distance between each log sample in the target log set and each initial centroid in the preset centroid set to obtain a cosine distance set, wherein the preset centroid set contains K initial centroids after sample clustering; a first selection module configured to select the minimum cosine distance in the cosine distance set and the corresponding initial centroid; and a first determination module configured to determine a sample label corresponding to the log sample based on the minimum cosine distance corresponding to the log sample and the initial centroid.
[0104] Optionally, the log sample screening device further comprises: a first receiving module, configured to receive a to-be-screened log set input by an external terminal before calculating the cosine distance between each log sample in the target log set and each centroid in the preset centroid set, and select a preset part of log samples in the to-be-screened log set, and take the selected preset part of log samples as the target log set; a clustering module, configured to perform sample clustering processing on the target log set to obtain K sample classes; and a first extracting module, configured to extract a sample center of each sample class, take the sample center as an initial centroid, and obtain the preset centroid set.
[0105] Optionally, the log sample screening device further comprises: a putting module, configured to read the preset centroid set after extracting the sample center of each sample class, taking the sample center as an initial centroid, and obtaining the preset centroid set, and put the preset centroid set into a preset distributed cache.
[0106] Optionally, the determining unit comprises: a second determining module, configured to determine that the sample label of the log sample is an invalid label in a case where the minimum cosine distance is less than a first preset threshold; a third determining module, configured to determine that the sample label of the log sample is a to-be-confirmed label in a case where the minimum cosine distance is greater than or equal to the first preset threshold and less than a second preset threshold, wherein the first preset threshold is less than the second preset threshold; and a fourth determining module, configured to determine that the sample label of the log sample is a valid label in a case where the minimum cosine distance is greater than the second preset threshold.
[0107] Optionally, the input unit comprises: a first obtaining module, configured to obtain all log samples corresponding to the invalid label in the target log set to obtain an invalid log sample set; a second calculating module, configured to calculate a clustering center of the invalid log sample set; and an input module, configured to input the clustering center of the invalid log sample set, all log samples corresponding to the to-be-confirmed label, and the corresponding sample label into a data integration model, and output a new centroid set.
[0108] Optionally, the log sample screening device further comprises: a first reading module, configured to read all log samples corresponding to the to-be-confirmed label before calculating the cosine distance value between the to-be-confirmed log sample corresponding to the to-be-confirmed label and each new centroid in the new centroid set, and select part of the log samples in the all log samples, and take the selected part of the log samples as the to-be-confirmed log sample set; and a fifth determining module, configured to take each log sample in the to-be-confirmed log sample set as the to-be-confirmed log sample corresponding to the to-be-confirmed label.
[0109] Optionally, the fifth determining module comprises: a first determining submodule, configured to determine that the sample label of the to-be-confirmed log sample is an invalid label when the minimum cosine distance is less than the third preset threshold; and a second determining submodule, configured to determine that the sample label of the to-be-confirmed log sample is a valid label when the minimum cosine distance is greater than or equal to the third preset threshold.
[0110] The log sample screening device described above can further comprise a processor and a memory, and the acquisition unit 20, the input unit 21, the calculation unit 22, the determining unit 23, and the output unit 24 are all stored in the memory as program units, and the corresponding functions are realized by the processor executing the program units stored in the memory.
[0111] The processor described above comprises a kernel, and the kernel retrieves the corresponding program units from the memory. The kernel can be one or more, and the kernel parameters are adjusted to screen the massive log samples and output all the log samples containing valid labels.
[0112] The memory described above can comprise a non-permanent memory in a computer readable medium, a random access memory (RAM), and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory comprises at least one memory chip.
[0113] The application further provides a computer program product adapted to execute the program of the following method steps when executed on a data processing device: acquiring each log sample in a target log set and a corresponding sample label, wherein the sample label is one of the following: an invalid label, a to-be-confirmed label, and a valid label; inputting all the log samples corresponding to the to-be-confirmed labels and the corresponding sample labels into a data integration model, and outputting a new centroid set, wherein the new centroid set contains K new centroids, and K is a positive integer; calculating the cosine distance value between each new centroid in the new centroid set and the to-be-confirmed log sample corresponding to the to-be-confirmed label, and selecting the minimum cosine distance corresponding to the to-be-confirmed log sample and the new centroid; determining the sample label corresponding to the to-be-confirmed log sample based on the minimum cosine distance corresponding to the to-be-confirmed log sample and the new centroid, wherein the sample label is one of the following: an invalid label and a valid label; and outputting all the log samples containing valid labels.
[0114] According to another aspect of the embodiments of the application, a computer readable storage medium is also provided, which comprises a stored computer program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to execute the log sample screening method described above when the computer program runs.
[0115] According to another aspect of the embodiments of the present application, an electronic device is also provided, which includes one or more processors and a memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the above-mentioned log sample screening method.
[0116] Figure 3 is a hardware structure block diagram of an electronic device (or a mobile device) for a log sample screening method according to an embodiment of the present application. As shown in Figure 3 , the electronic device can include one or more (as shown in Figure 3 , 302b, …, 302n) processors 302 (the processor 302 can include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 304 for storing data. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. Those skilled in the art can understand that Figure 3 , the structure shown is only schematic, and does not limit the structure of the above-mentioned electronic device. For example, the electronic device can also include more or fewer components than Figure 3 , or have a different configuration from Figure 3 .
[0117] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.
[0118] In the above-mentioned embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.
[0119] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between units or modules, which can be electrical or other forms.
[0120] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.
[0121] In addition, each functional unit in various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.
[0122] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical scheme of the present application or the part of the present application which contributes to the prior art or the whole or part of the technical scheme can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.
[0123] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.
Claims
1. A method for filtering log samples, characterized in that, include: Obtain each log sample and its corresponding sample tag from the target log set, wherein the sample tag is one of the following: invalid tag, tag to be confirmed, or valid tag; The step of obtaining each log sample and its corresponding sample label in the target log set includes: calculating the cosine distance between each log sample in the target log set and each initial centroid in a preset centroid set to obtain a cosine distance set, wherein the preset centroid set contains K initial centroids after sample clustering; selecting the minimum cosine distance and its corresponding initial centroid in the cosine distance set; and determining the sample label corresponding to the log sample based on the minimum cosine distance and the initial centroid corresponding to the log sample. Input all the log samples corresponding to the labels to be confirmed and their corresponding sample labels into the data integration model, and output a new centroid set, wherein the new centroid set contains K new centroids, and K is a positive integer; Calculate the cosine distance between the log sample to be confirmed corresponding to the label to be confirmed and each new centroid in the new centroid set, and select the minimum cosine distance and the new centroid corresponding to the log sample to be confirmed; Based on the minimum cosine distance and the new centroid corresponding to the log sample to be confirmed, the sample label corresponding to the log sample to be confirmed is determined, wherein the sample label is one of the following: invalid label or valid label; Output all the log samples containing the valid tags.
2. The method for filtering log samples according to claim 1, characterized in that, Before calculating the cosine distance between each log sample in the target log set and each centroid in the preset centroid set, the following steps are also included: Receive a set of logs to be filtered from an external terminal, select a preset portion of log samples from the set of logs to be filtered, and use the selected preset portion of log samples as the target log set; The target log set is subjected to sample clustering processing to obtain K sample classes; Extract the sample center of each sample class, and use the sample center as the initial centroid to obtain the preset centroid set.
3. The method for filtering log samples according to claim 2, characterized in that, After extracting the sample center of each sample class and using that sample center as the initial centroid to obtain the preset centroid set, the method further includes: Read the preset centroid set and put the preset centroid set into the preset distributed cache.
4. The method for filtering log samples according to claim 1, characterized in that, The step of determining the sample label corresponding to the log sample based on the minimum cosine distance and initial centroid of the log sample includes: If the minimum cosine distance is less than a first preset threshold, the sample label of the log sample is confirmed to be an invalid label; If the minimum cosine distance is greater than or equal to a first preset threshold and less than a second preset threshold, the sample label of the log sample is confirmed as a label to be confirmed, wherein the first preset threshold is less than the second preset threshold; If the minimum cosine distance is greater than the second preset threshold, the sample label of the log sample is confirmed as a valid label.
5. The method for filtering log samples according to claim 1, characterized in that, The steps of inputting all the log samples corresponding to the labels to be confirmed and their corresponding sample labels into the data integration model and outputting a new centroid set include: Obtain all log samples corresponding to invalid tags in the target log set to obtain an invalid log sample set; Calculate the cluster centers of the invalid log sample set; The cluster centers of the invalid log sample set, the log samples corresponding to all the labels to be confirmed, and the corresponding sample labels are input into the data integration model to output the new centroid set.
6. The method for filtering log samples according to claim 1, characterized in that, Before calculating the cosine distance between the log sample to be confirmed corresponding to the label to be confirmed and each new centroid in the new centroid set, the method further includes: Read all log samples corresponding to the tag to be confirmed, and select a portion of the log samples from all the log samples, and use the selected portion of the log samples as the set of log samples to be confirmed; Each log sample in the set of log samples to be confirmed is used as the log sample to be confirmed corresponding to the label to be confirmed.
7. The method for filtering log samples according to claim 1, characterized in that, The step of determining the sample label corresponding to the log sample to be confirmed based on the minimum cosine distance and the new centroid of the log sample to be confirmed includes: If the minimum cosine distance is less than a third preset threshold, the sample label of the log sample to be confirmed is determined to be an invalid label; If the minimum cosine distance is greater than or equal to a third preset threshold, the sample label of the log sample to be confirmed is determined to be a valid label.
8. A log sample screening device, characterized in that, include: The acquisition unit is used to acquire each log sample and its corresponding sample tag in the target log set, wherein the sample tag is one of the following: invalid tag, tag to be confirmed, or valid tag; The acquisition unit includes: a first calculation module, used to calculate the cosine distance between each log sample in the target log set and each initial centroid in a preset centroid set, to obtain a cosine distance set, wherein the preset centroid set contains K initial centroids after sample clustering; a first selection module, used to select the minimum cosine distance and the corresponding initial centroid in the cosine distance set; and a first determination module, used to determine the sample label corresponding to the log sample based on the minimum cosine distance and the initial centroid corresponding to the log sample. The input unit is used to input all the log samples corresponding to the tags to be confirmed and the corresponding sample tags into the data integration model and output a new centroid set, wherein the new centroid set contains K new centroids, and K is a positive integer; The calculation unit is used to calculate the cosine distance between the log sample to be confirmed corresponding to the label to be confirmed and each new centroid in the new centroid set, and to select the minimum cosine distance and the new centroid corresponding to the log sample to be confirmed. The determining unit is used to determine the sample label corresponding to the log sample to be confirmed based on the minimum cosine distance and the new centroid of the log sample to be confirmed, wherein the sample label is one of the following: invalid label or valid label; The output unit is used to output all the log samples containing the valid tags.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the log sample filtering method according to any one of claims 1 to 7.
10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the log sample filtering method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Picture deduplication method and device, storage medium and electronic device
CN110442738A