Log clustering method and device based on sample density and electronic equipment

Abstract

The invention discloses a log clustering method and device based on sample density and electronic equipment, and relates to the technical field of big data or other related technical fields, and the method comprises the steps: collecting abnormal operation logs of a financial business system in a target time period, and obtaining an abnormal operation log set; randomly sampling the abnormal operation log set to obtain a sample set; calculating the sample density of each log sample in the sample set, and obtaining a candidate centroid set based on the sample density; for each centroid candidate point in the candidate centroid set, acquiring a density neighborhood sample set of the centroid candidate point, and selecting an initial centroid corresponding to the centroid candidate point based on the density neighborhood sample set to obtain an initial centroid set; and clustering the abnormal operation log set based on the initial centroid set to obtain K types of abnormal operation log subsets. According to the method and the device, the technical problem of relatively low accuracy of a clustering result in a mode of randomly selecting the initial centroid in related technologies is solved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology or other related technical fields, and in particular to a log clustering method based on sample density, a device thereof, and an electronic device. Background Art

[0002] In the daily operations of financial institutions, log data is an important resource for system monitoring and problem diagnosis. In particular, for the use of applications, abnormal operation logs can reveal abnormal user behavior or system failures during operation, providing key fault location and solution basis for operation and maintenance personnel. However, as the scale of business continues to expand, the amount of log data is growing exponentially, which not only puts higher requirements on storage and processing capabilities, but also increases the difficulty of quickly and accurately extracting valuable information from the data.

[0003] Log clustering is one of the important means of log analysis. It aims to find similar logs from massive abnormal operation logs, analyze the abnormal problems that occur when each user uses the application or the abnormal conditions during the operation of the system, and classify the logs with the same abnormal type into a group. Then, we will dig out the causes and solutions of the abnormal conditions in different categories.

[0004] Although the log clustering method in the related technology plays an important role in data mining and information classification, it has obvious limitations in the processing of massive log data. The clustering algorithm randomly selects the initial centroid in the initialization stage. When the data distribution is uneven, this method may cause the centroid selection to be too close or too scattered, thereby affecting the convergence speed of clustering and the accuracy of the clustering results. Specifically, the random distribution of the initial centroid may cause the algorithm to require more iterations to reach a stable state, especially in the case of a large data set, which will significantly increase the processing time; if the centroid is initialized near discrete points or noise data, it will be difficult to correctly reflect the overall structure of the data, resulting in inaccurate or unsatisfactory clustering results; and as the amount of data increases, the above defects will be magnified, and the complexity of data preprocessing and clustering calculations will increase significantly, and the consumption of computing resources will also be greater.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiment of the present invention provides a log clustering method based on sample density, a device thereof, and an electronic device, so as to at least solve the technical problem in the related art that the clustering result has low accuracy when the initial centroid is randomly selected.

[0007] According to one aspect of an embodiment of the present invention, a log clustering method based on sample density is provided, comprising: collecting abnormal operation logs of a financial business system within a target time period to obtain an abnormal operation log set; randomly sampling the abnormal operation log set to obtain a sample set; calculating the sample density of each log sample in the sample set, and obtaining a candidate centroid set based on the sample density, wherein the candidate centroid set includes K centroid candidate points, K is the number of preset initial centroids, and K is a positive integer; for each centroid candidate point in the candidate centroid set, obtaining a density neighborhood sample set of the centroid candidate point, and selecting the initial centroid corresponding to the centroid candidate point based on the density neighborhood sample set to obtain an initial centroid set, wherein the initial centroid set includes K initial centroids; clustering the abnormal operation log set based on the initial centroid set to obtain K categories of abnormal operation log subsets.

[0008] Optionally, the step of calculating the sample density of each log sample in the sample set includes: taking each of the log samples in the sample set as a target log sample in turn, calculating the cosine distance between the target log sample and other log samples in the sample set, wherein the other log samples are the remaining log samples in the sample set except the target log sample; taking the other log samples whose cosine distance is less than a threshold radius as associated log samples of the target log sample, wherein the threshold radius is pre-set; counting the number of associated log samples of the target log sample; accumulating the cosine distances between the target log sample and all the associated log samples to obtain the sum of the cosine distances between the target log sample and all the associated log samples; calculating the ratio of the number of associated log samples to the sum of the cosine distances to obtain the sample density of the target log sample, thereby obtaining the sample density of each log sample in the sample set.

[0009] Optionally, the step of obtaining a set of candidate centroids based on the sample density includes: sorting the log samples in the sample set based on the sample density of each log sample in the sample set to obtain an ordered sample set; setting the number of the initial centroids to be selected; selecting a first subset of candidate centroids and a second subset of candidate centroids from the ordered sample set based on the number of the initial centroids; and obtaining the set of candidate centroids based on the first subset of candidate centroids and the second subset of candidate centroids.

[0010] Optionally, the step of selecting a first candidate centroid subset from the ordered sample set based on the number of initial centroids includes: step one, calculating the number of centroid candidate points in the first candidate centroid subset based on the number of initial centroids to obtain a first target number; step two, selecting the first log sample in the ordered sample set as the first centroid candidate point of the first candidate centroid subset; step three, for the remaining centroid candidate points in the first candidate centroid subset, calculating the position information of the i-th centroid candidate point based on a first position calculation formula, and selecting the i-th centroid candidate point of the first candidate centroid subset from the ordered sample set based on the position information; repeating step three until the number of centroid candidate points in the first candidate centroid subset meets the first target number.

[0011] Optionally, the first position calculation formula is expressed as: [A i ]=2ni / 3K, where [A i ] represents the position information of the i-th candidate centroid point in the first candidate centroid subset, [] represents rounding up to an integer when the value is not an integer, n represents the total number of log samples in the sample set, i represents the i-th candidate centroid point in the first candidate centroid subset, which is a positive integer, and K represents the number of initial centroids.

[0012] Optionally, the step of selecting a second candidate centroid subset from the ordered sample set based on the number of initial centroids includes: step one, calculating the number of centroid candidate points in the first candidate centroid subset based on the number of initial centroids to obtain a second target number; step two, selecting the first centroid candidate point of the second candidate centroid subset from the ordered sample set based on the total number of log samples in the sample set; step three, for the remaining centroid candidate points in the second candidate centroid subset, calculating the position information of the i-th centroid candidate point based on a second position calculation formula, and selecting the i-th centroid candidate point of the second candidate centroid subset from the ordered sample set based on the position information; repeat step three until the number of centroid candidate points in the second candidate centroid subset meets the second target number.

[0013] Optionally, the second position calculation formula is expressed as: [B i ]=n / 3+2(2i-1)n / 3K, where [B i ] represents the position information of the i-th candidate centroid point in the second candidate centroid subset, [] represents rounding up to an integer when the value is not an integer, n represents the total number of log samples in the sample set, i represents the i-th candidate centroid point in the second candidate centroid subset, which is a positive integer, and K represents the number of initial centroids.

[0014] Optionally, for each centroid candidate point in the candidate centroid set, a density neighborhood sample set of the centroid candidate point is obtained, and the step of selecting the initial centroid corresponding to the centroid candidate point based on the density neighborhood sample set includes: step one, calculating the sample density difference between the centroid candidate point and other ordered log samples in the ordered sample set, and selecting log samples whose sample density difference is less than a preset density difference threshold, to obtain the density neighborhood sample set of the centroid candidate point, wherein the other ordered log samples are the log samples remaining in the ordered sample set except the centroid candidate point; step two, Each of the log samples in the density neighborhood sample set of the centroid candidate point is taken as the target log sample in turn, and the sum of the cosine distances between the target log sample and other neighborhood log samples in the density neighborhood sample set is calculated to obtain the sum of the cosine distances of the target log sample, wherein the other neighborhood log samples are the remaining log samples in the density neighborhood sample set except the target log sample; step three, selecting the target log sample with the smallest sum of the cosine distance as the initial centroid corresponding to the centroid candidate point; repeating the above steps one to three to obtain the initial centroids corresponding to all the centroid candidate points.

[0015] According to another aspect of an embodiment of the present invention, a log clustering device based on sample density is also provided, including: a collection unit, used to collect abnormal operation logs of a financial business system within a target time period to obtain an abnormal operation log set; a sampling unit, used to randomly sample the abnormal operation log set to obtain a sample set; a calculation unit, used to calculate the sample density of each log sample in the sample set, and obtain a candidate centroid set based on the sample density, wherein the candidate centroid set includes K centroid candidate points, K is the number of preset initial centroids, and K is a positive integer; a selection unit, used to obtain a density neighborhood sample set of the centroid candidate point for each centroid candidate point in the candidate centroid set, and select the initial centroid corresponding to the centroid candidate point based on the density neighborhood sample set to obtain an initial centroid set, wherein the initial centroid set includes K initial centroids; a clustering unit, used to cluster the abnormal operation log set based on the initial centroid set to obtain K categories of abnormal operation log subsets.

[0016] Optionally, the calculation unit includes: a first calculation module, used to take each of the log samples in the sample set as a target log sample in turn, and calculate the cosine distance between the target log sample and other log samples in the sample set, wherein the other log samples are the remaining log samples in the sample set except the target log sample; a first as module, used to take the other log samples whose cosine distance is less than a threshold radius as associated log samples of the target log sample, wherein the threshold radius is pre-set; a first statistical module, used to count the number of associated log samples of the target log sample; a first accumulation module, used to accumulate the cosine distances between the target log sample and all the associated log samples to obtain the sum of the cosine distances between the target log sample and all the associated log samples; a second calculation module, used to calculate the ratio of the number of associated log samples to the sum of the cosine distances to obtain the sample density of the target log sample, thereby obtaining the sample density of each log sample in the sample set.

[0017] Optionally, the calculation unit includes: a first sorting module, used to sort the log samples in the sample set based on the sample density of each log sample in the sample set to obtain an ordered sample set; a first setting module, used to set the number of the initial centroids to be selected; a first selection module, used to select a first candidate centroid subset and a second candidate centroid subset from the ordered sample set based on the number of the initial centroids; a first acquisition module, used to obtain the candidate centroid set based on the first candidate centroid subset and the second candidate centroid subset.

[0018] Optionally, the first selection module includes: a first calculation submodule, used to execute step one, calculate the number of the centroid candidate points in the first candidate centroid subset based on the number of the initial centroids, and obtain a first target number; a first selection submodule, used to execute step two, select the first log sample in the ordered sample set as the first centroid candidate point of the first candidate centroid subset; a second calculation submodule, used to execute step three, for the remaining centroid candidate points in the first candidate centroid subset, calculate the position information of the i-th centroid candidate point based on a first position calculation formula, and select the i-th centroid candidate point of the first candidate centroid subset from the ordered sample set based on the position information; a first repetition submodule, used to repeatedly execute step three until the number of centroid candidate points in the first candidate centroid subset meets the first target number.

[0019] Optionally, the first position calculation formula is expressed as: [A i ]=2ni / 3K, where [A i] represents the position information of the i-th candidate centroid point in the first candidate centroid subset, [] represents rounding up to an integer when the value is not an integer, n represents the total number of log samples in the sample set, i represents the i-th candidate centroid point in the first candidate centroid subset, which is a positive integer, and K represents the number of initial centroids.

[0020] Optionally, the first selection module includes: a third calculation submodule, used to execute step one, calculate the number of the centroid candidate points in the first candidate centroid subset based on the number of the initial centroids, and obtain a second target number; a second selection submodule, used to execute step two, select the first centroid candidate point of the second candidate centroid subset from the ordered sample set based on the total number of log samples in the sample set; a fourth calculation submodule, used to execute step three, for the remaining centroid candidate points in the second candidate centroid subset, calculate the position information of the i-th centroid candidate point based on a second position calculation formula, and select the i-th centroid candidate point of the second candidate centroid subset from the ordered sample set based on the position information; a second repetition submodule, used to repeatedly execute step three until the number of centroid candidate points in the second candidate centroid subset meets the second target number.

[0021] Optionally, the second position calculation formula is expressed as: [B i ]=n / 3+2(2i-1)n / 3K, where [B i ] represents the position information of the i-th candidate centroid point in the second candidate centroid subset, [] represents rounding up to an integer when the value is not an integer, n represents the total number of log samples in the sample set, i represents the i-th candidate centroid point in the second candidate centroid subset, which is a positive integer, and K represents the number of initial centroids.

[0022] Optionally, the selection unit includes: a second selection module, used to execute step one, calculate the sample density difference between the centroid candidate point and other ordered log samples in the ordered sample set, and select log samples whose sample density difference is less than a preset density difference threshold to obtain a density neighborhood sample set of the centroid candidate point, wherein the other ordered log samples are the log samples remaining in the ordered sample set except the centroid candidate point; a third calculation module, used to execute step two, take each of the log samples in the density neighborhood sample set of the centroid candidate point as a target log sample in turn, calculate the sum of cosine distances between the target log sample and other neighborhood log samples in the density neighborhood sample set, and obtain the sum of cosine distances of the target log sample, wherein the other neighborhood log samples are the remaining log samples in the density neighborhood sample set except the target log sample; a third selection module, used to execute step three, select the target log sample with the smallest sum of cosine distances as the initial centroid corresponding to the centroid candidate point; a first repetition module, used to repeat steps one to three above to obtain the initial centroids corresponding to all the centroid candidate points.

[0023] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned log clustering methods based on sample density.

[0024] In the present application, the following steps are performed: abnormal operation logs of a financial business system within a target time period are collected to obtain an abnormal operation log set, and the abnormal operation log set is randomly sampled to obtain a sample set, the sample density of each log sample in the sample set is calculated, and a candidate centroid set is obtained based on the sample density, wherein the candidate centroid set contains K centroid candidate points, K is the number of preset initial centroids, and K is a positive integer, and then for each centroid candidate point in the candidate centroid set, a density neighborhood sample set of the centroid candidate point is obtained, and the initial centroid corresponding to the centroid candidate point is selected based on the density neighborhood sample set to obtain an initial centroid set, wherein the initial centroid set contains K initial centroids, and finally the abnormal operation log set is clustered based on the initial centroid set to obtain K categories of abnormal operation log subsets.

[0025] In this application, for the clustering of abnormal operation logs, a sample set for obtaining the initial centroid is first constructed, and the sample density of each log sample in the sample set is calculated. The candidate centroid point is selected according to the sample density, and then the initial centroid is selected from the density neighborhood set of the candidate centroid point to obtain multiple initial centroids for log clustering. Cluster analysis of abnormal operation logs is performed based on the initial centroid, and efficient and accurate clustering of abnormal operation logs of financial business systems is achieved, achieving the purpose of accurately selecting the initial centroid, accelerating the convergence speed and convergence effect of the clustering algorithm, and achieving the technical effect of improving the accuracy of clustering results. This solves the technical problem in the related art that the method of randomly selecting the initial centroid has low clustering result accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0027] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a log clustering method based on sample density is shown;

[0028] Figure 2 is a flowchart of an optional log clustering method based on sample density according to an embodiment of the present invention;

[0029] Figure 3 is a schematic diagram of an optional log clustering process based on sample density according to an embodiment of the present invention;

[0030] Figure 4 is a schematic diagram of an optional log clustering device based on sample density according to an embodiment of the present invention;

[0031] Figure 5 The hardware structure block diagram of an optional electronic device (or mobile device) for executing a log clustering method based on sample density according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] It should be noted that the sample density-based log clustering method and device thereof in the present application can be used in the field of big data technology when the initial centroid is accurately selected based on the sample density and log clustering is performed, and can also be used in any field other than the field of big data technology when the initial centroid is accurately selected based on the sample density and log clustering is performed. The application does not limit the application field of the sample density-based log clustering method and device thereof.

[0035] It should be noted that the collected information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with relevant laws, regulations and standards, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation entrances for users to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, the expert decision-making process will be entered.

[0036] The following embodiments of the present invention can be applied to various log clustering systems / applications / devices based on sample density. In view of the problems that the existing clustering methods have slow convergence speed and poor clustering effect when clustering a large number of logs, the present invention proposes a log clustering method based on sample density to accurately screen the initial centroid, thereby improving the accuracy of the clustering results.

[0037] The present invention is described in detail below in conjunction with various embodiments.

[0038] Embodiment 1

[0039] According to an embodiment of the present invention, an embodiment of a log clustering method based on sample density is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a log clustering method based on sample density. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0041] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0042] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the log clustering method based on sample density in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the log clustering method based on sample density is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0043] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0044] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0045] Under the above operating environment, this application provides Figure 2 The log clustering method based on sample density is shown. The implementation body of this method is a log clustering system based on sample density. This method can be applied to cluster abnormal operation logs of financial business systems, determine the causes of abnormal operation, and formulate application scenarios of abnormal resolution strategies.

[0046] Figure 2 is a flowchart of an optional log clustering method based on sample density according to an embodiment of the present invention, such as Figure 2 As shown, the method comprises the following steps:

[0047] Step S201, collecting abnormal operation logs of the financial business system within a target time period to obtain an abnormal operation log set.

[0048] It should be noted that during the operation of the financial business system, many abnormal situations are inevitable. The operation log can be used as a basis for tracing back the system anomalies and for analyzing the causes of the system anomalies. When the number of operation log files is large, manual analysis is inefficient and prone to anomalies. Therefore, clustering algorithms become one of the ways to analyze logs. The clustering algorithm updates the centroid through continuous iteration, thereby classifying a large number of abnormal operation logs, so that logs with similar anomalies are grouped together, thereby facilitating anomaly identification and formulating anomaly resolution strategies. In addition to being applied to financial business systems, the embodiments of the present invention can also be applied to log clustering analysis with other systems or applications.

[0049] It should be noted that when performing a clustering algorithm, the centroid is the key to the clustering algorithm. Before the clustering algorithm is iterated, the initial centroid needs to be selected. Generally, the initial centroid is selected randomly. There are many disadvantages in randomly selecting the initial centroid. On the one hand, the initial centroid is selected too close or scattered, not uniform enough, which will lead to slow convergence of the algorithm. On the other hand, if the initial centroid is selected as a discrete point or noise data, it is not conducive to the convergence of subsequent clustering, resulting in unsatisfactory clustering effect of the algorithm. Therefore, the selection of the initial centroid becomes one of the difficulties in improving clustering efficiency and clustering effect.

[0050] In response to the above-mentioned problems, an embodiment of the present invention proposes a technical solution for selecting high-quality initial centroids based on sample density. Specifically, when a financial business system is running, the financial business system is monitored in real time, real-time operation logs of the system are collected, and abnormal operation logs are identified through an abnormality recognition algorithm to obtain an abnormal log set. Cluster analysis is performed on the abnormal operation logs to obtain multiple abnormal groups, which facilitates the execution of solution strategies for similar abnormal situations.

[0051] Step S202: Randomly sample the abnormal operation log set to obtain a sample set.

[0052] It should be noted that when selecting the initial centroid, a representative sample set is first constructed from the abnormal operation log set by random sampling. The size of this sample set can be 15% of the abnormal operation log set. When sampling, ensure that the sample selection is uniform and random, avoid the introduction of any bias or systematic bias, and ensure that the sample set can fully and accurately reflect the main characteristics and distribution status of the abnormal operation log. The random sampling step is particularly important in big data analysis, especially when the abnormal operation log set is large in size. Directly processing all logs is not only computationally expensive, but also inefficient. By constructing a small but representative sample set, it is possible to reduce the amount of data processing while maintaining effective identification and analysis of abnormal patterns, optimize the use of computing resources, speed up the operation of the clustering algorithm, and reduce the requirements for storage and computing performance. Constructing a sample set can ensure the accuracy and stability of the clustering results. Even in the face of massive data, it can quickly locate and classify abnormal operation patterns, providing strong data support for the operation and maintenance, fault diagnosis and business optimization of financial institutions.

[0053] Step S203, calculating the sample density of each log sample in the sample set, and obtaining a candidate centroid set based on the sample density.

[0054] It should be noted that the embodiment of the present invention defines the concept of sample density for the log samples in the sample set. The sample density quantifies the combined impact of the number of samples and spatial distribution in the neighborhood of the log samples, and can effectively reflect the relative importance and density of the log samples in the sample set. According to the sample density, the centroid candidate points can be preliminarily screened to obtain a candidate centroid set, which includes K centroid candidate points, where K is the number of pre-set initial centroids and K is a positive integer.

[0055] Optionally, the step of calculating the sample density of each log sample in the sample set includes: taking each log sample in the sample set as a target log sample in turn, calculating the cosine distance between the target log sample and other log samples in the sample set, wherein the other log samples are the remaining log samples in the sample set except the target log sample; taking other log samples whose cosine distance is less than a threshold radius as associated log samples of the target log sample, wherein the threshold radius is pre-set; counting the number of associated log samples of the target log sample; accumulating the cosine distances between the target log sample and all associated log samples to obtain the sum of the cosine distances between the target log sample and all associated log samples; calculating the ratio of the number of associated log samples to the sum of the cosine distances to obtain the sample density of the target log sample, thereby obtaining the sample density of each log sample in the sample set.

[0056] It should be noted that the sample density is defined as the number of samples within the threshold radius Tr of the log sample X divided by the cumulative cosine distance between the relevant samples within the threshold radius Tr of the log sample X and the sample X. The threshold radius Tr is a preset positive number used to define the neighborhood range of sample X to ensure that the samples considered in the density calculation are samples that are closer to X.

[0057] Specifically, each log sample in the sample set is defined as a vector, where the dimensions of the vector may include features such as the timestamp, error code, and operation context of the log. Each log sample in the sample set is taken as the target log sample in turn, and the cosine distance between the target log sample and other log samples (the remaining log samples in the sample set except the target log sample) is calculated. The cosine distance is used to measure text similarity, and the degree of similarity in the feature space is expressed by comparing the angle between two vectors. Based on the pre-set threshold radius Tr, other log samples with a cosine distance less than Tr are identified as associated log samples of the target log sample. By screening other log samples that are highly similar to the target log sample in terms of features, it is determined which samples are closely related to the target sample in the log space. The selection of the threshold radius Tr needs to be adjusted according to the feature distribution of the log data and the expected clustering effect to ensure that the associated samples are not over- or under-identified. For each target log sample, the number of all its associated log samples is counted, which directly reflects the "proximity" or "clustering" of the target sample in the log sample set. The cosine distances between the target log sample and all associated log samples are accumulated to obtain a metric representing the overall similarity between log samples. Finally, the ratio of the number of associated log samples of the target log sample to the sum of the cosine distances between the target log sample and all associated log samples is calculated to obtain the sample density of the target log sample, which serves as the basis for subsequent screening of the initial centroid.

[0058] Through the above steps, the sample density of each log sample in the sample set can be accurately calculated, which provides a key basis for the selection of the initial centroid for the subsequent log clustering method based on density screening of the minimum distance. Compared with the traditional random centroid selection, this implementation method can quantify the log samples by calculating the sample density, thereby accurately screening out high-quality initial centroids, improving the convergence speed of the clustering algorithm, and optimizing the accuracy and stability of the clustering results.

[0059] Optionally, the step of obtaining a set of candidate centroids based on sample density includes: sorting the log samples in the sample set based on the sample density of each log sample in the sample set to obtain an ordered sample set; setting the number of initial centroids to be selected; selecting a first subset of candidate centroids and a second subset of candidate centroids from the ordered sample set based on the number of initial centroids; and obtaining a set of candidate centroids based on the first subset of candidate centroids and the second subset of candidate centroids.

[0060] It should be noted that the embodiment of the present invention is based on the centroid candidate points and selects the log sample with the optimal value in the density field space of the centroid candidate points as the initial centroid. Therefore, it is necessary to first determine the centroid candidate points of the sample set. The determination of the centroid candidate points includes two parts, one part is the determination of the first half of the centroid candidate points, that is, the first candidate centroid subset, and the other part is the determination of the latter general centroid candidate points, that is, the second candidate centroid subset. The hierarchical and diversified selection of candidate centroids effectively overcomes the defects of traditional clustering algorithms in the random selection of initial centroids.

[0061] Specifically, when selecting candidate centroid points, the log samples in the sample set are first sorted based on the sample density of each log sample in the sample set to construct an ordered sample set, and the number K of initial centroids to be selected is determined. This is pre-set according to the goal of cluster analysis. The selection of the number of centroids K directly affects the number of final clustering groups, as well as the accuracy and practicality of the clustering results. The number of candidate centroid points is the same as the number of initial centroids. According to the number of initial centroids K, the selection of candidate centroid points is divided into the first half and the second half. The basis for the division is the division value Q obtained by rounding up K / 2. The first candidate centroid subset is obtained based on the first Q centroid candidate points, and the second candidate centroid subset is constructed based on the last KQ centroid candidate points from Q to K, and the first candidate centroid subset and the second candidate centroid subset are merged to obtain the candidate centroid subset.

[0062] Optionally, the step of selecting a first candidate centroid subset from the ordered sample set based on the number of initial centroids includes: step one, calculating the number of centroid candidate points in the first candidate centroid subset based on the number of initial centroids to obtain a first target number; step two, selecting the first log sample in the ordered sample set as the first centroid candidate point in the first candidate centroid subset; step three, for the remaining centroid candidate points in the first candidate centroid subset, calculating the position information of the i-th centroid candidate point based on the first position calculation formula, and selecting the i-th centroid candidate point of the first candidate centroid subset from the ordered sample set based on the position information; repeating step three until the number of centroid candidate points in the first candidate centroid subset meets the first target number.

[0063] Specifically, for the selection of the first Q (i.e., the first target number) candidate centroid points, the first log sample of the ordered sample is used as the first centroid candidate point, and starting from the second candidate centroid point, the candidate centroid points are determined by calculating the position information.

[0064] Optionally, the first position calculation formula is expressed as: [A i ]=2ni / 3K, where [A i] represents the position information of the i+1th centroid candidate point in the first candidate centroid subset, [] indicates that the value is rounded up to an integer when it is not an integer, n represents the total number of log samples in the sample set, i represents the i-th centroid candidate point in the first candidate centroid subset, which is a positive integer, and K represents the number of initial centroids.

[0065] Specifically, the first sample in the ordered sample set is selected as the first centroid candidate point Cd 1 , starting from the second centroid candidate point, the selection method is [A i ]=2ni / 3K(i is a positive integer starting from 1, [] represents A i If it is not an integer, round it up. i Represents the candidate centroid point Cd i+1 (i.e., the position of the i+1th centroid candidate point in the first candidate centroid subset) in the ordered sample set, i.e., A1 represents the second centroid candidate point Cd 2 The position in the ordered sample set, that is, the A1th sample in the ordered sample set is the second centroid candidate point Cd 2 , where n is the total number of log samples in the sample set. Similarly, this continues until the Qth centroid candidate point Cd Q .

[0066] Optionally, the step of selecting a second candidate centroid subset from the ordered sample set based on the number of initial centroids includes: step one, calculating the number of centroid candidate points in the first candidate centroid subset based on the number of initial centroids to obtain a second target number; step two, selecting the first centroid candidate point of the second candidate centroid subset from the ordered sample set based on the total number of log samples in the sample set; step three, for the remaining centroid candidate points in the second candidate centroid subset, calculating the position information of the i-th centroid candidate point based on the second position calculation formula, and selecting the i-th centroid candidate point of the second candidate centroid subset from the ordered sample set based on the position information; repeat step three until the number of centroid candidate points in the second candidate centroid subset meets the second target number.

[0067] It should be noted that the selection of the remaining centroid candidate points after Q (i.e., the second target number) is different from the selection mechanism of the first half of the centroid candidate points. First, the Mth sample in the ordered sample set is selected as the first centroid candidate point of the second candidate centroid subset, M = [n / 2]. Starting from the second centroid candidate point, selection is performed by calculating the position information until KQ centroid candidate points are obtained, thereby constructing the second candidate centroid set.

[0068] Optionally, the second position calculation formula is expressed as: [B i ]=n / 3+2(2i-1)n / 3K, where [B i] represents the position information of the i+1th centroid candidate point in the second candidate centroid subset, [] indicates that the value is rounded up to an integer when it is not an integer, n represents the total number of log samples in the sample set, i represents the i-th centroid candidate point in the second candidate centroid subset, which is a positive integer, and K represents the number of initial centroids.

[0069] Specifically, starting from the Q+1th centroid candidate point in the candidate centroid set, the centroid selection method is to first select the Mth (n / 2 rounded up)th sample in the sample set as the first centroid candidate point Cd in the second candidate centroid subset. 1 , from the next centroid candidate point Cd 2 , start, select [B i ]=n / 3+2(2i-1)n / 3K(i is a positive integer starting from 1, [] represents B i If it is not an integer, round it up. i Represents the candidate centroid point Cd i+1 The position in the ordered sample set, B 1 Represents the candidate centroid point Cd 2 The position in the ordered sample set, that is, the B1th sample in the ordered sample set is the centroid candidate point Cd 2 Similarly, the KQth centroid candidate point Cd is selected. K-Q , and obtain the second candidate centroid subset.

[0070] The first candidate centroid subset obtained above is merged with the second candidate centroid subset to obtain a candidate centroid set, which contains Cd 1 、Cd 2 …Cd Q 、Cd Q+1 (i.e., Cd in the second candidate centroid subset 1 ), Cd Q+2 …Cd K .

[0071] Step S204, for each centroid candidate point in the candidate centroid set, obtain a density neighborhood sample set of the centroid candidate point, and select an initial centroid corresponding to the centroid candidate point based on the density neighborhood sample set to obtain an initial centroid set.

[0072] It should be noted that after calculating the K centroid candidates, for each centroid candidate point, the corresponding density neighborhood sample set is obtained, and the initial known centroid corresponding to the centroid candidate point is selected from the domain sample set, thereby obtaining K initial centroids, thereby constructing the initial centroid for log clustering. Further optimization of the candidate centroid set is achieved, ensuring that the initial centroid C of each centroid candidate point Cd not only has high density, but is also located at the center of its neighborhood samples, thereby significantly improving the clustering efficiency and clustering effect of the log clustering algorithm.

[0073] Optionally, for each centroid candidate point in the candidate centroid set, a density neighborhood sample set of the centroid candidate point is obtained, and the step of selecting an initial centroid corresponding to the centroid candidate point based on the density neighborhood sample set includes: step one, calculating the sample density difference between the centroid candidate point and other ordered log samples in the ordered sample set, and selecting log samples whose sample density difference is less than a preset density difference threshold, to obtain a density neighborhood sample set of the centroid candidate point, wherein the other ordered log samples are the remaining log samples in the ordered sample set except the centroid candidate point; step two, taking each log sample in the density neighborhood sample set of the centroid candidate point as a target log sample in turn, calculating the sum of cosine distances between the target log sample and other neighborhood log samples in the density neighborhood sample set, to obtain the sum of cosine distances of the target log sample, wherein the other neighborhood log samples are the remaining log samples in the density neighborhood sample set except the target log sample; step three, selecting the target log sample with the smallest sum of cosine distances as the initial centroid corresponding to the centroid candidate point; repeating the above steps one to three to obtain the initial centroids corresponding to all centroid candidate points.

[0074] Specifically, when calculating the density neighborhood sample set of the centroid candidate point, multiple sample logs whose density difference near the centroid candidate point is less than a preset threshold are selected as the density neighborhood sample set of the centroid candidate point, with the centroid candidate point Cd 1 Taking the density neighborhood sample set data_N_d1 as an example, in the ordered sample set, the centroid candidate point Cd 1 Neighbor samples of the nearby threshold Tn (since the centroid candidate points have been sorted based on the sample density in the ordered sample set, a certain number of centroid candidate points before and after the centroid candidate point can be directly selected from the ordered sample set as its density neighborhood sample set. Assume that the centroid candidate point Cd 1 In the ordered sample set, the ranking is rank1, then the neighbor samples near the threshold Tn are the log samples ranked rank1-Tn, ..., rank1-2, rank1-1, rank1+1, rank1+2, ..., rank1+Tn). The set of these qualified neighbor samples is the centroid candidate point Cd 1 The density neighborhood sample set data_N_d1.

[0075] Similarly, the remaining centroid candidate points Cd are obtained in this way 1 、Cd 2 …Cd K The density neighborhood sample set data_N_d2, data_N_d3,..., data_N_dK-1, data_N_dK.

[0076] Then, the initial centroid is selected from the density neighborhood sample set of each centroid candidate point, including:

[0077] Select the centroid C from the density neighborhood sample set data_N_d1 1 , calculate the sum of the cosine distances between each sample in data_N_d1 and all other remaining samples in data_N_d1, and find the sample with the smallest sum of cosine distances with other samples in data_N_d1 as the candidate centroid point Cd 1 The corresponding initial centroid C 1 ;

[0078] Similarly, select the centroid C from the density neighborhood sample set data_N_d2 2 , calculate the sum of the cosine distances between each sample in data_N_d2 and all other remaining samples in data_N_d2, and find the sample with the smallest sum of cosine distances with other samples in data_N_d2 as the candidate centroid point Cd 2 The corresponding initial centroid C 2 ;

[0079] ...(and so on)

[0080] Select the centroid C from the density neighborhood sample set data_N_dK K , calculate the sum of the cosine distances between each sample in data_N_dK and all other remaining samples in data_N_dK, and find the sample with the smallest sum of cosine distances with other samples in data_N_dK as the candidate centroid point Cd K The corresponding initial centroid C K .

[0081] At this point, we obtain K initial centroids, C 1 , C 2 , ..., C K-1 , C K .

[0082] Step S205 , clustering the abnormal operation log set based on the initial centroid set to obtain K categories of abnormal operation log subsets.

[0083] It should be noted that, using the initial centroid set generated by the above steps, for each log sample in the abnormal operation log set, the cosine distance between it and each centroid point in the initial centroid set is calculated, and the log sample is assigned to the category where the centroid point with the smallest cosine distance is located. Once all log samples are assigned, the centroid point is recalculated according to the current member log samples of each category, that is, the position of the centroid point is updated by calculating the average feature vector of all log samples in each category. Through iterative optimization, the log samples in each category are made as similar as possible, and the differences between different categories are maximized, so as to realize clustering analysis of the abnormal operation log set, and repeat the iterative allocation and update steps until the position change of the centroid point is less than the preset threshold or the maximum number of iterations is reached. The convergence of the clustering algorithm is ensured and unlimited iteration is avoided. Finally, K categories of abnormal operation log subsets are output, each of which contains the log samples closest to a specific centroid point, that is, log samples with similar abnormal characteristics.

[0084] The log clustering method based on sample density screening of minimum distance can perform fast and accurate clustering analysis on abnormal operation log sets, and obtain K abnormal operation log subsets with clear characteristics, which significantly improves the convergence speed of the clustering algorithm and the stability of the clustering effect, especially when processing massive log data.

[0085] Through the above steps, the abnormal operation logs of the financial business system within the target time period are collected to obtain an abnormal operation log set, and the abnormal operation log set is randomly sampled to obtain a sample set, and the sample density of each log sample in the sample set is calculated. Based on the sample density, a candidate centroid set is obtained, wherein the candidate centroid set contains K centroid candidate points, K is the number of preset initial centroids, and K is a positive integer. Then, for each centroid candidate point in the candidate centroid set, a density neighborhood sample set of the centroid candidate point is obtained, and the initial centroid corresponding to the centroid candidate point is selected based on the density neighborhood sample set to obtain an initial centroid set, wherein the initial centroid set contains K initial centroids. Finally, the abnormal operation log set is clustered based on the initial centroid set to obtain K categories of abnormal operation log subsets.

[0086] In this embodiment, for clustering of abnormal operation logs, a sample set for obtaining the initial centroid is first constructed, and the sample density of each log sample in the sample set is calculated. The candidate centroid point is selected according to the sample density, and then the initial centroid is selected from the density neighborhood set of the candidate centroid point to obtain multiple initial centroids for log clustering. Cluster analysis of abnormal operation logs is performed based on the initial centroid, and efficient and accurate clustering of abnormal operation logs of financial business systems within the target time period is achieved, achieving the purpose of accurately selecting the initial centroid, accelerating the convergence speed and convergence effect of the clustering algorithm, and achieving the technical effect of improving the accuracy of clustering results. This solves the technical problem in the related art that the method of randomly selecting the initial centroid has low clustering result accuracy.

[0087] Another optional specific implementation is described in detail below.

[0088] Figure 3 is a schematic diagram of an optional log clustering process based on sample density according to an embodiment of the present invention, such as Figure 3 As shown in the figure, the process of log clustering based on sample density specifically includes:

[0089] Step 1: Start by selecting a sample set for constructing an initial centroid from the initial abnormal operation log set;

[0090] Step 2: Calculate the sample density of each log sample in the sample set;

[0091] Step 3, sort the sample set according to the sample density to obtain an ordered sample set;

[0092] Step 4: Select the first half of the centroid candidate points to obtain the first candidate centroid subset;

[0093] Step 5, select the second half of the centroid candidate points to obtain the second candidate centroid subset;

[0094] Step 6, obtaining a density neighborhood sample set of each centroid candidate point;

[0095] Step 7, selecting the initial centroid corresponding to the centroid candidate point from the density neighborhood sample set;

[0096] Step 8, obtaining K initial centroids, iterating the clustering algorithm based on the K initial centroids, and dividing the original abnormal operation log set into K categories of abnormal operation log subsets;

[0097] Step nine, end.

[0098] In view of the problems that the existing clustering methods have slow convergence speed and poor clustering effect when clustering a large number of logs, the embodiment of the present invention proposes a log clustering method for accurately screening initial centroids based on sample density, thereby improving the accuracy of clustering results.

[0099] The following is a detailed description in conjunction with another embodiment.

[0100] Embodiment 2

[0101] The log clustering method based on sample density provided in this embodiment includes multiple implementation units, each of which corresponds to each implementation step in the above-mentioned embodiment 1. The specific implementation methods and beneficial effects thereof can be referred to the above-mentioned method embodiments, which will not be described in detail here.

[0102] Figure 4 is a schematic diagram of an optional log clustering device based on sample density according to an embodiment of the present invention, such as Figure 4 As shown, the log clustering device based on sample density may include: a collection unit 41, a sampling unit 42, a calculation unit 43, a selection unit 44, and a clustering unit 45, wherein:

[0103] A collection unit 41 is used to collect abnormal operation logs of the financial business system within a target time period to obtain an abnormal operation log set;

[0104] The sampling unit 42 is used to randomly sample the abnormal operation log set to obtain a sample set;

[0105] The calculation unit 43 is used to calculate the sample density of each log sample in the sample set, and obtain a candidate centroid set based on the sample density, wherein the candidate centroid set includes K centroid candidate points, K is the number of preset initial centroids, and K is a positive integer;

[0106] The selection unit 44 is used to obtain a density neighborhood sample set of each centroid candidate point in the candidate centroid set, and select an initial centroid corresponding to the centroid candidate point based on the density neighborhood sample set to obtain an initial centroid set, wherein the initial centroid set includes K initial centroids;

[0107] The clustering unit 45 is used to cluster the abnormal operation log set based on the initial centroid set to obtain K categories of abnormal operation log subsets.

[0108] The above-mentioned log clustering device based on sample density collects abnormal operation logs of the financial business system within the target time period through the collection unit 41 to obtain an abnormal operation log set; randomly samples the abnormal operation log set through the sampling unit 42 to obtain a sample set; calculates the sample density of each log sample in the sample set through the calculation unit 43, and obtains a candidate centroid set based on the sample density, wherein the candidate centroid set contains K centroid candidate points, K is the number of pre-set initial centroids, and K is a positive integer; obtains a density neighborhood sample set of the centroid candidate point for each centroid candidate point in the candidate centroid set through the selection unit 44, and selects the initial centroid corresponding to the centroid candidate point based on the density neighborhood sample set to obtain an initial centroid set, wherein the initial centroid set contains K initial centroids; clusters the abnormal operation log set based on the initial centroid set through the clustering unit 45 to obtain K categories of abnormal operation log subsets.

[0109] In this embodiment, for clustering of abnormal operation logs, a sample set for obtaining the initial centroid is first constructed, and the sample density of each log sample in the sample set is calculated. The candidate centroid point is selected according to the sample density, and then the initial centroid is selected from the density neighborhood set of the candidate centroid point to obtain multiple initial centroids for log clustering. Cluster analysis of abnormal operation logs is performed based on the initial centroid, and efficient and accurate clustering of abnormal operation logs of financial business systems within the target time period is achieved, achieving the purpose of accurately selecting the initial centroid, accelerating the convergence speed and convergence effect of the clustering algorithm, and achieving the technical effect of improving the accuracy of clustering results. This solves the technical problem in the related art that the method of randomly selecting the initial centroid has low clustering result accuracy.

[0110] Optionally, the calculation unit 43 includes: a first calculation module, used to take each log sample in the sample set as a target log sample in turn, and calculate the cosine distance between the target log sample and other log samples in the sample set, wherein the other log samples are the remaining log samples in the sample set except the target log sample; a first as module, used to take other log samples whose cosine distance is less than a threshold radius as associated log samples of the target log sample, wherein the threshold radius is pre-set; a first statistical module, used to count the number of associated log samples of the target log sample; a first accumulation module, used to accumulate the cosine distances between the target log sample and all associated log samples to obtain the sum of the cosine distances between the target log sample and all associated log samples; a second calculation module, used to calculate the ratio of the number of associated log samples to the sum of the cosine distances to obtain the sample density of the target log sample, thereby obtaining the sample density of each log sample in the sample set.

[0111] Optionally, the calculation unit 43 includes: a first sorting module, used to sort the log samples in the sample set based on the sample density of each log sample in the sample set to obtain an ordered sample set; a first setting module, used to set the number of initial centroids to be selected; a first selection module, used to select a first candidate centroid subset and a second candidate centroid subset from the ordered sample set based on the number of initial centroids; a first acquisition module, used to obtain a candidate centroid set based on the first candidate centroid subset and the second candidate centroid subset.

[0112] Optionally, the first selection module includes: a first calculation submodule, used to execute step one, calculate the number of centroid candidate points in the first candidate centroid subset based on the number of initial centroids, and obtain a first target number; a first selection submodule, used to execute step two, select the first log sample in the ordered sample set as the first centroid candidate point of the first candidate centroid subset; a second calculation submodule, used to execute step three, for the remaining centroid candidate points in the first candidate centroid subset, calculate the position information of the i-th centroid candidate point based on the first position calculation formula, and select the i-th centroid candidate point of the first candidate centroid subset from the ordered sample set based on the position information; a first repetition submodule, used to repeatedly execute step three until the number of centroid candidate points in the first candidate centroid subset meets the first target number.

[0113] Optionally, the first position calculation formula is expressed as: [A i ]=2ni / 3K, where [A i ] represents the position information of the i-th centroid candidate point in the first candidate centroid subset, [] indicates that the value is rounded up to an integer when it is not an integer, n represents the total number of log samples in the sample set, i represents the i-th centroid candidate point in the first candidate centroid subset, which is a positive integer, and K represents the number of initial centroids.

[0114] Optionally, the first selection module includes: a third calculation submodule, used to execute step one, calculate the number of centroid candidate points in the first candidate centroid subset based on the number of initial centroids, and obtain a second target number; a second selection submodule, used to execute step two, select the first centroid candidate point of the second candidate centroid subset from the ordered sample set based on the total number of log samples in the sample set; a fourth calculation submodule, used to execute step three, for the remaining centroid candidate points in the second candidate centroid subset, calculate the position information of the i-th centroid candidate point based on the second position calculation formula, and select the i-th centroid candidate point of the second candidate centroid subset from the ordered sample set based on the position information; a second repetition submodule, used to repeatedly execute step three until the number of centroid candidate points in the second candidate centroid subset meets the second target number.

[0115] Optionally, the second position calculation formula is expressed as: [B i]=n / 3+2(2i-1)n / 3K, where [B i ] represents the position information of the i-th centroid candidate point in the second candidate centroid subset, [] indicates that the value is rounded up to an integer when it is not an integer, n represents the total number of log samples in the sample set, i represents the i-th centroid candidate point in the second candidate centroid subset, which is a positive integer, and K represents the number of initial centroids.

[0116] Optionally, the selection unit 44 includes: a second selection module, which is used to execute step one, calculate the sample density difference between the centroid candidate point and other ordered log samples in the ordered sample set, and select log samples whose sample density difference is less than a preset density difference threshold to obtain a density neighborhood sample set of the centroid candidate point, wherein the other ordered log samples are the remaining log samples in the ordered sample set except the centroid candidate point; a third calculation module, which is used to execute step two, take each log sample in the density neighborhood sample set of the centroid candidate point as the target log sample in turn, calculate the sum of cosine distances between the target log sample and other neighborhood log samples in the density neighborhood sample set, and obtain the sum of cosine distances of the target log sample, wherein the other neighborhood log samples are the remaining log samples in the density neighborhood sample set except the target log sample; a third selection module, which is used to execute step three, select the target log sample with the smallest sum of cosine distances as the initial centroid corresponding to the centroid candidate point; a first repetition module, which is used to repeat steps one to three above to obtain the initial centroids corresponding to all centroid candidate points.

[0117] It should be noted that the above-mentioned acquisition unit 41, sampling unit 42, calculation unit 43, selection unit 44, and clustering unit 45 correspond to steps S201 to S205 in the first embodiment, and the above-mentioned units and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the first embodiment. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules or units can also be part of the device and can be run in the computer terminal 10 provided in the first embodiment.

[0118] The present invention is described below in conjunction with another optional embodiment.

[0119] Embodiment 3

[0120] An embodiment of the present invention may also provide an electronic device, Figure 5 is a hardware structure block diagram of an electronic device (or mobile device) for optionally executing a log clustering method based on sample density according to an embodiment of the present invention, such as Figure 5 As shown, the electronic device may include: one or more ( Figure 5(only one is shown) processor 502, memory 504, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0121] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0122] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: collect abnormal operation logs of the financial business system within the target time period to obtain an abnormal operation log set; randomly sample the abnormal operation log set to obtain a sample set; calculate the sample density of each log sample in the sample set, and obtain a candidate centroid set based on the sample density, wherein the candidate centroid set contains K centroid candidate points, K is the number of pre-set initial centroids, and K is a positive integer; for each centroid candidate point in the candidate centroid set, obtain a density neighborhood sample set of the centroid candidate point, and select the initial centroid corresponding to the centroid candidate point based on the density neighborhood sample set to obtain an initial centroid set, wherein the initial centroid set contains K initial centroids; cluster the abnormal operation log set based on the initial centroid set to obtain K categories of abnormal operation log subsets.

[0123] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: take each log sample in the sample set as the target log sample in turn, calculate the cosine distance between the target log sample and other log samples in the sample set, wherein the other log samples are the remaining log samples in the sample set except the target log sample; take other log samples whose cosine distance is less than the threshold radius as the associated log samples of the target log sample, wherein the threshold radius is preset; count the number of associated log samples of the target log sample; accumulate the cosine distances between the target log sample and all associated log samples to obtain the sum of the cosine distances between the target log sample and all associated log samples; calculate the ratio of the number of associated log samples to the sum of the cosine distances to obtain the sample density of the target log sample, thereby obtaining the sample density of each log sample in the sample set.

[0124] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: sort the log samples in the sample set based on the sample density of each log sample in the sample set to obtain an ordered sample set; set the number of initial centroids to be selected; select a first candidate centroid subset and a second candidate centroid subset from the ordered sample set based on the number of initial centroids; obtain a candidate centroid set based on the first candidate centroid subset and the second candidate centroid subset.

[0125] The processor may call the information and application program stored in the memory through the transmission device to perform the following steps:

[0126] Step one, calculate the number of centroid candidate points in the first candidate centroid subset based on the number of initial centroids to obtain the first target number; step two, select the first log sample in the ordered sample set as the first centroid candidate point of the first candidate centroid subset; step three, for the remaining centroid candidate points in the first candidate centroid subset, calculate the position information of the i-th centroid candidate point based on the first position calculation formula, and select the i-th centroid candidate point of the first candidate centroid subset from the ordered sample set based on the position information; repeat step three until the number of centroid candidate points in the first candidate centroid subset meets the first target number.

[0127] The processor can call the information and application program stored in the memory through the transmission device to perform the following steps: The first position calculation formula is expressed as: [A i ]=2ni / 3K, where [A i ] represents the position information of the i-th centroid candidate point in the first candidate centroid subset, [] indicates that the value is rounded up to an integer when it is not an integer, n represents the total number of log samples in the sample set, i represents the i-th centroid candidate point in the first candidate centroid subset, which is a positive integer, and K represents the number of initial centroids.

[0128] The processor may call the information and application program stored in the memory through the transmission device to perform the following steps:

[0129] Step 1: calculate the number of centroid candidate points in the first candidate centroid subset based on the number of initial centroids to obtain the second target number; step 2: select the first centroid candidate point of the second candidate centroid subset from the ordered sample set based on the total number of log samples in the sample set; step 3: for the remaining centroid candidate points in the second candidate centroid subset, calculate the position information of the i-th centroid candidate point based on the second position calculation formula, and select the i-th centroid candidate point of the second candidate centroid subset from the ordered sample set based on the position information; repeat step 3 until the number of centroid candidate points in the second candidate centroid subset meets the second target number.

[0130] The processor can call the information and application program stored in the memory through the transmission device to perform the following steps: The second position calculation formula is expressed as: [B i ]=n / 3+2(2i-1)n / 3K, where [B i ] represents the position information of the i-th centroid candidate point in the second candidate centroid subset, [] indicates that the value is rounded up to an integer when it is not an integer, n represents the total number of log samples in the sample set, i represents the i-th centroid candidate point in the second candidate centroid subset, which is a positive integer, and K represents the number of initial centroids.

[0131] The processor may call the information and application program stored in the memory through the transmission device to perform the following steps:

[0132] Step 1: Calculate the sample density difference between the candidate centroid point and other ordered log samples in the ordered sample set, and select log samples whose sample density difference is less than a preset density difference threshold to obtain a density neighborhood sample set of the candidate centroid point, wherein the other ordered log samples are the remaining log samples in the ordered sample set except the candidate centroid point; Step 2: Take each log sample in the density neighborhood sample set of the candidate centroid point as the target log sample in turn, calculate the sum of cosine distances between the target log sample and other neighborhood log samples in the density neighborhood sample set, and obtain the sum of cosine distances of the target log sample, wherein the other neighborhood log samples are the remaining log samples in the density neighborhood sample set except the target log sample; Step 3: Select the target log sample with the smallest sum of cosine distances as the initial centroid corresponding to the candidate centroid point; Repeat the above steps 1 to 3 to obtain the initial centroids corresponding to all the candidate centroid points.

[0133] By adopting the embodiment of the present invention, a scheme for clustering logs based on sample density is provided. By calculating the sample density of each log sample in the sample set, selecting a candidate centroid point according to the sample density, and then selecting the initial centroid from the density neighborhood set of the candidate centroid point, a plurality of initial centroids for log clustering are obtained, and clustering analysis is performed on abnormal operation logs based on the initial centroids, thereby realizing efficient and accurate clustering of abnormal operation logs of the financial business system within the target time period, achieving the purpose of accurately selecting the initial centroid, accelerating the convergence speed and convergence effect of the clustering algorithm, and achieving the technical effect of improving the accuracy of the clustering results. This solves the technical problem in the related art that the method of randomly selecting the initial centroid has a low accuracy of the clustering results.

[0134] It can be understood by those skilled in the art that Figure 5 The structure shown is for illustration only, and the electronic device may also be a smart phone, a tablet computer, a PDA, a mobile Internet device (MID), a PAD or other terminal device. Figure 5The structure of the electronic device is not limited. Figure 5 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 5 Different configurations are shown.

[0135] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0136] The present invention is described below in conjunction with another optional embodiment.

[0137] Embodiment 4

[0138] The embodiment of the present invention further provides a computer-readable storage medium. Optionally, in the embodiment of the present invention, the computer-readable storage medium can be used to store the program code executed by the log clustering method based on sample density provided in the first embodiment.

[0139] Optionally, in an embodiment of the present invention, the above-mentioned storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0140] An embodiment of the present invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of a log clustering method based on sample density: collecting abnormal operation logs of a financial business system within a target time period to obtain an abnormal operation log set; randomly sampling the abnormal operation log set to obtain a sample set; calculating the sample density of each log sample in the sample set, and obtaining a candidate centroid set based on the sample density, wherein the candidate centroid set includes K centroid candidate points, K is the number of preset initial centroids, and K is a positive integer; for each centroid candidate point in the candidate centroid set, obtaining a density neighborhood sample set of the centroid candidate point, and selecting the initial centroid corresponding to the centroid candidate point based on the density neighborhood sample set to obtain an initial centroid set, wherein the initial centroid set includes K initial centroids; clustering the abnormal operation log set based on the initial centroid set to obtain K categories of abnormal operation log subsets.

[0141] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0142] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0143] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0144] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0145] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0146] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0147] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A log clustering method based on sample density, characterized in that: include: Collect abnormal operation logs of the financial business system within the target time period to obtain an abnormal operation log set; Randomly sampling the abnormal operation log set to obtain a sample set; Calculate the sample density of each log sample in the sample set, and obtain a candidate centroid set based on the sample density, wherein the candidate centroid set includes K centroid candidate points, where K is the number of preset initial centroids and K is a positive integer; For each of the centroid candidate points in the candidate centroid set, a density neighborhood sample set of the centroid candidate point is obtained, and an initial centroid corresponding to the centroid candidate point is selected based on the density neighborhood sample set to obtain an initial centroid set, wherein the initial centroid set includes K initial centroids; The abnormal operation log set is clustered based on the initial centroid set to obtain K categories of abnormal operation log subsets.

2. The clustering method according to claim 1, characterized in that: The step of calculating the sample density of each log sample in the sample set includes: Taking each of the log samples in the sample set as a target log sample in turn, and calculating the cosine distance between the target log sample and other log samples in the sample set, wherein the other log samples are the remaining log samples in the sample set except the target log sample; Using the other log samples whose cosine distance is smaller than a threshold radius as associated log samples of the target log sample, wherein the threshold radius is preset; Counting the number of log samples associated with the target log sample; Accumulating the cosine distances between the target log sample and all the associated log samples to obtain the sum of the cosine distances between the target log sample and all the associated log samples; The ratio of the number of associated log samples to the sum of the cosine distances is calculated to obtain the sample density of the target log sample, thereby obtaining the sample density of each log sample in the sample set.

3. The clustering method according to claim 1, characterized in that: The step of obtaining a candidate centroid set based on the sample density comprises: Sorting the log samples in the sample set based on the sample density of each log sample in the sample set to obtain an ordered sample set; Setting the number of the initial centroids to be selected; Selecting a first subset of candidate centroids and a second subset of candidate centroids from the ordered sample set based on the number of the initial centroids; The candidate centroid set is obtained based on the first candidate centroid subset and the second candidate centroid subset.

4. The clustering method according to claim 3, characterized in that: The step of selecting a first subset of candidate centroids from the ordered sample set based on the number of the initial centroids comprises: Step 1: Calculate the number of the candidate centroid points in the first candidate centroid subset based on the number of the initial centroids to obtain a first target number; Step 2: Select the first log sample in the ordered sample set as the first centroid candidate point of the first candidate centroid subset; Step 3: for the remaining centroid candidate points in the first candidate centroid subset, calculate the position information of the i-th centroid candidate point based on the first position calculation formula, and select the i-th centroid candidate point in the first candidate centroid subset from the ordered sample set based on the position information; Repeat step 3 until the number of candidate centroid points in the first candidate centroid subset meets the first target number.

5. The clustering method according to claim 4, characterized in that: The first position calculation formula is expressed as: [A i ]=2ni / 3K, where [A i ] represents the position information of the i-th candidate centroid point in the first candidate centroid subset, [] represents rounding up to an integer when the value is not an integer, n represents the total number of log samples in the sample set, i represents the i-th candidate centroid point in the first candidate centroid subset, which is a positive integer, and K represents the number of initial centroids.

6. The clustering method according to claim 3, characterized in that: The step of selecting a second subset of candidate centroids from the ordered sample set based on the number of the initial centroids comprises: Step 1: Calculate the number of the candidate centroid points in the first candidate centroid subset based on the number of the initial centroids to obtain a second target number; Step 2: selecting a first centroid candidate point of the second candidate centroid subset from the ordered sample set based on the total number of the log samples in the sample set; Step 3: for the remaining centroid candidate points in the second candidate centroid subset, calculate the position information of the i-th centroid candidate point based on the second position calculation formula, and select the i-th centroid candidate point of the second candidate centroid subset from the ordered sample set based on the position information; Repeat step 3 until the number of candidate centroid points in the second candidate centroid subset meets the second target number.

7. The clustering method according to claim 6, characterized in that: The second position calculation formula is expressed as: [B i ]=n / 3+2(2i-1)n / 3K, where [B i ] represents the position information of the i-th candidate centroid point in the second candidate centroid subset, [] represents rounding up to an integer when the value is not an integer, n represents the total number of log samples in the sample set, i represents the i-th candidate centroid point in the second candidate centroid subset, which is a positive integer, and K represents the number of initial centroids.

8. The clustering method according to claim 3, characterized in that: For each centroid candidate point in the candidate centroid set, the steps of obtaining a density neighborhood sample set of the centroid candidate point, and selecting an initial centroid corresponding to the centroid candidate point based on the density neighborhood sample set include: Step 1: Calculate the sample density difference between the centroid candidate point and other ordered log samples in the ordered sample set, and select log samples whose sample density difference is less than a preset density difference threshold to obtain a density neighborhood sample set of the centroid candidate point, wherein the other ordered log samples are the remaining log samples in the ordered sample set except the centroid candidate point; Step 2: taking each of the log samples in the density neighborhood sample set of the centroid candidate point as the target log sample in turn, calculating the sum of cosine distances between the target log sample and other neighborhood log samples in the density neighborhood sample set, and obtaining the sum of cosine distances of the target log sample, wherein the other neighborhood log samples are the remaining log samples in the density neighborhood sample set except the target log sample; Step 3, selecting the target log sample with the smallest cosine distance as the initial centroid corresponding to the centroid candidate point; Repeat steps 1 to 3 above to obtain the initial centroids corresponding to all the centroid candidate points.

9. A log clustering device based on sample density, characterized in that: include: A collection unit, used to collect abnormal operation logs of the financial business system within a target time period to obtain an abnormal operation log set; A sampling unit, used for randomly sampling the abnormal operation log set to obtain a sample set; A calculation unit, used to calculate the sample density of each log sample in the sample set, and obtain a candidate centroid set based on the sample density, wherein the candidate centroid set includes K centroid candidate points, K is the number of preset initial centroids, and K is a positive integer; A selection unit is used to obtain, for each of the centroid candidate points in the candidate centroid set, a density neighborhood sample set of the centroid candidate point, and select an initial centroid corresponding to the centroid candidate point based on the density neighborhood sample set to obtain an initial centroid set, wherein the initial centroid set includes K initial centroids; A clustering unit is used to cluster the abnormal operation log set based on the initial centroid set to obtain K categories of abnormal operation log subsets.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the log clustering method based on sample density as described in any one of claims 1 to 8.

Images