A terminal security analysis method and system based on MapReduce parallel clustering technology
By adopting MapReduce parallel clustering technology in terminal security assessment, combined with natural language processing and TF-IDF algorithm, the problem of incomplete end security assessment in existing technologies has been solved, and the end security in the existing technology has achieved incomplete, accurate, and accurate, accurate, and accurate, and Efficient evaluation, identify new network risks, and ensure the security of network space.
Patent Information
- Application Number
- CN202110092700.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-01-25
- Publication Date
- 2025-01-28
- Estimated Expiration
- 2041-01-25
AI Technical Summary
The existing network terminal security assessment method based on penetration testing technology cannot comprehensively evaluate terminal security, resulting in low evaluation results; it relies on existing security vulnerability databases to be updated in time, and it cannot quickly identify new network risks; and it requires professional equipment and Tools are time-consuming and labor-intensive, and cannot reflect terminal security in a timely and efficient manner.
Adopt the end security analysis method based on MapReduce parallel clustering technology, by obtaining log data from the terminal, performing natural language processing and filtering, using the TF-IDF algorithm extraction feature, calculating log vector and pre-set cluster center Ou Shi distance. , Update the cluster center to identify new risks and automatically evaluate end security.
A comprehensive assessment of end security has been achieved, it can quickly identify new network risks, reduce the cost of manpower and material resources, improve the efficiency and accuracy of evaluation, timely reflect the security of the end space, and ensure the security of the network space.
Smart Images

Figure CN112765660B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of information security technology, and more specifically, relates to a terminal security analysis method and system based on MapReduce parallel clustering technology. Background Art
[0002] With the rapid development of computer network technology, the dependence of all sectors of society on the network has increased, and the number of virus, spyware and hacker intrusions has increased, and network and information security issues have become increasingly prominent. At present, the conventional security defense concept is often limited to the defense of the gateway level and network boundary (firewall, antivirus, vulnerability scanning), and important security protection equipment is mainly concentrated in the computer room or the entrance of the network. Under the monitoring of these security equipment, the security threats from the outside of the network are greatly reduced. On the contrary, the security threats from internal computer terminals have become a common problem. After years of network security construction, the development trend of network security has also shifted from the protection of core networks and backbone networks to the security management of internal terminals. The largest part of network security management workload has turned to terminal security. The risk assessment of terminal security is a key link in network security assurance. In order to prevent the network from being threatened, the terminal security status of the network is objectively and effectively analyzed and evaluated based on the terminal log indicators, and the security strategy and risk handling plan are improved based on the evaluation results to protect network information security to the greatest extent.
[0003] Traditional network terminal security assessment mainly uses penetration testing technology, which mainly relies on host-based scanning tools. By monitoring the security vulnerabilities of the network terminal system, an assessment value of the network terminal security status is obtained based on the system vulnerability information obtained, and then the terminal security is analyzed through qualitative, quantitative or a combination of quantitative and qualitative methods.
[0004] However, the above-mentioned network terminal security assessment method based on penetration testing technology still has some defects that cannot be ignored: First, since terminal security involves multiple levels and multiple factors, and is uncertain and complex, some traditional quantitative and qualitative analysis methods are contradictory, making it impossible to directly compare indicators. This method is difficult to conduct a comprehensive assessment of terminal security, so it will lead to a low terminal security assessment result in the end; Second, since the traditional network terminal security assessment method relies on the existing security vulnerability library, the update of the security vulnerability library will not be timely enough, so it cannot quickly identify new network risks; Third, since the traditional network terminal security assessment method mainly relies on the assessor's own experience and knowledge to grade, its assessment method needs to rely on professional equipment and tools, and takes a certain amount of time, so it requires a lot of manpower and material costs, and cannot reflect terminal security in a timely and efficient manner. Summary of the invention
[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a terminal security analysis method and system based on MapReduce parallel clustering technology, which aims to solve the technical problems that the existing network terminal security assessment method based on penetration testing technology cannot comprehensively assess the terminal security, resulting in a low terminal security assessment result, and the technical problem that it cannot quickly identify new network risks because it relies on an existing security vulnerability library and the security vulnerability library is not updated in a timely manner, and the technical problem that it needs to rely on professional equipment and tools, which is time-consuming and labor-intensive and cannot reflect the terminal security in a timely and efficient manner.
[0006] To achieve the above object, according to one aspect of the present invention, a terminal security analysis method based on MapReduce parallel clustering technology is provided, comprising the following steps:
[0007] (1) Obtain log data from the terminal and process the log data using a natural language processing library to obtain multiple word segments;
[0008] (2) filtering the multiple segmented words obtained in step (1) to obtain multiple filtered segmented words;
[0009] (3) The feature of each word segment after filtering in step (2) is extracted using the term frequency-inverse text frequency (TF-IDF) algorithm. All features constitute the log vector X corresponding to the log data, and X = (x1, x2, ..., x num ), x m Represents the mth feature among all features, num represents the total number of features of all word segmentations extracted, and m∈[1,num];
[0010] (4) Calculate the Euclidean distance between the log vector corresponding to the log data obtained in step (3) and each of the K pre-set cluster centers, and obtain the cluster center corresponding to the minimum value of all Euclidean distances, and determine the final security level of the terminal based on the cluster center.
[0011] Preferably, the step (1) of acquiring the log data is to acquire the log data of the computer by calling the Syslog interface;
[0012] Log data includes program module, severity, process name, generation time, and log content;
[0013] Severity includes error, information, warning, critical, etc.;
[0014] Program modules include kernel layer, user layer, mail system, authorization information, etc.
[0015] Preferably, the K cluster centers are established by the following steps:
[0016] (4-1) Obtain multiple log data from the terminal, and use a natural language processing library to process each log data to obtain multiple word segments corresponding to the log data;
[0017] (4-2) for each log data, filtering the multiple word segments corresponding to the log data to obtain the filtered multiple word segments;
[0018] (4-3) For each log data, use the TF-IDF algorithm to process the multiple word segments filtered in step (4-2) to obtain the log vector corresponding to the log data;
[0019] (4-4) Select K log vectors from all log vectors corresponding to all log data as cluster centers, and put these cluster centers into the global variable set (which is initially empty);
[0020] (4-5) For each log vector in the log vector set, use the MapReduce model to calculate the Euclidean distance from the log vector to each cluster center in the global variable set, and use the cluster center corresponding to the minimum value of all Euclidean distances as the key and the log vector as the value to establish a key-value pair;
[0021] (4-6) For each cluster center in the multiple key-value pairs established in step (4-5), the cluster center and all values corresponding to the cluster center as the key are formed into a set, and the average value of all log vectors in the set is calculated as the cluster center after the cluster center is updated;
[0022] (4-7) Determine whether the set consisting of all updated cluster centers is exactly the same as the global variable set obtained in step (4-4). If so, proceed to step (4-8). Otherwise, replace the global variable set with the set consisting of the updated K cluster centers and return to step (4-5).
[0023] (4-8) Determine whether new log data is obtained from the terminal. If so, return to step (4-2), otherwise the process ends.
[0024] Preferably, the process of selecting K log vectors from the log vector set includes the following sub-steps:
[0025] (4-4-1) Calculate the average Euclidean distance d of all log vectors in the log vector set avg ;
[0026] (4-4-2) The average Euclidean distance d obtained according to step (4-4-1) avg Compute the point density of each log vector in a collection of log vectors
[0027] (4-4-3) For each log vector in the log vector set, the shortest distance corresponding to the log vector is obtained according to the point density of the log vector obtained in step (4-4-2);
[0028] (4-4-4) Obtain The K log vector points with the largest values are taken as cluster centers.
[0029] Preferably, the point density of each log vector is equal to:
[0030]
[0031] in represents the log vector X i and the log vector X j The Euclidean distance between them, and both i and j ∈ [1, n], n represents the total number of log vectors in the log vector set;
[0032] Preferably, for the log vector X i For , if its point density is the maximum value of the point density corresponding to all log vectors, then the shortest distance corresponding to the log vector is is equal to the log vector X from i The maximum value of the Euclidean distances to each of the remaining log vectors in the log vector set;
[0033] For the log vector X i For , if its point density is not the maximum value of the point density corresponding to all log vectors, then the shortest distance corresponding to the log vector is is equal to the log vector X from i to the point where the density is greater than the log vector X i The minimum of the Euclidean distances of each log vector of the point density.
[0034] According to another aspect of the present invention, a terminal security analysis system based on MapReduce parallel clustering technology is provided, comprising:
[0035] The first module is used to obtain log data from the terminal and process the log data using a natural language processing library to obtain multiple word segments;
[0036] The second module is used to filter the multiple word segments obtained by the first module to obtain multiple filtered word segments;
[0037] The third module is used to extract the features of each word segment after filtering by the second module using the word frequency-inverse text frequency TF-IDF algorithm. All the features constitute the log vector X corresponding to the log data, and X = (x1, x2, ..., x num), x m Represents the mth feature among all features, num represents the total number of features of all word segmentations extracted, and m∈[1,num];
[0038] The fourth module is used to calculate the Euclidean distance between the log vector corresponding to the log data obtained in the third module and each of the K pre-set cluster centers, and obtain the cluster center corresponding to the minimum value of all Euclidean distances, and determine the final security level of the terminal based on the cluster center.
[0039] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:
[0040] (1) Since the present invention adopts steps (1) to (4), it combines the log characteristics of the terminal itself and adopts clustering technology to classify terminal security, thereby solving the technical problem of low accuracy of terminal security assessment results due to unclear quantitative and qualitative methods in the existing network terminal security assessment method based on penetration testing technology;
[0041] (2) Since the present invention adopts steps (4-2) to (4-8), the cluster center is updated according to the new log data obtained from the terminal, thereby solving the technical problem that the existing network terminal security assessment method based on penetration testing technology cannot quickly identify new network risks due to the insufficient update of the security vulnerability library;
[0042] (3) Since the present invention adopts steps (1) to (4), the evaluation process is automatically performed without human intervention, thus solving the technical problems in the existing network terminal security evaluation method based on penetration testing technology, that is, the evaluation process requires the evaluator to grade according to his own experience and knowledge, the evaluation method relies on professional equipment and tools and takes a certain amount of time, resulting in a large amount of manpower and material costs, and the terminal security cannot be reflected in a timely and efficient manner;
[0043] (4) Since the present invention adopts steps (4-4) to (4-7), it optimizes the selection of the initial cluster center and adopts the MapReduce parallel programming model, which speeds up the convergence speed and running speed of the clustering algorithm, improves efficiency and performance, and saves time cost;
[0044] (5) The present invention has a fast running speed, high efficiency, saves economic costs, provides an objective and efficient security division reference for the actual environment, and helps to improve the security of cyberspace. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 It is a flow chart of the terminal security analysis method based on MapReduce parallel clustering technology of the present invention. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0047] The basic idea of the present invention is to collect log data from the terminal, perform text preprocessing such as word segmentation and filtering on it, use the term frequency-inverse document frequency (TF-IDF) algorithm to vectorize the terminal log data, and use MapReduce parallel technology to perform parallel clustering processing on the log data, which fully accelerates the speed of clustering processing. By comparing the Euclidean distance between the newly collected log vector and the cluster center, a terminal security analysis method based on MapReduce parallel clustering technology is provided.
[0048] like Figure 1 As shown, the present invention provides a terminal security analysis method based on MapReduce parallel clustering technology, comprising the following steps:
[0049] (1) Obtain log data from the terminal and process the log data using a natural language processing library to obtain multiple word segments;
[0050] Specifically, the natural language processing library used in this step may be, for example, the jieba library, which is a Python third-party Chinese word segmentation library.
[0051] In this step, the time interval for obtaining log data is 1 minute to 10 minutes, preferably 5 minutes;
[0052] Specifically, the terminal in the present invention refers to a computer (which may be a personal computer or a laptop), and the log data is obtained in this step by calling the Syslog interface to obtain the log data of the computer.
[0053] Log data includes program module, severity, process name, generation time and log content, etc. Severity mainly includes error, information, warning, critical, etc. Program modules include kernel layer, user layer, mail system, authorization information, etc. In theory, the log data corresponding to each time period corresponds to a complete physical behavior.
[0054] (2) filtering the multiple segmented words obtained in step (1) to obtain multiple filtered segmented words;
[0055] Specifically, the filtering process in this step includes but is not limited to: deleting stop words (such as the auxiliary words "de", "le", "a", etc.), special symbols (such as punctuation marks, mathematical operators, etc.), and link addresses irrelevant to security judgment (such as advertisement links, etc.);
[0056] The advantage of this step is that the word segmentation and filtering process can remove the word segments irrelevant to the terminal security judgment, reducing the interference of them on the accuracy of the security judgment result.
[0057] (3) Use the Term frequency–inverse document frequency (TF-IDF) algorithm to extract the features of each word segment filtered in step (2). All the features constitute the log vector X corresponding to the log data, and there is X = (x1, x2,..., x num ), where x m represents the m-th among all the features, num represents the total number of features of all the extracted word segments, and m ∈ [1, num];
[0058] (4) Calculate the Euclidean distance between the log vector corresponding to the log data obtained in step (3) and each of the pre-set K clustering centers, and obtain the clustering center corresponding to the minimum value of all the Euclidean distances. Determine the final security level of the terminal according to this clustering center.
[0059] The advantages of the above steps (1) to (4) are as follows: By combining the log characteristics of the terminal itself, using the clustering result to evaluate the terminal security, the accuracy of the terminal security evaluation is guaranteed; In addition, by adopting the MapReduce parallel programming model, the running speed of the clustering algorithm is accelerated, and its efficiency and performance are improved;
[0060] Specifically, the value range of K is a natural number greater than or equal to 2. The larger its value, the more levels are used for security judgment; the smaller its value, the fewer levels are used for security judgment.
[0061] For example, assume K = 10. If the second clustering center finally has the minimum Euclidean distance in this step, it is considered that the final security level of the terminal is the lowest; if the tenth clustering center finally has the minimum Euclidean distance in this step, it is considered that the final security level of the terminal is the highest; if the fifth clustering center finally has the minimum Euclidean distance in this step, it is considered that the final security level of the terminal is medium.
[0062] The K clustering centers in this step are established through the following steps:
[0063] (4-1) Obtain multiple log data from the terminal, and use a natural language processing library to process each log data to obtain multiple word segments corresponding to the log data;
[0064] The process of this step is exactly the same as the above step (1), and will not be repeated here.
[0065] (4-2) for each log data, filtering the multiple word segments corresponding to the log data to obtain the filtered multiple word segments;
[0066] The process of this step is exactly the same as that of the above step (2), and will not be repeated here.
[0067] (4-3) For each log data, use the TF-IDF algorithm to process the multiple word segments filtered in step (4-2) to obtain the log vector corresponding to the log data;
[0068] (4-4) Select K log vectors from all log vectors corresponding to all log data as cluster centers, and put these cluster centers into the global variable set (which is initially empty);
[0069] Specifically, the process of selecting K log vectors from the log vector set in this step specifically includes the following sub-steps:
[0070] (4-4-1) Calculate the average Euclidean distance d of all log vectors in the log vector set avg ;
[0071] (4-4-2) The average Euclidean distance d obtained according to step (4-4-1) avg Compute the point density of each log vector in a collection of log vectors
[0072]
[0073] in, represents the log vector X i and the log vector X j The Euclidean distance between them, and both i and j ∈ [1, n], n represents the total number of log vectors in the log vector set;
[0074] (4-4-3) For each log vector in the log vector set, the shortest distance corresponding to the log vector is obtained according to the point density of the log vector obtained in step (4-4-2);
[0075] Specifically, for the log vector X i For , if its point density is the maximum value of the point density corresponding to all log vectors, then the shortest distance corresponding to the log vector is is equal to the log vector X from i The maximum value of the Euclidean distance to each of the remaining log vectors in the log vector set; for the log vector X i For , if its point density is not the maximum value of the point density corresponding to all log vectors, then the shortest distance corresponding to the log vector is is equal to the log vector X from i to the point where the density is greater than the log vector X i The minimum of the Euclidean distances of each log vector of the point density.
[0076] (4-4-4) Obtain The K log vector points with the largest values are taken as cluster centers;
[0077] The advantage of the above sub-steps (4-4-1) to (4-4-4) is that they optimize the selection of the initial cluster center and accelerate the convergence speed of the clustering algorithm.
[0078] (4-5) For each log vector in the log vector set, use the MapReduce model to calculate the Euclidean distance from the log vector to each cluster center in the global variable set, and use the cluster center corresponding to the minimum value of all Euclidean distances as the key (Key) and the log vector as the value (Value) to establish a key-value pair;
[0079] (4-6) For each cluster center in the multiple key-value pairs established in step (4-5), the cluster center and all values corresponding to the cluster center as the key are formed into a set, and the average value of all log vectors in the set is calculated as the cluster center after the cluster center is updated;
[0080] (4-7) Determine whether the set consisting of all updated cluster centers is exactly the same as the global variable set obtained in step (4-4). If so, proceed to step (4-8). Otherwise, replace the global variable set with the set consisting of the updated K cluster centers and return to step (4-5).
[0081] (4-8) Determine whether new log data is obtained from the terminal. If so, return to step (4-2), otherwise the process ends.
[0082] The advantage of this step is that the cluster center is updated with new log data obtained from the terminal, which can quickly identify new network risks, reflect terminal security in a timely manner, and ensure the security of the network space;
[0083] It should be noted that when returning to step (4-2) and executing subsequent steps, all log data in step (4-4) should include all log data before executing step (4-8) and the new log data obtained by (4-8); at the same time, each log data in steps (4-2) and (4-3) refers to the new log data obtained by step (4-8).
[0084] In summary, the terminal security analysis method based on MapReduce parallel clustering proposed in the present invention performs text preprocessing, text representation, log clustering model generation and terminal security judgment on terminal logs, which is very necessary for judging the terminal security in the network and improving network security in the actual network.
[0085] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A terminal security analysis method based on MapReduce parallel clustering technology, characterized in that: The following steps are involved: (1) Obtain log data from the terminal and process the log data using a natural language processing library to obtain multiple word segments; (2) filtering the multiple segmented words obtained in step (1) to obtain multiple filtered segmented words; (3) Use the term frequency-inverse text frequency (TF-IDF) algorithm to extract the features of each word segment after filtering in step (2). All features constitute the log vector X corresponding to the log data, and X = (x1, x2, …, x num ), x m represents the mth feature among all features, num represents the total number of features of all word segmentations extracted, and x∈[1,num]; (4) Calculate the Euclidean distance between the log vector corresponding to the log data obtained in step (3) and each of the K pre-set cluster centers, and obtain the cluster center corresponding to the minimum value of all Euclidean distances, and determine the final security level of the terminal according to the cluster center. The value range of K is a natural number greater than or equal to 2. The larger the value, the more levels are used for security judgment, and the smaller the value, the fewer levels are used for security judgment; K cluster centers are established through the following steps: (4-1) Obtain multiple log data from the terminal, and use a natural language processing library to process each log data to obtain multiple word segments corresponding to the log data; (4-2) for each log data, filtering the multiple word segments corresponding to the log data to obtain the filtered multiple word segments; (4-3) For each log data, use the TF-IDF algorithm to process the multiple word segments filtered in step (4-2) to obtain the log vector corresponding to the log data; (4-4) Select K log vectors from all log vectors corresponding to all log data as cluster centers, and put these cluster centers into an initially empty global variable set; The process of selecting K log vectors from the set of log vectors includes the following sub-steps: (4-4-1) Calculate the average Euclidean distance d of all log vectors in the log vector set avg ; (4-4-2) The average Euclidean distance d obtained according to step (4-4-1) avg Compute the point density of each log vector in a collection of log vectors (4-4-3) For each log vector in the log vector set, obtain the shortest distance corresponding to the log vector according to the point density of the log vector obtained in step (4-4-2) (4-4-4) Obtain The K log vector points with the largest values are taken as cluster centers; (4-5) For each log vector in the log vector set, use the MapReduce model to calculate the Euclidean distance from the log vector to each cluster center in the global variable set, and use the cluster center corresponding to the minimum value of all Euclidean distances as the key and the log vector as the value to establish a key-value pair; (4-6) For each cluster center in the multiple key-value pairs established in step (4-5), the cluster center and all values corresponding to the cluster center as the key are formed into a set, and the average value of all log vectors in the set is calculated as the cluster center after the cluster center is updated; (4-7) Determine whether the set consisting of all updated cluster centers is exactly the same as the global variable set obtained in step (4-4). If so, proceed to step (4-8). Otherwise, replace the global variable set with the set consisting of the updated K cluster centers and return to step (4-5). (4-8) Determine whether new log data is obtained from the terminal. If so, return to step (4-2), otherwise the process ends.
2. According to claim 1, the terminal security analysis method based on MapReduce parallel clustering technology is characterized in that: In step (1), the log data is obtained by calling the Syslog interface to obtain the log data of the computer; Log data includes program module, severity, process name, generation time, and log content; Severity includes Error, Information, Warning, Critical; The program modules include kernel layer, user layer, mail system, and authorization information.
3. The terminal security analysis method based on MapReduce parallel clustering technology according to claim 2 is characterized in that: The point density of each log vector is equal to: in d(X i ,X j ) represents the log vector X i and the log vector X j The Euclidean distance between them, and both i and j ∈ [1, n], n represents the total number of log vectors in the log vector set.
4. The terminal security analysis method based on MapReduce parallel clustering technology according to claim 3 is characterized in that: For the log vector X i For , if its point density is the maximum value of the point density corresponding to all log vectors, then the shortest distance corresponding to the log vector is is equal to the log vector X from i The maximum value of the Euclidean distances to each of the remaining log vectors in the log vector set; For the log vector X i For , if its point density is not the maximum value of the point density corresponding to all log vectors, then the shortest distance corresponding to the log vector is is equal to the log vector X from i to the point where the density is greater than the log vector X i The minimum of the Euclidean distances of each log vector of the point density.
5. A terminal security analysis system based on MapReduce parallel clustering technology, characterized in that: include: The first module is used to obtain log data from the terminal and process the log data using a natural language processing library to obtain multiple word segments; The second module is used to filter the multiple word segments obtained by the first module to obtain multiple filtered word segments; The third module is used to extract the features of each word segment after filtering by the second module using the word frequency-inverse text frequency TF-IDF algorithm. All the features constitute the log vector X corresponding to the log data, and X = (x1, x2, ..., x num ), x m Represents the mth feature among all features, num represents the total number of features of all word segmentations extracted, and m∈[1,num]; The fourth module is used to calculate the Euclidean distance between the log vector corresponding to the log data obtained in the third module and each of the K pre-set cluster centers, and obtain the cluster center corresponding to the minimum value of all Euclidean distances, and determine the final security level of the terminal according to the cluster center; K cluster centers are established through the following steps: (4-1) Obtain multiple log data from the terminal, and use a natural language processing library to process each log data to obtain multiple word segments corresponding to the log data; (4-2) for each log data, filtering the multiple word segments corresponding to the log data to obtain the filtered multiple word segments; (4-3) For each log data, use the TF-IDF algorithm to process the multiple word segments filtered in step (4-2) to obtain the log vector corresponding to the log data; (4-4) Select K log vectors from all log vectors corresponding to all log data as cluster centers, and put these cluster centers into an initially empty global variable set; The process of selecting K log vectors from the set of log vectors includes the following sub-steps: (4-4-1) Calculate the average Euclidean distance d of all log vectors in the log vector set avg ; (4-4-2) The average Euclidean distance d obtained according to step (4-4-1) avg Compute the point density of each log vector in a collection of log vectors (4-4-3) For each log vector in the log vector set, obtain the shortest distance corresponding to the log vector according to the point density of the log vector obtained in step (4-4-2) (4-4-4) Obtain The K log vector points with the largest values are taken as cluster centers; (4-5) For each log vector in the log vector set, use the MapReduce model to calculate the Euclidean distance from the log vector to each cluster center in the global variable set, and use the cluster center corresponding to the minimum value of all Euclidean distances as the key and the log vector as the value to establish a key-value pair; (4-6) For each cluster center in the multiple key-value pairs established in step (4-5), the cluster center and all values corresponding to the cluster center as the key are formed into a set, and the average value of all log vectors in the set is calculated as the cluster center after the cluster center is updated; (4-7) Determine whether the set consisting of all updated cluster centers is exactly the same as the global variable set obtained in step (4-4). If so, proceed to step (4-8). Otherwise, replace the global variable set with the set consisting of the updated K cluster centers and return to step (4-5). (4-8) Determine whether new log data is obtained from the terminal. If so, return to step (4-2), otherwise the process ends.
Citation Information
Patent Citations
Virtual human establishing method on MapReduce platform
CN104965846A
Big data parallel operation-based voice mail business data analysis method
CN105740397A