Intelligent analysis system and method for mass data
By performing text information clustering and historical anomaly event mining on massive amounts of data, combined with anomaly parameters and threshold judgment, the problem of low efficiency in anomaly analysis in existing technologies has been solved, and rapid and accurate anomaly data detection has been achieved.
Patent Information
- Application Number
- CN202511516476.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-22
- Publication Date
- 2026-03-10
AI Technical Summary
Existing technologies for anomaly analysis of massive amounts of data are inefficient and yield unsatisfactory results.
By using a threshold module to obtain text information for clustering, strong correlation rules of historical data anomalies are discovered, and combined with an early warning module to identify abnormal data.
It improves the efficiency of anomaly detection in the database, enabling rapid and accurate detection of abnormal data.
Smart Images
Figure CN121636707A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer, more particularly, to an intelligent analysis system and method for mass data. BACKGROUND
[0002] In the background of the digital era, the amount of data grows exponentially, the scale and complexity of information increase continuously, and various industries continue to generate and store mass data into various databases. In the face of this trend, efficient information analysis and management of data in the database have become an urgent need.
[0003] The prior art is limited to manual analysis based on feature files and other methods to analyze the data in the database, which is low in efficiency and unsatisfactory in abnormal analysis effect. SUMMARY
[0004] The present application provides an intelligent analysis system for mass data to solve the problem of low efficiency and unsatisfactory effect of data abnormal analysis in the prior art, comprising: A threshold module is configured to obtain text information of a current database, cluster each data according to content information of the text information, and determine an abnormal parameter threshold according to the clustering result; a mining module is configured to obtain historical data abnormal events, mine strong association rules of the historical data abnormal events, and determine an abnormal parameter of current data according to the strong association rules of the historical data abnormal events; and a warning module is configured to determine abnormal data in the database according to the abnormal parameter and the abnormal data threshold.
[0005] Further, the threshold module clusters each data according to content information of the text information, comprising: determining text keywords and text topics according to the text information of the data, vector encoding the text keywords and the text topics to obtain text keyword vectors and text topic vectors; establishing a sample data set according to the text keyword vectors and the text topic vectors of each data in the current database, and randomly selecting k initial clustering centers of the sample data set; calculating the Euclidean distance of sample data in the sample data set to the initial clustering centers, and dividing each data to a corresponding clustering partition according to the Euclidean distance of the sample data in the sample data set to the initial clustering centers; calculating the sample data mean in each clustering partition, and recalculating the clustering center according to the sample data mean in each clustering partition; repeating the above steps until the clustering center no longer changes or the number of iterations reaches a preset maximum number of iterations, and obtaining the clustering result of each data in the current database.
[0006] Further, the determining the abnormal parameter threshold according to the clustering result comprises: obtaining a preset database, calculating text keyword vectors and text theme vectors of each data in the preset database; obtaining preset abnormal type labels, labeling the text keyword vectors and the text theme vectors of each data in the preset database according to the preset abnormal type labels to obtain abnormal type labels corresponding to each text keyword vector and text theme vector; establishing a training sample set according to each data and the corresponding abnormal type label, training a model according to the training sample set to obtain a trained abnormal type evaluation model; determining a clustering center of each data corresponding to a clustering partition according to the clustering result, extracting text keyword vectors and text theme vectors of the clustering center; inputting the text keyword vectors and the text theme vectors of each clustering center into the trained abnormal type evaluation model to obtain abnormal data types of each clustering center, and determining the abnormal parameter threshold according to the abnormal data types.
[0007] Further, the mining module mines strong association rules of historical data abnormal events, and determines the abnormal parameter of the current data according to the strong association rules of the historical data abnormal events, comprising: determining text keyword vectors of text information corresponding to each data according to the historical data abnormal events, and establishing a feature data set according to the text keyword vectors of the text information corresponding to each data; performing standardization processing on the text keyword vectors, mining frequent item sets of the standardized operation feature set based on an apriori association rule algorithm, determining the strong association rules of the historical data abnormal events according to the frequent item sets of the standardized operation feature set; extracting text keyword vectors of data in the current database, and matching the text keyword vectors of the data in the current database with the strong association rules to obtain the abnormal parameter of the data in the current database.
[0008] Further, the mining module mines strong association rules of historical data abnormal events, and determines the abnormal parameter of the current data according to the strong association rules of the historical data abnormal events, comprising: performing discretization processing on the standardized operation feature set to obtain a discretized operation feature set; converting the discretized operation feature set into a transaction type data format to obtain a transaction type operation feature set; setting a minimum support threshold and a minimum confidence threshold, iteratively scanning the transaction type operation feature set by using an Apriori algorithm to generate a candidate item set; calculating the support of each candidate item set, and screening a frequent item set meeting the minimum support threshold from the candidate item set according to the support, and generating association rules according to the frequent item set; calculating the confidence of each association rule, and screening a strong association rule meeting the minimum confidence threshold from the association rules according to the confidence.
[0009] Further, the matching the text keyword vector of the data in the current database with the strong association rule to obtain the abnormal parameter of the data in the current database comprises: establishing a preset database according to the strong association rule of the historical data abnormal event, and obtaining the strong association rule of the data text keyword vector corresponding to different data abnormal events in the preset database; matching the data text keyword vector in the current database with the strong association rule of the data text keyword vector corresponding to different data abnormal events in the preset database to obtain the matching degree between the data text keyword vector in the current database and the strong association rule of the data text keyword vector corresponding to different data abnormal events in the preset database; and screening the data text keyword vector in the current database with a matching degree greater than a first preset threshold, and determining the abnormal parameter according to the corresponding matching degree.
[0010] Further, the pre-warning module determines the abnormal data in the database according to the abnormal parameter and the abnormal data threshold, comprising: calculating the difference between the abnormal parameter of the data in the current database and the abnormal data threshold, and determining whether the corresponding data is abnormal data according to the difference between the abnormal parameter of the data in the current database and the abnormal data threshold.
[0011] Further, the determining whether the corresponding data is abnormal data according to the difference between the abnormal parameter of the data in the current database and the abnormal data threshold comprises: determining whether the difference between the abnormal parameter of the data in the current database and the abnormal data threshold is greater than a second preset threshold, if the difference between the abnormal parameter of the data in the current database and the abnormal data threshold is greater than the second preset threshold, the corresponding data is determined as abnormal data; and if the difference between the abnormal parameter of the data in the current database and the abnormal data threshold is less than or equal to the second preset threshold, the corresponding data is determined as normal data.
[0012] In order to achieve the above-mentioned purpose, the application further provides an intelligent analysis method for mass data, comprising: obtaining the text information of the current database, clustering each data according to the content information of the text information, determining the abnormal parameter threshold according to the clustering result; obtaining the historical data abnormal event, mining the strong association rule of the historical data abnormal event, determining the abnormal parameter of the current data according to the strong association rule of the historical data abnormal event; and determining the abnormal data in the database according to the abnormal parameter and the abnormal data threshold.
[0013] The application has the advantages that: By applying the technical scheme, the threshold module is used to acquire text information of a current database, cluster each data according to content information of the text information, and determine an abnormal parameter threshold according to a clustering result; the mining module is used to acquire historical data abnormal events, mine strong association rules of the historical data abnormal events, and determine an abnormal parameter of current data according to the strong association rules of the historical data abnormal events; and the early warning module is used to determine abnormal data in the database according to the abnormal parameter and the abnormal data threshold. The abnormal detection efficiency of the database can be effectively improved, and the abnormal data can be quickly and accurately detected. BRIEF DESCRIPTION OF DRAWINGS
[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0015] Figure 1 A structure schematic diagram of an intelligent analysis system for mass data proposed by an embodiment of the present application is shown. Figure 2 A general flowchart of an intelligent analysis method for mass data proposed by an embodiment of the present application is shown. DETAILED DESCRIPTION
[0016] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.
[0017] An intelligent analysis system for mass data is provided in the embodiments of the present application, as shown in Figure 1 The intelligent analysis system comprises: A threshold module is used to acquire text information of a current database, cluster each data according to content information of the text information, and determine an abnormal parameter threshold according to a clustering result; a mining module is used to acquire historical data abnormal events, mine strong association rules of the historical data abnormal events, and determine an abnormal parameter of current data according to the strong association rules of the historical data abnormal events; and an early warning module is used to determine abnormal data in the database according to the abnormal parameter and the abnormal data threshold.
[0018] In the embodiment, the data are clustered by extracting the text information of each data in the database, and the abnormal parameter threshold corresponding to each data is determined according to the clustering result of each data. Meanwhile, the text information of each historical data when a historical data abnormal event occurs is collected to obtain the historical data abnormal event, strong association rules of the historical data abnormal event are mined, the data in the current database are matched with the strong association rules, thereby obtaining the abnormal parameter of the current data, and the abnormal data in the database is obtained based on the abnormal parameter and the abnormal data threshold.
[0019] In some embodiments of the present application, the threshold module clusters each data according to the content information of the text information, including: determining a text keyword and a text theme according to the text information of the data, and obtaining a text keyword vector and a text theme vector by vector encoding the text keyword and the text theme; establishing a sample data set according to the text keyword vector and the text theme vector of each data in the current database, and randomly selecting k initial clustering centers of the sample data set; calculating the Euclidean distance of the sample data in the sample data set to the initial clustering center, and dividing each data to the corresponding clustering partition according to the Euclidean distance of the sample data in the sample data set to the initial clustering center; calculating the sample data mean in each clustering partition, and recalculating the clustering center according to the sample data mean in each clustering partition; repeating the above steps until the clustering center no longer changes or the iteration number reaches the preset maximum iteration number, and obtaining the clustering result of each data in the current database.
[0020] In the embodiment, the text keyword and the text theme in the data text information are extracted and vector encoded to obtain the text keyword vector and the text theme vector, each data is clustered based on the k-means clustering algorithm according to the text keyword vector and the text theme vector, and the k value in the algorithm is set according to the data amount in the database. The larger the data amount is, the larger the corresponding k value is.
[0021] In some embodiments of the present application, the abnormal parameter threshold is determined according to the clustering result, including: obtaining a preset database, and calculating the text keyword vector and the text theme vector of each data in the preset database; obtaining a preset abnormal type label, and labeling the text keyword vector and the text theme vector of each data in the preset database according to the preset abnormal type label to obtain the abnormal type label corresponding to each text keyword vector and text theme vector; establishing a training sample set according to each data and the corresponding abnormal type label, training a model according to the training sample set to obtain a trained abnormal type evaluation model; determining the clustering center of each clustering partition corresponding to each data according to the clustering result, and extracting the text keyword vector and the text theme vector of the clustering center; inputting the text keyword vector and the text theme vector of each clustering center into the trained abnormal type evaluation model to obtain the abnormal data type of each clustering center, and determining the abnormal parameter threshold according to the abnormal data type.
[0022] In the embodiment, the preset database is established by the text information of the data in the historical data abnormal events, the abnormal type evaluation model is established and trained by the abnormal data in the preset database, the abnormal data types of each clustering center are output by the abnormal type evaluation model, and the abnormal parameter threshold is set according to the severity of the abnormal data type, and the higher the severity, the larger the corresponding abnormal parameter threshold.
[0023] In some embodiments of the present application, the mining module mines strong association rules of historical data abnormal events, and determines the abnormal parameters of the current data according to the strong association rules of the historical data abnormal events, including: determining the text keyword vector of the text information corresponding to each data according to the historical data abnormal events, and establishing a feature data set according to the text keyword vector of the text information corresponding to each data; performing standardization processing on the text keyword vector, mining the frequent item set of the standardized operation feature set based on the apriori association rule algorithm, and determining the strong association rules of the historical data abnormal events according to the frequent item set of the standardized operation feature set; extracting the text keyword vector of the data in the current database, matching the text keyword vector of the data in the current database with the strong association rules, and obtaining the abnormal parameters of the data in the current database.
[0024] In some embodiments of the present application, the mining module mines the frequent item set of the standardized operation feature set based on the apriori association rule algorithm, and determines the strong association rules of the historical data abnormal events according to the frequent item set of the standardized operation feature set, including: performing discretization processing on the standardized operation feature set to obtain a discretized operation feature set; converting the discretized operation feature set into a transaction type data format to obtain a transaction type operation feature set; setting a minimum support threshold and a minimum confidence threshold, iteratively scanning the transaction type operation feature set by using the Apriori algorithm to generate a candidate item set; calculating the support of each candidate item set, and screening the frequent item set meeting the minimum support threshold from the candidate item set according to the support, and generating association rules according to the frequent item set; calculating the confidence of each association rule, and screening the strong association rules meeting the minimum confidence threshold from the association rules according to the confidence.
[0025] In some embodiments of the present application, the matching of the text keyword vector of the data in the current database with the strong association rule to obtain the abnormal parameter of the data in the current database comprises: establishing a preset database according to the strong association rule of the historical data abnormal event, and obtaining the strong association rule of the data text keyword vector corresponding to different data abnormal events in the preset database; matching the data text keyword vector in the current database with the strong association rule of the data text keyword vector corresponding to different data abnormal events in the preset database to obtain the matching degree between the data text keyword vector in the current database and the strong association rule of the data text keyword vector corresponding to different data abnormal events in the preset database; screening the data text keyword vector in the current database with a matching degree greater than a first preset threshold, and determining the abnormal parameter according to the corresponding matching degree.
[0026] In the present embodiment, the strong association rule between each data in the historical data abnormal event is mined based on the apriori algorithm, the preset database is established through the data text keyword vector corresponding to different abnormal events and the strong association rule corresponding to each data, and the data with the strong association rule in the current database is obtained by matching the text keyword vector of the data in the current database with the strong association rule of the data text keyword vector in the preset database, and the corresponding matching degree is determined as the abnormal parameter.
[0027] In some embodiments of the present application, the warning module determines the abnormal data in the database according to the abnormal parameter and the abnormal data threshold, comprising: calculating the difference between the abnormal parameter of the data in the current database and the abnormal data threshold, and determining whether the corresponding data is abnormal data according to the difference between the abnormal parameter of the data in the current database and the abnormal data threshold.
[0028] In some embodiments of the present application, the determination of whether the corresponding data is abnormal data according to the difference between the abnormal parameter of the data in the current database and the abnormal data threshold comprises: determining whether the difference between the abnormal parameter of the data in the current database and the abnormal data threshold is greater than a second preset threshold, if the difference between the abnormal parameter of the data in the current database and the abnormal data threshold is greater than the second preset threshold, the corresponding data is determined as abnormal data; if the difference between the abnormal parameter of the data in the current database and the abnormal data threshold is less than or equal to the second preset threshold, the corresponding data is determined as normal data.
[0029] In the present embodiment, the abnormal degree of the data is determined by the difference between the abnormal parameter of the data in the current database and the abnormal data threshold, and when the abnormal degree is greater than the second preset threshold, the corresponding data is determined as abnormal data and abnormal warning is performed. The abnormal detection efficiency of the database can be effectively improved, and the rapid and accurate detection of abnormal data is realized.
[0030] Based on the same technical concept, the application also provides a method for intelligent analysis of mass data, comprising: Figure 2 As shown in the same technical concept, S101, obtaining text information of the current database, clustering each data according to the content information of the text information, and determining an abnormal parameter threshold according to the clustering result; S102, obtaining historical data abnormal events, mining strong association rules of the historical data abnormal events, and determining an abnormal parameter of the current data according to the strong association rules of the historical data abnormal events; S103, determining abnormal data in the database according to the abnormal parameter and the abnormal data threshold.
[0031] By applying the above technical solutions, the threshold module is used to obtain text information of the current database, cluster each data according to the content information of the text information, and determine an abnormal parameter threshold according to the clustering result; the mining module is used to obtain historical data abnormal events, mine strong association rules of the historical data abnormal events, and determine an abnormal parameter of the current data according to the strong association rules of the historical data abnormal events; and the early warning module is used to determine abnormal data in the database according to the abnormal parameter and the abnormal data threshold. The abnormal detection efficiency of the database can be effectively improved, and rapid and accurate detection of abnormal data can be realized.
[0032] Through the description of the above embodiments, those skilled in the art can clearly understand that the application can be implemented by hardware, or by means of software and necessary general hardware platform. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the application.
[0033] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing examples, those skilled in the art can understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the application.
Claims
1. An intelligent analysis system for mass data, characterized by, The method comprises the following steps: a threshold module is used to obtain text information of a current database, cluster each data according to content information of the text information, and determine an abnormal parameter threshold according to a clustering result; a mining module is used to obtain historical data abnormal events, mine strong association rules of the historical data abnormal events, and determine an abnormal parameter of current data according to the strong association rules of the historical data abnormal events; an early warning module is used to determine abnormal data in the database according to the abnormal parameter and the abnormal data threshold.
2. The intelligent analysis system for big data according to claim 1, wherein, The threshold module clusters each data according to content information of the text information, which comprises the following steps: text keywords and a text theme are determined according to text information of the data, the text keywords and the text theme are vector coded to obtain text keyword vectors and text theme vectors; a sample data set is established according to the text keyword vectors and the text theme vectors of each data in the current database, and k initial clustering centers of the sample data set are randomly selected; Euclidean distances of sample data in the sample data set to the initial clustering centers are calculated, and each data is divided into a corresponding clustering partition according to the Euclidean distances of the sample data in the sample data set to the initial clustering centers; sample data mean values in each clustering partition are calculated, and the clustering centers are recalculated according to the sample data mean values in each clustering partition; the above steps are repeated and iterated until the clustering centers no longer change or the number of iterations reaches a preset maximum number of iterations, and a clustering result of each data in the current database is obtained.
3. The intelligent analysis system for big data according to claim 2, wherein, The abnormal parameter threshold is determined according to the clustering result, which comprises the following steps: a preset database is obtained, and text keyword vectors and text theme vectors of each data in the preset database are calculated; preset abnormal type labels are obtained, and the text keyword vectors and the text theme vectors of each data in the preset database are labeled according to the preset abnormal type labels to obtain abnormal type labels corresponding to each text keyword vector and text theme vector; a training sample set is established according to each data and the corresponding abnormal type label, a model is trained according to the training sample set, and a trained abnormal type evaluation model is obtained; clustering centers of each data corresponding to a clustering partition are determined according to the clustering result, and text keyword vectors and text theme vectors of the clustering centers are extracted; the text keyword vectors and the text theme vectors of each clustering center are input into the trained abnormal type evaluation model to obtain abnormal data types of each clustering center, and an abnormal parameter threshold is determined according to the abnormal data types.
4. The intelligent analysis system for big data according to claim 1, wherein, The mining module mines strong association rules of the historical data abnormal events, and determines the abnormal parameter of the current data according to the strong association rules of the historical data abnormal events, which comprises the following steps: text keyword vectors of each data corresponding to text information are determined according to the historical data abnormal events, and a feature data set is established according to the text keyword vectors of each data corresponding to the text information; the text keyword vectors are standardized, frequent item sets of the standardized operation feature set are mined based on an apriori association rule algorithm, and strong association rules of the historical data abnormal events are determined according to the frequent item sets of the standardized operation feature set; The text keyword vector of the data in the current database is extracted, the text keyword vector of the data in the current database is matched with the strong association rule, and the abnormal parameter of the data in the current database is obtained.
5. The intelligent analysis system for big data according to claim 4, wherein, The standardized operation feature set is discretized to obtain a discretized operation feature set. The discretized operation feature set is converted into a transaction type data format to obtain a transaction type operation feature set. The minimum support threshold and the minimum confidence threshold are set, the transaction type operation feature set is iteratively scanned by using the Apriori algorithm, and a candidate item set is generated. The support of each candidate item set is calculated, and the frequent item set satisfying the minimum support threshold is screened from the candidate item set according to the support. The confidence of each association rule is calculated, and the strong association rule satisfying the minimum confidence threshold is screened from the association rule according to the confidence. The text keyword vector of the data in the current database is extracted, the text keyword vector of the data in the current database is matched with the strong association rule, and the abnormal parameter of the data in the current database is obtained.
6. The intelligent analysis system for big data according to claim 5, wherein, The strong association rule of the data text keyword vector corresponding to different data abnormal events in the preset database is obtained according to the strong association rule of the historical data abnormal event. The data text keyword vector in the current database is matched with the strong association rule of the data text keyword vector corresponding to different data abnormal events in the preset database to obtain the matching degree between the data text keyword vector in the current database and the strong association rule of the data text keyword vector corresponding to different data abnormal events in the preset database. The data text keyword vector in the current database is filtered out, and the abnormal parameter is determined according to the corresponding matching degree. The abnormal data in the database is determined according to the abnormal parameter and the abnormal data threshold by the early warning module.
7. The intelligent analysis system for big data according to claim 1, wherein, The difference between the abnormal parameter of the data in the current database and the abnormal data threshold is calculated, and it is judged whether the corresponding data is abnormal data according to the difference between the abnormal parameter of the data in the current database and the abnormal data threshold. The difference between the abnormal parameter of the data in the current database and the abnormal data threshold is calculated, and it is judged whether the corresponding data is abnormal data according to the difference between the abnormal parameter of the data in the current database and the abnormal data threshold.
8. The intelligent analysis system for big data according to claim 7, wherein, The difference between the abnormal parameter of the data in the current database and the abnormal data threshold is calculated, and it is judged whether the corresponding data is abnormal data according to the difference between the abnormal parameter of the data in the current database and the abnormal data threshold. The text information of the current database is obtained, the data is clustered according to the content information of the text information, and the abnormal parameter threshold is determined according to the clustering result. 9. An intelligent analysis method for massive data, characterized in that, Obtaining a historical data abnormal event, mining a strong association rule of the historical data abnormal event, and determining an abnormal parameter of current data according to the strong association rule of the historical data abnormal event; Determining abnormal data in the database according to the abnormal parameter and an abnormal data threshold.