Data analysis system driven by intelligent algorithm

Abstract

The invention relates to the technical field of intelligent data processing, and comprises an intelligent algorithm-driven data analysis system, which comprises a feature screening module, a topology backbone extraction module, an information transmission optimization module, a classification boundary adjustment module and a self-adaptive classification hierarchy module. According to the method, the conditional entropy of a data characteristic value is calculated, the fluctuation degree of characteristic information is analyzed, one-way dominant characteristics are screened, interference of low-contribution characteristics is reduced, the topological adjacency relation of data points is constructed, a data classification trunk is optimized, and attribution of the data points is adjusted in combination with local information entropy change, so that a classification structure better conforms to data transmission characteristics; by combining the information entropy fluctuation of the classification boundary and the connection strength of the core data points, the classification boundary division is optimized, the hierarchical stability change rate is combined, the data hierarchical division strategy is adjusted, the classification structure is optimized, the stability and the calculation efficiency of data classification are improved, the classification boundary adjustment is more accurate, and the classification hierarchy can be adaptively adjusted.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent data processing, and in particular to a data analysis system driven by an intelligent algorithm. Background Art

[0002] The field of intelligent data processing technology includes various methods and systems for collecting, storing, analyzing and processing data using computer technology. The core content includes data acquisition and preprocessing, data storage and management, data calculation and analysis, data mining and pattern recognition, and data visualization. Data acquisition and preprocessing involves the collection, denoising, standardization and conversion of raw data to ensure the integrity and consistency of the data. Data storage and management mainly involve database technology, distributed storage technology and data index optimization to support efficient data access and management. Data calculation and analysis covers a variety of calculation methods for parsing, classifying and associating data to extract valuable information. Data mining and pattern recognition involve statistical analysis, pattern matching and anomaly detection of large amounts of data to discover potential data relationships and laws. Data visualization intuitively presents data analysis results in a graphical way to facilitate user understanding and decision-making. Overall, this technical field is committed to improving the efficiency and accuracy of data processing through a systematic approach to meet the needs of different application scenarios.

[0003] Among them, the data analysis system driven by intelligent algorithms refers to a system that uses intelligent algorithms to process and analyze data, mainly involving technical matters such as data collection, feature extraction, data conversion, classification and cluster analysis. Data collection obtains data from different sources through sensors, interface devices or data import methods, and formats and converts them to meet subsequent processing requirements. Feature extraction uses mathematical calculation methods to extract key information from raw data to reduce redundant data and improve analysis efficiency. Data conversion converts data from one form of expression to another through specific calculation methods to facilitate analysis and modeling. Classification and cluster analysis uses data calculation methods to classify data according to specific rules to discover similarities or differences between data and achieve in-depth analysis and structured processing of data.

[0004] In the process of data feature screening, existing data analysis technologies mostly rely on traditional correlation calculation methods, and fail to effectively evaluate the impact of features on target classification variables, resulting in some highly correlated but low-contribution features entering the classification model, affecting classification accuracy. In the topology structure optimization link, the construction process mainly relies on static similarity measurements between data points, and fails to effectively evaluate the role of data points in global information propagation, resulting in a lack of pertinence in the data classification path, affecting classification efficiency. In the classification optimization process, the classification boundary adjustment method does not combine the direction of information transmission, resulting in the classification of some data points being easily affected by local features, making the classification rules unstable. The construction of the classification hierarchy mostly relies on static rules, and fails to adjust according to the actual hierarchical relationship of the data, making it difficult for the hierarchical division method to adapt to the dynamic changes in data distribution, affecting the rationality of the classification hierarchy, causing information redundancy in the data classification process, and reducing the adaptability and accuracy of the classification results. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a data analysis system driven by an intelligent algorithm.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: A data analysis system driven by an intelligent algorithm comprises: The feature screening module obtains data feature values, calculates conditional entropy, counts the degree of feature information fluctuation, calculates the information gain value and information gain deviation value of the feature for the target classification variable, screens unidirectional dominant features, and obtains the optimal feature set; The topological backbone extraction module calculates the topological adjacency relationship of the data points based on the preferred feature set, calculates the local connectivity, constructs a minimum spanning tree, calculates the information transfer centrality, and obtains the topological backbone network; The information transmission optimization module determines the classification influence degree based on the topological backbone network, counts the change of local information entropy, selects the classification reference benchmark and calculates the information contribution of weak connection points, and obtains data to optimize the classification structure; The classification boundary adjustment module calculates the direction of data point information transmission based on the data optimization classification structure, counts the change of classification boundary information entropy, selects core data points and calculates the connection strength, adjusts the classification boundary, and outputs the adjusted classification boundary; The adaptive classification hierarchy module calculates local density fluctuations based on the adjusted classification boundaries, determines hierarchical relationships, determines the initial classification hierarchy of the data, calculates the classification stability change rate, adjusts the hierarchy division strategy, and obtains an adaptive data classification hierarchy structure.

[0007] As a further scheme of the present invention, the preferred feature set includes information gain value, information gain deviation value, information gain dominant direction, feature priority ranking, and feature contribution threshold; the topological backbone network includes topological adjacency, local connectivity, minimum spanning tree, core connection path, information transmission centrality, and transmission threshold screening results; the data optimization classification structure includes information transmission directionality, local information entropy change analysis records, classification reference benchmarks, weak connection point information contribution, and data category attribution adjustment records; the adjusted classification boundaries include information propagation trend analysis results, consistency judgment results, local information entropy of classification boundaries, information propagation core data points, core data point connection strength, and classification boundary optimization results; the adaptive data classification hierarchical structure includes local density fluctuation analysis results, initial classification levels, classification stability change rate, classification entropy value change rate, stability threshold, and data hierarchy division strategy.

[0008] As a further solution of the present invention, the feature screening module includes: The feature information calculation submodule calculates the conditional entropy of each feature based on the data features, obtains the information fluctuation degree of each feature under the given other feature values, calculates the information gain value of each feature for the target classification variable, and obtains the feature information gain matrix; The information gain evaluation submodule calculates the information contribution value of the feature to the target feature based on the feature information gain matrix, using the formula: ; Calculate features Features The information gain deviation value , filter out the unidirectional dominant features and construct the information gain deviation matrix between features, where Representative features In Features The information gain value in a given situation is, Representative features In Features The information gain value for a given situation; The preferred feature set construction submodule prioritizes the features according to the information gain dominant direction based on the information gain deviation matrix between the features, eliminates the features whose information contribution is lower than the information gain deviation threshold, and obtains the preferred feature set.

[0009] As a further solution of the present invention, the topology backbone extraction module includes: The topological adjacency calculation submodule calculates the topological adjacency relationship between data points based on the preferred feature set, and calculates the adjacency matrix according to the spatial distance and attribute similarity of the data points, using the formula: ; Calculate data points The local connectivity , output the local connectivity data of the data point, where Representative data points and adjacent data points The strength of the relationship between Representative data points With data points The similarity score between Representative data points The set of adjacent points of Representative data points The total similarity weight of all neighboring points; The minimum spanning tree construction submodule determines the connection stability of the data points in the topological network based on the local connectivity data of the data points, and constructs a minimum spanning tree by sorting the connection weights. The connection weights are determined by the adjacency relationship and local connectivity between the data points, and the shortest path tree algorithm is used to construct the data core connection path; The information transmission screening submodule calculates the information transmission centrality of the data point in the minimum spanning tree according to the data core connection path. The centrality calculation adopts the weighted average transmission path method. The nodes with information transmission amount lower than the transmission threshold are screened and eliminated according to the calculated centrality value to obtain the topological backbone network.

[0010] As a further solution of the present invention, the information transmission optimization module includes: The influence calculation submodule calculates the connection strength of each data point with other data points based on the topological backbone network, and counts the change value of local information entropy using the formula: ; Calculate data points Information influence , obtain the data point information influence data, and Represents data points and The information status value of and is the local information entropy of the corresponding data point, Represents data points The set of neighbors of ; The information leading point screening submodule screens the information leading data points based on the information influence data of the data points, calculates the average influence threshold of the data points, and selects the data points with influence higher than the average influence threshold as the information leading points, and obtains the information leading point set; The optimized classification structure generation submodule calculates the contribution of the weak connection point according to the information dominant point set, and determines whether the contribution is lower than the information entropy change threshold. If it is lower than the threshold, the target data point is assigned to the adjacent high-contribution data category to obtain the data optimized classification structure.

[0011] As a further solution of the present invention, the classification boundary adjustment module includes: The information propagation trend analysis submodule obtains the information transmission direction of the data point based on the data optimization classification structure, calculates the information propagation path of adjacent data points, counts the information flow trend between different categories, determines whether the information propagation direction is consistent with the classification boundary direction, and calculates the information propagation trend deviation value; The classification boundary information entropy calculation submodule calculates the information entropy of the local classification boundary based on the information propagation trend deviation value, analyzes the change of information entropy in the difference area, determines whether there is a local anomaly in the information entropy change, and uses the formula: ; Calculate the information entropy gradient value , and output information entropy gradient data, where Representative The information entropy of a local area, represents the information entropy of the previous region, represents the distance between adjacent regions, Represents the deviation value of information dissemination trend, represents the mean of all deviation values, represents the total number of local areas, Represents the number of deviation values ​​calculated; The core data point screening and boundary adjustment submodule screens the information propagation core data points according to the information entropy gradient data, calculates the connection strength between the core data points and the surrounding categories, analyzes the degree of connection between the difference categories, adjusts the data classification boundary, and obtains the adjusted classification boundary.

[0012] As a further solution of the present invention, the adaptive classification hierarchy module includes: The topological local density fluctuation calculation submodule is based on the adjusted classification boundary and adopts the formula: ; Calculate the average density of each node and calculate the degree of fluctuation: ; Get local density fluctuation data, where Representative Node The local density of Representative Node The set of neighbor nodes of Representative Node With Node The connection weights between Representative Node The local density fluctuation value, Represents the average density of neighbor nodes; The data level initial classification submodule determines whether the data has a hierarchical relationship based on the local density fluctuation data, determines the hierarchical division method of the data, calculates the node cluster centers and their distribution relationships in each level, and performs cluster analysis to generate an initial classification hierarchical structure; The classification stability adjustment submodule calculates the classification stability change rate between different levels and the classification entropy change rate within the target level according to the initial classification hierarchy structure, and compares them with the stability threshold. If the change rate exceeds the stability threshold, the data partitioning strategy of the target level is adjusted to obtain an adaptive data classification hierarchy structure.

[0013] Compared with the prior art, the advantages and positive effects of the present invention are: In the present invention, by calculating the conditional entropy of data eigenvalues, analyzing the degree of characteristic information fluctuation and screening unidirectional dominant features, reducing the interference of low-contribution features, constructing the topological adjacency relationship of data points, optimizing the data classification backbone, reducing redundant computing overhead, adjusting the attribution of data points in combination with local information entropy changes, making the classification structure more in line with data transmission characteristics, calculating the information transmission direction of data points, combining the information entropy fluctuation of classification boundaries with the connection strength of core data points, optimizing the classification boundary division, calculating the local density fluctuation of the topological network, combining the level stability change rate, adjusting the data level division strategy, making the classification level more in line with the topological hierarchical relationship of the data, reducing the interference of irrelevant features, optimizing the classification structure, improving the stability and computing efficiency of data classification, reducing data computing redundancy, making the classification boundary adjustment more accurate, and ensuring that the classification level can be adaptively adjusted. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a system flow chart of the present invention; Figure 2 This is a flow chart of the feature screening module of the present invention; Figure 3 This is a flow chart of the topology backbone extraction module of the present invention; Figure 4 This is a flow chart of the information transmission optimization module of the present invention; Figure 5 This is a flow chart of the classification boundary adjustment module of the present invention; Figure 6 This is a flow chart of the adaptive classification hierarchy module of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0016] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0017] See also Figure 1 , an intelligent algorithm-driven data analysis system includes: The feature screening module obtains data feature values, calculates the conditional entropy of each data feature, counts the degree of information fluctuation of each feature given other feature values, calculates the information gain value of the data feature to the target classification variable, records the information contribution value of the target feature to other features, calculates the one-to-two information gain value and the two-to-one information gain value for features one and two, and calculates the information gain deviation value of the two, screens the one-way dominant feature, prioritizes the features according to the dominant direction of the information gain, eliminates the features whose information contribution is lower than the information gain deviation threshold, and obtains the preferred feature set; The topological backbone extraction module calculates the topological adjacency of data points based on the optimal feature set, calculates the local connectivity of data points, determines the connection stability of data points in the topological network, constructs a minimum spanning tree, determines the core connection path of the data, calculates the information transfer centrality of the data points in the minimum spanning tree, screens and removes nodes whose information transfer amount is lower than the transfer threshold, and obtains the topological backbone network; The information transmission optimization module is based on the topological backbone network, determines the degree to which the classification of data points is affected by other data points, calculates the local information entropy changes of data points, selects the information-dominant data points as the classification reference, and calculates the information contribution of weak connection points. If the contribution is lower than the information entropy change threshold, the target data point is classified into the adjacent high-contribution data category to obtain the data optimization classification structure; The classification boundary adjustment module calculates the information transmission direction of the data point according to the data optimization classification structure, determines the consistency of the classification boundary and the information propagation trend, counts the changes in the local information entropy of the classification boundary, selects the core data points of information propagation, calculates the connection strength between the core data points and the surrounding categories, adjusts the data classification boundary, and outputs the adjusted classification boundary; The adaptive classification hierarchy module calculates the local density fluctuation of the topological network based on the adjusted classification boundaries, determines whether the data has a hierarchical relationship, dynamically determines the initial classification hierarchy of the data, and calculates the classification stability change rate of the difference hierarchy. If the classification entropy value change rate within the target hierarchy is greater than the stability threshold, the data partitioning strategy of the target hierarchy is adjusted to obtain an adaptive data classification hierarchy structure.

[0018] The optimal feature set includes information gain value, information gain deviation value, information gain dominant direction, feature priority ranking, and feature contribution threshold; the topological backbone network includes topological adjacency, local connectivity, minimum spanning tree, core connection path, information transmission centrality, and transmission threshold screening results; the data optimization classification structure includes information transmission directionality, local information entropy change analysis records, classification reference benchmarks, weak connection point information contribution, and data category attribution adjustment records; the adjusted classification boundaries include information propagation trend analysis results, consistency judgment results, local information entropy of classification boundaries, information propagation core data points, core data point connection strength, and classification boundary optimization results; the adaptive data classification hierarchical structure includes local density fluctuation analysis results, initial classification levels, classification stability change rate, classification entropy value change rate, stability threshold, and data hierarchy division strategy.

[0019] See also Figure 2 , the feature screening module includes: The feature information calculation submodule calculates the conditional entropy of each feature based on the data features, obtains the information fluctuation degree of each feature under the given other feature values, calculates the information gain value of each feature for the target classification variable, and obtains the feature information gain matrix; Based on the original data set, each feature variable is extracted and conditional entropy is calculated to measure the randomness of each feature. The conditional entropy calculation method is based on the sum of the joint probability distribution of different feature values. For example, have possible values, the conditional entropy is calculated as: ; in, represent The number of possible values, i.e. the target categorical variable The number of categories, Representative features The number of possible values, i.e., features The number of different states of is the target variable, Features Value situation, represent Under the condition of value The probability distribution of .

[0020] Table 1.1 Example table of conditional entropy calculation: ; As shown in Table 1.1, the calculated conditional entropy value of each feature measures the randomness of the feature, and further calculates the information gain of each feature: ; in, The target variable The entropy is calculated as: ; The calculated information gain matrix is ​​as follows: Table 1.2 Information gain calculation results: ; As shown in Table 1.2, the larger the information gain value, the greater the contribution of the feature to the target variable, and finally the feature information gain matrix is ​​obtained.

[0021] The information gain evaluation submodule calculates the information contribution value of the feature to the target feature based on the feature information gain matrix, using the formula: ; Calculate features Features The information gain deviation value , filter out the unidirectional dominant features and construct the information gain deviation matrix between features, where Representative features In Features The information gain value in a given situation is, Representative features In Features The information gain value for a given situation; According to the feature information gain matrix, calculate the feature Features The information contribution value is an indicator to measure the degree to which a feature provides information to other features. In order to measure the dominant direction of information between two features, the one-to-two information gain is calculated. and two-to-one information gain , and further calculate the information gain deviation.

[0022] Setting Features and Features The information gain is as follows: ; ; Enter the formula to calculate: ; This gives us the information gain deviation matrix between features.

[0023] Table 1.3 Information gain deviation matrix between features: ; As shown in Table 1.3, information gain deviation is used to screen out features with a unidirectional dominant effect. If the deviation value of a feature to other features exceeds the set threshold, the feature is regarded as the dominant feature in information contribution.

[0024] The optimal feature set construction submodule prioritizes features according to the information gain dominant direction based on the information gain deviation matrix between features, removes features whose information contribution is lower than the information gain deviation threshold, and obtains the optimal feature set; According to the information gain deviation matrix between features, the features are prioritized according to the dominant direction of information gain, and features whose information contribution is lower than the information gain deviation threshold are eliminated. The setting of the information gain deviation threshold is based on the entropy distribution of the target variable and the mean information gain between features. The specific value is calculated as follows: The mean information gain between features is: ; The information gain deviation threshold is 65% of the mean information gain between features, that is: ; The basis for this setting is that when the information gain deviation shows a large difference between features, it often leads to an increase in redundant features. Therefore, the threshold is set to 65% of the mean information gain, that is, some features with higher information gain contributions are retained, while features with smaller contributions are discarded, making feature screening more representative.

[0025] According to the calculated threshold value of 0.18, the preferred feature set is selected from Table 1.3: The deviation of feature AB is 0.20, which is higher than the threshold, so feature A is retained; The feature BC deviation is 0.15, which is lower than the threshold, so feature B is eliminated; The final optimal feature set is {A}.

[0026] See also Figure 3 ,The topological backbone extraction module includes: The topological adjacency calculation submodule calculates the topological adjacency relationship between data points based on the optimal feature set, and calculates the adjacency matrix based on the spatial distance and attribute similarity of the data points, using the formula: ; Calculate data points The local connectivity , output the local connectivity data of the data point, where Representative data points and adjacent data points The strength of the relationship between Representative data points With data points The similarity score between Representative data points The set of adjacent points of Representative data points The total similarity weight of all neighboring points; Based on the optimal feature set, we first need to extract the feature information of each data point from the data set, such as spatial coordinates, attribute similarity, etc. For example, in a geographic information system (GIS), each data point can represent a city, and its feature information includes key indicators such as longitude and latitude, population density, and GDP. Then, we calculate the topological adjacency relationship between the data points, which requires the construction of an adjacency matrix, in which each element Representative data points With data points The strength of the relationship between is determined by the inverse of the spatial distance, that is: ; in, For data points and For example, suppose there are two data points A(2,3) and B(5,7), and their distance is calculated as follows: ; Substitute into the weight calculation formula: ; After calculating the entire data set, the adjacency matrix can be obtained. Next, the local connectivity of the data points is calculated by local density estimation, that is: ; in, Representative data points With data points The similarity score between them can be calculated by the cosine similarity between attribute features, for example: ; Assume there are data points A and B, whose attribute vectors are , ,but: ; After calculating the adjacency weights and similarity scores of all points, substitute them into the final calculation of the local connectivity of each data point, as shown in Table 2.1: Table 2.1 Data point local connectivity calculation table: ; As shown in Table 2.1, the local connectivity of the data points can be obtained by calculation.

[0027] The minimum spanning tree construction submodule determines the connection stability of data points in the topological network based on the local connectivity data of data points, and constructs a minimum spanning tree by sorting the connection weights. The connection weights are determined by the adjacency relationship and local connectivity between data points, and the shortest path tree algorithm is used to construct the data core connection path; According to the local connectivity of the data points, firstly, the connection stability of the data points in the topological network is judged based on the sorting of the local connectivity. For the connection stability in the topological network, the weight distribution method can be used to calculate, that is, if the data point Connect to data point When the connection stability It can be calculated as: ; For example, the local connectivity of data points A and B are , ,but: ; After calculating all possible connections, they are sorted from high to low according to the weight value, and a minimum spanning tree is constructed. The minimum spanning tree uses the Kruskal or Prim algorithm to connect data points according to the weight priority and gradually form a data core connection path, as shown in Table 2.2.

[0028] Table 2.2 Spanning tree core connection path: ; As shown in Table 2.2, the data core connection path is calculated.

[0029] The information transmission screening submodule calculates the information transmission centrality of the data point in the minimum spanning tree according to the data core connection path. The centrality calculation adopts the weighted average transmission path method. According to the calculated centrality value, the nodes with information transmission amount lower than the transmission threshold are screened and eliminated to obtain the topological backbone network. Based on the data core connection path, the information transfer centrality of the data point in the minimum spanning tree is calculated. Information transfer centrality can be calculated by weighted path: ; in, For data points The transitive centrality of Representative data points For example, suppose data point A is connected to B, B is connected to C, and the path weights are as follows: ; ; ; Then the transitive centrality of A is: ; Similarly, the information transfer centrality of all data points is calculated, as shown in Table 2.3: Table 2.3 Data point information transmission centrality: ; The information transmission threshold is set based on the average transmission centrality and standard deviation of the data points to ensure that the selected data points can effectively undertake the information transmission task in the network. When setting the information transmission threshold, first calculate the mean transmission centrality of all data points. and standard deviation , and determine the screening threshold based on the data distribution.

[0030] Assume that the transitive centrality distribution of data points is as follows: ; ; ; ; ; Calculate the mean: ; Calculate the standard deviation: ; ; ; Information transmission threshold Set to ,Right now: ; According to the calculation results, the information transmission threshold is finally set to 0.35 , nodes below this threshold will be removed. For example, data point A , so it is removed from the topological network. Finally, data points B and C are retained to form the final topological backbone network.

[0031] See also Figure 4 , the information transmission optimization module includes: The influence calculation submodule calculates the connection strength of each data point with other data points based on the topological backbone network, and counts the change value of local information entropy using the formula: ; Calculate data points Information influence , obtain the data point information influence data, and Represents data points and The information status value of and is the local information entropy of the corresponding data point, Represents data points The set of neighbors of ; Based on the topological backbone network, first obtain the data point set , where each data point Has a specific location information and initial information state value , and define the connection relationship between data points ,in Representative data points and The strength of the connection is determined by the information flow characteristics. If there is information transfer between two data points, the connection weight is defined. , whose value can be calculated based on communication frequency, similarity or shared information. For each data point , and calculate its local information entropy , can be calculated by the following formula: ; in Represents data points In category The probability in , which can be estimated from historical data or the distribution of similar data points. For example, for a data point , if it may belong to the category There are three types, and their probabilities can be calculated based on the training data: , and thus calculate the information entropy: ; After calculating the information entropy of all data points, it is necessary to evaluate the degree of influence between the data points. The influence degree calculation formula is used to calculate the data points After calculating the value of , we can get the information influence of the data point. In practical applications, such as analyzing the spread of social network information, if a user Status information , its neighbor user The status are , and their information entropy are , then calculate its influence: ; Finally, the influence of data point information is obtained for subsequent screening process.

[0032] The information leading point screening submodule screens the information leading data points based on the information influence data of the data points, calculates the average influence threshold of the data points, and selects the data points with influence higher than the average influence threshold as the information leading points to obtain the information leading point set; Based on the influence of data point information, first calculate the average influence threshold of the data point , assuming there are five data points, their influences are ,but: ; Filter out those with influence higher than The data point of is used as the information leading point, that is, the data point in this example and Selected as the information dominant point, and obtain the information dominant point set.

[0033] The optimized classification structure generation submodule calculates the contribution of weak connection points based on the set of information dominant points, and determines whether the contribution is lower than the information entropy change threshold. If it is lower than the threshold, the target data point is assigned to the adjacent high-contribution data category to obtain the data optimized classification structure. Call the information dominant point set to calculate the contribution of weak connection points : ; in, Representative data points With its neighbors The connection weights between is the influence of neighboring data points. Weight The setting is based on the data point and The information transmission intensity between them is calculated from the historical data flow, interaction frequency and similarity. Normalization is usually used to make the weight value The more frequent the information exchange between data points, the closer the weight is to 1. For points with less information exchange, the weight tends to be in the lower range of 0.1-0.2. For example, suppose a data point and its adjacent data points and ,like In the past 24 hours 120 interactions with If the interaction is 80 times, you can set .

[0034] Obtained from the above , then calculate: ; If the contribution Below the information entropy change threshold , then the data point belongs to the adjacent high-contribution data category. Threshold The setting refers to the global information entropy distribution. Usually, the mean of the information entropy of all data points is selected as the benchmark value, and an adjustable scaling factor is set. Make fine-tuning, namely: ; in, It reflects the strictness of information screening and is usually , to adapt to the discreteness of different data sets. Assuming that the mean information entropy in a data set is 0.65, if you choose , then In this case, ,therefore Belongs to the neighboring data point with the highest contribution The category in which the data is located is finally obtained to optimize the classification structure.

[0035] See also Figure 5 , the classification boundary adjustment module includes: The information propagation trend analysis submodule obtains the information transmission direction of data points based on the data optimization classification structure, calculates the information propagation path of adjacent data points, counts the information flow trend between different categories, determines whether the information propagation direction is consistent with the classification boundary direction, and calculates the information propagation trend deviation value; Obtain the information transmission direction of the data points, analyze the information transmission path between data points based on historical data and real-time data. For example, in urban traffic flow analysis, the real-time traffic flow of each traffic intersection can be obtained, the traffic migration between each intersection can be calculated, the information propagation direction can be statistically analyzed, and the information propagation relationship matrix between data points can be established to further analyze the classification boundaries, calculate the information propagation path of adjacent data points, and set the propagation weight between data points. The weight value is set based on the information transmission rate and propagation stability between data points. The flow change rate between data points in the past 10 minutes is used as the basic weight calculation item. If the flow change rate exceeds the set threshold (such as greater than 50 vehicles / minute), the weight value is increased, otherwise the weight is reduced. The calculation method is: ; in, Representative data points and The propagation weight between for arrive The data transfer rate, is the global maximum data transmission rate, is the weight scaling factor, set to 2, so that the weight value is distributed in the interval [0, 2]. For example, if the average flow rate between two intersections in the past 10 minutes is 40 vehicles / minute, and the global maximum flow rate is 80 vehicles / minute, then ; Statistics are collected on the information flow trends between different categories. By calculating the information flow ratio between the inside and outside of each category, it is determined whether the direction of information propagation is consistent with the direction of the classification boundary. If the information flow ratio is within a specific range (such as 0.8-1.2), the information propagation trend is determined to be stable. Otherwise, it is determined that there may be problems with the classification boundary. The information propagation trend deviation value is calculated, and the threshold of the deviation value is set to 0.2. This value comes from the mean calculation of historical data and is set after variance analysis of the information propagation trend over multiple time periods, so that the offset trend of the data point can be controlled. If the deviation value of a data point is higher than the threshold (such as 0.3), it indicates that there is an abnormality in its classification boundary and further adjustment is required.

[0036] The classification boundary information entropy calculation submodule calculates the information entropy of the local classification boundary based on the information propagation trend deviation value, analyzes the change of information entropy in the difference area, determines whether there is a local anomaly in the change of information entropy, and uses the formula: ; Calculate the information entropy gradient value , output information entropy gradient data, where Representative The information entropy of a local area, represents the information entropy of the previous region, represents the distance between adjacent regions, Represents the deviation value of information dissemination trend, represents the mean of all deviation values, represents the total number of local areas, Represents the number of deviation values ​​calculated; Based on the information propagation trend deviation value, the information entropy of the local classification boundary is calculated. First, the information entropy calculation range of the data point is set. For example, in the manufacturing data classification, the sensor data of the adjacent area is selected as the local range, and the information entropy of each category in the area is calculated. For example, if a production area has three types of abnormal conditions (low temperature, high pressure, vibration), the information entropy is calculated as: ; in, is the probability of occurrence of each category, for example, the proportions of low temperature, high pressure, and vibration are 0.4, 0.3, and 0.3 respectively, then:

[0037] Furthermore, the changes in information entropy in different regions are analyzed, the local information entropy gradient is calculated by difference, and the information entropy gradient value of the classification boundary is obtained by formula operation. For example, if the continuous information entropy of a certain region is , the corresponding area spacing is 1, then calculate: ; The information entropy gradient value is 0.18, and the local information entropy change trend is analyzed, which can be used to adjust the classification boundary.

[0038] The core data point screening and boundary adjustment submodule screens the core data points of information dissemination according to the information entropy gradient data, calculates the connection strength between the core data points and the surrounding categories, analyzes the degree of connection between different categories, adjusts the data classification boundary, and obtains the adjusted classification boundary; According to the trend of local information entropy change, the core data points of information propagation are screened. The screening criteria can be based on the peak area of ​​the local information entropy gradient. For example, when the information entropy change exceeds 0.15, the core data points in the area are identified, and the connection strength between the core data points and the surrounding categories is calculated. For example, the distribution of each core data point in different categories is counted, and the degree of belonging between categories is calculated. For example, if the connection degree of a core data point in category A is 0.7 and the connection degree in category B is 0.3, then its belonging category is category A. If the connection strength is less than 0.6, it is determined to be a boundary point and classification adjustment is required. The degree of connection between different categories is analyzed. The connection strength is calculated using the normalized category density ratio, and its calculation formula is: ; in, Representative data points The strength of the connection in a certain category, is the number of neighbors of the data point in the category, is the total number of neighbors, and the connection strength threshold is set to 0.6. This value is derived from the distribution density statistics of the boundary points in the classification data set, so that the data points between low-density categories can be reasonably classified. For example, if the number of neighbors of a data point in category A is 15 and the total number of neighbors is 25, then its connection strength is: ; If the classification threshold is reached, and if it is lower than 0.6, such as 0.55, the point is considered to be a boundary point and the classification needs to be adjusted. If the proportion of core data points of a certain category transferred to other categories exceeds 30%, the boundary is adjusted to finally obtain the adjusted classification boundary.

[0039] See also Figure 6 , the adaptive classification hierarchy module includes: The topological local density fluctuation calculation submodule is based on the adjusted classification boundary and adopts the formula: ; Calculate the average density of each node and calculate the degree of fluctuation: ; Get local density fluctuation data, where Representative Node The local density of Representative Node The set of neighbor nodes of Representative Node With Node The connection weights between Representative Node The local density fluctuation value, Represents the average density of neighbor nodes; Based on the adjusted classification boundary, the local density of each node in the topological network needs to be calculated. First, the topological network consists of a series of nodes and its adjacent nodes Each node is composed of The connection weight represents the interaction strength between two nodes, and then the standard deviation of each node density is calculated. , which is used to measure the degree of fluctuation of the local density of the node. Assuming that the local densities of adjacent nodes are 3.2, 2.8, 4.1 and 3.5 respectively, the average value is: ; Substitute to calculate the degree of volatility: ; when If it exceeds the set threshold (such as 0.5), it means that the local density fluctuation of the node is large, which may belong to an abnormal node or hierarchical junction in the topological structure, thus affecting the classification hierarchy and calculating the local density fluctuation value.

[0040] The data hierarchical initial classification submodule determines whether the data has a hierarchical relationship based on the local density fluctuation data, determines the hierarchical division method of the data, calculates the node cluster centers and their distribution relationships in each level, and performs cluster analysis to generate the initial classification hierarchical structure; Based on the local density fluctuation data, it is judged whether the data has a hierarchical relationship. When classifying the data in layers, the threshold method can be used, that is, the threshold of the local density fluctuation value can be set. To determine whether the hierarchy needs to be divided, for example, , if a node , it is considered to be located at the intersection of the levels and needs to be used as a new level node to guide the subsequent classification strategy. The setting of is based on the overall density fluctuation range in the network topology. Usually, the local density fluctuation mean of all nodes plus 10% of the standard deviation is taken as the value to ensure the stability of the hierarchical division. Assuming that the local density fluctuation mean of 100 nodes in a network is 0.35 and the standard deviation is 0.08, then the calculation as follows: ; The final value is rounded up to two decimal places, so set .

[0041] In the classification hierarchy, the classification of nodes is based on topological characteristics. The cluster centers of nodes in each level and their distribution relationships are calculated. The cluster centers are determined using the mean method. For example, assuming that a level contains 5 nodes with densities of 3.2, 3.5, 3.8, 4.0, and 4.3, the center density of the level can be calculated as follows: ; In order to avoid the influence of outliers on the density of cluster centers, the weight coefficient needs to be set. To adjust the contribution of each node to the cluster center, the weight coefficient depends on the local density fluctuation value of the node. The weight is calculated as follows: ; in, is the maximum density fluctuation value in the current level. If The value range is , then the maximum value , then the weight of a node is calculated as follows: ; Finally, the center density of this level is calculated by weighting: ; According to the density fluctuation value and the cluster center density, the classification level is further adjusted. For example, if the density of a node is more than 15% higher than the hierarchical cluster center density, its classification level needs to be adjusted to obtain the initial classification hierarchy structure.

[0042] The classification stability adjustment submodule calculates the classification stability change rate between the difference levels according to the initial classification hierarchy structure, calculates the classification entropy value change rate within the target level, and compares it with the stability threshold. If the change rate exceeds the stability threshold, the data partitioning strategy of the target level is adjusted to obtain an adaptive data classification hierarchy structure. Call the initial classification hierarchy structure, calculate the classification stability change rate between different levels, and set the classification entropy value To measure the stability of the hierarchy, the calculation formula is as follows: ; in, Represents the proportion of data points in a certain category. Assuming that a certain level contains 50 data points, 30 of which belong to category A and 20 belong to category B, its entropy value is calculated as follows: ; Calculate the rate of change of classification entropy within the target level: ; Assuming that the entropy value before adjustment is 0.2923 and the entropy value after adjustment is 0.2500, the rate of change is calculated as follows: ; Entropy change rate The threshold is set to 0.1. The selection of this threshold is based on the statistical analysis of multiple groups of classification level data. The setting method is to calculate the entropy change rate of 100 groups of different classification levels and take the 90% quantile value as the stability threshold. Assuming that the mean entropy change rate of 100 groups of data is 0.085 and the standard deviation is 0.02, the threshold is calculated as follows: ; The final value is rounded up to two decimal places, so the stability threshold is set , if the entropy change rate , then the hierarchical structure is unstable and the data partitioning strategy needs to be adjusted.

[0043] If the rate of change of entropy is greater than the set stability threshold, it is necessary to adjust the data partitioning strategy of the target level, such as reallocating some data points to other levels to optimize classification stability. The reallocation of the data points is based on their information entropy contribution, and the contribution weight is set. , the calculation formula is as follows: ; Assume that the information entropy contribution of 30 data points in category A is calculated as follows: ; Similarly, calculate the contribution of category B: ; If the contribution of a category If it exceeds 0.5, it means that the influence of this classification is significant, and it can be used as the focus of hierarchical stability adjustment. By reducing the number of data points or adjusting the distribution of data points, the hierarchical division strategy can be optimized to finally obtain an adaptive data classification hierarchical structure.

[0044] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A data analysis system driven by an intelligent algorithm, characterized in that: The system comprises: The feature screening module obtains data feature values, calculates conditional entropy, counts the degree of feature information fluctuation, calculates the information gain value and information gain deviation value of the feature for the target classification variable, screens unidirectional dominant features, and obtains the optimal feature set; The topological backbone extraction module calculates the topological adjacency relationship of the data points based on the preferred feature set, calculates the local connectivity, constructs a minimum spanning tree, calculates the information transfer centrality, and obtains the topological backbone network; The information transmission optimization module determines the classification influence degree based on the topological backbone network, counts the change of local information entropy, selects the classification reference benchmark and calculates the information contribution of weak connection points, and obtains data to optimize the classification structure; The classification boundary adjustment module calculates the direction of data point information transmission based on the data optimization classification structure, counts the change of classification boundary information entropy, selects core data points and calculates the connection strength, adjusts the classification boundary, and outputs the adjusted classification boundary; The adaptive classification hierarchy module calculates local density fluctuations based on the adjusted classification boundaries, determines hierarchical relationships, determines the initial classification hierarchy of the data, calculates the classification stability change rate, adjusts the hierarchy division strategy, and obtains an adaptive data classification hierarchy structure.

2. The data analysis system driven by the intelligent algorithm according to claim 1, characterized in that: The preferred feature set includes information gain value, information gain deviation value, information gain dominant direction, feature priority ranking, and feature contribution threshold; the topological backbone network includes topological adjacency, local connectivity, minimum spanning tree, core connection path, information transmission centrality, and transmission threshold screening results; the data optimization classification structure includes information transmission directionality, local information entropy change analysis records, classification reference benchmarks, weak connection point information contribution, and data category attribution adjustment records; the adjusted classification boundaries include information propagation trend analysis results, consistency judgment results, local information entropy of classification boundaries, information propagation core data points, core data point connection strength, and classification boundary optimization results; the adaptive data classification hierarchical structure includes local density fluctuation analysis results, initial classification levels, classification stability change rate, classification entropy value change rate, stability threshold, and data hierarchy division strategy.

3. The data analysis system driven by the intelligent algorithm according to claim 1, characterized in that: The feature screening module includes: The feature information calculation submodule calculates the conditional entropy of each feature based on the data features, obtains the information fluctuation degree of each feature under the given other feature values, calculates the information gain value of each feature for the target classification variable, and obtains the feature information gain matrix; The information gain evaluation submodule calculates the information contribution value of the feature to the target feature based on the feature information gain matrix, using the formula: ； Calculate features Features The information gain deviation value , filter out the unidirectional dominant features and construct the information gain deviation matrix between features, where Representative features In Features The information gain value in a given situation is, Representative features In Features The information gain value for a given situation; The preferred feature set construction submodule prioritizes the features according to the information gain dominant direction based on the information gain deviation matrix between the features, eliminates the features whose information contribution is lower than the information gain deviation threshold, and obtains the preferred feature set.

4. The data analysis system driven by the intelligent algorithm according to claim 1, characterized in that: The topology backbone extraction module comprises: The topological adjacency calculation submodule calculates the topological adjacency relationship between data points based on the preferred feature set, and calculates the adjacency matrix according to the spatial distance and attribute similarity of the data points, using the formula: ； Calculate data points The local connectivity , output the local connectivity data of the data point, where Representative data points and adjacent data points The strength of the relationship between Representative data points With data points The similarity score between Representative data points The set of adjacent points of Representative data points The total similarity weight of all neighboring points; The minimum spanning tree construction submodule determines the connection stability of the data points in the topological network based on the local connectivity data of the data points, and constructs a minimum spanning tree by sorting the connection weights. The connection weights are determined by the adjacency relationship and local connectivity between the data points, and the shortest path tree algorithm is used to construct the data core connection path; The information transmission screening submodule calculates the information transmission centrality of the data point in the minimum spanning tree according to the data core connection path. The centrality calculation adopts the weighted average transmission path method. The nodes with information transmission amount lower than the transmission threshold are screened and eliminated according to the calculated centrality value to obtain the topological backbone network.

5. The data analysis system driven by the intelligent algorithm according to claim 1, characterized in that: The information transmission optimization module includes: The influence calculation submodule calculates the connection strength of each data point with other data points based on the topological backbone network, and counts the change value of local information entropy using the formula: ； Calculate data points Information influence , obtain the data point information influence data, and Represents data points and The information status value of and is the local information entropy of the corresponding data point, Represents data points The set of neighbors of ; The information leading point screening submodule screens the information leading data points based on the information influence data of the data points, calculates the average influence threshold of the data points, and selects the data points with influence higher than the average influence threshold as the information leading points, and obtains the information leading point set; The optimized classification structure generation submodule calculates the contribution of the weak connection point according to the information dominant point set, and determines whether the contribution is lower than the information entropy change threshold. If it is lower than the information entropy change threshold, the target data point is classified into the adjacent high-contribution data category to obtain the data optimized classification structure.

6. The data analysis system driven by the intelligent algorithm according to claim 1, characterized in that: The classification boundary adjustment module includes: The information propagation trend analysis submodule obtains the information transmission direction of the data point based on the data optimization classification structure, calculates the information propagation path of adjacent data points, counts the information flow trend between different categories, determines whether the information propagation direction is consistent with the classification boundary direction, and calculates the information propagation trend deviation value; The classification boundary information entropy calculation submodule calculates the information entropy of the local classification boundary based on the information propagation trend deviation value, analyzes the change of information entropy in the difference area, determines whether there is a local anomaly in the information entropy change, and uses the formula: ； Calculate the information entropy gradient value , and output information entropy gradient data, where Representative The information entropy of a local area, represents the information entropy of the previous region, represents the distance between adjacent regions, Represents the deviation value of information dissemination trend, represents the mean of all deviation values, represents the total number of local areas, Represents the number of deviation values ​​calculated; The core data point screening and boundary adjustment submodule screens the information propagation core data points according to the information entropy gradient data, calculates the connection strength between the core data points and the surrounding categories, analyzes the degree of connection between the difference categories, adjusts the data classification boundary, and obtains the adjusted classification boundary.

7. The data analysis system driven by the intelligent algorithm according to claim 1, characterized in that: The adaptive classification hierarchy module comprises: The topological local density fluctuation calculation submodule is based on the adjusted classification boundary and adopts the formula: ； Calculate the average density of each node and calculate the degree of fluctuation: ； Get local density fluctuation data, where Representative Node The local density of Representative Node The set of neighbor nodes of Representative Node With Node The connection weights between Representative Node The local density fluctuation value, Represents the average density of neighbor nodes; The data level initial classification submodule determines whether the data has a hierarchical relationship based on the local density fluctuation data, determines the hierarchical division method of the data, calculates the node cluster centers and their distribution relationships in each level, and performs cluster analysis to generate an initial classification hierarchical structure; The classification stability adjustment submodule calculates the classification stability change rate between different levels and the classification entropy change rate within the target level according to the initial classification hierarchy structure, and compares them with the stability threshold. If the change rate exceeds the stability threshold, the data partitioning strategy of the target level is adjusted to obtain an adaptive data classification hierarchy structure.

Images