Artificial intelligence big data processing and management system

Abstract

The invention discloses an artificial intelligence big data processing and management system and a data acquisition module, belongs to the field of big data processing, and comprises a data acquisition module, a preprocessing distribution module and a data analysis module. The method comprises the following steps: performing data acquisition on a plurality of data sources, and monitoring a data acquisition process to obtain logs and data acquisition time; preprocessing and privacy judgment are carried out on received collected data by utilizing a trained random forest model, and metadata is uploaded to a private chain or a public chain according to a label in a judgment result; an original data set is grouped by constructing a K-Means clustering algorithm model, the similar distance between each feature vector and a cluster center is calculated, and the consistency of a clustering result and real classification is evaluated; by analyzing the data transmission process, the data transmission interference ratio and the data transmission rate are obtained, and the data transmission effect is evaluated.

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and in particular to an artificial intelligence big data processing and management system. Background Art

[0002] Artificial intelligence (AI) technology can enhance the ability to process and analyze big data. For enterprises, the use of AI technology enables them to gain deeper insights from big data, make smarter decisions, improve competitiveness, and realize business data intelligence.

[0003] When processing data, most existing big data processing and management systems fail to optimize the data collection and processing process to ensure data quality and protect sensitive data from leakage; they do not further evaluate the consistency between clustering results and true results, thereby failing to improve the accuracy and robustness of clustering effects, resulting in an inability to more accurately mine the potential value in the data; they also do not analyze the data transmission process, thereby failing to improve the overall efficiency and response speed of the system. Summary of the invention

[0004] The purpose of the present invention is to provide an artificial intelligence big data processing and management system to address the deficiencies in the background technology.

[0005] The object of the present invention can be achieved by the following technical solutions: An artificial intelligence big data processing and management system, including a data acquisition module, for collecting data from multiple data sources, monitoring the data acquisition process and recording logs and data acquisition time; The preprocessing allocation module performs preprocessing and privacy judgment on the received collected data using the trained random forest model; and uploads the metadata to the private chain or public chain according to the labels in the judgment results; The data analysis module builds a K-Means clustering algorithm model to group the original data set, calculates the similarity distance between each feature vector and the cluster center, and evaluates the consistency between the clustering results and the true classification; By analyzing the data transmission process, the data transmission interference ratio and data transmission rate are obtained, and the effect of data transmission is evaluated; among which, the frame length, signal width and modulation level number of the data transmitted at the current moment are obtained, and the transmission delay is calculated; The calculation formula is: ; In the formula, t is the transmission delay, Q is the frame length of the data transmitted at the current moment, c represents the signal width of the transmitted data, and W represents the number of modulation levels; Calculate the data transmission anomaly coefficient: ; Where YC is the data transmission anomaly coefficient, α is the weight coefficient of the data transmission interference ratio, and β is the weight coefficient of the transmission delay; The acquired data transmission anomaly coefficient is compared with the data transmission threshold. If the data transmission anomaly coefficient is ≥ the data transmission threshold, the data transmission interference ratio and the transmission delay are analyzed to determine the value higher than the normal value, and measures are taken to optimize the network configuration according to the determination result. If the data transmission abnormality coefficient is less than the data transmission threshold, it means that the data transmission process is normal and no upgrade is required temporarily; the data transmission threshold is preset according to the actual situation of the system.

[0006] Preferably, the data acquisition module is used to simultaneously access multiple types of data sources to ensure that the required data content is obtained from different channels, and the collected data is integrated into a unified format for subsequent processing and analysis; wherein the multiple data sources include but are not limited to sensors, databases, and API interfaces.

[0007] Preferably, the preprocessing allocation module preprocesses the received collected data, arranges and combines the collected data in the order of collection time, cleans the collected data, and uses the trained random forest model to identify and remove abnormal data; Input preprocessed data, based on the random forest model containing m decision trees, for each tree, based on the path information of the data point from the root node to the leaf node, the random forest model will calculate the number of edges from the root node of the tree to the leaf node where the data point is located, that is, the path length of the data point , using the average path length of each trained tree To normalize this path length, get the anomaly score ; The calculation formula is: ; In the formula, Indicates the path length expectations; According to the data points of each tree The corresponding anomaly score , calculate the data points of all decision trees The corresponding sum of anomaly scores; The calculation formula is: ; In the formula, Represents the data points of all decision trees The corresponding sum of anomaly scores; represents the i-th data point; Set a threshold, marked as H, if >H, then the corresponding data point Mark as abnormal; if ≤H, then the corresponding data point Marked as normal; where the threshold H is calculated based on the actual normal data; Traverse the collection logs to classify normal data according to data type, and integrate normal data of the same type into metadata, including data content, source, and format.

[0008] Preferably, when implementing privacy judgment, the data content of the metadata is obtained and matched with the preset sensitive information. If the match is successful, the metadata corresponding to the data content is marked with a privacy label; if the match fails, the metadata corresponding to the data content is marked with a public label; wherein sensitive information includes personal privacy information and protected health information; Public labels or private labels constitute the judgment result; The judgment results are traversed, and the metadata corresponding to the privacy label is marked as private data and uploaded to the private chain for data storage, and the metadata corresponding to the public label is marked as public data and uploaded to the public chain for data update.

[0009] Preferably, the data analysis module extracts the preprocessed data set from the data warehouse, constructs a K-Means clustering algorithm model to group the data set, combines the elbow method to determine the optimal K value and the initial cluster center point, and verifies the consistency of the clustering result with the actual classification; Select the k value range, k ≥ 1 and is a positive integer, randomly select k data points as the initial cluster centers, form k clusters, establish the feature vector corresponding to each data, where the feature vector is composed of the quantitative attributes of the data point, and calculate the similarity distance between each feature vector and the k cluster center; The calculation formula is: ; Where x is the currently calculated feature vector, c is the preset feature vector corresponding to the current cluster center, dist(x,c) represents the similarity distance between the feature vector and the cluster center, Ed(x,c) represents the Euclidean distance between the feature vector and the cluster center, Md(x,c) represents the Manhattan distance between the feature vector and the cluster center, Cd(x,c) represents the cosine distance between the feature vector and the cluster center, ω1, ω2, and ω3 are different weight coefficients respectively; The calculation formula of Euclidean distance is: ; In the formula, is the jth cluster center, is the i-th data point; The calculation formula of Manhattan distance is: ; The calculation formula for cosine distance is: ; In the formula, represents the dot product of vectors x and c, , denote the Euclidean norm of vectors x and c respectively.

[0010] Preferably, each data is assigned to the nearest cluster center according to the obtained similarity distance, and for each k value, the sum of squares of the Euclidean distances SSE(k) from each data to the cluster center to which it belongs is calculated; The calculation formula is: ; Where N is the total number of data points x, C is the set of cluster centers under the current k value, is the jth cluster center, It’s data To its nearest cluster center The square of the Euclidean distance; Repeat the above steps until the cluster center no longer changes significantly or the pre-set number of iterations is reached; Draw a line graph of the k value and the corresponding SSE(k), and find the "elbow" point where SSE(k) decreases rapidly. The k value at this point is the optimal number of clusters. Use the selected optimal k value to execute the K-means algorithm to obtain the final clustering result.

[0011] Preferably, the final clustering result is compared with the true classification, the consistency between the clustering result and the true classification is evaluated, and the RI is calculated; The calculation formula is: ; In the formula, TP means two data of the same class are assigned to the same cluster, TN means two data of different classes are assigned to different clusters; C(N,2) is the number of combinations of two data points selected from N data points; RI is used to measure the consistency between the clustering result and the true classification; The RI value is between 0 and 1. The closer the value is to 1, the better the clustering effect is, and the better the consistency between the clustering result and the true classification is; Normalize RI and calculate ARI; The calculation formula is: ; Where E(RI) is the expected RI value of random clustering, max(RI) is the maximum RI value; ARI is an adjustment of RI, which takes into account the expected value in random clustering to standardize the actual RI value; The value of ARI is between -1 and 1, where 0 means that the clustering algorithm did not find any structure in the data, a value close to 1 means that the clustering effect is good, and a negative value means that the clustering effect is poor. That is, the larger the ARI value, the more consistent the clustering result is with the true classification, and the better the clustering effect.

[0012] Preferably, a big data integration platform is created to facilitate communication and data exchange between different modules, and the data transmission process is evaluated based on the abnormal coefficients obtained during the data transmission process; Monitor the data transmission process and determine the degree of interference in data transmission efficiency during the data transmission process: obtain the actual data transmission volume S during the data transmission process and establish a data transmission volume set , n is a positive integer, get the maximum data transmission value , minimum data transfer value , calculate the average data transfer value , calculate the data transmission interference ratio; The calculation formula is: ; Where G is the data transmission interference ratio.

[0013] Compared with the existing solutions, the present invention achieves the following beneficial effects: By collecting data from multiple data sources and monitoring the data collection process to obtain logs and data collection time, it is helpful to break the singleness of data sources; By using the trained random forest model to pre-process and determine privacy for the collected data, it helps to screen out sensitive data and protect it. At the same time, metadata is uploaded to the corresponding blockchain based on the data label to ensure data security and compliance. By building a K-Means clustering algorithm model to group the original data set, the similarity distance between each feature vector and the cluster center is calculated, and the consistency between the clustering results and the true classification is evaluated, thereby verifying the effectiveness of feature selection and model construction and improving the accuracy and stability of the clustering results; By analyzing the data transmission process and evaluating the effect of data transmission, the network configuration can be optimized and the efficiency and quality of data transmission can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention will be further described below in conjunction with the accompanying drawings.

[0015] Figure 1 This is a module block diagram of an artificial intelligence big data processing and management system of the present invention. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0017] Embodiment 1, as Figure 1As shown, an artificial intelligence big data processing and management system includes a data acquisition module, a preprocessing allocation module and a data analysis module; The data collection module is used to simultaneously access multiple types of data sources to ensure that the required data is obtained from different channels, which can be business data from the enterprise database; monitor the entire data collection process and record the collection log, including data type, collection time, data source, and data content; Convert the collected data information into a unified format or structure to facilitate subsequent processing and analysis, such as data type conversion, unit unification, and coding standardization; among them, multiple data sources include but are not limited to sensors, databases, and API interfaces.

[0018] The preprocessing allocation module preprocesses the received collected data, arranges and combines the collected data in the order of collection time, cleans the collected data, and uses the trained random forest model to identify and eliminate abnormal data; The method of building a random forest model is as follows: select a high-quality historical data set without too many missing values ​​or outliers, set as the training data set D, which contains n samples, each with g features; randomly extract n samples from the training data set D with replacement to form a new data set D', which is used to train each decision tree; For each tree , randomly select k features from g features, find the best split feature e and the corresponding split point c, split the node into two child nodes according to feature e and split point c, and stop splitting when the tree meets the stopping condition, such as reaching the maximum depth or the minimum number of node samples; each tree gives a prediction value, and the prediction values ​​of all trees are integrated and the average is taken as the final prediction; Use a test data set that has not participated in the training to evaluate the performance of the model and calculate the accuracy of the model. If the accuracy of the model does not meet the requirements, adjust the model parameters until the model meets the requirements. If the accuracy of the model meets the requirements, use the tuned random forest model to predict the actual collected data. The model parameters can be the number of trees, the maximum depth, and the number of features selected for each split. Input preprocessed data, based on the random forest model containing m decision trees, for each tree, based on the path information of the data point from the root node to the leaf node, the random forest model will calculate the number of edges from the root node of the tree to the leaf node where the data point is located, that is, the path length of the data point , using the average path length of each trained tree To normalize this path length, get the anomaly score ; The calculation formula is: ; In the formula, Indicates the path length expectations; According to the data points of each tree The corresponding anomaly score , calculate the data points of all decision trees The corresponding sum of anomaly scores; The calculation formula is: ; In the formula, Represents the data points of all decision trees The corresponding sum of anomaly scores; represents the i-th data point; Set a threshold, marked as H, if >H, then the corresponding data point Mark as abnormal; if ≤H, then the corresponding data point Marked as normal; where the threshold H is calculated based on the actual normal data; Traverse the collection logs to classify normal data according to data type, and integrate normal data of the same type into metadata, including data content, source, and format; When implementing privacy judgment, the data content of the metadata is obtained and matched with the preset sensitive information. If the match is successful, the metadata corresponding to the data content is marked with a privacy label; if the match fails, the metadata corresponding to the data content is marked with a public label; where sensitive information includes personal privacy information and protected health information; Public labels or private labels constitute the judgment result; The purpose of implementing privacy judgment here is to classify sensitive information of the acquired metadata. While ensuring data privacy, it can effectively improve the accuracy of subsequent data analysis and provide reliable data support for the implementation of data sharing. The judgment results are traversed, and the metadata corresponding to the privacy label is marked as private data and uploaded to the private chain for data storage, and the metadata corresponding to the public label is marked as public data and uploaded to the public chain for data update.

[0019] The data analysis module extracts the preprocessed data set from the data warehouse, builds a K-Means clustering algorithm model to group the data set, so that the data in the same group has high similarity, while the data in different groups has low similarity. The elbow method is combined to determine the best K value and the initial cluster center point, and verify the consistency of the clustering result with the actual classification; Select a k value range, k ≥ 1 and is a positive integer, randomly select k data points as the initial cluster centers, form k clusters, establish the feature vector corresponding to each data point, where the feature vector is composed of the quantitative attributes of the data point, such as size, color or any feature that can be represented numerically, and calculate the similarity distance between each feature vector and the k cluster centers; The calculation formula is: ; Where x is the currently calculated feature vector, c is the preset feature vector corresponding to the current cluster center, dist(x,c) represents the similarity distance between the feature vector and the cluster center, Ed(x,c) represents the Euclidean distance between the feature vector and the cluster center, Md(x,c) represents the Manhattan distance between the feature vector and the cluster center, Cd(x,c) represents the cosine distance between the feature vector and the cluster center, ω1, ω2, and ω3 are different weight coefficients, depending on the specific application scenario and the importance attached to different distance metrics; The calculation formula of Euclidean distance is: ; In the formula, is the jth cluster center, is the i-th data point; The calculation formula of Manhattan distance is: ; The calculation formula for cosine distance is: ; In the formula, represents the dot product of vectors x and c, , denote the Euclidean norm of vectors x and c respectively; Assign each data to the nearest cluster center according to the obtained similarity distance. For each k value, calculate the sum of squares of the Euclidean distances SSE(k) from each data to the center of its cluster. The calculation formula is: ; Where N is the total number of data points x, C is the set of cluster centers under the current k value, is the jth cluster center, It’s data To its nearest cluster center The square of the Euclidean distance; Repeat the above steps until the cluster center no longer changes significantly or the pre-set number of iterations is reached; Draw a line graph of the k value and the corresponding SSE(k), and find the "elbow" point where the SSE(k) decreases rapidly. The k value at this point is the optimal number of clusters. Use the selected optimal k value to execute the K-means algorithm to obtain the final clustering result. Compare the final clustering results with the true classification, evaluate the consistency between the clustering results and the true classification, and calculate the RI; The calculation formula is: ; In the formula, TP means two data of the same class are assigned to the same cluster, TN means two data of different classes are assigned to different clusters; C(N,2) is the number of combinations of two data points selected from N data points; RI is used to measure the consistency between the clustering result and the true classification; The RI value is between 0 and 1. The closer the value is to 1, the better the clustering effect is, and the better the consistency between the clustering result and the true classification is; Normalize RI and calculate ARI; The calculation formula is: ; Where E(RI) is the expected RI value of random clustering, max(RI) is the maximum RI value; ARI is an adjustment of RI, which takes into account the expected value in random clustering to standardize the actual RI value; The value of ARI is between -1 and 1, where 0 means that the clustering algorithm does not find any structure in the data, close to 1 means good clustering effect, and negative value means poor clustering effect. That is, the larger the ARI value, the more consistent the clustering result is with the true classification, and the better the clustering effect. It should be noted that ARI is an improvement on RI. It is more accurate and robust in evaluating clustering effects because it takes into account the similarities that may be generated by random clustering and improves the ability to distinguish between different clustering results.

[0020] Example 2: Creating a big data integration platform to facilitate communication and data exchange between different modules, and evaluating the data transmission process based on the abnormal coefficients obtained during the data transmission process; Monitor the data transmission process and determine the degree of interference in data transmission efficiency during the data transmission process: obtain the actual data transmission volume S during the data transmission process and establish a data transmission volume set , n is a positive integer, get the maximum data transmission value , minimum data transfer value , calculate the average data transfer value , calculate the data transmission interference ratio; The calculation formula is: ; Where G is the data transmission interference ratio; The larger the data transmission interference ratio is, the greater the interference to the data transmission is, the more the time it takes for the data to reach the destination will be affected, and the worse the real-time performance of the data transmission during the transmission process will be; Obtain the frame length, signal width and modulation level of the data being transmitted at the current moment, and calculate the transmission delay; The calculation formula is: ; In the formula, t is the transmission delay, Q is the frame length of the data transmitted at the current moment, c represents the signal width of the transmitted data, and W represents the number of modulation levels; Calculate the data transmission anomaly coefficient: ; Where YC is the data transmission anomaly coefficient, α is the weight coefficient of the data transmission interference ratio, and β is the weight coefficient of the transmission delay, all of which are obtained according to the system settings; The acquired data transmission anomaly coefficient is compared with the data transmission threshold. If the data transmission anomaly coefficient is ≥ the data transmission threshold, the data transmission interference ratio and the transmission delay are analyzed to determine the value higher than the normal value, and measures are taken to optimize the network configuration according to the determination result. If the data transmission abnormality coefficient is less than the data transmission threshold, it means that the data transmission process is normal and no upgrade is required temporarily; the data transmission threshold is preset according to the actual situation of the system.

[0021] In the several embodiments provided by the present invention, it should be understood that the disclosed system can be implemented in other ways. For example, the above-described embodiments of the invention are only illustrative, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation.

[0022] The modules described as separate accessories may or may not be physically separated, and the accessories shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0023] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0024] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0025] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. An artificial intelligence big data processing and management system, including a data acquisition module, for collecting data from multiple data sources, monitoring the data acquisition process and recording logs and data acquisition time; The preprocessing allocation module performs preprocessing and privacy judgment on the received collected data using the trained random forest model; and uploads the metadata to the private chain or public chain according to the labels in the judgment results; The data analysis module builds a K-Means clustering algorithm model to group the original data set, calculates the similarity distance between each feature vector and the cluster center, and evaluates the consistency between the clustering results and the true classification; By analyzing the data transmission process, the data transmission interference ratio and data transmission rate are obtained, and the effect of data transmission is evaluated; among which, the frame length, signal width and modulation level number of the data transmitted at the current moment are obtained, and the transmission delay is calculated; The calculation formula is: ; In the formula, t is the transmission delay, Q is the frame length of the data transmitted at the current moment, c represents the signal width of the transmitted data, and W represents the number of modulation levels; Calculate the data transmission anomaly coefficient: ; Where YC is the data transmission anomaly coefficient, α is the weight coefficient of the data transmission interference ratio, and β is the weight coefficient of the transmission delay; The acquired data transmission anomaly coefficient is compared with the data transmission threshold. If the data transmission anomaly coefficient is ≥ the data transmission threshold, the data transmission interference ratio and the transmission delay are analyzed to determine the value higher than the normal value, and measures are taken to optimize the network configuration according to the determination result. If the data transmission abnormality coefficient is less than the data transmission threshold, it means that the data transmission process is normal and no upgrade is required temporarily; wherein the data transmission threshold is preset according to the actual situation of the system.

2. The big data processing and management system of artificial intelligence according to claim 1, characterized in that: The data acquisition module is used to simultaneously access multiple types of data sources to ensure that the required data content is obtained from different channels, and to integrate the collected data into a unified format for subsequent processing and analysis; among them, multiple data sources include but are not limited to sensors, databases, and API interfaces.

3. The big data processing and management system of artificial intelligence according to claim 2, characterized in that: The preprocessing allocation module preprocesses the received collected data, arranges and combines the collected data in the order of collection time, cleans the collected data, and uses the trained random forest model to identify and eliminate abnormal data; Input preprocessed data, based on the random forest model containing m decision trees, for each tree, based on the path information of the data point from the root node to the leaf node, the random forest model will calculate the number of edges from the root node of the tree to the leaf node where the data point is located, that is, the path length of the data point , using the average path length of each trained tree To normalize this path length, get the anomaly score ; The calculation formula is: ; In the formula, Indicates the path length expectations; According to the data points of each tree The corresponding anomaly score , calculate the data points of all decision trees The corresponding sum of anomaly scores; The calculation formula is: ; In the formula, Represents the data points of all decision trees The corresponding sum of anomaly scores; represents the i-th data point; Set a threshold, marked as H, if >H, then the corresponding data point Mark as abnormal; like ≤H, then the corresponding data point Marked as normal; where the threshold H is calculated based on the actual normal data; Traverse the collection logs to classify normal data according to data type, and integrate normal data of the same type into metadata, including data content, source, and format.

4. The big data processing and management system of artificial intelligence according to claim 3, characterized in that: When implementing privacy judgment, the data content of the metadata is obtained and matched with the preset sensitive information. If the match is successful, the metadata corresponding to the data content is marked with a privacy label; If the match fails, the metadata corresponding to the data content will be marked with a public label; sensitive information includes personal privacy information and protected health information; Public labels or private labels constitute the judgment result; The judgment results are traversed, and the metadata corresponding to the privacy label is marked as private data and uploaded to the private chain for data storage, and the metadata corresponding to the public label is marked as public data and uploaded to the public chain for data update.

5. The big data processing and management system of artificial intelligence according to claim 4, characterized in that: The data analysis module extracts the preprocessed data set from the data warehouse, builds a K-Means clustering algorithm model to group the data set, combines the elbow method to determine the optimal K value and the initial cluster center point, and verifies the consistency of the clustering result with the actual classification; Select the k value range, k ≥ 1 and is a positive integer, randomly select k data points as the initial cluster centers, form k clusters, establish the feature vector corresponding to each data, where the feature vector is composed of the quantitative attributes of the data point, and calculate the similarity distance between each feature vector and the k cluster center; The calculation formula is: ; Where x is the currently calculated feature vector, c is the preset feature vector corresponding to the current cluster center, dist(x,c) represents the similarity distance between the feature vector and the cluster center, Ed(x,c) represents the Euclidean distance between the feature vector and the cluster center, Md(x,c) represents the Manhattan distance between the feature vector and the cluster center, Cd(x,c) represents the cosine distance between the feature vector and the cluster center, ω1, ω2, and ω3 are different weight coefficients respectively; The calculation formula of Euclidean distance is: ; In the formula, is the jth cluster center, is the i-th data point; The calculation formula of Manhattan distance is: ; The calculation formula for cosine distance is: ; In the formula, represents the dot product of vectors x and c, , denote the Euclidean norm of vectors x and c respectively.

6. The big data processing and management system of artificial intelligence according to claim 5, characterized in that: Assign each data to the nearest cluster center according to the obtained similarity distance. For each k value, calculate the sum of squares of the Euclidean distances SSE(k) from each data to the center of its cluster. The calculation formula is: ; Where N is the total number of data points x, C is the set of cluster centers under the current k value, is the jth cluster center, It’s data To its nearest cluster center The square of the Euclidean distance; Repeat the above steps until the cluster center no longer changes significantly or the pre-set number of iterations is reached; Draw a line graph of the k value and the corresponding SSE(k), and find the "elbow" point where SSE(k) decreases rapidly. The k value at this point is the optimal number of clusters. Use the selected optimal k value to execute the K-means algorithm to obtain the final clustering result.

7. The big data processing and management system of artificial intelligence according to claim 6, characterized in that: Compare the final clustering results with the true classification, evaluate the consistency between the clustering results and the true classification, and calculate the RI; The calculation formula is: ; In the formula, TP means two data of the same class are assigned to the same cluster, TN means two data of different classes are assigned to different clusters; C(N,2) is the number of combinations of two data points selected from N data points; RI is used to measure the consistency between the clustering result and the true classification; The RI value is between 0 and 1. The closer the value is to 1, the better the clustering effect is, and the better the consistency between the clustering result and the true classification is; Normalize RI and calculate ARI; The calculation formula is: ; Where E(RI) is the expected RI value of random clustering, max(RI) is the maximum RI value; ARI is an adjustment of RI, which takes into account the expected value in random clustering to standardize the actual RI value; The value of ARI is between -1 and 1, where 0 means that the clustering algorithm did not find any structure in the data, a value close to 1 means that the clustering effect is good, and a negative value means that the clustering effect is poor. That is, the larger the ARI value, the more consistent the clustering result is with the true classification, and the better the clustering effect.

8. The big data processing and management system of artificial intelligence according to claim 1, characterized in that: Create a big data integration platform to facilitate communication and data exchange between different modules, and evaluate the data transmission process based on the anomaly coefficients obtained during the data transmission process; Monitor the data transmission process and determine the degree of interference in data transmission efficiency during the data transmission process: obtain the actual data transmission volume S during the data transmission process and establish a data transmission volume set , n is a positive integer, get the maximum data transmission value , minimum data transfer value , calculate the average data transfer value , calculate the data transmission interference ratio; The calculation formula is: ; Where G is the data transmission interference ratio.

Images