A dimensionality reduction and visualization processing method for high-dimensional theater data based on data mining

Abstract

A method for dimensionality reduction and visualization processing of high-dimensional theater data based on data mining belongs to the technical field of theater equipment information management. It includes the following steps: 1. Dimensionality reduction processing of high-dimensional theater equipment operation big data; 2. Information fusion of heterogeneous data and extraction of frequent sequence patterns from the fused high-dimensional data information; 3. Clustering processing of theater equipment operation data; 4. Training theater equipment operation data vectors, and performing text classification based on sentence vectors constructed from word vectors, and performing semantic visualization of the optimized vectors in a two-dimensional semantic space. The present invention performs dimensionality reduction processing on massive theater equipment high-dimensional data, converges similar texts to a processable level, and combines relevant semi-supervised learning and isolation forest algorithms to classify the dimensionality reduction data files, designs a data classification model algorithm and performs visualization, optimizes the management of theater equipment operation and maintenance data, and achieves efficient and accurate operation and maintenance.

Description

Technical Field

[0001] The present invention belongs to the technical field of theater equipment information management, and in particular relates to a high-dimensional theater data dimensionality reduction visualization processing method based on data mining. Background Art

[0002] At present, theaters of all sizes have fully launched the work of mining the value of theater equipment data. Based on the characteristics of theater equipment data formats, relevant technical research and application are carried out around data collection, data processing and model creation. Massive theater equipment information is analyzed and processed, and the potential value of theater equipment information is deeply mined. In combination with visualization technology, data development trends and correlations are presented in an intuitive graphical manner, providing important data information for specific work. With the continuous deepening of information construction and application, theater equipment managers' demand for high-dimensional theater equipment data value mining and visualization continues to grow. The service scope of theater equipment operation systems has expanded from theater equipment information management departments to all businesses. Data mining and visualization are no longer just for a single information system, but need to grasp the comprehensive status of data, timely discover hidden faults of theater equipment, and improve the intelligent management level of theater equipment operation systems.

[0003] The theater equipment management department has accumulated a massive amount of high-dimensional data on theater equipment, laying the foundation for big data visualization. However, the theater equipment management department's information management system primarily manages the system's high-dimensional data format from a professional perspective. The data content and frequency in the information management system only consider the current business needs of the department, and do not consider the needs of future large-scale data analysis applications and multi-dimensional data visualization. This leads to data quality issues such as inconsistent high-dimensional data standards and missing data between systems, which makes it difficult to mine the value of log data. As the business of the theater equipment information management system gradually expands, the complexity of its business support system has also increased significantly. The current operation, maintenance, management, and monitoring systems supporting the system only determine the system's operating status based on underlying system indicators, providing system visualization monitoring and fault diagnosis services, but lack evaluation standards for the system's overall performance and health. Summary of the Invention

[0004] In response to the technical problems in the existing technology that there is a lack of dimensionality reduction processing for high-dimensional theater data and the visualization process is cumbersome and inaccurate, the purpose of the present invention is to provide a high-dimensional theater data dimensionality reduction and visualization processing method based on data mining, which can perform dimensionality reduction processing on massive high-dimensional data of theater equipment, optimize the management of theater equipment operation and maintenance data, and achieve efficient and accurate operation and maintenance.

[0005] The present invention provides the following technical solutions:

[0006] A high-dimensional theater data dimensionality reduction and visualization processing method based on data mining includes the following steps:

[0007] (1) The multi-source heterogeneous theater equipment operation data was preliminarily sorted out. Based on the multi-source fuzzy conflict information, the isolation forest anomaly detection and classification technology based on the imprecise inference theory and the isometric mapping (ISOMAP) nonlinear dimensionality reduction algorithm in manifold learning were used to reduce the dimensionality of the high-dimensional theater equipment operation big data while maintaining the local geometric features.

[0008] (2) In view of the heterogeneity and multi-source of the theater equipment operation data, the information of the heterogeneous data is extracted and fused, multi-source data is fused, and conflict fusion is performed respectively. Frequent sequence patterns are extracted from the fused high-dimensional data information, the correlation between high-dimensional data is discovered from the log information, and the log information sequence is constructed;

[0009] (3) Based on the log analysis method of deep learning and semi-supervised learning, the theater equipment operation data is clustered: based on the isometric mapping algorithm, the high-dimensional data with a similarity greater than 0.75 is converged to a processable level, and the corresponding semi-supervised learning and deep learning methods are combined to classify the log files;

[0010] (4) The Skip-gram model is used to train the theater equipment operation data vector, and sentence vectors are constructed based on word vectors for text classification. This is used to measure the quality of word vectors, optimize the number of iterations and dimensions of training word vectors, and adjust the size of the training model and corpus; the optimized word vectors are used as the input of the CNN model and are semantically visualized in a two-dimensional semantic space.

[0011] Preferably, the specific process of step (2) is as follows:

[0012] 2.1. Log data aggregation: sort log information by time series and format it in a standard way;

[0013] 2.2. Frequent pattern mining: using time sliding windows to mine frequent patterns;

[0014] 2.3. Event pattern detection: After mining the frequent sequence pattern set from the previous stage, perform correlation analysis on the sequences in the set;

[0015] 2.4. Association rule prediction: predict association rules and generate prediction rules;

[0016] 2.5. Update the causal relationship matrix: add the new association patterns mined to the causal relationship matrix.

[0017] Preferably, the specific process of step 2.2 is as follows:

[0018] By utilizing the characteristics of the sliding window, the sequence is divided into subsequences of equal width in time. When the active time window enters the next time interval, a candidate set with a width of i is mined from it, a frequent set with a width of i is constructed, and then a candidate set with a width of i+1 is constructed until the new frequent sequence is empty.

[0019] Preferably, the specific process of step (3) is as follows:

[0020] 3.1. Divide the log data into training, validation, and test sets. Divide the training set into labeled and unlabeled training data. Generate a random forest for the labeled training data and learn to label the unlabeled training data. Calculate the confidence of the samples during the labeling process.

[0021] 3.2. Take out the sample with the lowest confidence and eliminate it;

[0022] 3.3. Retrain the random forest until the confidence values in the unlabeled training set are within the confidence threshold.

[0023] Preferably, the specific process of step (4) is as follows:

[0024] 4.1. Use Skip-gram model to predict context target words to maximize the entire corpus Among them, P(w / w j ) represents the multinomial distribution of the wth word generated by the softmax function, w j Represents a word sequence [w1,w2,…,w j ], w is any word in the dictionary, c is the context word of w, and D is the dictionary extracted from the scheduling corpus. Each time, the model selects n words from the context of the target word and maps their word vectors to the corresponding TF-IDF vectors as the input of the model;

[0025] 4.2. Use the Jaccard algorithm to calculate the similarity of log information: Use the Jaccard distance for preprocessing to filter out log data with a similarity less than 0.25, and then use the edit distance for judgment to achieve clustering convergence of the logs and assign a unique ID to all data.

[0026] By adopting the above technology, compared with the prior art, the beneficial effects of the present invention are as follows:

[0027] The present invention uses the big data of theater equipment operation as the data warehouse, and adopts the high-dimensional theater data dimensionality reduction visualization processing method based on data mining to cluster and group the data, converge similar texts to a processable level, and combines semi-supervised learning and isolation forest algorithm to classify the dimensionality reduction data files and perform visual display, thereby solving the problem of dimensionality reduction processing of theater equipment data, optimizing the management of theater equipment operation and maintenance data, and realizing efficient and accurate operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the method of the present invention;

[0029] Figure 2 This is a diagram of the visualization process of dimensionality reduction of high-dimensional theater data in the present invention;

[0030] Figure 3 This is a diagram of the high-dimensional theater data log association mining model of the present invention;

[0031] Figure 4 This is a flow chart of the high-dimensional theater data anomaly detection and classification process of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0033] On the contrary, the present invention covers any alternatives, modifications, equivalents, and solutions that fall within the spirit and scope of the present invention as defined by the claims. Furthermore, to facilitate a better understanding of the present invention, certain specific details are described in detail below in the detailed description of the present invention. Those skilled in the art will be able to fully understand the present invention without these details.

[0034] See also Figure 1-4 The present embodiment provides a method for visualizing and reducing the dimension of high-dimensional theater data based on data mining, comprising the following steps:

[0035] S1. Preliminary sorting of multi-source heterogeneous theater equipment operation data. Aiming at the multi-source fuzzy conflict information, the isolation forest anomaly detection and classification technology based on the imprecise inference theory and the isometric mapping (ISOMAP) nonlinear dimensionality reduction algorithm in manifold learning are used to reduce the dimensionality of the high-dimensional theater equipment operation big data while maintaining the local geometric features.

[0036] Specifically include:

[0037] S1.1. Construct a neighborhood graph G: For all data points x in the observation space dataset Xi , i=1,2,…,n,x i ∈R D , calculate the Euclidean distance d between all pairs of data points E (x i ,x j ), define the neighborhood relationship between points with k nearest neighbors, and get x i Neighborhood Points N i The neighborhood relationship between sampling points is represented as a neighborhood graph G with sampling points as nodes and Euclidean distances as edges.

[0038] S1.2. Calculate the geodesic distance matrix D: Based on the neighborhood graph, let d G (x i x j )=d E (x i ,x j ) Initialize the geodesic distance matrix and apply the shortest path algorithm to calculate the x between any two points in the neighborhood graph G. i , x j The shortest path d G (x i x j ),Right now:

[0039]

[0040] Get the geodesic distance matrix D, where N j Represents x j Neighborhood points.

[0041] S1.3. Apply the MDS algorithm to reduce the dimension of X: construct the matrix B = HDH to centralize the geodesic distance matrix D, where I is the n-order unit matrix, and L is the n-dimensional vector whose elements are 1. Thus, we can get the matrix element b ij for:

[0042]

[0043] Among them, a ij For the elements in the geodesic distance matrix D, the spectral decomposition is B = HDH = XX T ΛΛ T , where Λ=diag{λ1, λ2,…,λ d},λ1,λ2,…,λ d is the d positive eigenvalues of B, and the corresponding eigenvector Y={y i}, i = 1, 2, ..., n, y i ∈R D (d<D) is the obtained low-dimensional embedded manifold.

[0044] S2. Considering the heterogeneity and multi-source nature of theater equipment operating data, we perform extraction and fusion on heterogeneous data, multi-source data fusion, and conflict fusion. We extract frequent sequence patterns from the fused high-dimensional data, discover correlations between high-dimensional data, and construct log information sequences.

[0045] Specifically include:

[0046] S2.1. Log data aggregation: sort the log information according to time series and format it in a standard way.

[0047] S2.2, frequent pattern mining, using time sliding window to mine frequent patterns.

[0048] S2.3, event pattern detection, after mining the frequent sequence pattern set from the previous stage, the sequences in the set are subjected to correlation analysis.

[0049] S2.4. Association rule prediction: predict association rules and generate prediction rules.

[0050] S2.5. Update the causal relationship matrix and add the new association patterns mined into the causal relationship matrix.

[0051] S3. Log analysis methods based on deep learning and semi-supervised learning cluster massive amounts of theater equipment operating data. Based on an isometric mapping algorithm, similar high-dimensional data is converged to a manageable level. Log files are then classified using related semi-supervised learning and deep learning methods.

[0052] Specifically include:

[0053] S3.1. Divide the log data into training set, validation set and test set in a ratio of 6:3:1, and divide the training set into labeled training data and unlabeled training data in a ratio of 6:4. Learn to generate a random forest for the labeled training data, learn to label the unlabeled training data, and calculate the confidence of the samples during the labeling process; the validation set is used to verify whether the trained random forest model can converge high-dimensional data to a processable level; the test set is used to test whether the proposed log analysis method based on deep learning and semi-supervised learning can classify log files.

[0054] S3.2. Take out the samples with the lowest confidence and eliminate them.

[0055] S3.3. Retrain the random forest until the confidence values in the unlabeled training set are within the confidence threshold.

[0056] S4. Use the Skip-gram model to train theater equipment operation data vectors. Use the word vectors to construct sentence vectors for text classification. This measures the quality of the word vectors, optimizes the number of training iterations and dimensions, and adjusts the training model and corpus size. The optimized vectors serve as input to the CNN model and are visualized in a two-dimensional semantic space.

[0057] The processing results include:

[0058] 1. Use the Skip-gram model to predict the context target word to maximize the entire corpus Among them, P(w / w j ) represents the multinomial distribution of the wth word generated by the softmax function, w j Represents a word sequence [w1,w2,…,w j ], w is any word in the dictionary, c is the context word of w, and D is the dictionary extracted from the scheduling corpus. Each time, the model selects n words from the context of the target word and maps their word vectors to corresponding TF-IDF vectors as the model input.

[0059] 2. Use the Jaccard algorithm to calculate the similarity of log information: Use the Jaccard distance for preprocessing to filter out log data with a similarity less than 0.25, and then use the edit distance for judgment to achieve clustering convergence of the logs and assign unique IDs to all data.

[0060] 3. A comprehensive comparison of dimensionality reduction visualization analysis results using traditional models such as SVM and KNN shows that the ISOMAP-based nonlinear dimensionality reduction visualization model designed in this paper achieves higher dimensionality reduction accuracy. Furthermore, the CNN model performs relatively stably on both the training and test sets, demonstrating that deep learning models can learn temporal variations in the data. Furthermore, the model's accuracy improves further with increasing amounts of training data.

[0061] Using a high-dimensional theater data dimensionality reduction and visualization method based on the ISOMAP nonlinear algorithm, this method performs nonlinear simplification on the original feature information while preserving the data's topological structure. This method deeply mines the system's nonlinear dynamic information and reduces computational complexity. This method, combined with a deep learning network for feature extraction and visualization, accurately identifies key information about theater equipment, enabling real-time dynamic detection and status updates, ensuring optimal equipment operation and safeguarding the normal operation of the theater.

[0062] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A high-dimensional theater data dimensionality reduction and visualization processing method based on data mining, characterized in that: The following steps are involved: (1) The multi-source heterogeneous theater equipment operation data was preliminarily sorted out. Based on the multi-source fuzzy conflict information, the isolation forest anomaly detection and classification technology based on the imprecise inference theory and the isometric mapping (ISOMAP) nonlinear dimensionality reduction algorithm in manifold learning were used to reduce the dimensionality of the high-dimensional theater equipment operation big data while maintaining the local geometric features. (2) In view of the heterogeneity and multi-source of the theater equipment operation data, the information of the heterogeneous data is extracted and fused, multi-source data is fused, and conflict fusion is performed respectively. Frequent sequence patterns are extracted from the fused high-dimensional data information, the correlation between high-dimensional data is discovered from the log information, and the log information sequence is constructed; (3) Based on the log analysis method of deep learning and semi-supervised learning, the theater equipment operation data is clustered: based on the isometric mapping algorithm, the high-dimensional data with a similarity greater than 0.75 is converged to a processable level, and the corresponding semi-supervised learning and deep learning methods are combined to classify the log files; (4) Use the Skip-gram model to train the theater equipment operation data vector, and construct sentence vectors based on word vectors for text classification, so as to measure the quality of word vectors, optimize the number of iterations and dimensions of training word vectors, and adjust the size of the training model and corpus; the optimized word vectors are used as the input of the CNN model and are semantically visualized in a two-dimensional semantic space; The specific process of step (4) is as follows: 4.

1. Use Skip-gram model to predict context target words, thereby maximizing the entire corpus ∑ (w,c)∈D ∑w j∈C logP(w / w j ), where P(w / w j ) represents the multinomial distribution of the wth word generated by the softmax function, w j Represents a word sequence [w1,w2,…,w j ], w is any word in the dictionary, c is the context word of w, and D is the dictionary extracted from the scheduling corpus; The model selects n words from the context of the target word each time and maps their word vectors to corresponding TF-IDF vectors as the input of the model; 4.

2. Use the Jaccard algorithm to calculate the similarity of log information: Use the Jaccard distance for preprocessing to filter out log data with a similarity less than 0.25, and then use the edit distance for judgment to achieve clustering convergence of the logs and assign a unique ID to all data.

2. The method for dimensionality reduction and visualization of high-dimensional theater data based on data mining according to claim 1, characterized in that: The specific process of step (2) is as follows: 2.

1. Log data aggregation: sort log information by time series and format it in a standard way; 2.

2. Frequent pattern mining: using time sliding windows to mine frequent patterns; 2.

3. Event pattern detection: After mining the frequent sequence pattern set from the previous stage, perform correlation analysis on the sequences in the set; 2.

4. Association rule prediction: predict association rules and generate prediction rules; 2.

5. Update the causal relationship matrix: add the new association patterns mined to the causal relationship matrix.

3. The method for dimensionality reduction and visualization of high-dimensional theater data based on data mining according to claim 2, characterized in that: The specific process of step 2.2 is as follows: By utilizing the characteristics of the sliding window, the sequence is divided into subsequences of equal width in time. When the active time window enters the next time interval, a candidate set with a width of i is mined from it, a frequent set with a width of i is constructed, and then a candidate set with a width of i+1 is constructed until the new frequent sequence is empty.

4. A high-dimensional theater data dimensionality reduction and visualization processing method based on data mining according to claim 1 or 3, characterized in that: The specific process of step (3) is as follows: 3.

1. Divide the log data into training, validation, and test sets. Divide the training set into labeled and unlabeled training data. Generate a random forest for the labeled training data and learn to label the unlabeled training data. Calculate the confidence of the samples during the labeling process. 3.

2. Take out the sample with the lowest confidence and eliminate it; 3.

3. Retrain the random forest until the confidence values in the unlabeled training set are within the confidence threshold.

Images