Multi-source heterogeneous data intelligent cleaning processing method and system

Abstract

The invention discloses an intelligent cleaning processing method and system for multi-source heterogeneous data, and belongs to the technical field of data cleaning processing methods, industrial sensor time sequence data, long text log data and other high-dimensional industrial data are obtained, the obtained data are subjected to cleaning, missing value processing, standardization, text processing and timestamp alignment, and the obtained data are subjected to data cleaning, missing value processing, standardization, text processing and timestamp alignment. The method comprises the following steps: preprocessing time sequence data of an industrial sensor, inputting the preprocessed time sequence data of the industrial sensor into a Transform auto-encoder, and in a low-dimensional feature space output by the Transform, applying a density-sensitive clustering method, setting a neighborhood radius and a minimum point number, and identifying a data cluster formed in a data dense region and noise points in a sparse region.

Description

Technical Field

[0001] The present invention relates to a data cleaning and processing method, in particular to an intelligent cleaning and processing method and system for multi-source heterogeneous data, belonging to the technical field of data cleaning and processing methods. Background Art

[0002] In the context of industrial intelligentization and digital transformation, the processing and analysis of multi-source heterogeneous data have become the core requirements for improving production efficiency and equipment reliability. In industrial scenarios, data exhibits significant multi-source and heterogeneity, including time-series data (such as temperature, vibration, current signals) collected in real time by sensors, unstructured device log texts (such as fault records, operation logs), and high-dimensional process parameter data (such as PLC configurations, production process indicators). These data contain key information such as equipment operating status and fault modes. However, due to diverse data formats, uneven data quality, and significant semantic differences, traditional data cleaning methods are difficult to process efficiently. The specific pain points are as follows: Low efficiency in processing multi-source heterogeneous data. Traditional methods adopt independent processing flows for industrial sensor time-series data and text log data: Rely on manually designed time-domain / frequency-domain features (such as kurtosis of vibration signals, threshold judgment of temperature signals), which require in-depth domain knowledge and have weak feature generalization ability, making it difficult to adapt to dynamic scenarios such as equipment aging and working condition changes.

[0003] Rely on regular expressions or rule engines for parsing, with low parsing accuracy for unstructured descriptions (such as fuzzy fault statements), and the need to repeatedly develop parsing rules for different log formats, resulting in high costs and poor scalability.

[0004] The curse of dimensionality makes it difficult for traditional dimensionality reduction methods (such as PCA) to retain density-distinguishing features, and subsequent clustering effects are poor.

[0005] Noise and outlier processing: Industrial sensors are vulnerable to electromagnetic interference and mechanical vibration, resulting in frequent data outliers; Log data has problems such as missing information and inconsistent descriptions. Traditional methods are prone to misjudging or missing key anomalies through simple filtering or interpolation processing.

[0006] Poor adaptability to complex data distributions: Industrial data clusters often exhibit irregular shapes (such as strip clusters formed by gradual changes in features due to equipment aging). Traditional clustering algorithms (such as K-means) rely on the assumption of spherical clusters. Although DBSCAN is sensitive to density, the definition of "density" is ambiguous in high-dimensional spaces and requires efficient low-dimensional feature representations for support.

[0007] In traditional data cleaning processes, dimensionality reduction and clustering are independent steps: Dimensionality reduction (such as PCA) only retains features with the largest variance, ignoring non-linear relationships that are crucial for density discrimination; Clustering relies on manual features and cannot utilize the deep semantics of data (such as the causal relationship of faults in log texts and the long-term dependence of time series data). This fragmentation leads to the loss of key information. Especially in the early fault detection of equipment, weak abnormal features are easily filtered, delaying the warning time. Therefore, a multi-source heterogeneous data intelligent cleaning and processing method and system are designed to solve the above problems. Summary of the Invention

[0008] The main purpose of the present invention is to provide a multi-source heterogeneous data intelligent cleaning and processing method and system.

[0009] The object of the present invention can be achieved by adopting the following technical solutions: A multi-source heterogeneous data intelligent cleaning and processing method includes the following steps: Step 1: Obtain industrial sensor time series data, long text log data, and other high-dimensional industrial data; Step 2: Clean, handle missing values, standardize, process text, and align timestamps for the obtained data; Step 3: Input the preprocessed industrial sensor time series data into a Transformer autoencoder; Step 4: In the low-dimensional feature space output by the Transformer, apply a density-sensitive clustering method, set the neighborhood radius and the minimum number of points, and identify data clusters formed in the data dense area and noise points in the sparse area.

[0010] Preferably, in Step 1, the industrial sensor time series data includes time series signals collected in real time by temperature, pressure, vibration, current, and voltage sensors; the long text log data includes equipment fault logs, operation record logs, and system alarm logs; The other high-dimensional industrial data includes equipment parameter configuration data, production process data, and quality inspection data.

[0011] Preferably, in Step 2, the cleaning includes removing duplicate records, correcting error values, and filtering redundant information; The missing value processing includes using interpolation or model-based filling for sensor time series data, and completing or marking special values for long text log data through rule inference; The standardization includes performing Z-score standardization or normalization on sensor numerical data; The text processing includes word segmentation, removing stop words, word stemming, and text vectorization; The timestamp alignment includes resampling or interpolating asynchronously collected sensor data.

[0012] Preferably, in step three, a low-dimensional feature vector containing the core pattern is generated through sliding window sampling, positional encoding, multi-head self-attention mechanism, feed-forward neural network, and bottleneck layer compression; The preprocessed long text log data is input into a text-specific Transformer autoencoder, and a low-dimensional semantic feature vector is generated through word vector embedding, multi-head self-attention mechanism, and bottleneck layer compression.

[0013] Preferably, the sliding window sampling divides the time series data into sequence samples according to a fixed window length, and the windows can overlap to capture local dependencies; The positional encoding generates a position vector for each time step through sine / cosine functions, embedding the time order information into the feature space.

[0014] Preferably, the multi-head self-attention mechanism maps the input features into query matrix, key matrix, and value matrix, calculates the dependencies in multiple subspaces in parallel, and captures the dependencies between time steps or semantic associations between words in multiple dimensions.

[0015] Preferably, the bottleneck layer compresses the high-dimensional features into low-dimensional feature vectors through a fully connected layer, combines mean pooling to process variable-length sequences, and eliminates the influence of sequence length differences.

[0016] Preferably, the density-sensitive clustering method sets the neighborhood radius and the minimum number of points, starts from the points in the dense area to expand the search and merge data clusters, determines the isolated points that cannot be attributed as noise points, and attributes the boundary points to the neighboring clusters according to the distance.

[0017] A multi-source heterogeneous data intelligent cleaning and processing system includes: a data acquisition module: used to acquire industrial sensor time series data, long text log data, and other high-dimensional industrial data; A data preprocessing module: used to clean, handle missing values, standardize, process text, and align timestamps for the acquired data; A feature generation module: including a time series data feature generation unit and a text data feature generation unit; The time series data feature generation unit is used to input the preprocessed industrial sensor time series data into a Transformer autoencoder to generate a low-dimensional feature vector containing the core pattern; the text data feature generation unit is used to input the preprocessed long text log data into a text-specific Transformer autoencoder to generate a low-dimensional semantic feature vector; A clustering analysis module: used to apply a density-sensitive clustering method in the low-dimensional feature space output by the Transformer to identify data clusters and noise points.

[0018] Preferably, the time series data feature generation unit includes a sliding window sampling module, a position encoding module, a multi-head self-attention module, a feed-forward neural network module, and a bottleneck layer compression module; The sliding window sampling module is used to segment the time series data according to a fixed window length; The position encoding module is used to generate time step position vectors; The multi-head self-attention module is used to calculate multi-dimensional dependencies; The feed-forward neural network module is used to enhance non-linear feature expression; The bottleneck layer compression module is used to compress high-dimensional features into low-dimensional feature vectors; The clustering analysis module includes a density definition unit, a core cluster construction unit, a noise point determination unit, and a boundary point processing unit; The density definition unit is used to set the neighborhood radius and the minimum number of points; The core cluster construction unit is used to expand the search and merge data clusters starting from the points in the dense area; The noise point determination unit is used to determine isolated points as noise points; The boundary point processing unit is used to assign boundary points to neighboring clusters according to the distance; The system further includes a model training module, which is used to perform self-supervised training on the Transformer autoencoder, minimize the reconstruction error, introduce contrastive loss to improve the feature discrimination ability, and add noise to improve the model robustness.

[0019] Advantageous technical effects of the present invention: A multi-source heterogeneous data intelligent cleaning and processing method and system provided by the present invention. Traditional methods rely on manually designed time domain / frequency domain features to process industrial sensor time series data, which not only requires in-depth domain knowledge but also has weak feature generalization ability. The present invention uses a Transformer autoencoder to automatically capture dependencies at different time steps from time series data through techniques such as sliding window sampling, position encoding, multi-head self-attention mechanism, and feed-forward neural network, and generate low-dimensional feature vectors containing core patterns. This avoids cumbersome manual feature engineering, greatly improves the feature extraction efficiency, and can better adapt to dynamic scenarios such as equipment aging and working condition changes.

[0020] For long text log data, traditional regular expression or rule engine parsing methods have low parsing accuracy for unstructured descriptions and also require repeated development of parsing rules for different log formats. The present invention uses a text-specific Transformer autoencoder to automatically extract low-dimensional semantic feature vectors of text through word vector embedding, multi-head self-attention mechanism, and bottleneck layer compression, without the need for manual writing of complex rules, improving the efficiency and accuracy of text processing.

[0021] Traditional dimensionality reduction methods (such as PCA) are difficult to retain density discrimination features, and dimensionality reduction and clustering are independent of each other, resulting in the loss of key information. The present invention combines a Transformer autoencoder with a density-sensitive clustering method (such as DBSCAN) to achieve end-to-end unsupervised learning. The low-dimensional features generated by the Transformer autoencoder are optimized for the clustering target, retaining features valuable for density discrimination, reducing information loss, and improving the overall efficiency of data processing.

[0022] Industrial sensors are vulnerable to interference and generate outliers, and log data has problems such as missing information and inconsistent descriptions. In the data preprocessing stage of the present invention, various methods are adopted, such as interpolation methods for missing values in sensor time series data or filling based on models, and rule inference completion or marking of special values for missing key information in long text log data. In clustering analysis, by utilizing the density-sensitive characteristics of DBSCAN, data clusters formed in data-dense areas and noise points in sparse areas can be accurately identified. In this way, outliers such as instantaneous interference of sensors and data acquisition errors are effectively filtered, reducing misjudgment and missed judgment, and improving the accuracy of data cleaning.

[0023] Industrial data clusters often exhibit irregular shapes. Traditional clustering algorithms (such as K-means) rely on the spherical cluster assumption and cannot effectively handle them. The DBSCAN algorithm of the present invention combines the low-dimensional features generated by the Transformer autoencoder and can adapt to the irregular distribution of industrial data and automatically identify complex shapes such as strip clusters formed by the gradual change of features caused by equipment aging.

[0024] At the same time, by setting the neighborhood radius and the minimum number of points, data clusters are expanded and searched for merging starting from the points in the dense area, accurately dividing data clusters and noise points, and improving the adaptability to complex data distributions and clustering accuracy. Brief Description of the Drawings

[0025] Figure 1 It is an overall three-dimensional structural decomposition diagram of the device according to a preferred embodiment of a multi-source heterogeneous data intelligent cleaning processing method and system of the present invention. Detailed Embodiment

[0026] To make the technical solutions of the present invention clearer and more definite for those skilled in the art, the present invention will be further described in detail below in conjunction with embodiments and the accompanying drawings, but the implementation manners of the present invention are not limited thereto.

[0027] Input the original high-dimensional data into the Transformer autoencoder, and generate low-dimensional feature representations through the encoding layer.

[0028] For time series data, the Transformer can capture the dependencies of different time steps and compress them into feature vectors containing core patterns.

[0029] In the low-dimensional feature space output by the Transformer, the DBSCAN algorithm is applied. Utilizing the density-sensitive characteristic of DBSCAN, data clusters and noise points are identified.

[0030] In user behavior analysis, first, the user operation sequence is converted into a feature vector through a Transformer autoencoder, and then DBSCAN clustering is used. User groups with different behavior patterns can be discovered, and at the same time, abnormal operations are filtered.

[0031] In traditional methods, dimensionality reduction (such as PCA) and clustering are independent steps, which may lose key information. The fusion of the Transformer autoencoder and DBSCAN realizes end-to-end unsupervised learning. The encoding process is optimized for the clustering objective, and features valuable for density discrimination are retained.

[0032] For non-Euclidean space data (such as social network relationships, molecular structure data), the Transformer autoencoder can capture non-linear relationships, and DBSCAN clusters based on the density of the encoded features, breaking through the assumption of spherical clusters in traditional clustering algorithms (such as K-means).

[0033] In the present invention, before the original high-dimensional data is input into the Transformer autoencoder, the types of industrial data involved and the preprocessing are as follows: Industrial sensor time-series data is obtained, including time-series signals such as temperature, pressure, vibration, current, and voltage collected in real-time by sensors, reflecting the dynamic changes in the operating state of the equipment. For example, the vibration frequency data of a motor and the temperature change data of a reactor.

[0034] Long text log data such as equipment fault logs (recording the time of fault occurrence, phenomena, codes, etc.), operation record logs (manual operation steps, parameter adjustment records), system alarm logs (abnormal state descriptions), etc. are obtained to store key information in text form.

[0035] Other high-dimensional industrial data such as equipment parameter configuration data (such as the parameter settings of a PLC), production process data (the combination of process parameters for each link), and quality inspection data (multi-dimensional inspection indicators) are obtained. These data have high dimensions and complex structures, and deep features need to be extracted through the Transformer autoencoder.

[0036] Perform preprocessing on the obtained data; Remove duplicate records, correct error values (such as abnormal values when the sensor exceeds the range), and filter out irrelevant redundant information (such as invalid placeholders in the log).

[0037] For missing values in sensor time-series data, interpolation methods (such as linear interpolation, cubic spline interpolation) or model-based methods (such as KNN interpolation) are used for filling; for long text logs with missing key information, it can be inferred through rules (such as completing according to the content of the previous and subsequent logs) or marked as special values (such as "UNKNOWN").

[0038] Standardize (such as Z-score standardization) or normalize (scale to the [0,1] interval) the numerical data of sensors to ensure that data with different dimensions are on the same scale and improve the training efficiency of the Transformer.

[0039] Word segmentation (splitting Chinese logs into words), removing stop words (such as meaningless words like "de", "le", etc.), word lemmatization (such as restoring "running" to "run"), and if necessary, text vectorization (such as TF-IDF, Word2Vec) to convert the text into a numerical form that can be processed by the model.

[0040] Ensure the alignment of timestamps of different sensor data, resample or interpolate asynchronously collected data to ensure the consistency of the time-series sequence input to the Transformer autoencoder.

[0041] Through the above preprocessing, the data quality and normativity can be improved, enabling the Transformer autoencoder to capture data features more efficiently and avoiding the degradation of model performance caused by raw data problems.

[0042] After data collection and data cleaning and processing are completed, low-dimensional feature representations are generated through the encoding layer, which specifically includes the following steps: Perform sliding window sampling on industrial sensor time-series data (such as temperature, vibration), split the data according to a fixed window length (such as 100 time steps) to form sequence samples, and the windows can overlap (such as a step size of 50) to capture local dependencies.

[0043] Perform standardization. By calculating the data mean and standard deviation, scale the data to a unified scale to avoid the influence of dimensional differences on model training.

[0044] Introduce positional encoding. Generate positional vectors for each time step through sine / cosine functions and incorporate the time-series order information to solve the problem that the Transformer is insensitive to the sequence order.

[0045] Construct the Transformer autoencoder; Encoder layer: Adopt the multi-head self-attention mechanism to calculate multiple attention heads in parallel to capture dependency relationships in different dimensions; combine with a feed-forward neural network to enhance the feature expression ability, and prevent overfitting through layer normalization and Dropout.

[0046] Bottleneck layer: Add a fully connected layer (e.g., 128 dimensions) after the encoder to compress high-dimensional features into low-dimensional representations, achieving information dimensionality reduction and core feature extraction.

[0047] Decoder: Mirror the encoder structure and perform self-supervised training by reconstructing the original input. The loss function uses mean squared error (MSE) to ensure that the low-dimensional features retain the essential information of the data.

[0048] Training and feature extraction: During model training, use the original time series data as the input and supervision signal to minimize the reconstruction error. After training, intercept the encoder part as the feature extractor, input the preprocessed time series data, and the output is the low-dimensional feature representation, which can be used for subsequent anomaly detection or clustering analysis.

[0049] Text preprocessing: Clean the text (remove punctuation and convert to lowercase), and use pre-trained word vectors (such as Universal Sentence Encoder) to convert the text into a sequence of vectors to capture semantic information.

[0050] Chunk the ultra-long text into fixed-length blocks (e.g., each block has 128 words), and ensure context continuity for the overlapping part (e.g., 32 words).

[0051] Text-specific Transformer autoencoder: The input is a sequence of word vectors. The encoder layer structure is the same as that of the time series model. It captures the semantic associations between words through multi-head self-attention, and the feed-forward network further abstracts the features.

[0052] The bottleneck layer compresses the feature dimension (e.g., 128 dimensions), the decoder reconstructs the word vectors, and the training loss uses mean squared error to ensure that the semantic information is retained.

[0053] Text feature extraction: After training, use the encoder to convert the text word vectors into low-dimensional features, which can be used for tasks such as text clustering and fault log classification to improve the efficiency of semantic analysis.

[0054] After generating the low-dimensional feature representation through the encoding layer, for time series data, Transformer can capture the dependencies at different time steps and compress them into a feature vector containing the core patterns, specifically as follows: Convert univariate / multivariate time series data (such as time series of device vibration, temperature, current) into a matrix input of sequence length x feature dimension; Use sliding window sampling for long time series data (e.g., window size 100, step size 50) to ensure that each input sample contains sufficient time context; Generate sine / cosine position encoding for each time step, through the formula: ; where pos is the time step position, i is the dimension index, dmodel It is the dimension of the Transformer hidden layer, which encodes the chronological information into the feature space. This encoding embeds the chronological information into the feature space to solve the problem that Transformer is insensitive to sequence order.

[0055] First, the dimension of the original time series features is increased to d through a fully connected layer model (such as from 5-dimensional sensor data to 256-dimensional), and then added to the positional encoding so that the features at each time step contain both data values and chronological information.

[0056] Parallel dependency capture by multi-head attention: The input features are mapped into query matrix Q, key matrix K, and value matrix V, and the dependency relationships in h subspaces (such as 8 heads) are calculated in parallel through multi-head attention.

[0057] Each head independently calculates the attention weights, and the formula is: ; where is the key dimension, and the scaled dot product avoids gradient vanishing. Finally, the results of each head are concatenated to capture the time step dependencies in multiple dimensions (such as short-term fluctuations and long-term trends).

[0058] Causal mask to ensure temporal causality: When calculating attention, future time steps are masked through an upper triangular mask (that is, the current time step can only focus on past or current information), and the formula is: ; Ensure that the model conforms to the causal relationship of time series data and is suitable for prediction tasks (such as predicting future states based on historical data).

[0059] Transformer encoder and core feature generation; Encoder layer structure: After the input of the multi-head self-attention layer is calculated through causal mask multi-head attention, the output is the features containing time step dependencies.

[0060] The feed-forward neural network enhances the non-linear feature expression through two fully connected layers (with the activation function GELU) to capture local patterns (such as the periodicity of vibration signals).

[0061] Layer normalization and residual connection stabilize the training process, solve the problem of gradient vanishing in deep networks, and enhance the ability to capture long-term trends (such as the temperature rise caused by equipment aging).

[0062] Bottleneck layer feature compression compresses the high-dimensional features (such as 256-dimensional) output by the encoder into low-dimensional feature vectors (such as 128-dimensional) through a fully connected layer, and the formula is: ; where h is the encoder output and z is the low-dimensional feature containing the core pattern.

[0063] For variable-length sequences, a fixed-length feature vector is generated through mean pooling (taking the average of the features at all time steps), eliminating the influence of sequence length differences.

[0064] The original time series data is restored through the decoder, and the mean squared error (MSE) is used as the loss function to ensure that the low-dimensional features retain the essential information of the data: ; where is the original sequence, is the output of the decoder.

[0065] The contrastive loss is introduced to force the features of similar time series (such as different periods of normal operation of the same device) to be close in the low-dimensional space and far from different types of time series (such as normal and faulty). The formula is: ; where is the sample feature, is the positive sample, is the negative sample, is the temperature parameter, which enhances the special Classifying ability of features for classification.

[0066] During training, Gaussian noise is added or some time steps are masked (such as randomly setting 10% of the time points to zero), improving the model's tolerance to sensor noise.

[0067] Processing of long sequences: For extremely long sequences (such as > 500 time steps), a sparse attention mechanism (such as sliding window attention) is adopted, reducing the computational complexity from to and improving the training efficiency.

[0068] Through the above scheme, the Transformer can effectively capture the long-term and short-term dependencies of time series and generate low-dimensional feature vectors containing core patterns.

[0069] After obtaining the feature vectors, how to implement in the low-dimensional feature space output by the Transformer. Applying the DBSCAN algorithm and using the density-sensitive characteristics of DBSCAN to identify data clusters and noise points, the specific scheme is as follows: Through the Transformer autoencoder, multi-source high-dimensional data such as industrial sensor time series data and device log texts are compressed into low-dimensional feature vectors (such as 128-dimensional) containing core patterns, retaining the essential differences in the data (such as the feature distinction between normal operation and abnormal states).

[0070] Data standardization unifies the scale of feature vectors (such as Z-score standardization) to ensure the comparability of data in different dimensions (for example, the eigenvalues of device temperature and vibration signals are in the same order of magnitude).

[0071] Define "density neighborhood": Set a "neighborhood radius" (which can be understood as "search range"). In the feature space, if there are enough neighboring points (such as more than 5) within the neighborhood of each point, then this area is considered a "data-dense area", forming potential data clusters.

[0072] Conversely, the area with insufficient number of points in the neighborhood is a "sparse area", which may contain noise points (such as temporary fluctuations of the device, abnormal sensor data); Core cluster construction: Starting from any unlabeled point in the dense area, expand the search around it, and merge all neighboring points that meet the density condition into a data cluster (for example, different operating conditions during normal operation of the device form multiple dense clusters).

[0073] The shape of the cluster is not restricted and can adapt to the common irregular distributions in industrial data (such as the strip-shaped clusters formed by the gradual change of features caused by device aging).

[0074] Identify isolated points: Points that cannot be assigned to any dense cluster (that is, the number of points in the neighborhood is always insufficient) are determined as noise points. These points usually correspond to abnormal states of the device (such as sudden failures, sensor failure data).

[0075] Boundary point processing: Points located at the edge of multiple dense clusters (the number of points in the neighborhood just meets the condition) are assigned to the nearest cluster according to their distance from the nearest cluster to avoid misjudgment as noise.

[0076] Embodiment

[0077] Industrial scenario application: Taking device anomaly detection as an example; Normal mode modeling: Cluster the feature vectors generated from the historical normal operation data of the device to form "normal clusters" (such as dense areas of different operating modes like low load and high load).

[0078] Real-time data matching: Input the real-time collected device feature vectors into the feature space and determine whether they belong to the existing clusters: Belonging to a certain normal cluster: The device is in a known operating mode (such as "high load but stable").

[0079] Falling into the sparse area or unable to match any cluster: Marked as a noise point, triggering an anomaly warning (such as the device may have an early failure).

[0080] Result verification and feedback: Verify the noise points in combination with the device maintenance records. If they correspond to actual failure cases multiple times, then update the clustering model to enhance the ability to identify similar anomalies Density-sensitive characteristic: Without presetting the number of clusters (such as the K value that needs to be specified in K-means), it can automatically identify the irregular clusters naturally existing in industrial data (such as multiple sub-clusters formed by the progressive aging of equipment).

[0081] Noise robustness: Directly determine the points in the sparse area as noise, effectively filtering out outliers such as instantaneous interference of sensors and data acquisition errors.

[0082] Dynamic adaptation: As new data is continuously input, the neighborhood radius and density point threshold can be updated to adapt to the feature drift during the long-term operation of the equipment (such as the slow change of the position of normal clusters caused by aging).

[0083] Through the logic of "density definition → cluster construction → noise screening", in the low-dimensional feature space generated by Transformer, the core patterns (dense clusters) of industrial data and abnormal information (noise points) are intuitively separated, which is applicable to scenarios such as equipment status monitoring and fault prediction, providing clear data support for subsequent maintenance decisions.

[0084] As described above, these are only further embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the scope disclosed by the present invention, according to the technical solution and concept of the present invention, makes equivalent substitutions or changes, all belong to the protection scope of the present invention.

Claims

1. A method for intelligent cleaning and processing of multi-source heterogeneous data, characterized in that: It includes the following steps: Step 1: Obtain industrial sensor time-series data, long text log data, and other high-dimensional industrial data; Step 2: Clean, handle missing values, standardize, process text, and align timestamps for the obtained data; Step 3: Input the preprocessed industrial sensor time-series data into a Transformer autoencoder; Step 4: In the low-dimensional feature space output by the Transformer, apply a density-sensitive clustering method, set the neighborhood radius and the minimum number of points, and identify data clusters formed in data-dense areas and noise points in sparse areas.

2. The intelligent cleaning and processing method for multi-source heterogeneous data according to claim 1, wherein: The industrial sensor time-series data described in Step 1 includes time-series signals collected in real time by temperature, pressure, vibration, current, and voltage sensors; the long text log data includes equipment failure logs, operation record logs, and system alarm logs; The other high-dimensional industrial data includes equipment parameter configuration data, production process data, and quality inspection data.

3. A multi-source heterogeneous data intelligent cleaning and processing method according to claim 1, characterized in that: The cleaning described in Step 2 includes removing duplicate records, correcting error values, and filtering redundant information; The handling of missing values includes using interpolation or model-based filling for sensor time-series data, and completing or marking special values for long text log data through rule inference; The standardization includes performing Z-score standardization or normalization on sensor numerical data; The text processing includes word segmentation, removing stop words, lemmatization, and text vectorization; The timestamp alignment includes resampling or interpolating asynchronously collected sensor data.

4. A multi-source heterogeneous data intelligent cleaning and processing method according to claim 1, characterized in that: In Step 3, through sliding window sampling, positional encoding, multi-head self-attention mechanism, feed-forward neural network, and bottleneck layer compression, a low-dimensional feature vector containing core patterns is generated; Input the preprocessed long text log data into a text-specific Transformer autoencoder, and generate a low-dimensional semantic feature vector through word vector embedding, multi-head self-attention mechanism, and bottleneck layer compression.

5. A method for intelligent cleaning and processing of multi-source heterogeneous data according to claim 4, characterized in that: The sliding window sampling divides the time-series data into sequence samples according to a fixed window length, and the windows can overlap to capture local dependencies; The positional encoding generates a position vector for each time step through sine / cosine functions, and embeds the time order information into the feature space.

6. A method for intelligent cleaning and processing of multi-source heterogeneous data according to claim 4, characterized in that: The multi-head self-attention mechanism maps the input features into query matrix, key matrix, and value matrix, and parallelly calculates the dependencies in multiple subspaces to capture multi-dimensional time step dependencies or semantic associations between words.

7. A multi-source heterogeneous data intelligent cleaning and processing method according to claim 4, characterized in that: The bottleneck layer compresses high-dimensional features into low-dimensional feature vectors through a fully connected layer, combines mean pooling to process variable-length sequences, and eliminates the influence of sequence length differences.

8. A multi-source heterogeneous data intelligent cleaning and processing method according to claim 1, characterized in that: The density-sensitive clustering method sets the neighborhood radius and the minimum number of points, starts from points in the dense area to expand the search and merge data clusters, determines isolated points that cannot be attributed as noise points, and attributes boundary points to neighboring clusters according to the distance.

9. A multi-source heterogeneous data intelligent cleaning and processing system, based on the multi-source heterogeneous data intelligent cleaning and processing method according to any one of claims 1-8, characterized in that: It includes: Data acquisition module: Used to obtain industrial sensor time-series data, long text log data, and other high-dimensional industrial data; Data preprocessing module: Used to clean, handle missing values, standardize, process text, and align timestamps for the obtained data; Feature generation module: Includes a time-series data feature generation unit and a text data feature generation unit; The timing data feature generation unit is used to input the preprocessed industrial sensor timing data into a Transformer autoencoder to generate a low-dimensional feature vector containing core patterns; the text data feature generation unit is used to input the preprocessed long text log data into a text-specific Transformer autoencoder to generate a low-dimensional semantic feature vector; The clustering analysis module: is used to apply a density-sensitive clustering method in the low-dimensional feature space output by the Transformer to identify data clusters and noise points.

10. A multi-source heterogeneous data intelligent cleaning and processing system according to claim 9, characterized in that: The timing data feature generation unit includes a sliding window sampling module, a position encoding module, a multi-head self-attention module, a feed-forward neural network module, and a bottleneck layer compression module; The sliding window sampling module is used to segment the timing data according to a fixed window length; The position encoding module is used to generate time step position vectors; The multi-head self-attention module is used to calculate multi-dimensional dependencies; The feed-forward neural network module is used to enhance non-linear feature expression; The bottleneck layer compression module is used to compress high-dimensional features into low-dimensional feature vectors; The clustering analysis module includes a density definition unit, a core cluster construction unit, a noise point determination unit, and a boundary point processing unit; The density definition unit is used to set the neighborhood radius and the minimum number of points; The core cluster construction unit is used to expand the search and merge data clusters starting from the points in the dense area; The noise point determination unit is used to determine isolated points as noise points; The boundary point processing unit is used to assign boundary points to neighboring clusters according to the distance; The system further includes a model training module, which is used to perform self-supervised training on the Transformer autoencoder, minimize the reconstruction error, introduce contrastive loss to improve the feature discrimination ability, and add noise to improve the model robustness.

Images