A method for data cleaning in construction of an industrial vertical domain corpus

Abstract

The application provides a kind of industrial vertical field corpus construction data cleaning method, belong to industrial big data technical field, the application is through collecting multi-modal industrial data and recording original time stamp, unified time reference is generated using dynamic time warping algorithm for time alignment, cross-modal fusion feature vector is generated by multi-modal attention mechanism learning feature interaction weight, key data section is identified based on event-driven trigger rule and is associated as event data package, quality evaluation and abnormal calibration are carried out through step response consistency detection, multi-modal knowledge graph is constructed and cross-modal alignment model based on dual optimization constraint satisfaction framework is used for feature completion, finally, data quality is ensured through adaptive verification process, solve the technical problems that multi-modal industrial data is difficult to ensure alignment accuracy and original information integrity in time dimension alignment and semantic level fusion process.

Description

Technical Field

[0001] This invention belongs to the field of industrial big data technology, and more specifically, it relates to a method for data cleaning in the construction of an industrial vertical domain corpus. Background Technology

[0002] In the field of intelligent industrial production monitoring, it is necessary to simultaneously collect multimodal data such as text logs, numerical sensors, image monitoring, and audio signals to achieve comprehensive perception of equipment status. Traditional technologies typically use fixed timestamp alignment methods to map data sequences with different sampling frequencies to a unified time axis and achieve multimodal information fusion through feature splicing. However, due to clock synchronization errors, network transmission delays, and differences in sensor response characteristics among different modal data acquisition systems, simple timestamp alignment methods cannot handle dynamic time deviation patterns, leading to misalignment of different modal data at critical event moments. Furthermore, traditional feature splicing fusion methods lack explicit modeling of semantic relationships between modalities, easily causing information loss or introducing noise during alignment. Especially when some modal data is of poor quality or missing, a rigid alignment strategy can propagate errors to other modalities, affecting the reliability of the overall dataset. In other words, existing technologies face the technical challenge of simultaneously ensuring alignment accuracy and the integrity of original information during the alignment of multimodal industrial data at the time dimension and the fusion at the semantic level. Summary of the Invention

[0003] In view of this, the present invention provides a method for data cleaning in the construction of an industrial vertical domain corpus, which can solve the technical problem in the prior art that it is difficult to simultaneously ensure alignment accuracy and original information integrity in the process of aligning multimodal industrial data in the time dimension and fusing it at the semantic level.

[0004] This invention is implemented as follows: It provides a method for data cleaning and construction of an industrial vertical domain corpus. This includes collecting text data, numerical sensor data, image data, and audio data from industrial scenarios, recording the original timestamps and original sampling frequencies of each modality to establish an original multimodal dataset, performing time alignment processing on the original multimodal dataset, identifying timestamp asynchrony patterns by calculating the instantaneous synchronization deviation matrix, and generating a unified time reference using a dynamic time warping algorithm to output an aligned multimodal dataset. The aligned multimodal dataset is then input into a multimodal feature extractor, which calculates the long-term association strength matrix and uses a multimodal attention mechanism to generate a cross-modal fusion feature vector set. An event-driven triggering rule base is established based on a cross-modal fusion feature vector set. Multimodal data within the time window before and after the key event trigger point is associated as event data packets by calculating a continuous rate of change matrix. The event data packets are then subjected to quality assessment and cleaning by calculating a step response consistency matrix. A multimodal knowledge graph is constructed, and feature completion is performed by calculating the intermodal association transfer matrix and using a cross-modal alignment model. The cleaned multimodal dataset is then input into the quality verification process. The cross-modal alignment model adopts a constraint satisfaction solution framework based on dual optimization. By introducing Lagrange multipliers, the constraints are incorporated into the objective function, and the primal problem and dual problem are iteratively optimized alternately until convergence.

[0005] The text data includes equipment log text and operation record text; the numerical sensor data includes temperature sensor data, pressure sensor data, and vibration sensor data; the image data includes equipment status monitoring image sequences; and the audio data includes equipment operation audio signals.

[0006] The time alignment process is applied to time series data with different original sampling frequencies, and the dynamic time warping algorithm performs flexible matching of data sequences with different modalities.

[0007] The multimodal feature extractor performs feature vectorization processing, the long-term association strength matrix quantifies the semantic association degree between different modalities, and the multimodal attention mechanism learns the interaction weights between text feature vectors, numerical feature vectors, image feature vectors, and audio feature vectors.

[0008] The continuous rate of change matrix identifies stationary data segments and abrupt data segments in the data sequence.

[0009] The step response consistency matrix detects the response synchronization of multimodal data when an event occurs.

[0010] The intermodal association transfer matrix establishes the topological structure of entity nodes and relational edges. The entity nodes include device entity nodes, state entity nodes, and event entity nodes. The relational edges include causal relational edges, temporal relational edges, and spatial relational edges. The cross-modal alignment model completes the features of incomplete nodes in the multimodal knowledge graph.

[0011] The constraint-satisfaction solution framework based on dual optimization describes the modality alignment problem as a constrained optimization problem. The primal problem minimizes the distance between feature vectors of different modalities in the semantic space. The constraints ensure that the alignment process satisfies temporal consistency, feature integrity, and semantic fidelity. The dual problem incorporates the constraints into the objective function by introducing Lagrange multipliers. While updating the model parameters in the primal problem to make the feature vectors more closely aligned, the dual problem updates the constraint weights to minimize the degree of constraint violation.

[0012] Specifically, the steps for establishing the instantaneous synchronization deviation matrix are as follows: extract the original timestamp of the i-th mode at time t from the original multimodal dataset containing M modes, calculate the time difference between any two modes i and j at the corresponding data points, divide the time difference by the maximum time span within the time period for normalization to obtain a dimensionless deviation value, and construct an M×M dimensional instantaneous synchronization deviation matrix.

[0013] Specifically, the steps for establishing the long-term correlation strength matrix are as follows: select a time window length W to cover a sufficiently long observation period, extract feature sequences for each modality data, calculate the cross-correlation function values ​​of the feature sequences of modality i and modality j within the time window length W, normalize the peak value of the cross-correlation function by dividing it by the product of the standard deviations of their respective feature sequences to obtain the correlation strength value, and traverse all modality pairs to construct an M×M dimensional long-term correlation strength matrix.

[0014] Specifically, the steps for establishing the continuous rate of change matrix involve calculating the first-order difference between adjacent sampling points in the numerical sensor data sequence to obtain the instantaneous change, dividing the instantaneous change by the time interval to obtain the rate of change, and then normalizing the rate of change by the maximum value of the numerical sensor data measurement range to obtain the normalized rate of change. The normalized rates of change of each mode in the time series are then organized into a continuous rate of change matrix.

[0015] Specifically, the steps for establishing the step response consistency matrix are as follows: identifying step input events in the industrial process, extracting the response curves of each modality before and after the step input event, calculating the rise time, overshoot, and settling time of each mode from steady state to new steady state as response feature parameters, normalizing the response feature parameters by dividing them by their respective standard reference values, calculating the cosine similarity between the normalized response feature parameter vectors of any mode pair i and j as a consistency metric, and constructing an M×M dimensional step response consistency matrix.

[0016] Specifically, the steps for establishing the intermodal association transfer matrix are as follows: in the multimodal knowledge graph, count the number of all relational paths from the entity node of modality i to the entity node of modality j through relational paths; calculate the reciprocal of the relational path length as the initial weight of the association strength; accumulate the normalized weights of all relational paths to obtain the total association transfer strength from modality i to modality j; divide the total association transfer strength by the maximum association strength in the multimodal knowledge graph for normalization; and construct an M×M dimensional intermodal association transfer matrix.

[0017] The cross-modal alignment model is structured as follows: the input layer receives feature vectors from multiple modalities; the encoder layer performs feature extraction and representation learning for each modality; the alignment layer calculates the similarity matrix between the encoded feature vectors of different modalities through a cross-modal attention mechanism and learns a transformation function that maps heterogeneous modalities to a unified semantic space; the fusion layer concatenates or weights the aligned multimodal feature vectors; and the output layer generates a unified feature representation for downstream tasks.

[0018] In the training iteration, the dual variables are first fixed to solve the original problem and update the cross-modal alignment model parameters to reduce the alignment loss. Then, the cross-modal alignment model parameters are fixed to solve the dual problem and update the Lagrange multipliers. For the violated constraints, the weight of the Lagrange multipliers is increased to make the next round of original optimization pay more attention to the constraints. For the constraints that have been satisfied, the weight of the Lagrange multipliers is decreased to release the optimization degrees of freedom.

[0019] Specifically, when the normalized rate of change of numerical sensor data exceeds the stationarity threshold, it is marked as a critical event trigger point; when the normalized inter-frame variability of image data exceeds the stability threshold, it is marked as a visual event trigger point; when the time difference between the text log recording time and the abnormal time of numerical sensor data is greater than the tolerance time threshold, it is determined to be asynchronous data and time calibration is performed; when the phase difference between the event response of image data and audio data exceeds the synchronization phase threshold, it is determined to be inconsistent data and marked as a sample to be verified; when the missing rate of a certain modality data exceeds the integrity ratio threshold, the inter-modal knowledge transfer mechanism is activated to infer and complete the missing data through other modal information; when the data repeatability error index exceeds the normalization error threshold, the number of cross-validation samplings is increased to the number of enhanced samplings to improve the reliability of the results.

[0020] This invention identifies dynamic temporal deviation patterns in multimodal data by establishing an instantaneous synchronization deviation matrix, employs a dynamic time warping algorithm to achieve flexible temporal alignment, combines a long-term association strength matrix to quantify semantic dependencies between modes, and utilizes a multimodal attention mechanism to learn feature interaction weights to generate cross-modal fusion feature vectors. Building upon this, the invention introduces a constraint satisfaction solution framework based on dual optimization, formalizing the multimodal alignment problem as an optimization problem with temporal consistency constraints, feature integrity constraints, and semantic fidelity constraints. By alternately optimizing model parameters and constraint weights using the Lagrange dual method, the alignment process searches for the optimal solution within the feasible region, avoiding overfitting to a single modality while neglecting information from other modalities. This dual optimization mechanism achieves an adaptive balance between alignment accuracy and information integrity by dynamically adjusting constraint weights. When the quality of a certain type of data is poor, its weight in the alignment is automatically reduced to prevent error propagation, thereby ensuring high-quality cross-modal alignment of the cleaned multimodal dataset while maintaining the integrity of the original information of each modality. In summary, this invention solves the technical problem mentioned in the background art of simultaneously ensuring alignment accuracy and original information integrity in the process of aligning multimodal industrial data in the time dimension and fusing it at the semantic level. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method of the present invention.

[0022] Figure 2 This is a multimodal response diagram of the event trigger point distribution and rate of change matrix identified in the embodiment.

[0023] Figure 3 This is a quality verification diagram of the data repeatability error index and parameter adaptive adjustment process in the embodiment. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0025] like Figure 1 The diagram shown is a flowchart of a data cleaning method for constructing an industrial vertical domain corpus provided by the present invention. This method includes the following steps: S01. Collect text data, numerical sensor data, image data, and audio data from an industrial scene and record the original timestamps and original sampling frequencies of each modal data to establish an original multimodal dataset. The text data includes equipment log text and operation record text, the numerical sensor data includes temperature sensor data, pressure sensor data, and vibration sensor data, the image data includes equipment status monitoring image sequences, and the audio data includes equipment operation audio signals. S02. Time alignment processing is performed on time series data with different original sampling frequencies in the original multimodal dataset. The timestamp synchronization deviation matrix of each modality data is calculated to identify the timestamp synchronization mode. The dynamic time warping algorithm is used to elastically match different modality data sequences to generate a unified time reference and output the aligned multimodal dataset. S03. Input the aligned multimodal dataset into the multimodal feature extractor for feature vectorization. Quantify the semantic association between different modalities by calculating the long-term association strength matrix. Use the multimodal attention mechanism to learn the interaction weights between text feature vectors, numerical feature vectors, image feature vectors, and audio feature vectors to generate a cross-modal fusion feature vector set. S04. Establish an event-driven triggering rule base based on cross-modal fusion feature vector set. Identify stationary data segments and abrupt data segments in the data sequence by calculating the continuous rate of change matrix. When the normalized rate of change of numerical sensor data exceeds the stationarity threshold, it is marked as a key event trigger point. When the normalized inter-frame difference of image data exceeds the stability threshold, it is marked as a visual event trigger point. Associate the multimodal data within the time window before and after the key event trigger point as an event data packet. S05. Perform quality assessment and cleaning on the event data packets. Detect the response synchronization of multimodal data when the event occurs by calculating the step response consistency matrix. When the time difference between the text log recording time and the abnormal time of the numerical sensor data is greater than the tolerance time threshold, it is determined to be asynchronous data and time calibration is performed. When the event response phase difference between image data and audio data exceeds the synchronization phase threshold, it is determined to be inconsistent data and marked as a sample to be verified. S06. Construct a multimodal knowledge graph to represent the cleaned dataset. Establish the topological structure of entity nodes and relational edges by calculating the intermodal association transfer matrix. The entity nodes include device entity nodes, state entity nodes, and event entity nodes. The relational edges include causal relational edges, temporal relational edges, and spatial relational edges. Use a cross-modal alignment model to complete the features of incomplete nodes in the multimodal knowledge graph. When the missing data rate of a certain modality exceeds the integrity ratio threshold, the intermodal knowledge transfer mechanism is activated to complete the missing data through inference from other modal information. S07. Input the cleaned multimodal dataset into the quality verification process. When the data repeatability error index exceeds the normalization error threshold, increase the number of cross-validation samplings to enhance the reliability of the results. When the data repeatability error index is in the range of [0, normalization error threshold], adopt the standard single-validation mode to save resources. If the correlation coefficient of the multimodal data sequence is greater than the strong correlation coefficient threshold, use the multiple linear fitting algorithm to establish a prediction model. When the correlation coefficient of the multimodal data sequence is in the range of [weak correlation coefficient threshold, strong correlation threshold], switch to the adaptive piecewise fitting method to handle nonlinear relationships. When the data variation degree is less than the stable variation threshold, maintain the stable operation of the existing cleaning strategy to ensure system reliability. If the data variation degree is greater than the fluctuation variation threshold, start the parameter adaptive adjustment mechanism to cope with changes in data distribution.

[0026] The process of establishing the instantaneous synchronization deviation matrix is ​​as follows: for a matrix containing... The original multimodal dataset of the first modality is used to extract the 1st modality. The mode at time The original timestamp is Calculate any two modes and The time difference between corresponding data points is denoted as . Time difference Divide by the maximum time span within the specified time period Normalization is performed to obtain the dimensionless deviation value. , build The instantaneous synchronization deviation matrix, wherein the dimensional instantaneous synchronization deviation matrix contains the dimensional instantaneous synchronization deviation matrix. Line number The elements of the column represent modalities. Relative to mode The instantaneous time deviation normalized value, the instantaneous synchronization deviation matrix is ​​used to identify the real-time synchronization error mode in different modal data acquisition processes.

[0027] The process of establishing the long-term correlation strength matrix is ​​as follows: selecting the time window length. Covering a sufficiently long observation period, feature sequence extraction is performed on each modality data, and modalities are calculated. Feature sequences and modes The feature sequence in the time window length cross-correlation function values ​​within cross-correlation function values The peak value divided by the product of the standard deviations of their respective characteristic sequences Normalization is performed to obtain the correlation strength value. traverse all modal pairs to construct A long-term association strength matrix, wherein the elements of the long-term association strength matrix reflect the semantic dependencies and information complementarity of different modal data over a long time scale.

[0028] The process of establishing the continuous rate of change matrix is ​​as follows: The instantaneous change is obtained by calculating the first-order difference between adjacent sampling points of the numerical sensor data sequence; the instantaneous change is divided by the time interval to obtain the rate of change; the rate of change is then divided by the maximum value of the measurement range of the numerical sensor data for normalization to obtain the normalized rate of change; the change in the structural similarity index between adjacent frames of image data is calculated as the visual rate of change; the temporal changes in short-time energy and zero-crossing rate of audio data are calculated as the acoustic rate of change; and the normalized rates of change of each mode in the time series are organized into a continuous rate of change matrix. The rows of the continuous rate of change matrix correspond to different modes, and the columns correspond to the time series index. The continuous rate of change matrix is ​​used to identify stationary data segments and abrupt data segments to support event-driven data segmentation.

[0029] The process of establishing the step response consistency matrix is ​​as follows: Identify step input events in the industrial process; extract the response curves of each modal data before and after the step input event; calculate the rise time, overshoot, and settling time of each mode from steady state to a new steady state as response characteristic parameters; normalize the response characteristic parameters by dividing them by their respective standard reference values; and apply this normalization method to any mode pair. and Calculate the cosine similarity between their normalized response feature parameter vectors as a consistency measure, and construct... The step response consistency matrix represents the degree of synchronization and consistency of multimodal data during the dynamic response process. The step response consistency matrix is ​​used to detect sensor failures, data delays, or abnormal response modes.

[0030] The process of establishing the intermodal correlation transfer matrix is ​​as follows: In the multimodal knowledge graph, statistical modes... Entity nodes are connected to modalities via relational paths. The number of all relational paths to the entity node is used to calculate the reciprocal of the relational path length as the initial weight for the association strength. For a length of... The relational path, whose initial weight is the unit weight divided by The modality is obtained by summing the normalized weights of all relational paths. To mode The total association transmission strength is normalized by dividing the total association transmission strength by the maximum association strength in the multimodal knowledge graph, and then constructing... A modal transfer matrix is ​​defined, wherein the elements of the modal transfer matrix represent the strength of the ability to transfer and reason knowledge between different modalities, and the modal transfer matrix is ​​used to guide the direction of missing modal data completion and transfer learning strategies.

[0031] The dynamic time warping algorithm is an elastic matching method for aligning time series of different lengths and original sampling frequencies. It calculates the distance between each pair of data points in the two sequences by constructing a cost matrix, and uses dynamic programming to find the matching path with the minimum cumulative cost. The matching path allows nonlinear scaling of the time axis to compensate for differences in original sampling frequencies and time delays, thereby achieving a precise correspondence between different modal data in the time dimension.

[0032] The multimodal attention mechanism is a computational framework for learning the interaction weights of features from different modalities. It performs a linear transformation on the feature vector of each modality to generate a query vector, a key vector, and a value vector. It calculates the dot product of the query vector and all modal key vectors and normalizes it to obtain the attention weight distribution. It uses the attention weight distribution to perform a weighted summation on the value vector to obtain the fused feature vector. The multimodal attention mechanism enables the model to automatically learn to focus on important modalities and feature dimensions according to task requirements.

[0033] The event-driven triggering rule base is a set of rules built on expert knowledge in the industrial field and data-driven analysis. It includes triggering conditions, time window parameters, and associated modal ranges as rule elements. When a data pattern that meets the predefined conditions is detected, data association and packaging operations are automatically triggered, avoiding high-cost full processing of data at all times and improving the targeting and efficiency of data cleaning.

[0034] The multimodal knowledge graph is a graph structure that abstracts data from different modalities into a unified semantic representation. Entity nodes represent entity concepts extracted from data from each modality, and relation edges represent semantic relationships between entity nodes. The multimodal knowledge graph structure explicitly models cross-modal semantic connections, supporting data completion, anomaly detection, and knowledge transfer based on graph reasoning.

[0035] The cross-modal alignment model has the following structure: the input layer receives feature vectors from multiple modalities; the encoder layer performs feature extraction and representation learning for each modality; the text encoder uses a bidirectional long short-term memory network structure, the image encoder uses a convolutional neural network structure, the numerical sequence encoder uses a temporal convolutional network structure, and the audio encoder uses a combination of one-dimensional convolutional and recurrent neural networks; the alignment layer calculates the similarity matrix between the encoded feature vectors of different modalities through a cross-modal attention mechanism and learns a transformation function that maps heterogeneous modalities to a unified semantic space; the fusion layer concatenates or weights the aligned multimodal feature vectors; and the output layer generates a unified feature representation for downstream tasks. The number of heads in the multi-head attention mechanism in the cross-modal alignment model depends on the number of modalities. Data set size and feature dimensions Confirmed, the head count is calculated as the number of modes. With feature dimension The product of dividing by 64 and rounding up, then dividing by 2 and rounding down, is used to solve the cross-modal alignment model using a constraint-based solution framework based on dual optimization. The modal alignment problem is expressed as a constrained optimization problem. The primal problem minimizes the distance between feature vectors of different modalities in the semantic space. The constraints ensure that the alignment process satisfies temporal consistency, feature integrity, and semantic fidelity. The dual problem incorporates the constraints into the objective function by introducing Lagrange multipliers. While updating the model parameters in the primal problem to make the feature vectors more closely aligned, the dual problem updates the constraint weights to minimize the degree of constraint violation. The primal problem and the dual problem are iteratively optimized alternately until convergence.

[0036] The dual optimization mechanism theoretically ensures that the optimization process always searches within the feasible region, avoiding overfitting of the cross-modal alignment model to a single modality while ignoring information from other modalities. Simultaneously, the dynamic adjustment of constraints allows the cross-modal alignment model to achieve a balance between alignment accuracy and semantic fidelity. Compared to simple unconstrained optimization methods, the dual optimization framework explicitly models the structural relationships and semantic constraints between modalities, enabling high-quality cross-modal alignment of the cleaned multimodal dataset while maintaining the integrity of the original information of each modality. This improves the accuracy and robustness of subsequent multimodal knowledge graph construction and data analysis tasks. The dual optimization mechanism formalizes the multimodal data alignment problem into a constrained optimization problem. The goal of the original problem is to learn the parameters of the cross-modal alignment model so that the feature vectors of different modalities are as close as possible in the common semantic space. Constraints include temporal constraints requiring the aligned data to maintain the original temporal order, and integrity constraints requiring the information entropy of each modality to remain within a preset range before and after alignment. Within the scope, the correlation constraint requires that modal data pairs with known causal relationships maintain a strong correlation after alignment. Lagrange multipliers are introduced to assign weight coefficients to each constraint, transforming the constrained optimization problem into an unconstrained Lagrange function optimization problem. The Lagrange function contains a weighted sum of the original objective term and constraint penalty terms. The optimal dual variable that makes the Lagrange function reach a saddle point is solved by solving the dual problem. In the training iterations, the dual variable is first fixed to solve the original problem and update the cross-modal alignment model parameters to reduce the alignment loss. If a constraint is violated at this point, the corresponding penalty term increases. Then, the cross-modal alignment model parameters are fixed to solve the dual problem and update the Lagrange multipliers. For violated constraints, the weight of the Lagrange multiplier is increased to make the next round of original optimization pay more attention to the constraint. For satisfied constraints, the weight of the Lagrange multiplier is decreased to release the optimization freedom. The alternating optimization strategy of the dual optimization mechanism theoretically guarantees that the algorithm converges to the optimal solution or near-optimal solution of the original constrained optimization problem.

[0037] The dual optimization mechanism avoids time inversion or information loss during alignment by explicitly modeling constraints, ensuring the physical interpretability and semantic consistency of the cleaned multimodal dataset. The dynamic balancing of the dual optimization mechanism allows the cross-modal alignment model to adaptively adjust among multiple competing optimization objectives. When the quality of a certain type of data is poor, its weight in the alignment is automatically reduced to prevent error propagation. When certain constraints are difficult to satisfy simultaneously under operating conditions, Pareto optimal solutions are sought through adjustments to Lagrange multipliers. Compared to traditional hard constraint methods, the dual optimization framework provides a flexible constraint satisfaction mechanism, ensuring strict enforcement of core constraints while allowing secondary constraints to be appropriately relaxed when necessary. From the perspective of algorithm convergence, the dual optimization mechanism decomposes the originally non-convex constraint optimization problem into two relatively simple subproblems. Each subproblem has better mathematical properties when the other set of variables is fixed, increasing the probability of the optimization algorithm finding the global optimum in the high-dimensional parameter space. Ultimately, the cleaned multimodal dataset meets industrial application requirements in multiple quality dimensions such as alignment accuracy, completeness, and consistency, providing a reliable data foundation for subsequent data analysis, model training, and decision support.

[0038] The steps for establishing the training dataset of the cross-modal alignment model specifically include: collecting multiple batches of raw multimodal data under various working conditions from industrial scenarios; manually labeling different modal data sample pairs corresponding to the same time or the same event; for modal data combinations where it is difficult to directly determine the correspondence, a dual acquisition method is adopted, that is, repeatedly collecting data under the same working conditions and recording the precise event trigger time as the alignment anchor point; obtaining multimodal data combinations under different perturbation factors by systematically changing the acquisition conditions through orthogonal experimental design method; and using the alignment anchor point to perform interpolation labeling and temporal alignment labeling on the original data groups that are difficult to determine, forming a training dataset containing positive sample pairs and negative sample pairs, wherein the positive sample pairs are semantically related and temporally aligned multimodal data combinations, and the negative sample pairs are semantically unrelated or temporally misaligned data combinations.

[0039] The specific steps for training the cross-modal alignment model include: initializing the parameters of each layer of the cross-modal alignment model; dividing the training dataset into a training subset and a validation subset in an 8:2 ratio; performing batch training on the training subset, with each batch containing several positive and negative sample pairs; calculating the modality-encoded feature vectors and the aligned fused feature vectors through forward propagation; constructing a contrastive loss function to minimize the feature vector distance between positive sample pairs and maximize the feature vector distance between negative sample pairs; introducing a temporal consistency constraint loss to penalize alignment results that violate causality and temporal order; introducing a feature integrity constraint loss to prevent excessive information compression during the alignment process; and integrating the constraint loss terms into the total loss using the Lagrange duality method. In the loss function, the dual variables, i.e., the constraint weight coefficients, are initialized. In the primal optimization step, the dual variables are fixed and the gradient descent method is used to update the cross-modal alignment model parameters to minimize the total loss function. In the dual optimization step, the cross-modal alignment model parameters are fixed and the dual variables are updated according to the degree of constraint violation. The primal optimization step and the dual optimization step are executed alternately and iteratively. After each complete dual optimization cycle, the alignment accuracy and constraint satisfaction of the cross-modal alignment model are evaluated on the validation subset. When the performance of the validation subset does not improve for 5 consecutive cycles, the learning rate is reduced. The training process continues until the loss of the validation subset converges or the preset iteration limit is reached. The cross-modal alignment model parameters with the best performance of the validation subset are saved as the final cross-modal alignment model.

[0040] The source of the stationarity threshold is: statistically analyzing the normalized rate of change distribution of numerical sensor data under normal operating conditions in the training dataset, and calculating the 85th percentile of the normalized rate of change distribution as the stationarity threshold. The stationarity threshold is set to 0.15. The stationarity threshold is used to distinguish between stationary segments and abnormal abrupt changes in the numerical sensor data sequence.

[0041] The stability threshold is derived from the following: the normalized inter-frame difference distribution of image data under normal monitoring conditions is statistically analyzed in the training dataset, and the 90th percentile of the normalized inter-frame difference distribution is calculated as the stability threshold. The stability threshold is set to 0.22. The stability threshold is used to identify significant scene changes or device status changes in the image data sequence.

[0042] The tolerance time threshold is derived from the measurement results of clock synchronization accuracy and network transmission delay of the industrial field data acquisition system. The mean of the clock synchronization accuracy plus twice the standard deviation is taken as the tolerance time threshold. The tolerance time threshold is 3 seconds. The tolerance time threshold is used to determine whether the time alignment between the text log text and the numerical sensor data meets the synchronization requirements.

[0043] The synchronization phase threshold is obtained by analyzing the phase relationship between image data and audio data when the same event is triggered in the training dataset, and calculating the 95th percentile of the phase relationship deviation as the synchronization phase threshold. The synchronization phase threshold is set to 0.4. The synchronization phase threshold is used to detect whether the responses of image data and audio data occur synchronously.

[0044] The integrity ratio threshold is derived from the minimum data integrity requirements constructed based on the multimodal knowledge graph, combined with the reliability boundary of the data completion algorithm in industrial applications. The integrity ratio threshold is set at 18%. When the missing data rate of a certain modality exceeds the integrity ratio threshold, simply relying on the remaining data can no longer guarantee the reliability of the analysis results, and it is necessary to activate the intermodal knowledge transfer mechanism for completion.

[0045] The source of the standard error threshold is as follows: based on the data quality specifications of the industrial field and the error tolerance of downstream application tasks, the upper limit of the data repeatability error index is set at 12%, and the standard error threshold is set to 12%. When the data repeatability error index exceeds the standard error threshold, the number of samplings needs to be increased to improve measurement reliability.

[0046] The source of the enhanced sampling number is: when the data repeatability error index exceeds the standard error threshold, the sampling number is increased from the standard 1 to 5 cross-validation sampling numbers through statistical analysis. The enhanced sampling number is set to 5 times. The enhanced sampling number is used to reduce the impact of random error by taking multiple measurements under high error conditions.

[0047] The strong correlation coefficient threshold is derived from the following: analyzing the correlation distribution between multimodal data sequences in the training dataset, defining modal pairs with a correlation coefficient greater than 0.78 as strongly correlated, and setting the strong correlation coefficient threshold to 0.78. The strong correlation coefficient threshold is used to determine whether it is suitable to use a multiple linear fitting algorithm to establish a prediction model.

[0048] The weak correlation coefficient threshold is derived from the following: analyzing the correlation distribution between multimodal data sequences in the training dataset, defining modal pairs with a correlation coefficient less than 0.45 as weakly correlated or nonlinearly correlated, and setting the weak correlation coefficient threshold to 0.45. When the correlation coefficient of the multimodal data sequence is in the interval [0.45, 0.78], it indicates the existence of a nonlinear relationship that requires an adaptive piecewise fitting method.

[0049] The source of the stable variation threshold is: based on the stability requirements of industrial process control, the upper limit of stable operation of data variation is set to 8%. The stable variation threshold is 8%. When the data variation is less than the stable variation threshold, it indicates that the data distribution is stable and the existing cleaning strategy can meet the requirements.

[0050] The source of the fluctuation and variation threshold is: based on the experience value of industrial process anomaly detection, the lower limit of the high fluctuation of data variation is set to 25%, and the fluctuation and variation threshold is set to 25%. When the data variation is greater than the fluctuation and variation threshold, it indicates that the data distribution has changed significantly, and the parameter adaptive adjustment mechanism needs to be activated to re-optimize the cleaning parameters.

[0051] Optionally, the present invention also provides a computer-based approach to form an industrial vertical domain multimodal corpus data cleaning system, wherein the computer is equipped with a readable storage medium that stores program instructions, which execute the above-described method when the computer is run.

[0052] The specific implementation methods of the above steps are described in detail below.

[0053] The specific implementation of step S01 is to first deploy multiple types of data acquisition devices at the industrial production site, including a text log recording system to collect equipment operation logs and operator records, numerical acquisition devices such as temperature sensors, pressure sensors, and vibration sensors to monitor the physical state of the equipment in real time, high-definition cameras to collect visual image sequences of the equipment status, and acoustic sensors to collect audio signals during equipment operation. During the data acquisition process, the timestamp information of each data point and the original sampling frequency parameters of various sensors are recorded simultaneously. The above heterogeneous multimodal data are uniformly stored to form an original multimodal dataset. The purpose of this step is to establish an industrial scene data foundation covering multi-dimensional information such as text, numerical values, images, and audio, providing a complete data source for subsequent cross-modal analysis and knowledge extraction.

[0054] The specific implementation of step S02 addresses the time alignment problem caused by inconsistent sampling frequencies of different sensors in the original multimodal dataset. First, the timestamp sequences of each modality's data are extracted. The time difference between any two modalities at corresponding data points is calculated. The time difference is normalized by dividing it by the maximum time span within the observation period to obtain the instantaneous synchronization deviation matrix. By analyzing this matrix, systematic time delay patterns and random synchronization errors are identified. Then, a dynamic time warping algorithm is used to elastically match the time series of different modalities. This algorithm measures the distance between sequence point pairs by constructing a cost matrix and uses dynamic programming to find the matching path with the minimum cumulative cost. It allows nonlinear scaling of the time axis to compensate for sampling frequency differences. Finally, a unified time benchmark is established and the time-aligned multimodal dataset is output. This step ensures the accurate correspondence of different modal data in the time dimension, laying the foundation for subsequent cross-modal feature extraction.

[0055] The specific implementation of step S03 involves inputting time-aligned multimodal data into a multimodal feature extractor for vectorization representation. Text data is processed using natural language processing to extract semantic feature vectors, numerical sensor data is processed using time series analysis to extract statistical feature vectors, image data is processed using a convolutional neural network to extract visual feature vectors, and audio data is processed using acoustic analysis to extract spectral feature vectors. The cross-correlation function values ​​of each modality feature sequence within a set time window are calculated and normalized by dividing by the product of their respective standard deviations to obtain a long-term association strength matrix. This matrix quantifies the semantic dependencies of different modalities over a long time scale. Using a multimodal attention mechanism, the attention weight distribution is obtained by calculating the dot product of the query vector and the key vector. The value vectors are weighted and summed to generate a fusion feature vector. This mechanism enables the model to automatically learn the interaction weights between different modalities and highlight important modal feature dimensions. Finally, a cross-modal fusion feature vector set is output. This step realizes the mapping of heterogeneous modal data to a unified semantic space and establishes the association relationship between modalities.

[0056] The specific implementation of step S04 is based on establishing an event-driven data segmentation strategy using a fused feature vector set. For numerical sensor data, the first-order difference between adjacent sampling points is calculated, divided by the time interval to obtain the rate of change, and then normalized by dividing by the maximum value of the measurement range. For image data, the change in the structural similarity index between adjacent frames is calculated as the visual rate of change. For audio data, the temporal changes in short-time energy and zero-crossing rate are calculated as the acoustic rate of change. The normalized rates of change for each modality are organized into a continuous rate of change matrix to identify stationary and abrupt transition segments. When the normalized rate of change of numerical sensor data exceeds the stationarity threshold of 0.15, it is marked as a critical event trigger point. When the normalized inter-frame difference of image data exceeds the stability threshold of 0.22, it is marked as a visual event trigger point. Multimodal data within the preceding and following time windows are extracted around the trigger point and associated as an event data packet. This step avoids full processing of all data at all times but focuses on key change periods in the industrial process, improving the targeting and computational efficiency of data cleaning.

[0057] The specific implementation of step S05 involves performing a multi-dimensional quality assessment of the event data packets, identifying step input events in the industrial process, extracting the response curves of each mode, calculating response characteristic parameters such as rise time, overshoot, and settling time from steady state to new steady state, and normalizing them by dividing by a standard reference value. For any mode pair, the cosine similarity between the normalized response characteristic vectors is calculated to construct a step response consistency matrix. This matrix reflects the degree of synchronization and consistency of multimodal data in the dynamic response process. When the time difference between the text log recording time and the numerical sensor abnormal time is greater than the tolerance time threshold of 3 seconds, it is determined to be asynchronous data and time calibration is performed. When the event response phase difference between image data and audio data exceeds the synchronization phase threshold of 0.4, it is determined to be inconsistent data and marked as a sample to be verified. This step ensures the internal consistency and reliability of the event data packets by detecting sensor failures, data delays, and abnormal response modes.

[0058] The specific implementation of step S06 involves abstracting the cleaned multimodal data into a knowledge graph structure. Device entities, state entities, and event entities are extracted from each modality's data as graph nodes. Causal relationship edges, temporal relationship edges, and spatial relationship edges between entities are established to form a topological structure. The number of all paths connecting modalities through relationship paths is counted, and the reciprocal of the path length is calculated as the initial weight of the association strength. The normalized weights are accumulated to obtain the total association transmission strength, which is then divided by the maximum value to construct the intermodal association transmission matrix. This matrix represents the strength of the ability of knowledge to be transmitted and reasoned between different modalities. When the missing rate of a certain modality's data exceeds the integrity ratio threshold of 18%, the intermodal knowledge transfer mechanism is activated. The cross-modal alignment model is used to infer and complete the missing data through information from other modalities. This model includes a text bidirectional long short-term memory network encoder, an image convolutional neural network encoder, a numerical temporal convolutional network encoder, and an audio one-dimensional convolutional recurrent neural network encoder. The similarity matrix between heterogeneous modalities is calculated through a cross-modal attention mechanism, and a transformation function mapping to a unified semantic space is learned. This step supports data completion and anomaly detection based on graph reasoning by explicitly modeling cross-modal semantic connections.

[0059] The specific implementation of step S07 involves quality verification of the cleaned multimodal dataset. When the data repeatability error index exceeds the standard error threshold of 12%, the number of sampling times is increased from the standard 1 to 5 cross-validation samplings to reduce the impact of random errors. When the data repeatability error index is in the range of 0 to 12%, the standard single-validation mode is used to save computational resources. If the correlation coefficient of the multimodal data sequence is greater than the strong correlation coefficient threshold of 0.78, a multiple linear fitting algorithm is used to establish a prediction model. When the correlation coefficient is in the range of weak correlation coefficient threshold of 0.45 to strong correlation coefficient threshold of 0.78, the adaptive piecewise fitting method is switched to handle nonlinear relationships. When the data variation is less than the stable variation threshold of 8%, the existing cleaning strategy is kept running stably. If the data variation is greater than the fluctuation variation threshold of 25%, the parameter adaptive adjustment mechanism is activated to re-optimize the cleaning parameters to cope with changes in data distribution. This step ensures the reliability and adaptability of the cleaning results by dynamically adjusting the verification strategy and fitting method.

[0060] The training of the cross-modal alignment model employs a dual optimization framework, formulating the modal alignment problem as a constrained optimization problem. The primal problem objective is to minimize the distance between feature vectors of different modalities in the semantic space. Constraints include temporal constraints requiring the preservation of the original temporal order, integrity constraints requiring information entropy to remain within a preset range before and after alignment, and correlation constraints requiring strongly correlated modal pairs with known causal relationships. Lagrange multipliers are introduced to assign weight coefficients to each constraint, transforming the constrained optimization problem into an unconstrained Lagrangian function optimization problem. The dual problem is solved to find the optimal dual variable that brings the Lagrangian function to a saddle point. During training, the dual variable is first fixed to solve the primal problem and update the model parameters to reduce the alignment loss. Then, the dual variable is fixed... The model parameter solution involves updating the Lagrange multipliers in the dual problem. For violated constraints, their weights are increased to make the next round of optimization prioritize those constraints. For satisfied constraints, their weights are decreased to release optimization freedom. This alternating optimization strategy theoretically guarantees that the algorithm converges to the optimal solution of the original constraint optimization problem. This mechanism avoids time inversion or information loss and ensures physical interpretability through explicit constraint modeling. Dynamic balancing allows the model to adaptively adjust among multiple competing objectives. When the quality of a certain modality data is poor, its weight is automatically reduced to avoid error propagation. Compared to hard constraint methods, this provides a flexible constraint satisfaction mechanism that ensures strict enforcement of core constraints while allowing for moderate relaxation of secondary constraints, thereby increasing the probability of finding the global optimum in the high-dimensional parameter space.

[0061] It should be noted that the key technical approach of this invention includes a multimodal time alignment mechanism based on dynamic time warping. This mechanism identifies systematic delays and random errors by constructing an instantaneous synchronization deviation matrix and compensates for different sampling frequencies and time delays using elastic matching paths. Compared to traditional fixed-rate resampling methods, this avoids information distortion caused by linear interpolation, preserves the original temporal characteristics of each modality's data, and provides accurate time correspondences for subsequent cross-modal analysis. The event-driven data segmentation strategy identifies stationary and abrupt segments using a continuous rate-of-change matrix, and only associates and packages time windows around key event trigger points. Compared to full data processing, this significantly reduces computational costs while improving the targeting of data cleaning, ensuring complete recording of important industrial events while filtering redundant, stable operational data. The cross-modal alignment model based on dual optimization formalizes the modal alignment problem into a constrained optimization framework. By dynamically adjusting the constraint weights through Lagrange multipliers, it achieves a balance between alignment accuracy and semantic fidelity. Compared with unconstrained optimization methods, it explicitly models the structural relationships between modes, avoiding overfitting to a single mode. By alternately optimizing and decomposing the non-convex problem, it improves the global optimal solution search capability, ensuring that the cleaned data meets the requirements of industrial applications in multiple dimensions such as alignment accuracy, integrity, and consistency.

[0062] The synergistic effect of the above technical approaches is reflected in the fact that dynamic time warping provides an accurate time reference for cross-modal alignment, the event-driven segmentation strategy focuses on key time periods to reduce the computational burden of dual optimization, and the dual optimization framework ensures the semantic consistency of time alignment and event association through constraint mechanisms. The three form a progressive data cleaning system from low-level time synchronization to mid-level event recognition and then to high-level semantic alignment. Compared with traditional methods that independently process each modality or simply splice multimodal data, this invention achieves deep fusion of cross-modal information by explicitly modeling the temporal dependencies, semantic associations and physical constraints between modalities, improving the integrity and reliability of the cleaned dataset, and providing a high-quality multimodal corpus foundation for industrial intelligent analysis and decision support.

[0063] It should be noted that this invention also addresses the following technical problem: In the process of industrial multimodal data acquisition, there are complex temporal dependencies and causal relationships between different modal data. However, traditional data cleaning methods lack explicit modeling of these relationships, making it difficult to accurately identify asynchronous and inconsistent data during data quality assessment. This invention identifies stationary and abrupt segments of data sequences by establishing a continuous rate of change matrix, constructs an event-driven triggering rule base to associate multimodal data at key event moments into event data packets, and detects the degree of synchronization of multimodal data in the dynamic response process through a step response consistency matrix. This mechanism utilizes the physical characteristics of industrial processes, namely that although there is a time difference in the responses of different sensors to the same event, they should conform to causal logic. By quantifying the similarity of response characteristic parameters, it judges whether the data is consistent, thereby accurately locating abnormal data that needs calibration or labeling, improving the targeting and accuracy of data cleaning.

[0064] Furthermore, this invention addresses the technical challenge of effectively completing data in multimodal datasets when certain modalities are missing or of poor quality. Traditional methods typically employ independent single-modal data imputation algorithms, neglecting the complementarity of information and the transferability of knowledge between different modalities. This invention constructs a multimodal knowledge graph to establish a topological structure of entity nodes and relational edges. It utilizes an intermodal association transfer matrix to quantify the strength of knowledge transfer capabilities between different modalities. When the missing data rate of a certain modality exceeds a completeness threshold, an intermodal knowledge transfer mechanism is activated. This mechanism, based on graph reasoning, uses information from existing modalities to reason along relational paths in the knowledge graph to complete the missing modality data, achieving cross-modal knowledge transfer and feature completion. Compared to single-modal imputation methods, it can utilize richer contextual information, improving the accuracy and reliability of data completion and ensuring the completeness and consistency of the cleaned dataset.

[0065] Specifically, the principle of this invention is as follows: The solution to the aforementioned technical problems lies in decoupling the time synchronization problem from the semantic alignment problem through a multi-layered alignment and fusion mechanism. First, in the time dimension, a dynamic time warping algorithm is used to compensate for the differences in sampling frequency and time delays between different modalities, constructing a unified time benchmark. Then, at the semantic level, a multi-modal attention mechanism is used to learn the dependencies between modalities and automatically allocate feature interaction weights. More importantly, the dual optimization framework integrates domain knowledge from engineering practice into the optimization objective by explicitly modeling constraints such as temporal consistency, feature integrity, and semantic fidelity. The original problem optimizes model parameters to align features of different modalities in the semantic space, while the dual problem dynamically adjusts penalty weights according to the degree of constraint violation. The alternating iteration of both ensures that the algorithm converges to the optimal solution that satisfies all constraints. This constrained optimization mechanism avoids the problem of sacrificing information integrity for improved alignment accuracy in traditional unconstrained methods. Simultaneously, the adaptive adjustment of Lagrange multipliers achieves a flexible balance between different quality objectives, making the data cleaning process more robust and interpretable, conforming to the objective laws of multi-dimensional data quality requirements in industrial applications.

[0066] The following provides a specific embodiment 1 of the present invention, and the specific implementation of each step in this embodiment 1 is described in detail below.

[0067] In this embodiment, the specific implementation of step S01 is the same as described above, and will not be repeated in detail here.

[0068] The specific implementation of step S02 addresses the time alignment problem caused by inconsistent sampling frequencies of different sensors in the original multimodal dataset. First, the timestamp sequence of each modality's data is extracted. The formula for establishing the instantaneous synchronization deviation matrix is ​​as follows: ; ; ; In the formula, For the first The mode at time The original timestamp, in seconds; For the first The mode at time The original timestamp, in seconds; For modality and modality The time difference between corresponding data points, in seconds; The maximum time span within the observation period, in seconds; This is a dimensionless deviation value; for 3D instantaneous synchronization deviation matrix; The total number of modes; This serves as a time index. By analyzing this matrix, systematic time delay patterns and random synchronization errors are identified. Then, a dynamic time warping algorithm is used to elastically match time series of different modalities. This algorithm measures the distance between sequence point pairs by constructing a cost matrix and uses dynamic programming to find the matching path with the minimum cumulative cost. It allows nonlinear scaling of the time axis to compensate for sampling frequency differences, and finally establishes a unified time benchmark and outputs a time-aligned multimodal dataset.

[0069] The specific implementation of step S03 involves inputting the time-aligned multimodal data into a multimodal feature extractor for vectorization representation. The formula for establishing the long-term correlation strength matrix is ​​as follows: ; ; ; In the formula, The time window length is expressed in units of the number of sampling points. For modality In the Normalized feature values ​​of each sampling point; For modality In the Normalized feature values ​​of each sampling point; For sampling point index; For modality With mode The cross-correlation function value; For modality The standard deviation of the characteristic sequence; For modality The standard deviation of the characteristic sequence; This represents the correlation strength value. for The long-term correlation strength matrix is ​​given. The calculation formula for the multimodal attention mechanism is as follows: ; ; ; ; ; In the formula, For modality The query vector; For modality The query transformation matrix; For modality The input feature vector; For modality The key vector; For modality The key transformation matrix; For modality The input feature vector; For modality The value vector; For modality The value transformation matrix; The dimension of the key vector; It is a natural exponential function; For modality For modes Attention weights; For modality fused feature vectors; Key vector transpose; This is the modality index. The formula for calculating the number of heads in a multi-head attention mechanism is as follows: ; In the formula, The number of heads in a multi-head attention mechanism; The dimension of the feature vector; It is a rounding function; This is the floor function.

[0070] The specific implementation of step S04 is to establish an event-driven data segmentation strategy based on the fused feature vector set. The formula for establishing the continuous rate of change matrix is ​​expressed as follows: ; ; ; ; In the formula, For modality At any moment The sensor data values ​​are in units determined by the sensor type. The time interval is in seconds. It is an instantaneous change; The rate of change; For modality The maximum value of the measurement range; Normalized rate of change; for A continuous rate of change matrix; The length of the time series; For each moment in the time series, the formula for calculating the visual rate of change of image data is as follows: ; ; In the formula, For a moment Image frames; For a moment Image frames; For a moment Image frames; This is a structural similarity index function; For image The mean; For image The mean; For image Standard deviation; For image Standard deviation; For image and covariance; and These are stability constants, with empirical values ​​of 0.01 and 0.03, respectively. This represents the maximum change in the structural similarity index, typically set to 1. The normalized visual rate of change of the image; This is an absolute value function. The formula for calculating the acoustic rate of change of audio data is as follows: ; ; ; In the formula, For the first Short-time energy of frame audio; For audio frame indexing; For the first The first frame of audio Amplitude of each sampling point; For sampling point index; The number of sampling points per frame of audio; For the first Zero-crossing rate of frame audio; This is a sign function; it outputs 1 when the input is greater than 0, -1 when the input is less than 0, and 0 when the input is equal to 0. This represents the maximum value of the short-time energy. This represents the maximum value of the zero-crossing rate. This represents the normalized acoustic rate of change of the audio signal. For the first Short-time energy of frame audio; For the first Zero-crossing rate of frame audio.

[0071] The specific implementation of step S05 involves performing a multi-dimensional quality assessment of the event data packets. The formula for establishing the step response consistency matrix is ​​as follows: ; ; ; ; ; ; In the formula, For modality The rise time, in seconds; For modality The standard reference rise time, in seconds, is obtained through historical data statistics. Normalized rise time; For modality The overshoot is expressed as a percentage. For modality The standard reference overshoot is expressed as a percentage and is obtained through historical data statistics. This is the normalized overshoot. For modality The settling time, in seconds; For modality The standard reference stabilization time, in seconds, is obtained through historical data statistics. For normalized steady-state time; For modality The normalized response feature parameter vector; For modality The normalized response feature parameter vector; This is a vector transpose operation; For modality With mode Consistency measure; for 3D step response consistency matrix; Let be the Euclidean norm of a vector, calculated using the formula: .

[0072] The specific implementation of step S06 involves abstracting the cleaned multimodal data into a knowledge graph structure. The formula for establishing the intermodal association transfer matrix is ​​as follows: ; ; ; ; In the formula, For relational path indexing; For the first The length of the relationship path, i.e., the number of edges in the path; For the first The initial weights of the paths; For modality To mode The total number of relational paths; For modality To mode Total correlation propagation strength; The maximum correlation strength among all modality pairs in a multimodal knowledge graph; This represents the normalized total correlation transitivity strength. for Intermodal correlation transfer matrix.

[0073] The specific implementation method of step S07 is the same as described above, and will not be repeated in detail here.

[0074] It should be explained that the specific implementation of the dual optimization mechanism is to formalize the multimodal data alignment problem into a constrained optimization problem. The objective function formula of the original problem is expressed as follows: ; In the formula, For alignment loss; For modality Feature vectors in a common semantic space; For modality Feature vectors in a common semantic space; Let be the Euclidean norm of the vector; express Not equal to The constraints are expressed in the following formula: ; ; ; ; ; ; In the formula, For time-constrained loss; Alignment of model parameters across modalities; This is a time sequence function, representing the number of data points that satisfy the temporal sequence relationship before a certain time. For time indexing; This is an indicator function that takes the value 1 when the condition is met and 0 otherwise. Symbols indicating temporal sequence; The number of samples in the training set; For modality At any moment Alignment feature vectors; This is a function to find the maximum value. Loss due to integrity constraints; The information entropy function; This is the number of intervals after discretization of the feature vector, with an empirical value of 100. For modality The eigenvectors in the th The probability of each interval; For modality The original input feature vector; For correlation constraint loss; A set of modal pairs with known causal relationships; For set The number of elements; The threshold for strong correlation is empirically set at 0.78. This is the Pearson correlation coefficient function; For modality The mean of the aligned feature vectors on the training set is calculated using the following formula: ; For modality The mean of the aligned feature vectors on the training set. The formula for the Lagrange function is as follows: ; In the formula, It is a Lagrange function; Lagrange multipliers with time constraints; Lagrange multipliers for integrity constraints; Let be the Lagrange multipliers for the correlation constraints. The update formula for optimizing the primal problem is expressed as follows: ; In the formula, For the first Model parameters for the next iteration; For the first Model parameters for the next iteration; Index for iteration count; The learning rate is 0.001 (empirical value). Let be the gradient operator with respect to the model parameters. The update formula for the dual problem optimization is expressed as follows: ; ; ; In the formula, For the first Temporal constraint Lagrange multipliers for each iteration; For the first Temporal constraint Lagrange multipliers for each iteration; The dual learning rate has an empirical value of 0.01. For the first Integrity constraints on Lagrange multipliers in each iteration; For the first Integrity constraints on Lagrange multipliers in each iteration; For the first The correlation constraint Lagrange multipliers in the next iteration; For the first The correlation-constrained Lagrange multipliers of the next iteration.

[0075] The principle and effect of the self-generated formula for the dual optimization mechanism are explained as follows: this mechanism achieves a dynamic balance between cross-modal alignment accuracy and semantic fidelity by decomposing the constrained optimization problem into alternating optimization of the primal and dual problems. The primal problem promotes the alignment of heterogeneous modalities in the common semantic space by minimizing the distance between the feature vectors of different modalities. Its objective function is... Euclidean distance is used to measure the difference in feature vectors between modalities, ensuring that the aligned feature vectors are as close as possible in the semantic space.

[0076] ; Temporal Constraint Loss Through time sequence function The system counts the number of violations of temporal relationships before and after alignment, and penalizes data points whose temporal order is reversed after alignment, ensuring that the alignment process does not destroy the original temporal causal relationship.

[0077] ; Integrity constraint loss Through the information entropy function The change in information content of feature vectors before and after alignment is measured to prevent excessive compression or loss of information during the alignment process and to ensure that the original information of each modality is preserved.

[0078] ; Correlation constraint loss Through the Pearson correlation coefficient function Measure the correlation strength of modal pairs with known causal relationships after alignment to ensure that modalities with causal relationships remain strongly correlated after alignment, and avoid the alignment process from destroying the semantic connections between modalities.

[0079] ; The dual problem ensures that temporal consistency, information integrity, and relevance constraints are satisfied by adjusting the Lagrange multipliers. This alternating optimization strategy avoids the problem of the model overfitting a single modality while ignoring information from other modalities. At the same time, explicit modeling of constraints prevents temporal inversion or information loss during the alignment process, ensuring the physical interpretability and semantic consistency of the cleaned multimodal dataset. Compared with traditional unconstrained optimization methods, this dual optimization framework significantly improves the robustness and accuracy of cross-modal alignment, enabling high-quality cross-modal fusion of data cleaning results while maintaining the integrity of the original information of each modality. This provides a reliable data foundation for subsequent multimodal knowledge graph construction and industrial data analysis.

[0080] To better understand and implement this invention, the following is a specific application scenario of this invention, Example 2: A technical team used the method proposed in this invention for data cleaning and fusion when processing multimodal data from a chemical production process monitoring system. This system involves multiple data sources, including temperature, pressure, and vibration monitoring of reaction vessels, as well as equipment operation audio, operation logs, and real-time video monitoring.

[0081] In phase S01, the technical team collected 72 hours of raw multimodal dataset. The text data included 4,850 equipment operation logs and 2,340 fault diagnosis records; the numerical sensor data included temperature sensor (sampling frequency of 10 Hz), pressure sensor (sampling frequency of 8 Hz), and vibration sensor (sampling frequency of 12 Hz); the image data consisted of 960×720 resolution video sequences (frame rate of 25 fps) captured by the equipment monitoring camera; and the audio data consisted of equipment operating noise records (sampling rate of 16 kHz).

[0082] During the S02 time alignment phase, the technical team discovered significant discrepancies in the original timestamps of different modal data. The synchronization deviation between the temperature and pressure sensors reached a maximum of 0.18, and the synchronization deviation between video frames and audio data reached 0.22. By applying a dynamic time warping algorithm, the team aligned all modal data to a unified 100 ms time base, ultimately reducing the synchronization deviation between all modalities to below 0.03.

[0083] In the S03 feature extraction and fusion stage, as shown in Table 1, the calculated cross-modal long-term association strength matrix displays the semantic association degree between different modalities. The association strength between temperature and pressure sensor data is 0.82, indicating a strong correlation between the two; the association strength between temperature and audio data is 0.56, indicating a moderate semantic dependency. Through a multimodal attention mechanism, text features, numerical features, image features, and audio features are mapped into a unified 512-dimensional semantic space.

[0084] Table 1. Long-term correlation strength matrix across modes

[0085] In the S04 event-driven segmentation phase, such as Figure 2 As shown, the technical team identified a stationarity threshold of 0.15 and a stability threshold of 0.22. During the 72-hour monitoring period, a total of 128 key event trigger points were identified, including 42 temperature rise events, 38 pressure drop events, 31 abnormal vibration events, and 17 equipment alarm events. Multimodal data within a 60-second window before and after each event were correlated into event data packets, generating a total of 128 complete multimodal event data packets.

[0086] During the S05 quality assessment and cleaning phase, as shown in Table 2, the step response consistency matrix detected the response synchronization of the multimodal data. The consistency metric for the temperature and pressure sensors was 0.91, indicating a high degree of synchronization between their responses; the consistency metric for the text logs and numerical sensors was 0.73, indicating an average time delay of approximately 2 seconds. The technical team performed time calibration on 18 of the text log records and marked 12 asynchronous data entries as samples to be verified.

[0087] Table 2. Step Response Consistency Matrix

[0088] During the S06 multimodal knowledge graph construction phase, the technical team established a knowledge graph containing 125 device entity nodes, 87 state entity nodes, and 128 event entity nodes. Through intermodal association transitivity matrix calculations, the association transitivity strength from text modality to numerical modality was 0.84, and from visual modality to audio modality was 0.67. For modal data with a missing rate exceeding the 18% integrity threshold, the technical team initiated an intermodal knowledge transfer mechanism, utilizing information from other modalities to complete 38 incomplete data segments.

[0089] During the S07 quality verification phase, such as Figure 3As shown, the initial data repeatability error was 14.2%, exceeding the 12% normed error threshold. The technical team increased the cross-validation sampling to 5 times, ultimately reducing the error to 8.6%. Regarding the correlation coefficient distribution of multimodal data sequences, the team found that mode pairs with strong correlation coefficients above 0.78 accounted for 38%, and a prediction model was established using multiple linear fitting; mode pairs with correlation coefficients between 0.45 and 0.78 accounted for 47%, and an adaptive piecewise fitting method was used; mode pairs with correlation coefficients below 0.45 accounted for 15%, and nonparametric methods were used. 72% of the time periods had a data variation rate below 8%, and the existing cleaning strategy was maintained; 8% of the time periods had a variation rate above 25%, and an adaptive parameter adjustment mechanism was initiated.

[0090] Using the method of this invention, the technical team ultimately output a high-quality corpus containing 128 complete event data packets, 115 completed multimodal knowledge graph entity nodes, and a data consistency rate of 93.7%. Compared to traditional single-modal data processing methods, this invention fundamentally solves the asynchronous acquisition problem through a dynamic time warping algorithm, fundamentally ensures semantic consistency between different modalities through a cross-modal alignment model and a dual optimization constraint framework, fundamentally achieves targeted data cleaning rather than blind full processing through an event-driven multi-matrix framework, fundamentally makes missing data completion more interpretable through explicit modeling of the multimodal knowledge graph, and fundamentally ensures the multi-dimensional reliability of the cleaning results through a hierarchical quality verification mechanism.

[0091] It should be noted that the variables involved in this invention are explained in detail in Tables 3, 4, and 5.

[0092] Table 3. Variable Explanation Table (Part 1)

[0093] Table 4. Variable Explanation Table (Part Two)

[0094] Table 5. Variable Explanation Table (Part 3)

[0095] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for data cleaning in constructing an industrial vertical domain corpus, characterized in that, The process includes collecting text data, numerical sensor data, image data, and audio data from industrial scenarios, recording the original timestamps and original sampling frequencies of each modality to establish an original multimodal dataset, performing time alignment processing on the original multimodal dataset, identifying timestamp desynchronization patterns by calculating the instantaneous synchronization deviation matrix, and generating a unified time reference using a dynamic time warping algorithm to output an aligned multimodal dataset, inputting the aligned multimodal dataset into a multimodal feature extractor, calculating the long-term association strength matrix, and generating a cross-modal fusion feature vector set using a multimodal attention mechanism, establishing an event-driven triggering rule base based on the cross-modal fusion feature vector set, associating multimodal data within the time window before and after the key event trigger point into event data packets by calculating the continuous rate of change matrix, performing quality assessment and cleaning of the event data packets by calculating the step response consistency matrix, constructing a multimodal knowledge graph, calculating the intermodal association transfer matrix, and performing feature completion using a cross-modal alignment model, and inputting the cleaned multimodal dataset into a quality verification process. The cross-modal alignment model adopts a constraint satisfaction solution framework based on dual optimization. By introducing Lagrange multipliers, the constraints are incorporated into the objective function, and the primal problem and the dual problem are iteratively optimized alternately until convergence.

2. The method according to claim 1, characterized in that, The text data includes equipment log text and operation record text; the numerical sensor data includes temperature sensor data, pressure sensor data, and vibration sensor data; the image data includes equipment status monitoring image sequences; and the audio data includes equipment operation audio signals.

3. The method according to claim 2, characterized in that, The time alignment process is applied to time series data with different original sampling frequencies, and the dynamic time warping algorithm performs flexible matching of data sequences with different modalities.

4. The method according to claim 3, characterized in that, The multimodal feature extractor performs feature vectorization processing, the long-term association strength matrix quantifies the semantic association degree between different modalities, and the multimodal attention mechanism learns the interaction weights between text feature vectors, numerical feature vectors, image feature vectors, and audio feature vectors.

5. The method according to claim 4, characterized in that, The continuous rate of change matrix identifies stationary and abrupt data segments in the data sequence.

6. The method according to claim 5, characterized in that, The step response consistency matrix detects the response synchronization of multimodal data when an event occurs.

7. The method according to claim 6, characterized in that, The intermodal association transfer matrix establishes the topological structure of entity nodes and relation edges. The entity nodes include device entity nodes, state entity nodes, and event entity nodes. The relation edges include causal relation edges, temporal relation edges, and spatial relation edges. The cross-modal alignment model completes the features of incomplete nodes in the multimodal knowledge graph.

8. The method according to claim 7, characterized in that, The constraint-satisfaction solution framework based on dual optimization describes the modality alignment problem as a constrained optimization problem. The primal problem minimizes the distance between feature vectors of different modalities in the semantic space. The constraints ensure that the alignment process satisfies temporal consistency, feature integrity, and semantic fidelity. The dual problem incorporates the constraints into the objective function by introducing Lagrange multipliers. While updating the model parameters in the primal problem to make the feature vectors more closely aligned, the dual problem updates the constraint weights to minimize the degree of constraint violation.

9. The method according to claim 8, characterized in that, The steps for establishing the instantaneous synchronization deviation matrix are as follows: for the original multimodal dataset containing M modes, extract the original timestamp of the i-th mode at time t, calculate the time difference between any two modes i and j at the corresponding data points, divide the time difference by the maximum time span within the time period for normalization to obtain the dimensionless deviation value, and construct the M×M-dimensional instantaneous synchronization deviation matrix.

10. The method according to claim 9, characterized in that, The steps for establishing the long-term correlation strength matrix are as follows: select a time window length W to cover a sufficiently long observation period, extract feature sequences for each mode of data, calculate the cross-correlation function values ​​of the feature sequences of mode i and mode j within the time window length W, normalize the peak value of the cross-correlation function by dividing it by the product of the standard deviations of their respective feature sequences to obtain the correlation strength value, and traverse all mode pairs to construct an M×M dimensional long-term correlation strength matrix.

Images