Double-door sentry type entrance dirty data cleaning and error classification method and system

Abstract

The invention relates to the technical field of data preprocessing, and provides a double-door sentry type entrance dirty data cleaning and error classification method and system. The double-door sentry type entrance dirty data cleaning and error classification method comprises the following steps: calculating the root mean square of each window; calculating spectrum dispersion; if the minimum value of the root-mean-square is smaller than the idle calibration value of the production line or the spectrum dispersion is larger than a spectrum dispersion threshold value, the event window is discarded; calculating a vibration peak value or a root-mean-square of the offline industrial sensing signal data, and if the vibration peak value or the root-mean-square exceeds a first threshold value, determining that the data is signal distortion data; performing consistency judgment on the sampling rate of the off-line industrial sensing signal data, and screening sampling rate error data; and calculating a cross correlation coefficient and an energy difference between the current event window and the previous frame event window, and if the cross correlation coefficient is smaller than a second threshold value and the energy difference is more than K times, judging that the data is packet loss data. The closed loop of training data self-cleaning and error reutilization in the offline environment of the local area network is realized.

Description

Technical Field

[0001] This invention relates to the field of data preprocessing technology in industrial big data, and in particular to a method and system for cleaning and classifying dirty data with a dual-gate sentinel-type entry point. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Industrial fault prediction and health management systems have been widely deployed in high-end equipment such as rail transit, wind power, and petrochemicals. To balance real-time performance and model accuracy, the mainstream architecture adopts a three-tier model: "large model trained in the cloud—small model distilled at the edge—multimodal data collected at the edge." However, critical industries such as military, energy, and rail transit are constrained by data confidentiality regulations, requiring core production lines to be physically isolated and data leakage prohibited. This prevents the large cloud model from continuously acquiring and iterating on-site data, and edge gateways can only perform lightweight fine-tuning based on local data within the local area network. Otherwise, the model's performance will degrade as operating conditions change.

[0004] Given the rigid requirement of offline data, the industry generally adopts two technical approaches: one is to first train a main model of hundreds of megabytes in the cloud with hundreds of millions of samples, and then obtain sub-models of ≤30MB through pruning / quantization, and then distribute them to the edge; the other is to store nearly a week's worth of data on the edge side and perform 1-3 rounds of incremental fine-tuning daily using methods such as LightGBM, TinyCNN, or LoRA. Both approaches rely on locally generated high-quality training data, but the field environment inevitably has "industrial contamination" such as sensor disconnections, electromagnetic interference, resonance flooding, and timestamp misalignment. If this data is directly fed into the fine-tuning stage, it will amplify model bias and accelerate performance degradation. Summary of the Invention

[0005] To address the technical problems mentioned above, this invention provides a dual-gate sentinel-style entry point dirty data cleaning and error classification method and system. This invention can complete the entire chain of self-cleaning at the edge entry point, from alignment to lightweight signal-to-noise ratio (SNR) gate to physical gate to reuse. This improves the purity of the training set for fine-tuning of offline industrial prognostics and health management (PHM) systems, reduces the false alarm rate of sensor disconnection, and requires zero depth, zero external network, and zero hardware modification throughout the process. It provides a reliable data foundation for lightweight fine-tuning of large industrial models in physically isolated data environments.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a method for cleaning and classifying dirty data at a dual-gate sentinel-type entrance.

[0007] A dual-gate sentinel-style entry dirty data cleaning and error classification method includes: Acquire offline industrial sensor signal data, detect impact events, extract data of a fixed length before and after the impact event, and form several event windows; Each event window is divided into several segments, and the root mean square of each segment is calculated. A fast Fourier transform is performed on the entire event window to calculate the spectral dispersion. If the minimum root mean square value is less than the production line idling calibration value or the spectral dispersion is greater than the spectral dispersion threshold, the event window is discarded. Calculate the vibration peak value or root mean square value of offline industrial sensor signal data. If the vibration peak value or root mean square value exceeds the first threshold, it is determined to be signal distortion data. Perform consistency judgment on the sampling rate of offline industrial sensor signal data and filter out data with sampling rate errors. Calculate the cross-correlation coefficient and energy difference between the current event window and the previous frame event window. If the cross-correlation coefficient is less than the second threshold and the energy difference is more than K times, it is determined to be packet loss data. Signal distortion data, sampling rate error data, and packet loss data are entered into an independent FIFO queue according to their error codes, quantified, analyzed, and stored.

[0008] Furthermore, offline industrial sensor signal data is acquired to detect impact events; the method includes: acquiring offline industrial sensor signal data and detecting impact events based on short-time energy and peak factor.

[0009] Furthermore, the spectral dispersion is calculated using the following formula:

[0010]

[0011] Where SP represents the spectral dispersion. Indicates the first n Power of each frequency component; Indicates signal x The n Complex representation of each frequency component.

[0012] Furthermore, the spectral dispersion threshold is calculated using the following formula:

[0013] in, Indicates the threshold of spectral dispersion; This indicates the number of sample points used when performing the Fast Fourier Transform.

[0014] Furthermore, the first threshold is the mean of the peak value or RMS under normal working conditions plus a certain number of times the standard deviation.

[0015] Furthermore, the method for performing a consistency judgment on the sampling rate of offline industrial sensor signal data and filtering out data with erroneous sampling rates includes: For modes applicable to frequency analysis, check whether the main peak frequency exceeds half of the sampling rate; if so, determine that the sampling rate is incorrect. For modes that are not suitable for frequency analysis, different methods are used to evaluate the consistency of the sampling rate; these include: for infrared signals, evaluating whether the temporal resolution of the data meets the application requirements and filtering out data with incorrect sampling rates; for video signals, using video quality evaluation metrics to evaluate whether the sampling rate is appropriate and filtering out data with incorrect sampling rates.

[0016] Furthermore, data from independent FIFO queues is used as a standard for detecting device health and as error data to adjust the model; correct data is used for model training.

[0017] Further, quantitative analysis is performed and the data is stored; methods include: Calculate the peak factor of the signal distortion data, the main frequency drift of the sampling rate error data, and the cross-correlation peak of the adjacent windows of the lost packet data; input the peak factor of the signal distortion data, the main frequency drift of the sampling rate error data, and the cross-correlation peak of the adjacent windows of the lost packet data into the model for training and adjust the model parameters. Error data is stored in the local dirty data cache in the form of feature vectors, timestamps, and error codes. The feature vectors include: peak factor of signal distortion data, main frequency drift of sampling rate error data, cross-correlation peak of adjacent windows of lost packet data, and raw offline industrial sensor signal data.

[0018] A second aspect of the present invention provides a dual-gate sentinel-type entrance dirty data cleaning and error classification system.

[0019] A dual-gate sentinel-style entry dirty data cleaning and error classification system includes: The data alignment module is configured to: acquire offline industrial sensor signal data, detect impact events, extract data of a fixed length before and after the impact event, and form several event windows; The front door module is configured to: divide each event window into several segments and calculate the root mean square of each segment; perform a fast Fourier transform on the entire event window and calculate the spectral dispersion; if the minimum root mean square value is less than the production line idling calibration value or the spectral dispersion is greater than the spectral dispersion threshold, then discard the event window. The backdoor module is configured to: calculate the vibration peak value or root mean square value of offline industrial sensor signal data; if the vibration peak value or root mean square value exceeds a first threshold, it is determined to be signal distorted data; perform consistency judgment on the sampling rate of offline industrial sensor signal data and filter out data with incorrect sampling rate; calculate the cross-correlation coefficient and energy difference between the current event window and the previous frame event window; if the cross-correlation coefficient is less than a second threshold and the energy difference is more than K times, it is determined to be packet loss data. The error data utilization module is configured to: input signal distortion data, sampling rate error data, and packet loss data into an independent FIFO queue according to the error code, perform quantitative analysis, and store the data.

[0020] A third aspect of the present invention provides a computer device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the dual-gate sentinel-type entry dirty data cleaning and error classification method as described in the first aspect above.

[0021] Compared with the prior art, the beneficial effects of the present invention are: This invention adds a dual-gate sentinel mechanism to the data entry point: the front gate uses segmented energy and spectral dispersion dual thresholds to promptly remove low signal-to-noise ratio (SNR) segments; the back gate uses a three-layer physical consistency verification tree (value range, spectrum, cross-correlation) to classify and triage signal distortion, sampling rate errors, and device disconnections at the millisecond level, which can be directly embedded into the MCU acquisition firmware. Intercepted disconnection or low SNR samples are automatically labeled and archived for subsequent training of the sensor health monitoring sub-model, enabling the reuse of erroneous samples. This solution achieves a closed loop of training data self-cleaning and error reuse for the first time in a local area network offline environment, improving the F1 score of the fine-tuned model, reducing the false alarm rate of sensor disconnections, and requiring no modification to existing hardware, resulting in zero deployment cost. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a flowchart illustrating the method for cleaning and classifying dirty data at a dual-gate sentinel-type entrance, as shown in an embodiment of the present invention. Figure 2 This is a flowchart of another embodiment of the dual-gate sentinel-type entrance dirty data cleaning and error classification method shown in the embodiments of the present invention; Figure 3 This is a waveform diagram of simulated multimodal industrial vibration signals shown in an embodiment of the present invention; Figure 4 This is a waveform diagram of simulated multimodal industrial current signal shown in an embodiment of the present invention; Figure 5 This is a schematic diagram of a simulated sound wave signal shown in an embodiment of the present invention; Figure 6 This is an embodiment of the present invention shown in Figure 5 A schematic diagram of the window after adding 5dB of noise to the signal; Figure 7 This is a comparison chart of cleaning data and noise data shown in an embodiment of the present invention; Figure 8 This is a schematic diagram illustrating the mixing of dirty data into the training set before cleaning, as shown in an embodiment of the present invention. Figure 9 This is a schematic diagram illustrating a significant reduction in false alarms related to disconnections after cleaning, as shown in an embodiment of the present invention. Figure 10 This is a structural diagram of the dual-gate sentinel-type entrance dirty data cleaning and error classification system shown in an embodiment of the present invention; Figure 11 This is a structural diagram of a computer device shown in an embodiment of the present invention. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0027] To facilitate understanding of the technical solutions of this invention, some technical terms involved in this invention will be introduced below.

[0028] Spectral Purity (SP): A metric that measures the uniformity of energy distribution in the spectrum of a signal. It is commonly used to analyze the spectral characteristics of a signal, especially in signal processing and communications.

[0029] Signal distortion: This refers to errors in sensor data that are affected at a certain frame or moment, or common high-frequency information distortion.

[0030] Sampling rate error: Different sampling rates should be set for different devices. This situation will occur if the sampling rate is set incorrectly.

[0031] Packet loss: Packet loss caused by network transmission issues.

[0032] As introduced in the background technology, for PHM systems deployed offline within a local area network, it is necessary to use on-site multimodal sensor data to fine-tune lightweight models online to adapt to specific equipment operating conditions. However, the quality of the generated training data directly affects the fine-tuning effect. Currently, there are two main problems: (1) Low signal-to-noise ratio pollution: On-site electromagnetic interference, equipment resonance and other factors cause some modes to have extremely low signal-to-noise ratios, and the signals are submerged by noise. If such segments are not cleaned before entering the training set, they will lower the overall feature distribution, increase the training difficulty and cause overfitting, and ultimately weaken the model's ability to identify real fault features. (2) Modal disconnection misleading: Sensors often go offline or communication is interrupted in industrial sites, resulting in the complete loss of a certain modality data such as vibration, current, infrared. Although the correct judgment can be given through other modalities during the inference stage, if these samples with missing modalities but correct labels are directly used for fine-tuning, it will significantly reduce the model's sensitivity to real disconnection scenarios, causing the misjudgment rate to increase when actual disconnection occurs later.

[0033] To address at least one of the aforementioned technical problems, this invention provides a dual-gate sentinel-style entry dirty data cleaning and error classification method and system. It employs a training data self-cleaning mechanism that can operate at the edge, automatically identifying and removing the two types of dirty data before fine-tuning, ensuring data reliability and model robustness during the fine-tuning phase of the offline PHM system. The following describes the solution in detail through several embodiments.

[0034] Figure 1 This is a flowchart illustrating the dual-gate sentinel-type inlet dirty data cleaning and error classification method according to an embodiment of the present invention; see reference. Figure 1 The method includes: Acquire offline industrial sensor signal data, detect impact events, extract data of a fixed length before and after the impact event, and form several event windows; Each event window is divided into several segments, and the root mean square of each segment is calculated. A fast Fourier transform is performed on the entire event window to calculate the spectral dispersion. If the minimum root mean square value is less than the production line idling calibration value or the spectral dispersion is greater than the spectral dispersion threshold, the event window is discarded. Calculate the vibration peak value or root mean square value of offline industrial sensor signal data. If the vibration peak value or root mean square value exceeds the first threshold, it is determined to be signal distortion data. Perform consistency judgment on the sampling rate of offline industrial sensor signal data and filter out data with sampling rate errors. Calculate the cross-correlation coefficient and energy difference between the current event window and the previous frame event window. If the cross-correlation coefficient is less than the second threshold and the energy difference is more than K times, it is determined to be packet loss data. Signal distortion data, sampling rate error data, and packet loss data are entered into an independent FIFO queue according to their error codes, quantified, analyzed, and stored.

[0035] This invention adds a dual-gate sentinel mechanism to the data entry point: the front gate uses segmented energy and spectral dispersion dual thresholds to promptly remove low signal-to-noise ratio (SNR) segments; the back gate uses a three-layer physical consistency verification tree (value range, spectrum, cross-correlation) to classify and triage signal distortion, sampling rate errors, and device disconnections at the millisecond level, which can be directly embedded into the MCU acquisition firmware. Intercepted disconnection or low SNR samples are automatically labeled and archived for subsequent training of the sensor health monitoring sub-model, enabling the reuse of erroneous samples. This solution achieves a closed loop of training data self-cleaning and error reuse for the first time in a local area network offline environment, improving the F1 score of the fine-tuned model, reducing the false alarm rate of sensor disconnections, and requiring no modification to existing hardware, resulting in zero deployment cost.

[0036] Figure 2 This is a flowchart of another embodiment of the dual-gate sentinel-type entrance dirty data cleaning and error classification method shown in this invention; see also... Figure 2 The dual-gate sentinel-style entrance dirty data cleaning and error classification method includes the following steps: Step 1: Data alignment.

[0037] The sampling rates of the various sensors on site vary greatly (vibration 25 kHz, current 1 kHz, infrared 10 Hz, video 60 Hz). In this embodiment, the vibration clock synchronized with IEEE-1588 is used as the main axis. The UTC timestamps of other modes are converted into vibration indices. An event anchoring mechanism is adopted—using short-time energy and peak factor as dual thresholds to detect impact events in the vibration channel and generate a unified event number Event_ID. Then, with Event_ID as the center, fixed lengths are taken before and after to form an event window, ensuring that all subsequent cleaning and verification are completed within the same physical time period, and eliminating false positive dirty data introduced by time misalignment. Figure 3 This is a waveform diagram of simulated multimodal industrial vibration signals shown in an embodiment of the present invention; Figure 4 This is a waveform diagram of a simulated multimodal industrial current signal shown in an embodiment of the present invention; as follows: Figure 3 , Figure 4 As shown, the analysis of vibration signals and industrial current signals is very important for mechanical health monitoring, as it can predict mechanical failures and assess the operating status of equipment.

[0038] Step 2: Front door, i.e., signal-to-noise ratio gate.

[0039] Once the event window is in memory, a lightweight signal-to-noise ratio evaluation is performed immediately.

[0040] (1) Divide the window into four equal segments and calculate the root mean square of each segment:

[0041] in, Indicates the first in the window i One sampling point; This indicates the number of sampling points within a single window; the total number of sampling points within the event window is 4. ; Indicates the first k The root mean square value of a segment signal reflects the overall energy of that segment.

[0042] This implementation uses the root mean square (RMS) to describe the amplitude characteristics of the signal. The RMS can provide the effective value or equivalent DC value of the signal, which in many cases is more representative of the average power of the signal than the peak value.

[0043] (2) Perform a 128-point Fast Fourier Transform (FFT) on the entire window and calculate the spectral dispersion SP:

[0044]

[0045] in, Indicates the first n Power of each frequency component; Indicates signal x The n SP represents the complex number representation of a frequency component; SP represents the spectral dispersion, which measures the uniformity of the signal spectrum; an SP value close to 1 indicates that the spectral energy distribution of the signal is very uniform, close to the characteristics of white noise; white noise is an idealized random signal whose spectrum has the same power at all frequencies; an SP value far from 1 indicates that the spectral energy distribution of the signal is not uniform, and the power of some frequency components is much greater than that of other components, which usually means that there may be specific frequency components or noise in the signal.

[0046] Figure 5 This is a schematic diagram of an analog sound wave signal shown in an embodiment of the present invention; as follows: Figure 5 As shown, a simulated acoustic signal has clear spectral characteristics in the absence of noise interference. Figure 6 This is an embodiment of the present invention shown in Figure 5 A schematic diagram of the window after adding 5dB of noise to the signal; as shown. Figure 6 As shown, in Figure 5 The signal was augmented with 5dB of noise to simulate electromagnetic interference or equipment resonance that may occur in real industrial environments. Figure 7 This is a comparison chart of cleaning data and noise data shown in an embodiment of the present invention; as follows: Figure 7 As shown, the spectral dispersion (SP) of the signal decreases significantly after adding noise. Spectral dispersion is an indicator of the uniformity of the signal's spectrum; the closer the SP value is to 1, the closer the signal is to white noise. In this example, the increase in noise causes a decrease in the SP value, indicating that the signal's spectral characteristics are overwhelmed by noise, thus reducing the signal quality.

[0047] (3) If Below the production line idling calibration value or If a window is deemed to have a low signal-to-noise ratio, it is immediately discarded, not written to the training cache, and not involved in any subsequent calculations.

[0048] in, The mean of the root mean square of M ≥ 1000 windows is continuously collected when the production line is idle. ,according to Settings; SP threshold only applies to 128-point real-value FFT, press The calculated value is 0.85. If the FFT points change, the value will be adjusted proportionally. This indicates the number of sample points used when performing the Fast Fourier Transform (FFT), which converts a time-domain signal into a frequency-domain signal. This determines the resolution of the converted frequency domain signal.

[0049] In this embodiment Taking 0.85 is just one case; in actual use, different values ​​will be used for different devices. .

[0050] This embodiment will Below the production line idling calibration value or Data is cleaned up as a front-door judgment condition. The value is lower than the production line idling calibration value. If the device fails to start or the data collection service goes offline, the energy level will be very low. It can be determined; This indicates a very low signal-to-noise ratio. The low signal-to-noise ratio may be caused by the sensor not being installed properly or by electrical problems leading to sensor acquisition errors. These situations can be directly filtered out.

[0051] This step uses only IIR low-pass, sliding window RMS, and small-point FFT, with minimal computation and RAM usage, yet it can remove common high-noise segments such as broadband electromagnetic interference, resonance flooding, and quantization noise.

[0052] Step 3: Backdoor, perform physical consistency verification of the three-layer hard rule tree.

[0053] (1) L1 range gate - vibration peak or RMS exceeding the first threshold is considered signal distortion.

[0054] The first threshold is determined based on statistical analysis under normal operating conditions. The first threshold is typically set as the mean of the peak value or RMS under normal operating conditions plus a certain number of standard deviations. For example, it can be set as the mean plus three times the standard deviation, i.e.:

[0055] in, Indicates the first threshold; μ This represents the peak value or the mean of the RMS. This represents the corresponding standard deviation. Setting the first threshold ensures that signals under normal operating conditions are not misinterpreted as distorted. If the calculated peak value or RMS exceeds the first threshold, the signal is considered distorted.

[0056] (2) L2 Spectrum Gate - Sampling Rate Consistency Test.

[0057] The L2 spectrum gate is designed to verify whether the sampling rate of multimodal sensor data is appropriate to prevent information loss or aliasing. This test is based on the Nyquist sampling theorem, which states that the sampling rate should be at least twice the highest frequency component of the signal.

[0058] For modes such as vibration and sound signals that can be analyzed by frequency, check whether the peak frequency exceeds half the sampling rate or is below 1 Hz. If the peak frequency exceeds half the sampling rate, it may cause signal distortion; if the peak frequency is below 1 Hz, it may indicate that the sampling rate is set too high, affecting the efficiency and accuracy of data acquisition.

[0059] For modes such as infrared and video signals, which are not directly applicable to frequency analysis, different methods are used to evaluate the consistency of the sampling rate. For infrared signals, the temporal resolution of the data is evaluated to see if it meets application requirements, such as whether it can capture rapidly changing temperatures or reflectivity. For video signals, video quality evaluation metrics, such as PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity), or VMAF (Video Multimodal Evaluation Fusion), are used to evaluate whether the sampling rate is appropriate.

[0060] In this way, the L2 spectrum gate provides a flexible sampling rate consistency check for different types of multimodal signals, ensuring that the sampled data can accurately reflect the characteristics of the original signal, thereby improving the quality and reliability of data analysis.

[0061] (3) L3 cross-correlation gate - cross-correlates with the previous frame window, if ρ If the energy difference exceeds the second threshold by more than K times, the device is determined to be offline or experiencing packet loss.

[0062] First, calculate the cross-correlation coefficient between the current window and the previous frame window. ρ This is used to assess the similarity between two consecutive data windows. The formula for calculating the cross-correlation coefficient is as follows:

[0063] in, and These are the sampled point values ​​of the current window and the previous frame window, respectively. N This is the number of sampling points within the window. Cross-correlation coefficient. ρ The value ranges between -1 and 1. The closer the value is to ±1, the more similar the two windows are. The closer the value is to 0, the lower the similarity.

[0064] Secondly, the energy difference between the current window and the previous frame window is compared to detect potential data loss or device malfunction. The formula for calculating the energy difference is as follows:

[0065] in, This represents the energy difference, used to measure the energy change between two consecutive windows. If ΔE is much greater than 1, it may indicate that data was lost in the previous window or that the data in the current window is abnormal.

[0066] Combining cross-correlation coefficients ρ and energy difference If the setting is ρ <Second threshold and Δ E If the similarity between two consecutive windows is low (the second threshold can be 0.2) and the energy difference is large (ΔE exceeds K times, where K can be 3), then the data is considered to have been lost due to equipment failure or communication problems.

[0067] If any level fails, the corresponding error code is immediately assigned (1-signal distortion, 2-sampling rate error, 3-packet loss), and the entire window is moved into the dirty data bucket. The bucket retains the original label and error code to ensure traceability and reusability. All three levels of rules involve algebraic operations and do not involve floating-point matrices.

[0068] In this embodiment, the front door cleans up data with low signal-to-noise ratio, and the back door performs physical rule verification on the data and marks erroneous data.

[0069] This invention relies solely on sliding window statistics, FFT, and 3-σ thresholding, without deep learning. It requires less than 512 bytes of RAM and less than 0.3 ms of CPU time, and can be directly embedded into a microcontroller or edge gateway without additional hardware.

[0070] Step 4: Reuse of erroneous samples.

[0071] Steps 1-3 have achieved data alignment, low signal-to-noise ratio data cleaning, and error data labeling. Now, data classification is needed, dividing it into correct data and dirty data (error data). Correct data is used for model training, while error data can be used for device health detection standards or as error data to adjust the model. For example, if a sensor of a certain modality goes offline, the model can automatically mark, delete, or even repair it. This embodiment mainly focuses on data cleaning and labeling, classifying the collected data with minimal performance.

[0072] The three types of dirty data—distorted signals, incorrect sampling rates, and lost packets—that are blocked at the back door are no longer discarded directly, but instead enter an independent FIFO queue according to their error codes.

[0073] During the feature calculation stage, different types of data errors are quantitatively analyzed, and the model parameters are fine-tuned based on the analysis results.

[0074] (1) For signal distortion, calculate the peak factor:

[0075] in, It represents the maximum absolute value of the signal, while RMS represents the root mean square value.

[0076] (2) For sampling rate errors, statistical main frequency drift is calculated:

[0077] in, Indicates the actual main frequency; Indicates the nominal frequency.

[0078] (3) In the case of data packet loss, record the peak value of cross-correlation between adjacent windows. ρ .

[0079] The local dirty data buffer is designed as an efficient data structure for storing and aligning erroneous data records from multiple modal sensors. This buffer employs a 3×1000-row circular FIFO (First-In-First-Out) structure to ensure ordered data management and fast access. Each record includes: a feature vector, a timestamp, and an error code.

[0080] The construction of feature vectors is a core component of this invention, integrating key error features extracted from data from multiple modal sensors. These features not only encompass statistical indicators such as peak factor, dominant frequency drift, and cross-correlation peaks, but also include raw waveform data or key segments thereof. This design aims to provide rich information for the model training phase, enabling it to accurately identify and learn normal and abnormal patterns in different modal data.

[0081] Each erroneous data record is accompanied by a timestamp to precisely record when the error occurred. This timestamp ensures the time alignment of the data, allowing for analysis of data errors over time and thus a better understanding of the context in which the error occurred.

[0082] Each record also contains an error code to identify the error type, such as signal distortion, sampling rate error, or data loss. Error codes provide a quick way to identify and classify errors, helping to simplify the error handling process.

[0083] This approach ensures that error information from multimodal data is effectively captured and stored, supporting subsequent error analysis and diagnosis. This buffer design not only improves data processing efficiency but also provides a solid foundation for the quality management of multimodal data.

[0084] For anomaly detection and alarms, if the count of any type of error suddenly increases by more than 3 times the standard deviation (based on statistics from the past 24 hours) within the sliding window, the MCU will immediately output a sensor health anomaly signal and upload the error code and start and end times via communication protocols such as Modbus-TCP, ensuring a delay of less than 1 second. This step ensures that potential sensor problems can be detected and responded to in a timely manner.

[0085] This invention enables the reuse of erroneous results. Offline or low signal-to-noise ratio samples that are rejected are automatically labeled and archived, and subsequently used to train a dedicated sensor failure detection sub-model, turning waste into treasure and maximizing the value of the data.

[0086] Through the above four steps, this invention completes the entire chain of self-cleaning—"alignment-SNR gate-physical gate-reuse"—at the edge entry point, thereby improving the purity of the offline PHM fine-tuning training set, reducing the false alarm rate of sensor disconnection, and achieving zero depth, zero external network, and zero hardware modification throughout the entire process. This provides a reliable data foundation for lightweight fine-tuning of large industrial models in a data physically isolated environment.

[0087] Figure 8 This is a schematic diagram illustrating the mixing of dirty data into the training set before cleaning, as shown in an embodiment of the present invention; Figure 8As shown, the classification accuracy of different fault types in the dataset before data cleaning is as follows. Due to the influence of dirty data (such as noise, sensor disconnection, etc.), the classification accuracy of some fault types is low; for example, the accuracy of sensor disconnection is only 65%. This indicates that dirty data significantly reduces the quality of the dataset. Figure 9 This is a schematic diagram illustrating a significant reduction in false alarms related to dropped connections after cleaning, as shown in an embodiment of the present invention; Figure 9 As shown, after processing with the data cleaning method described in this invention, the classification accuracy of the dataset was significantly improved. In particular, the classification accuracy for sensor disconnection increased to 93%, and the classification accuracy for other fault types also improved. This indicates that data cleaning effectively removed dirty data and significantly improved the overall quality of the dataset.

[0088] The above combination Figure 1 The dual-gate sentinel-type entrance dirty data cleaning and error classification method provided in the embodiments of the present invention has been described in detail. Next, the dual-gate sentinel-type entrance dirty data cleaning and error classification system provided in the embodiments of the present invention will be described in conjunction with the accompanying drawings.

[0089] Figure 10 This is a schematic diagram of the structure of a dual-gate sentinel-type entrance dirty data cleaning and error classification system according to an embodiment of the present invention, with reference to... Figure 10 The system described in this invention includes: The data alignment module is configured to: acquire offline industrial sensor signal data, detect impact events, extract data of a fixed length before and after the impact event, and form several event windows; The front door module is configured to: divide each event window into several segments and calculate the root mean square of each segment; perform a fast Fourier transform on the entire event window and calculate the spectral dispersion; if the minimum root mean square value is less than the production line idling calibration value or the spectral dispersion is greater than the spectral dispersion threshold, then discard the event window. The backdoor module is configured to: calculate the vibration peak value or root mean square value of offline industrial sensor signal data; if the vibration peak value or root mean square value exceeds a first threshold, it is determined to be signal distorted data; perform consistency judgment on the sampling rate of offline industrial sensor signal data and filter out data with incorrect sampling rate; calculate the cross-correlation coefficient and energy difference between the current event window and the previous frame event window; if the cross-correlation coefficient is less than a second threshold and the energy difference is more than K times, it is determined to be packet loss data. The error data utilization module is configured to: input signal distortion data, sampling rate error data, and packet loss data into an independent FIFO queue according to the error code, perform quantitative analysis, and store the data.

[0090] In some embodiments, acquiring offline industrial sensor signal data and detecting impact events includes: acquiring offline industrial sensor signal data and detecting impact events based on short-time energy and peak factor.

[0091] In some embodiments, the spectral dispersion is calculated using the following formula:

[0092]

[0093] Where SP represents the spectral dispersion. Indicates the first n Power of each frequency component; Indicates signal x The n Complex representation of each frequency component.

[0094] In some embodiments, the spectral dispersion threshold is calculated using the following formula:

[0095] in, Indicates the threshold of spectral dispersion; This indicates the number of sample points used when performing the Fast Fourier Transform.

[0096] In some embodiments, the first threshold is the mean of the peak value or RMS under normal operating conditions plus a certain number of times the standard deviation.

[0097] In some embodiments, the process of determining the consistency of the sampling rate of offline industrial sensor signal data and filtering out data with erroneous sampling rates includes: for modes applicable to frequency analysis, detecting whether the main peak frequency exceeds half of the sampling rate; if so, determining that the data with erroneous sampling rates is incorrect; for modes not applicable to frequency analysis, using different methods to evaluate the consistency of the sampling rate includes: for infrared signals, evaluating whether the temporal resolution of the data meets the application requirements and filtering out data with erroneous sampling rates; for video signals, using video quality evaluation metrics to evaluate whether the sampling rate is appropriate and filtering out data with erroneous sampling rates.

[0098] In some embodiments, data in independent FIFO queues are used as a criterion for detecting device health and as error data to adjust the model; correct data is used for model training.

[0099] In some embodiments, quantitative analysis is performed and stored, including: calculating the peak factor of the signal distortion data, calculating the main frequency drift of the sampling rate error data, and calculating the cross-correlation peak of the adjacent windows of the lost packet data; inputting the peak factor of the signal distortion data, the main frequency drift of the sampling rate error data, and the cross-correlation peak of the adjacent windows of the lost packet data into the model for training, and adjusting the model parameters; storing the error data in the structure of feature vector, timestamp, and error code in the local dirty data cache; wherein, the feature vector includes: the peak factor of the signal distortion data, the main frequency drift of the sampling rate error data, the cross-correlation peak of the adjacent windows of the lost packet data, and the original offline industrial sensor signal data.

[0100] According to embodiments of the present invention, the dual-gate sentinel-type inlet dirty data cleaning and error classification system can correspond to the execution of the methods described in the embodiments of the present invention, and the above and other operations and / or functions of each module of the dual-gate sentinel-type inlet dirty data cleaning and error classification system are respectively for implementing Figure 1 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0101] See Figure 11 The diagram shows the structure of a computer device, which includes a processor, a communication interface, and a computer-readable storage medium. The processor, communication interface, and computer-readable storage medium are connected via a bus or other means. The communication interface is used to receive and send data. The computer-readable storage medium can be stored in the computer device's memory. The computer-readable storage medium stores computer programs, including program instructions, and the processor executes the program instructions stored in the computer-readable storage medium. The processor (or CPU, Central Processing Unit) is the computing and control core of the computer device, adapted to implement one or more instructions, specifically adapted to load and execute one or more instructions to implement the corresponding steps in the dual-gate sentinel-style entry dirty data cleaning and error classification method embodiment.

[0102] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0103] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0105] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0106] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0107] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for cleaning and classifying dirty data at a dual-gate sentinel-type entrance, characterized in that, include: Acquire offline industrial sensor signal data, detect impact events, extract data of a fixed length before and after the impact event, and form several event windows; Each event window is divided into several segments, and the root mean square of each segment is calculated. A fast Fourier transform is performed on the entire event window to calculate the spectral dispersion. If the minimum root mean square value is less than the production line idling calibration value or the spectral dispersion is greater than the spectral dispersion threshold, the event window is discarded. Calculate the vibration peak value or root mean square value of offline industrial sensor signal data. If the vibration peak value or root mean square value exceeds the first threshold, it is determined to be signal distorted data. Perform a consistency judgment on the sampling rate of offline industrial sensor signal data and filter out data with incorrect sampling rate. Calculate the cross-correlation coefficient and energy difference between the current event window and the previous frame event window. If the cross-correlation coefficient is less than the second threshold and the energy difference is more than K times, it is determined to be lost data. Signal distortion data, sampling rate error data, and packet loss data are entered into an independent FIFO queue according to their error codes, quantified, analyzed, and stored.

2. The method for cleaning and classifying dirty data at a dual-gate sentinel-type entrance according to claim 1, characterized in that, Acquire offline industrial sensor signal data to detect impact events; the method includes: acquiring offline industrial sensor signal data and detecting impact events based on short-time energy and peak factor.

3. The method for cleaning and classifying dirty data at a dual-gate sentinel-type entrance as described in claim 1, characterized in that, The spectral dispersion is calculated using the following formula: Where SP represents the spectral dispersion. Indicates the first n Power of each frequency component; Indicates signal x The n Complex representation of each frequency component.

4. The method for cleaning and classifying dirty data at a dual-gate sentinel-type entrance according to claim 1, characterized in that, The spectral dispersion threshold is calculated using the following formula: in, Indicates the threshold of spectral dispersion; This indicates the number of sample points used when performing the Fast Fourier Transform.

5. The method for cleaning and classifying dirty data at a dual-gate sentinel-type entrance according to claim 1, characterized in that, The first threshold is the mean of the peak value or RMS under normal working conditions plus a certain number of standard deviations.

6. The method for cleaning and classifying dirty data at a dual-gate sentinel-type entrance according to claim 1, characterized in that, The method for determining the consistency of the sampling rate of offline industrial sensor signal data and filtering out data with incorrect sampling rates includes: For modes applicable to frequency analysis, check whether the main peak frequency exceeds half of the sampling rate; if so, determine that the sampling rate is incorrect. For modes that are not suitable for frequency analysis, different methods are used to evaluate the consistency of the sampling rate; these include: for infrared signals, evaluating whether the temporal resolution of the data meets the application requirements and filtering out data with incorrect sampling rates; for video signals, using video quality evaluation metrics to evaluate whether the sampling rate is appropriate and filtering out data with incorrect sampling rates.

7. The method for cleaning and classifying dirty data at a dual-gate sentinel-type entrance according to claim 1, characterized in that, Data from independent FIFO queues is used as a standard for detecting device health and as error data to adjust the model; correct data is used for model training.

8. The method for cleaning and classifying dirty data at a dual-gate sentinel-type entrance according to claim 1, characterized in that, Perform quantitative analysis and store the data; methods include: Calculate the peak factor of the signal distortion data, the main frequency drift of the sampling rate error data, and the cross-correlation peak of the adjacent windows of the lost packet data; input the peak factor of the signal distortion data, the main frequency drift of the sampling rate error data, and the cross-correlation peak of the adjacent windows of the lost packet data into the model for training and adjust the model parameters. Error data is stored in the local dirty data cache in the form of feature vectors, timestamps, and error codes. The feature vectors include: peak factor of signal distortion data, main frequency drift of sampling rate error data, cross-correlation peak of adjacent windows of lost packet data, and raw offline industrial sensor signal data.

9. A dual-gate sentinel-type entrance dirty data cleaning and error classification system, characterized in that, include: The data alignment module is configured to: acquire offline industrial sensor signal data, detect impact events, extract data of a fixed length before and after the impact event, and form several event windows; The front door module is configured to: divide each event window into several segments and calculate the root mean square of each segment; perform a fast Fourier transform on the entire event window and calculate the spectral dispersion; if the minimum root mean square value is less than the production line idling calibration value or the spectral dispersion is greater than the spectral dispersion threshold, then discard the event window. The backdoor module is configured to: calculate the vibration peak value or root mean square value of offline industrial sensor signal data; if the vibration peak value or root mean square value exceeds a first threshold, it is determined to be signal distorted data; and perform consistency judgment on the sampling rate of offline industrial sensor signal data to filter out data with incorrect sampling rate. Calculate the cross-correlation coefficient and energy difference between the current event window and the previous frame event window. If the cross-correlation coefficient is less than the second threshold and the energy difference is more than K times, it is determined to be lost data. The error data utilization module is configured to: input signal distortion data, sampling rate error data, and packet loss data into an independent FIFO queue according to the error code, perform quantitative analysis, and store the data.

10. A computer device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps of the dual-gate sentinel-type entry dirty data cleaning and error classification method as described in any one of claims 1-8.

Images