A liquid level meter production data processing system based on data cleaning

Abstract

The present application relates to liquid level meter production and industrial data processing field, specifically to a kind of liquid level meter production data processing system based on data cleaning, comprising: data interface module and data normalization module, utilize reference mean and scaling coefficient to carry out normalization to effective data;Time synchronization module, calculate unified time, form single echo data set and calculate completeness rate;Waveform inspection module: analyze waveform characteristics, calculate equipment and environmental deviation and process compliance rate;Data cleaning module, calculate deviation score and data efficiency, output retention, delete or repair instruction according to threshold value;Result output and update module, output cleaning result and index, update parameter and standard value according to inspection test record;The present application can realize the unified access of multi-source data, associated arrangement and cleaning determination, avoid disorder data, partial packet loss and interference waveform directly into subsequent quality judgment process.

Description

Technical Field

[0001] This invention relates to the field of level gauge production and industrial data processing, specifically to a level gauge production data processing system based on data cleaning. Background Technology

[0002] The production process of level gauges usually requires the simultaneous collection of multi-source data such as echo waveforms, production environment, equipment status, and process parameters. Based on the relevant data, support is provided for subsequent calibration tests and quality inspections. Therefore, the production data processing results directly affect the accuracy of production judgment and the stability of testing of level gauges. As the automation level of level gauge production lines increases, the sources of data on the production site gradually increase. There are differences between different data in terms of communication protocols, time recording methods, fluctuation scales, and process correlations. Under the existing processing methods, various types of data are often received independently or simply spliced ​​based on a fixed time window, and abnormal data is removed by static thresholds. This can easily lead to problems such as misaligned data correlations, missing contextual information, accidental deletion of real waveform features, and difficulty in adjusting judgment criteria with changes in production status. As a result, the data cleaning effect and subsequent quality judgment results are difficult to guarantee. Summary of the Invention

[0003] The purpose of this invention is to provide a liquid level gauge production data processing system based on data cleaning, and to solve the following technical problems: This avoids problems that are prone to occur under existing processing methods, such as misaligned data associations, missing contextual information, accidental deletion of real waveform features, and difficulty in adjusting judgment criteria as production status changes. It also makes it easier to achieve unified access, association, organization, and cleaning judgment of multi-source data during the liquid level gauge production process, preventing out-of-order data, partial missing data, and interference waveforms from directly entering the subsequent quality judgment process.

[0004] The objective of this invention can be achieved through the following technical solutions: The data interface module receives process parameters containing current values, reference values, and allowable error ranges, as well as echo waveforms, production environment, and equipment status data, and converts them into raw datasets in a unified format. The data normalization module calls the baseline mean and scaling factor to effectively normalize the original dataset, resulting in a unified format dataset. The time synchronization module calculates a unified time based on the transmission and reception time, calibration value, and delay compensation, and generates a time sequence table according to the maximum delay and process correspondence table; it extracts corresponding data based on single echo waveform data to build a single echo dataset, and calculates the data integrity rate based on the expected total number of associated data. The waveform inspection module performs frequency analysis and reference comparison on waveform data to obtain the main echo amplitude ratio, echo width and waveform matching degree; and calculates the operating deviation, environmental change and process compliance rate according to equipment status, environment and process parameters. The data cleaning module calculates data deviation scores and data effectiveness based on preset indicator weights; it sets qualified standard values, deletion standard values, and a minimum data collection quantity as a lower limit, where the deletion standard value is less than the qualified standard value, and outputs instructions to retain, delete, or repair accordingly. The results output and update module outputs the cleaning signal sequence, timestamp, single dataset number, processing label and score, data effectiveness rate, and completeness rate; based on the test records containing calibration deviation, the inspection records containing pass marks, and the sampling records, it updates the normalization parameters, time synchronization parameters, reference waveform, weight, standard value, expected total number of associated data, and minimum data acquisition amount.

[0005] Furthermore, the data interface module is used for: Receive echo waveform data stream, production environment data stream, equipment status data stream, and process parameter data stream; Extract the data number, receiving time, generation time, valid data, and data quality markers for each data item; Among them, the valid data for echo waveform data, production environment data and process parameter data are measured values, and the valid data for equipment status data are status values. Among them, the production environment data includes at least temperature, pressure and humidity, and the equipment status data includes at least equipment current, power supply voltage and equipment start-up and shutdown status; The original dataset is formed by combining the data number, receipt time, generation time, valid data, and data quality tags.

[0006] Furthermore, the data normalization module is used for: Read valid data from the original dataset; Retrieve the corresponding baseline mean and scaling factor according to the data source category; The effective data is normalized based on the baseline mean and scaling factor; The calculation results are combined with the corresponding data number, receiving time, generation time and data quality marker to form a unified format dataset.

[0007] Furthermore, the time synchronization module is used for: Read the reception time, generation time, and data source category from a uniform format dataset; Determine time calibration values ​​and communication delay compensation for various data sources; Calculate the unified time of each data based on the reception time, generation time, time calibration value, and communication delay compensation; Data pairs with time differences within the set allowable range are filtered based on the preset maximum delay, and data pairs with sequential relationships are filtered based on the production process correspondence table. Generate a time sequence table based on the unified time sequence and the sequence of processes; Based on the waveform data of each echo, the associated data before and after the echo is generated are extracted to form a single echo dataset. The data integrity rate is calculated based on the number of data points within a single echo dataset and the expected total number of associated data points.

[0008] Furthermore, the waveform inspection module is used for: Read echo waveform data from a single echo dataset; The echo waveform data is divided into frequency bands, and the amplitude ratio and echo width of the main echo are calculated based on the energy values ​​of each frequency band. The echo waveform data is compared with a preset reference waveform to calculate the waveform matching degree; Read the device status data from the single echo dataset and calculate the operating deviation based on the degree of deviation from the normal operating state; Read the production environment data from the single echo dataset and calculate the environmental change based on the difference from the standard environmental parameters; Read the process parameter data from the single echo dataset, and calculate the process compliance rate based on the relationship between the degree of deviation of the current value from the reference value and the allowable error range.

[0009] Furthermore, the data cleaning module is used for: Received main echo amplitude ratio, echo width, waveform matching degree, operating deviation, environmental change, process compliance rate, and data integrity rate; Based on the preset indicator weights, calculate the data deviation score and data effectiveness rate; The deletion standard value, the pass standard value, and the minimum data collection quantity are dynamically set based on the amount of operational deviation, environmental changes, process compliance rate, and data integrity rate. The deletion standard value is less than the pass standard value, and the minimum data collection quantity is determined based on the expected total number of related data and the preset proportional coefficient. When the data integrity rate is lower than the minimum data collection amount, the corresponding single echo dataset is identified as data to be repaired. When the data integrity rate is not lower than the minimum data collection amount, the data deviation score is not lower than the deletion standard value, and the data effectiveness rate is lower than the qualified standard value, the corresponding single echo dataset will be judged as contaminated data. When the data integrity rate is not lower than the minimum data collection amount, the data effectiveness rate is not lower than the qualified standard value, and the data deviation score is lower than the deletion standard value, the corresponding single echo dataset will be judged as normal test data. In other cases, the corresponding single echo dataset is identified as data to be repaired.

[0010] Furthermore, the data cleaning module is used to: Extract the original echo waveform data corresponding to the data to be repaired; Based on the waveform distortion points of the original echo waveform data and the filtered data, waveform points are filled in to generate a preliminary repaired waveform; The preliminary repair waveform is tested based on the preset amplitude change limit, smoothness lower limit and matching degree lower limit to obtain the final repaired echo waveform data; The final repaired echo waveform data replaces the original data to be repaired, and the cleaning result containing the repair instructions is output.

[0011] Furthermore, the results output and update module is used for: Receive the cleaning results for retention, deletion, and repair; The cleaned echo signal sequence, timestamp, single echo dataset number, data deviation score, data effectiveness rate, data integrity rate, and processing mark are saved accordingly and output to the subsequent calibration test and quality inspection stages. Receive subsequent calibration test results, factory inspection results, and manual sampling records; Update the normalization parameters and reference waveforms based on the product data that has been confirmed as qualified by the factory inspection. Update time synchronization parameters based on delay statistics from various data sources; Based on the erroneous deleted records, erroneous retained records, corrected misjudged records, and matching error records in the manual sampling records, update the indicator weights, standard values, expected total number of related data, and minimum data collection volume; The updated parameters are sent to the data normalization module, time synchronization module, waveform inspection module, and data cleaning module, respectively.

[0012] Furthermore, the result output and update module is used when updating parameters to: Update the reference waveform based on the echo waveform data that passed the factory inspection in the most recent production batch and the historical template; The total number of misjudgments and the total number of repairs were calculated based on manual sampling records. The parameter modification amount for the current batch is calculated based on the total number of misjudgments and the total number of repairs. The parameter modification amount for different parameters is either a numerical increase or decrease or an adjustment ratio coefficient. The parameter modification amount of the current batch is added to the historical parameters according to their weights to obtain the updated standard value, time calibration value, communication delay compensation, expected total number of associated data and minimum data collection amount.

[0013] The beneficial effects of this invention are: 1. This invention establishes a unified original data structure around the liquid level gauge production process by uniformly receiving and converting echo waveform data, production environment data, equipment status data, and process parameter data. It realizes the centralized organization of data with different communication protocols and recording methods, and can integrate the originally scattered and heterogeneous production data into a unified data foundation, thereby enhancing the consistency of multi-source data access and avoiding data field loss, misalignment of associations, or distortion of quality marks caused by inconsistent interface formats. 2. This invention normalizes valid data by calling the benchmark mean and scaling factor according to the data source category, and maps data with different dimensions and fluctuation scales such as temperature, pressure, current, echo characteristics and process parameters to a unified comparable interval. It establishes a clear basis for joint calculation of multi-source data, and provides a unified scale basis for subsequent waveform inspection and cleaning judgment, thereby effectively solving the problem that it is difficult to integrate multiple indicators and the joint judgment is prone to distortion in the prior art. 3. This invention performs unified time calculation on the receiving time, generation time, time calibration value and communication delay compensation, and generates a time sequence table by combining the maximum delay and the correspondence between production processes. This enables the time sequence reorganization of out-of-order data or data with transmission jitter, avoiding the misassociations and context mismatches caused by simple splicing of traditional fixed time windows, and improving the accuracy and process rationality of single echo dataset construction. 4. This invention extracts environmental data, equipment status data, and process parameter data related to each echo waveform and further calculates the data integrity rate, ensuring that each echo is analyzed within a data set with contextual constraints. This effectively limits aggressive judgments in cases of local missing packets or insufficient correlation, and enhances the robustness of subsequent quality judgments. Attached Figure Description

[0014] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of a liquid level gauge production data processing system based on data cleaning, as described in an embodiment of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] This embodiment provides a liquid level gauge production data processing system based on data cleaning. Its main processing line can be summarized as follows: First, receive echo waveform data, production environment data, equipment status data and process parameter data during the liquid level gauge production process, and organize data from different sources and different protocols into a unified format; then, normalize the effective data and complete time synchronization, so that various types of data establish a corresponding relationship around a single echo waveform. Then, a joint inspection is performed on the echo waveform, equipment status, production environment, and process parameters, and cleaning instructions are output to retain, delete, or repair the data. The cleaning results are then output to the subsequent calibration and quality inspection stages, and the relevant parameters are updated using the subsequent feedback results. Through the above processing, unified access, correlation, and cleaning judgment of multi-source data can be achieved during the production of level gauges, avoiding out-of-order data, partial missing data, and interference waveforms from directly entering the subsequent quality judgment process.

[0017] Please see Figure 1 A liquid level gauge production data processing system based on data cleaning, comprising: The data interface module receives process parameters containing current values, reference values, and allowable error ranges, as well as echo waveforms, production environment, and equipment status data, and converts them into raw datasets in a unified format. The data normalization module calls the baseline mean and scaling factor to effectively normalize the original dataset, resulting in a unified format dataset. The time synchronization module calculates a unified time based on the transmission and reception time, calibration value, and delay compensation, and generates a time sequence table according to the maximum delay and process correspondence table; it extracts corresponding data based on single echo waveform data to build a single echo dataset, and calculates the data integrity rate based on the expected total number of associated data. The waveform inspection module performs frequency analysis and reference comparison on waveform data to obtain the main echo amplitude ratio, echo width and waveform matching degree; and calculates the operating deviation, environmental change and process compliance rate according to equipment status, environment and process parameters. The data cleaning module calculates data deviation scores and data effectiveness based on preset indicator weights; it sets qualified standard values, deletion standard values, and a minimum data collection quantity as a lower limit, where the deletion standard value is less than the qualified standard value, and outputs instructions to retain, delete, or repair accordingly. Based on test records containing calibration deviations, inspection records containing pass marks, and sampling records, update the normalization parameters, time synchronization parameters, reference waveforms, weights, standard values, expected total number of associated data, and minimum data acquisition quantity.

[0018] In this embodiment, echo waveform data stream, production environment data stream, equipment status data stream, and process parameter data stream from the level gauge production line are acquired. According to the communication protocol of each data source, the format of each data stream is converted to generate a basic data set, i.e., the original dataset, after unified encapsulation. The original dataset contains at least the data number, reception time, generation time, valid data, and data quality marker. The effective data for echo waveform data, production environment data, and process parameter data are usually measured values, such as echo amplitude sequences, temperature values, pressure values, humidity values, and current process parameter values. The effective data for equipment status data are usually status values, such as equipment current, power supply voltage, and equipment start / stop status. This embodiment first encapsulates heterogeneous data in a unified manner, so that subsequent time synchronization, waveform inspection and cleaning judgment are all carried out around the same data structure, avoiding data field loss, misalignment or quality mark distortion caused by inconsistent interface formats. In practice, the process parameter data should at least include the current value, the reference value, and the allowable error range, so as to directly determine whether the process execution deviates from the standard state. The reference value here refers to the standard parameter value predetermined in the corresponding production process, such as the standard torque in the assembly stage and the standard setting in the debugging stage. The allowable error range refers to the allowable fluctuation range around the reference value. For example, the tolerance band established around the reference value integrates the absolute degree of deviation of the current parameter value from the reference value with the allowable error range to provide a basis for the calculation of the subsequent process compliance rate.

[0019] The data interface module is used to receive echo waveform data streams, production environment data streams, equipment status data streams, and process parameter data streams. Extract the data number, receiving time, generation time, valid data, and data quality markers for each data item; Among them, the valid data for echo waveform data, production environment data and process parameter data are measured values, and the valid data for equipment status data are status values. Among them, the production environment data includes at least temperature, pressure and humidity, and the equipment status data includes at least equipment current, power supply voltage and equipment start-up and shutdown status; The original dataset is formed by combining the data number, receipt time, generation time, valid data, and data quality tags.

[0020] In this embodiment, valid data from the original dataset is obtained, and the corresponding benchmark mean, which represents the typical level of the corresponding data category in the benchmark production stage, and the scaling factor, which represents the allowable scale of change of the data in normal production, are called according to the data source category. The valid data is then normalized to generate a unified format dataset. For example, ambient temperature data can correspond to a set of temperature baseline averages and temperature scaling factors, and equipment current data can correspond to a set of current baseline averages and current scaling factors. The normalization calculation logic is as follows: subtract the baseline mean of the data under stable production conditions from the measured or state values ​​collected within a certain system cycle, divide the deviation by the corresponding scaling factor, thereby compressing data of different dimensions and different fluctuation scales into a comparable range, and obtaining a normalized result that represents the degree of deviation of the current data from the stable reference level. Specifically, the normalization calculation uses the formula Execution refers to the process of collecting valid data measurement values ​​or status values. Subtract the baseline mean Then divide by a scaling factor that is set to be greater than 0 to prevent the denominator from being zero. ,in This is the result of normalization; This calculation process maps numerical values ​​of different dimensions to a unified dimensionless interval, which typically ensures that normal data within the allowable error range fall within the bounds of the given interval. Within the relative deviation range; Preferably, the scaling factor can be taken as the standard scale of this type of data in the benchmark sample, and the value can be obtained from the statistical results of continuous qualified workpiece samples in the trial production stage. In this embodiment, the normalization calculation is used to pull data of different dimensions such as temperature, pressure, current, and waveform characteristics into the same calculation range, so as to avoid the other key changes being overshadowed by the large dimension of a certain type of value.

[0021] The data normalization module is used to: read valid data from the original dataset; Retrieve the corresponding baseline mean and scaling factor according to the data source category; The effective data is normalized based on the baseline mean and scaling factor; The calculation results are combined with the corresponding data number, receiving time, generation time and data quality marker to form a unified format dataset.

[0022] After completing the construction of the unified format dataset, the receiving time, generation time, time calibration value for correcting the time recording deviation of the source device are obtained from the unified format dataset, and communication delay compensation for eliminating the arrival time difference caused by network jitter, buffer forwarding or interface transmission differences are pre-set for different data sources. The unified time is calculated according to different data source categories. A time sequence table is established based on the maximum delay that represents the upper limit of the time difference allowed to establish the relationship between different data sources and the production process correspondence table that reflects the sequential relationship of different types of data in the process. Other data in the corresponding time period are extracted around the single echo waveform to form a single echo dataset. The calculation logic for unified time is as follows: correct the clock deviation of the data source, force all edge computing nodes and source devices to access the globally unified network time protocol or precise time protocol, and under this absolute common clock domain constraint, add the time calibration value to the generation time of the data generated by the source device to form the feedforward correction time. The average delay along the communication path is deducted, and the reception time of the edge node receiving the data is subtracted from the communication delay compensation to form the feedback calculation time. Multiply each of these two moments by its corresponding fusion weight, and sum the two weights to obtain the standard time for unified sorting across data sources. Among them, the allocation rules for the fusion weight can be dynamically set according to the communication jitter and device clock stability of the corresponding data source: when the network transmission delay variance is less than or equal to the preset safety threshold, the first fusion weight is allocated to the time when the feedback calculation occurs. When the network transmission delay variance is greater than the preset safety threshold and the source device clock recording accuracy meets the requirements, the first fusion weight is assigned to the feedforward correction time, and the value of the first fusion weight is greater than the value of the second fusion weight. In the specific implementation, the fusion weight at the time of feedback conversion is defined as The fusion weight at the time of feedforward correction is defined as follows: and make both satisfy The system presets a safety threshold for network latency variance based on the maximum latency variance statistically analyzed under historical stable communication conditions. It then performs structured rule-based judgment by real-time monitoring of the latency variance of the communication path: if the current latency variance is less than or equal to this safety threshold, then... It is 0.7. It is 0.3; If the current delay variance is greater than the safety threshold, then set... It is 0.3. The value is 0.7; through the binary judgment logic tree determined above, the dynamic assignment of weights is completed, avoiding physical distortion caused by simply relying on the generation time and ignoring the transmission timeliness or simply relying on the reception time and being affected by network jitter. After establishing the time sequence table, environmental data, equipment status data, and process parameter data within the preceding and following time periods are extracted based on each single echo waveform data to form a single echo dataset. The data integrity rate is then calculated based on the number of associated data and the expected total number of associated data. The data integrity rate is calculated as follows: the number of associated data entries obtained in the current single echo dataset is divided by the total number of associated data entries that should be matched for this type of echo under normal production conditions; The completeness rate is used to characterize whether the current echo context information is sufficient. In terms of value, the completeness rate is usually between 0 and 1. The closer it is to 1, the more complete the associated context corresponding to the single echo is. In this embodiment, instead of simply storing time-adjacent data into the same cache, the sequential relationship of the process and the maximum delay constraint are superimposed for filtering, so that the associated data satisfies both the temporal proximity and the sequential logic of the production process. After completing the construction of the single echo dataset, the echo waveform data in the single echo dataset is obtained. Based on the frequency analysis and reference waveform comparison results, the echo waveform is checked, and the main echo amplitude ratio, which represents the proportion of the main echo peak energy in the total echo energy, the echo width, which represents the width of the sampling interval covered by the main echo under the predetermined amplitude threshold, and the waveform matching degree, which represents the consistency between the current echo and the reference waveform, are generated. Simultaneously, acquire equipment status data, production environment data, and process parameter data from the single echo dataset, and calculate the operating deviation, which characterizes the degree of deviation of equipment status data from normal operating conditions; the environmental change, which characterizes the degree of change of temperature, pressure, and humidity relative to standard environmental parameters; and the process compliance rate, which characterizes the degree to which the current values ​​of each process parameter meet the constraints of reference values ​​and allowable error ranges. The calculation logic for the process compliance rate is as follows: calculate the absolute value of the difference between the process parameter value collected in the current system cycle and the standard execution value under the corresponding process, divide the absolute value by the allowable tolerance band of the process parameter, and introduce a very small positive number to prevent the denominator from being zero, for example, take a value of 0.0001, to obtain the deviation ratio; Take the smaller value between the deviation ratio and 1, average all process parameters included in the analysis, and then subtract 1 to obtain the process compliance rate, which characterizes the degree of consistency between the current process execution and the standard process. The closer the value is to 1, the more stable the process execution is. This embodiment performs joint feature analysis on waveform features with three types of contextual information: equipment, environment, and process, in order to distinguish whether the amplitude change signal originates from the physical production process or from physical disturbances in the equipment and environment. After obtaining the above indicators, the main echo amplitude ratio, echo width, waveform matching degree, operating deviation, environmental change, process compliance rate, and data integrity rate are acquired. Based on the preset indicators, the weights are calculated, and the data deviation score, which represents the degree to which the current single echo dataset deviates from the normal test state, and the data efficiency, which represents the degree to which the current single echo dataset retains effective information, are calculated. According to the preset rules, a reference standard (deletion standard value) is set for when the data is below the threshold, and a reference standard (qualification standard value) is set for when the data is above the threshold. The minimum amount of data collected is used as the lower limit of the number of associated data in the single echo dataset to constrain whether the context information is sufficient to support the current judgment. If the deletion standard value is less than the qualification standard value, a cleaning instruction to retain, delete, or repair is output. The data deviation score is calculated as follows: the opposing values ​​of the main echo amplitude ratio, echo width, waveform matching degree, operating deviation, environmental change, process compliance rate, and data integrity rate are multiplied by their respective weights and summed. These weights are used to dynamically balance the contribution ratio of each indicator in the current liquid level gauge production, and the sum of the weights can be set to 1. Accordingly, data effectiveness can be obtained by combining indicators that are positively correlated with effective information: multiply the main echo amplitude ratio, echo width offset, waveform matching degree, process compliance rate, and data integrity rate by the corresponding weights used to measure the contribution to fidelity and sum them up; when performing the above weighted calculation, in order to unify the dimensions and avoid scale conflicts, it is necessary to ensure that all input indicators have been mapped to the normalized dimensionless interval. For indicators whose logic is already between 0 and 1, their opposite value is defined as 1 minus the current value of the indicator; for indicators with absolute dimensions, the system pre-sets a physical upper limit benchmark, performs division mapping, and then calculates the opposite value; through strict boundary and dimensional constraints, it is ensured that the scoring calculation will not be distorted due to differences in absolute values ​​when each indicator is substituted into the weighted summation. The deletion standard value, the qualified standard value, and the minimum data collection amount can be dynamically set according to the amount of operational deviation, environmental change, process compliance rate, and data integrity rate, so as to trigger the dynamic adjustment action of the standard value when the amount of environmental change is higher than the preset environmental threshold or the amount of related data is lower than the preset integrity threshold. This embodiment adjusts the index weights and standard values ​​in a linked manner, so that the same type of waveform has different judgment boundaries under different equipment disturbances and different process states, transforming the original problem of insufficient or excessive sensitivity to abnormal changes into a judgment process that can be quantified and evaluated. After outputting the cleaning command, obtain the cleaned echo signal sequence, synchronization timestamp, single echo dataset number, data deviation score, data effectiveness rate, data integrity rate, and processing result label; The system saves and outputs the results, and receives subsequent calibration test results including calibration deviation values ​​reflecting the degree of measurement deviation after the data enters the calibration stage, factory inspection results including conformity marks reflecting whether the product finally meets the factory requirements, and manual sampling records as the basis for correction after manual review of the automatic cleaning results. It also updates the normalization parameters, time synchronization parameters, reference waveforms, index calculation weights, standard values, total number of associated data and minimum data collection amount. By applying subsequent quality results in reverse to the front-end parameters, the normalization scale, synchronization compensation, and cleaning standard values ​​can be gradually adjusted with each production batch, rather than remaining in a fixed parameter state for a long time. In the field of liquid level gauge production data processing, commonly used evaluation benchmarks typically include the accuracy of out-of-order data correlation, the degree of retention of effective waveforms, and the stability of outlier data removal. When network jitter, equipment start-up and shutdown disturbances, and slow environmental drift occur simultaneously on the production line, traditional fixed time window averaging or static threshold elimination methods are prone to two types of problems: First, irrelevant environmental data may be spliced ​​onto a certain echo due to time misalignment, resulting in incorrect context interpretation. Secondly, treating all sudden changes as noise weakens the key peak shape in the true calibration echo. This embodiment uses unified format access, normalized mapping, time sequence table construction, single echo dataset association, and multi-index joint scoring to ensure that each single echo completes the cleaning judgment in a traceable and quantifiable context. For operating conditions where partial missing data frequently occurs, this embodiment can also use data integrity rate constraints to limit aggressive deletion; for operating conditions where equipment operation fluctuates greatly, this embodiment can use operating deviation and environmental change to adjust the judgment criteria, thereby ensuring that contaminated data is processed while preserving as much of the true production test characteristics as possible. In this application example, the system can be deployed on the edge computing node of the level gauge production line and connected to the echo acquisition unit, environmental acquisition unit, equipment monitoring unit, and process parameter recording unit. The data newly entering the system cycle is encapsulated into a raw dataset, then normalized and converted to a unified time. The system generates a corresponding dataset for each single echo, completes waveform inspection, joint scoring, and outputs cleaning instructions; the cleaning results continue to flow to the subsequent calibration test and quality inspection stages, while the calibration deviation value, factory qualified mark, and manual sampling record are returned to this system to update the normalization parameters, synchronization parameters, and judgment parameters; In this way, new data continuously entering the liquid level gauge production process can be repeatedly incorporated into the same processing flow, enabling continuous application of data cleaning, result output, and parameter updates during production.

[0023] In a preferred embodiment of the present invention, the time synchronization module is used to: read the reception time, generation time, and data source category from the uniform format dataset; Determine time calibration values ​​and communication delay compensation for various data sources; Calculate the unified time of each data based on the reception time, generation time, time calibration value, and communication delay compensation; Data pairs with time differences within the set allowable range are filtered based on the preset maximum delay, and data pairs with sequential relationships are filtered based on the production process correspondence table. Generate a time sequence table based on the unified time sequence and the sequence of processes; Based on the waveform data of each echo, the associated data before and after the echo is generated are extracted to form a single echo dataset. The data integrity rate is calculated based on the number of data points within a single echo dataset and the expected total number of associated data points.

[0024] In this embodiment, the receiving time, generation time and data source category used to distinguish echo waveform data, production environment data, equipment status data and process parameter data in the unified format dataset are obtained. Based on the compensation amount, i.e. time calibration value, which is preset to correct the clock deviation of the corresponding data source category, and the compensation amount, i.e. communication delay compensation, which is offset to cancel the average arrival delay of the data in the transmission link, the unified time calculation is performed on each data to determine the unified time result of each data. The conversion logic for unified time is as follows: based on the recording time when the source device generates the data, add the compensation amount for the feedforward correction of clock deviation to form the feedforward correction time; Based on the recorded time when the edge node receives the data, subtract the compensation amount for deducting the average delay of the communication path from the arrival side to form the reverse calculation time; take the average of these two times to obtain the standard time when data from different sources can jointly participate in sorting and association; Preferably, different compensation parameters can be used for different types of data to adapt to the transmission differences between the echo sampling link, the environmental acquisition link, and the device status acquisition link; in this embodiment, time calibration value and communication delay compensation are introduced into the unified time calculation so that data with network jitter, buffer forwarding, or device clock deviation can still fall into a more accurate relative order; Obtain the unified time of each data, filter data pairs whose time difference is within the set allowable range according to the preset maximum delay, and filter data pairs with process sequence relationship according to the production process correspondence table to generate a time sequence table; The time of convergence for the two data points is denoted as follows: and The maximum allowable delay between corresponding categories is denoted as the maximum response time difference characterizing the maximum possible association between data category a and data category b within the production cycle time. When satisfied Furthermore, if these two types of data have a sequential relationship in the production process correspondence table, the data pair can be included in the time sequence table; Should The value can be preset based on the production line cycle characteristics, interface refresh cycle, or equipment response characteristics. An example of the value can be several sampling intervals within a system cycle. This embodiment adds the screening condition of process sequence relationship, so that data that are close in time but unrelated in process will not be incorrectly classified into the same echo context. Acquire waveform data for each echo, extract related data before and after the echo based on the time sequence table, and form a local related data set around a certain echo waveform event, namely a single echo dataset. This dataset includes at least environmental data, equipment status data, and process parameter data that are adjacent to the echo time and satisfy the process relationship. The data integrity rate is calculated based on the number of data items that have been successfully associated in the single echo dataset, i.e., the number of data items, and the number of associated data items that should be matched for this type of echo under stable production conditions, i.e., the expected total number of associated data items. The calculation logic for data integrity rate is as follows: divide the number of data that were actually acquired and successfully paired before and after the current echo by the number of baseline references used for integrity context judgment to obtain a value used to characterize whether the echo has sufficient context support in the current system cycle; Its value is usually between 0 and 1. The lower the value, the less complete the current echo context is, and the more cautious one should be in proceeding with subsequent judgments. In this embodiment, waveform analysis conditions are constrained based on the data integrity of the associated context to ensure that waveform analysis has corresponding data support.

[0025] The waveform inspection module is used to: read echo waveform data from a single echo dataset; The echo waveform data is divided into frequency bands, and the amplitude ratio and echo width of the main echo are calculated based on the energy values ​​of each frequency band. The echo waveform data is compared with a preset reference waveform to calculate the waveform matching degree; Read the device status data from the single echo dataset and calculate the operating deviation based on the degree of deviation from the normal operating state; Read the production environment data from the single echo dataset and calculate the environmental change based on the difference from the standard environmental parameters; Read the process parameter data from the single echo dataset, and calculate the process compliance rate based on the relationship between the degree of deviation of the current value from the reference value and the allowable error range.

[0026] After completing the construction of the single echo dataset, the echo waveform data in the single echo dataset is obtained. The echo waveform data is converted into frequency domain data using fast Fourier transform. The effective bandwidth is divided into multiple analysis frequency bands at equal intervals according to a fixed frequency step size. Based on the energy accumulation results in the corresponding frequency band, i.e., the energy value of each frequency band, the echo waveform data is analyzed to generate the main echo amplitude ratio, which represents the ratio of the energy in the frequency band where the main echo is located to the total energy, and the echo width, which represents the width of the expansion interval of the main echo under a preset energy or amplitude threshold. The calculation logic for the main echo amplitude ratio is as follows: divide the energy of the frequency band corresponding to the main peak of the target echo by the total energy of all analyzed frequency bands of the current echo segment, and introduce a very small positive number to prevent the denominator from being zero, for example, take a value of 0.0001, so as to characterize whether the main components of the echo are sufficiently concentrated. Correspondingly, the echo width can be determined based on the sampling range of the main echo continuously distributed under the threshold condition. Its physical meaning is whether the echo peak is sharp and whether there is obvious tailing or diffusion. In this embodiment, the echo structure is analyzed by frequency band energy distribution, so that the concentration and diffusion of the echo peak can be quantified, making it easier to distinguish between the real main echo and broadband noise interference. The echo waveform data and the standard waveform template extracted from the echo sample of the qualified workpiece are obtained, which is the preset reference waveform. Based on the comparison results of the two, the waveform matching degree, which represents the consistency between the current echo waveform and the reference waveform, is calculated. The calculation logic of waveform matching degree is as follows: calculate the sum of the normalized amplitudes of the current echo waveform and the reference waveform at each sampling position, divide by the square root product of the sum of squares of their respective amplitudes, and introduce a very small constant to prevent the denominator from being zero, for example, a value of 0.0001, thereby characterizing the degree of consistency between the current echo shape and the standard echo shape. The higher the value, the closer the current waveform is to the qualified template. Preferably, the template length can cover the main peak of the echo and its adjacent transition section to take into account both the main peak shape and local details; this embodiment additionally introduces reference waveform comparison so that echoes that have abrupt changes but are consistent with the standard template shape can be identified, rather than being simply classified as abnormal. Acquire the equipment status data in the single echo dataset, and calculate the degree of deviation of the equipment status from the stable operating benchmark, which can be characterized by at least the equipment current, supply voltage and equipment start-up and shutdown status, based on the degree of deviation from the normal operating state. Obtain production environment data and calculate the degree of deviation of temperature, pressure, and humidity from standard environmental parameters, i.e., environmental change, based on the difference from standard environmental parameters. Acquire process parameter data, and calculate the process compliance rate, which represents the overall degree to which the current process execution falls within the allowable error range, based on the relationship between the degree to which the current value deviates from the reference value and the allowable error range. The quantification logic of the operating deviation is as follows: obtain the degree of change of the device current relative to the reference current, the degree of change of the power supply fluctuation relative to the reference voltage, and the device start-stop state jump variable in adjacent states. When a state switch occurs, the value is 1, otherwise it is 0. The system performs uniform interval truncation processing on the relative change of power supply and current to align the scale, and then multiplies the above parameters by the corresponding weights of the contributions of the three types of state variables in dynamic balance and sums them up. Accordingly, the environmental change is formed by weighting temperature offset, pressure offset, and humidity offset. It should be noted that, to avoid dimensional conflicts caused by directly adding different physical units, such as degrees Celsius, Pascals, and relative humidity percentages, the system forcibly divides each of the above offsets by its corresponding maximum permissible deviation limit before weighting, converting them into dimensionless relative deviation rates. Then, the system performs the product and summation of the corresponding weights. The specific mathematical formula is as follows: in, For environmental change, These are the offsets for temperature, pressure, and humidity, respectively. An absolute value operator is introduced to prevent positive and negative drifts from canceling each other out. This corresponds to the maximum permissible deviation limit. The corresponding weights are preset and their sum is 1; Its physical meaning is whether the current echo environment deviates from the standard test conditions according to a unified dimensionless scale; the process compliance rate is used to quantify the stability of the current process execution; this embodiment directly converts these data into quantitative indicators that can participate in echo judgment, so that whether the echo anomaly is related to equipment disturbance, environmental drift or process deviation can be evaluated simultaneously. In signal processing and industrial timing correlation, common extreme conditions include communication disorder, partial packet loss, instantaneous equipment start-up and shutdown disturbances, and the superimposed effects of gradual environmental changes. If a conventional fixed-window caching method is used, although it can quickly stitch together data that are close in time, it is difficult to distinguish between data that are merely close in time and data that are actually related in terms of process. If only waveform amplitude and simple filtering are used for judgment, it is easy to regard pollution caused by equipment fluctuations and the real main echo as anomalies. In contrast, this embodiment first establishes a basis-based association using a unified time, maximum delay, and process correspondence table. Then, it uses the main echo amplitude ratio, echo width, waveform matching degree, operating deviation, environmental change, and process compliance rate to jointly characterize a single echo dataset, enabling subsequent modules to make judgments in a more accurate context. In the application example, an edge computing device can be configured at the level gauge assembly and calibration station. The edge computing device periodically receives echo sampling data, environmental sampling data, equipment status data, and process record data. After the system performs a unified time calculation on each data, it completes the correlation between the echo before and after according to the maximum delay and the process correspondence, and generates a single echo dataset. The system then performs frequency band analysis, template comparison, and context index calculation on the dataset and outputs the results to the subsequent cleaning module. For new echo events, the system repeats the same process to continuously complete timing rectification and waveform inspection during the production process.

[0027] In a preferred embodiment of the present invention, the data cleaning module is used to: receive the main echo amplitude ratio, echo width, waveform matching degree, operating deviation, environmental change, process compliance rate and data integrity rate; Based on the preset indicator weights, calculate the data deviation score and data effectiveness rate; The deletion standard value, the pass standard value, and the minimum data collection quantity are dynamically set based on the amount of operational deviation, environmental changes, process compliance rate, and data integrity rate. The deletion standard value is less than the pass standard value, and the minimum data collection quantity is determined based on the expected total number of related data and the preset proportional coefficient. When the data integrity rate is lower than the minimum data collection amount, the corresponding single echo dataset is identified as data to be repaired. When the data integrity rate is not lower than the minimum data collection amount, the data deviation score is not lower than the deletion standard value, and the data effectiveness rate is lower than the qualified standard value, the corresponding single echo dataset will be judged as contaminated data. When the data integrity rate is not lower than the minimum data collection amount, the data effectiveness rate is not lower than the qualified standard value, and the data deviation score is lower than the deletion standard value, the corresponding single echo dataset will be judged as normal test data. In other cases, the corresponding single echo dataset is identified as data to be repaired; The data cleaning module is used to: extract the original echo waveform data corresponding to the data to be repaired when processing the data to be repaired; Based on the waveform distortion points of the original echo waveform data and the filtered data, waveform points are filled in to generate a preliminary repaired waveform; The preliminary repair waveform is tested based on the preset amplitude change limit, smoothness lower limit and matching degree lower limit to obtain the final repaired echo waveform data; The final repaired echo waveform data replaces the original data to be repaired, and the cleaning result containing the repair instructions is output.

[0028] In this embodiment, the main echo amplitude ratio, echo width, waveform matching degree, operating deviation, environmental change, process compliance rate and data integrity rate are obtained. The weights are calculated according to the preset indicators, and the indicators are weighted to generate data deviation score and data effectiveness rate. The calculation logic for the data deviation score is as follows: extract the quantitative indicators corresponding to the main echo energy concentration, echo peak diffusion, waveform consistency, equipment operation disturbance, environmental deviation, process compliance, and echo context sufficiency. Directly take the values ​​of the indicators positively correlated with the deviation state, and take the opposite values ​​of the indicators negatively correlated with the deviation state. Multiply them by the corresponding weights used for dynamic balancing of each quantity's contribution and sum them up. Accordingly, data effectiveness can be obtained by positively combining the relevant indicators of fidelity: the indicators that play a positive role in effectiveness are directly assigned values, and the indicators that play a negative role are assigned their opposite values, and then multiplied by the weights of each factor used to balance the contribution of fidelity and summed. This embodiment simultaneously constructs a data deviation score and a data effectiveness rate, so that the two dimensions of large deviation and still retaining useful information are expressed separately, avoiding the direct deletion of some data that, although it has local anomalies, still has analytical value. Acquire operational deviations, environmental changes, process compliance rates, and data integrity rates. Dynamically set deletion standard values, pass standard values, and minimum data collection amounts based on these indicators, ensuring that the deletion standard value is less than the pass standard value. The calculation logic for the minimum data acquisition amount is as follows: multiply the total number of data that this type of echo should be associated with under normal test conditions by a coefficient used to control the minimum context retention ratio, with a value between 0 and 1, to obtain the minimum number of associated data required to support the current echo to complete a valid determination, and perform an up rounding operation on the calculation result to ensure that the minimum data acquisition amount is a meaningful integer at the physical level; Furthermore, when executing the cleaning instruction judgment logic, since the data integrity rate is essentially a ratio of 0 to 1 and the minimum data collection amount is the number of data records, the system will make an integer judgment on the same dimension between the number of extracted related data and the minimum data collection amount when performing the above comparison; thus completing and self-consistently handling the comparison action when the data integrity rate is lower than the minimum data collection amount. The deletion standard value and the pass standard value can be adaptively adjusted according to the amount of operational deviation, environmental change, process compliance rate and data integrity rate. When the amount of operational deviation and environmental change are higher than the preset first threshold, the value of the deletion standard value is reduced; when the process compliance rate and data integrity rate are higher than the preset second threshold, the value of the pass standard value is reduced. The dynamically set quantitative execution rules are as follows: preset the baseline deletion standard value, the baseline pass standard value, and the baseline data collection ratio; calculate the arithmetic mean of the current dataset's running deviation and environmental changes as the environmental disturbance coefficient; Determine whether the environmental disturbance coefficient is greater than the preset disturbance warning threshold: if it is greater, multiply the baseline deletion standard value by the preset compression coefficient, such as 0.8, as the current deletion standard value, and multiply the baseline data collection ratio by the preset strict coefficient, such as 1.2, to increase the minimum data collection amount; if it is less than or equal to, directly use the baseline value. If the threshold is not exceeded, the baseline value is directly adopted. This embodiment dynamically sets the standard value and the minimum data collection amount, so that the judgment boundary under different production conditions is adjusted according to the context conditions, avoiding the use of overly aggressive deletion rules when there are large environmental fluctuations or a lot of missing data. The system obtains data integrity rate, data deviation score, data effectiveness rate, minimum data collection volume, deletion standard value, and qualification standard value. Based on the preset judgment rules, it classifies the corresponding single echo dataset and determines the cleaning instructions. Among them, when the data integrity rate is lower than the minimum data collection amount, the corresponding single echo dataset is judged as data that is neither suitable for direct deletion nor suitable for direct retention as normal test data, i.e., data to be repaired. When the data integrity rate is not lower than the minimum data collection amount, the data deviation score is not lower than the deletion standard value, and the data effectiveness rate is lower than the qualified standard value, the corresponding single echo dataset is judged as a data state with obvious deviation characteristics and insufficient effective information, i.e., polluted data. When the data integrity rate is not lower than the minimum data collection amount, the data effectiveness rate is not lower than the qualified standard value, and the data deviation score is lower than the deletion standard value, the corresponding single echo dataset is judged as normal test data when the data deviation score is lower than the preset deviation threshold and the data effectiveness rate reaches the preset effective threshold. In other cases, the corresponding single echo dataset is identified as data to be repaired; this embodiment adds a repair branch so that data in the boundary area will not be simply classified as deleted or retained, thereby transferring the dataset in the judgment benchmark boundary to the subsequent repair process; When corresponding data is identified as data to be repaired, for example when the data integrity rate is lower than the minimum data acquisition amount, it indicates that the current echo lacks sufficient contextual constraints and cannot clearly rule out the influence of transient equipment disturbances or sudden environmental changes. Therefore, based on the anomaly avoidance logic under the condition of missing data, the system transfers data with unclear status due to missing associated data to the repair branch, extracts amplitude change points below the conventional judgment threshold as suspected distortion points, and uses a sliding window mean filtering algorithm. The calculation formula is as follows: in, The first one after filtering The amplitude of each sampling point, i.e., the filtered data. For the first in the sliding window One original sampled amplitude, This is the index of the current sampling point. The offset index within the window. The preset smooth window half-width, This is the total length of the sampling sequence; To prevent data structure crashes due to sequence index out-of-bounds errors, when or At this time, the boundary truncation compensation method is forcibly adopted, and the filtered sequence is automatically reduced. The original echo waveform is smoothed to obtain a reference waveform result, i.e., filtered data, which is used to provide a proximity trend basis for the point filling position. Waveform point filling is then performed to generate a preliminary repair waveform. Under this defense strategy, for data lacking context, the system will actively tighten the waveform distortion recognition threshold and forcibly extract potentially hidden tiny jumps as suspected distortion points to ensure that the waveform itself can still be smoothed and has noise resistance even under unknown environmental or equipment conditions. Within the set of waveform anomaly locations that need repair, the initial repair waveform will directly use the reference filter amplitude obtained by using adjacent trends and smoothing constraints; while at non-distorted points, the amplitude of the original echo remains unchanged; this embodiment uses filtered data to fill in the local distortion locations, so that the main peak and the original trend are preserved as much as possible, so as to avoid the overall discarding of the corresponding echo waveform data; The preliminary repair waveform is obtained. The preliminary repair waveform is checked according to the preset amplitude change limit to limit the amplitude change degree between adjacent sampling points to avoid retaining unreasonable change characteristics after repair, the smoothness lower limit to limit the minimum requirement of local continuity of the repair waveform, and the matching degree lower limit to reflect the minimum matching requirement required for the repair waveform to be basically consistent with the reference waveform. The final repaired echo waveform data is determined, and the original data to be repaired is replaced by the final repaired echo waveform data. The cleaning result containing the repair instructions is output. Among them, smoothness characterizes the continuity of change between adjacent sampling points, and the matching degree lower limit characterizes the minimum value at which the repair result maintains the basic contour of the normal test echo. Preferably, the amplitude mutation limit, smoothness lower limit, and matching degree lower limit can all be preset based on historical qualified waveform samples or production debugging results, and their values ​​are used in the form of logical judgment conditions, without limiting specific absolute values. To ensure the reliability of the above threshold acquisition and the clarity of the quantitative boundary, the specific basis for setting it is as follows: extract the maximum statistical value of the amplitude difference between adjacent sampling points in the most recent batches of consecutive qualified waveform samples as the amplitude mutation limit, and extract the lower baseline of the smoothness distribution and matching degree distribution of historical qualified waveform samples. For example, the mean of the corresponding index minus three times the standard deviation is used as the specific value of the lower limit of smoothness and the lower limit of matching degree, thereby clarifying the defensive evaluation index as a quantitative judgment boundary with evidence. This embodiment adds an inspection step after patching to screen the repair results again, so that the repair action is neither mechanical interpolation nor unconditional replacement, but quantitatively constrained by waveform continuity and template consistency.

[0029] The results output and update module is used to: receive, retain, delete, and repair the corresponding cleaning results; The cleaned echo signal sequence, timestamp, single echo dataset number, data deviation score, data effectiveness rate, data integrity rate, and processing mark are saved accordingly and output to the subsequent calibration test and quality inspection stages. Receive subsequent calibration test results, factory inspection results, and manual sampling records; Update the normalization parameters and reference waveforms based on the product data that has been confirmed as qualified by the factory inspection. Update time synchronization parameters based on delay statistics from various data sources; Based on the erroneous deleted records, erroneous retained records, corrected misjudged records, and matching error records in the manual sampling records, update the indicator weights, standard values, expected total number of related data, and minimum data collection volume; The updated parameters are sent to the data normalization module, time synchronization module, waveform inspection module, and data cleaning module, respectively.

[0030] The corresponding cleaning results for retention, deletion, and repair are obtained. The cleaned echo signal sequence, the synchronization timestamp indicating the corresponding position of the echo in a unified time system, the single echo dataset number used to locate the source of the original data in subsequent feedback, the data deviation score, the data effectiveness rate, the data integrity rate, and the processing result mark used to identify whether the data is determined to be retained, deleted, or repaired in the current processing cycle are saved accordingly and output to the subsequent calibration test and quality inspection stages. With this setting, subsequent calibration test results, factory inspection results and manual sampling records can all flow back to the system around the same number, without losing their correspondence with the original echo before cleaning. Obtain subsequent calibration test results, factory inspection results, and manual sampling records; update normalization parameters and reference waveforms based on product data confirmed as qualified by factory inspection. The time synchronization parameters are updated based on the delay statistics of various data sources. The data records that should have been retained after manual confirmation are identified as follows: data records that should have been retained after automatic system deletion but manual confirmation (i.e., erroneously deleted records); data records that should have been deleted after manual confirmation but automatic system retention (i.e., erroneously retained records); data records that should have been removed after automatic system entry into the repair branch but were ultimately found to be improperly selected repair paths (i.e., repair misjudgment records); and data records where the template matching results are inconsistent with the manual review conclusions (i.e., matching error records). Update the indicator calculation weights, standard values, total number of associated data and minimum data collection volume, and send the updated parameters to the data normalization module, time synchronization module, waveform inspection module and data cleaning module respectively; This embodiment directly transforms subsequent quality conclusions into the basis for parameter updates, enabling the normalization scale, time synchronization compensation, template, and judgment threshold to be adjusted as production batches change.

[0031] The results output and update module is used to update the reference waveform based on the echo waveform data that has passed factory inspection in the most recent production batch and the historical template when updating parameters. The total number of misjudgments and the total number of repairs were calculated based on manual sampling records. The parameter modification amount for the current batch is calculated based on the total number of misjudgments and the total number of repairs. The parameter modification amount for different parameters is either a numerical increase or decrease or an adjustment ratio coefficient. The parameter modification amount of the current batch is added to the historical parameters according to their weights to obtain the updated standard value, time calibration value, communication delay compensation, expected total number of associated data and minimum data collection amount.

[0032] During the parameter update process, the echo waveform data and historical templates that have passed the factory inspection in the most recent production batch are obtained, and the reference waveform is updated according to the data. At the same time, the total number of records such as error deletion, error retention, repair misjudgment and matching error are summarized according to the manual sampling records, which is the total number of misjudgments, and the total number of data in the current batch that enters the repair process, which is the total number of repairs. Based on the total number of misjudgments and the total number of repairs, the parameter modification amount for the current batch, which can be expressed as a numerical increase or decrease or a proportional coefficient adjustment, is calculated. The parameter modification amount for the current batch is then added to the reference waveform template used in the previous processing stage, i.e., the historical parameters, according to weights, to obtain the updated standard value, time calibration value, communication delay compensation, total number of associated data, and minimum data acquisition amount. The update logic of the reference waveform is as follows: at each sampling position, the historical template amplitude that has been used for waveform comparison is multiplied by a weighting coefficient that balances historical stability, and the new template amplitude obtained from the statistical analysis of the current batch of qualified echo data is multiplied by the corresponding supplementary weighting coefficient that reflects the current change, so as to obtain a new reference waveform for waveform matching in the next processing stage. The update logic for the associated parameters is as follows: the adjustment range of the current batch parameters derived from the total number of misjudgments and the total number of repairs is multiplied by a coefficient that controls the smoothness of parameter changes. The coefficient is set between 0 and 1 to keep the change of parameters between adjacent batches within a preset range and is then superimposed on the historical parameter values. The specific rules for deriving the adjustment range of the parameters are as follows: taking the standard value as an example, statistically analyze the manual sampling records of the current batch, calculate the proportion of incorrectly deleted records to the total number of misjudged records, and the proportion of incorrectly retained records to the total number of misjudged records; If the percentage of incorrectly deleted records is higher than the percentage of incorrectly retained records, it indicates that the current judgment is too strict. In this case, the parameter adjustment range is defined as a preset fixed step size for lowering the standard value, such as a decrease of 0.05. Conversely, the parameter adjustment range is defined as an increase of the preset fixed step size. For time synchronization compensation, the average arrival time difference that caused the mismatch in the matching error record is directly extracted and used as the corresponding parameter adjustment range; This embodiment uses batch statistical results to weight and adjust the template and multiple parameters, so that parameter updates have both historical continuity and can absorb the effective experience of the current batch. In industrial data cleaning and quality assessment, conventional evaluation benchmarks typically include the false deletion rate, the false retention rate, and the usability rate after repair. If a traditional static threshold removal scheme is used, under the conditions of equipment start-up and shutdown disturbances and environmental drift, the system is prone to directly deleting boundary waveforms as contaminated data, resulting in an increase in the false deletion rate. If only a conservative retention approach is adopted, the contaminated waveform will continue to flow into the calibration and factory inspection process, resulting in an increase in the false retention rate. This embodiment combines data deviation scoring and data effectiveness with the minimum data collection constraint to guide boundary samples into the repair path, thereby achieving a parameter balance between data contamination removal rate and effective feature retention rate. Furthermore, in this embodiment, the subsequent qualified marks, total number of false positives and total number of repairs are continuously used to update the template, time compensation and judgment parameters, so that the system can still maintain good adaptability under the conditions of batch changes, equipment status changes and network latency changes; In this application example, this embodiment can be deployed in the data processing node of the level gauge production line. When the system receives a new single echo dataset, it first calculates the data deviation score and data effectiveness rate, and then generates retention, deletion or repair instructions by combining the minimum data acquisition amount, deletion standard value and qualified standard value. For the data to be repaired, the system automatically performs waveform patching and verification and outputs the repair results. The cleaning results, along with the score, number, and timestamp, are sent to the calibration and testing and quality inspection stages. As new calibration deviation values, factory qualification marks, and manual sampling records are returned, the system updates parameters such as the reference waveform, time synchronization parameters, standard values, and the number of associated data, and continues to use them for processing new echo data in subsequent system cycles.

[0033] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A liquid level gauge production data processing system based on data cleaning, characterized in that... include: The data interface module receives process parameters containing current values, reference values, and allowable error ranges, as well as echo waveforms, production environment, and equipment status data, and converts them into raw datasets in a unified format. The data normalization module calls the baseline mean and scaling factor to effectively normalize the original dataset, resulting in a unified format dataset. The time synchronization module calculates a unified time based on the transmission and reception time, calibration value, and delay compensation, and generates a time sequence table according to the maximum delay and process correspondence table. Based on the single echo waveform data, corresponding data are extracted to form a single echo dataset, and the data integrity rate is calculated according to the expected total number of associated data. The waveform inspection module performs frequency analysis and reference comparison on the waveform data to obtain the echo amplitude ratio, echo width and waveform matching degree. Calculate the operating deviation, environmental change, and process compliance rate based on equipment status, environment, and process parameters respectively; The data cleaning module calculates data bias scores and data effectiveness based on preset indicator weights. Set a pass / fail standard value, a delete standard value, and a minimum data collection quantity as a lower limit, wherein the delete standard value is less than the pass / fail standard value, and output retain, delete, or repair instructions accordingly; The results output and update module outputs the cleaning signal sequence, timestamp, single dataset number, processing label and score, data effectiveness rate, and completeness rate. Based on test records containing calibration deviations, inspection records containing pass marks, and sampling records, update the normalization parameters, time synchronization parameters, reference waveforms, weights, standard values, expected total number of associated data, and minimum data acquisition quantity.

2. The liquid level gauge production data processing system based on data cleaning according to claim 1, characterized in that, The data interface module is used for: Receive echo waveform data stream, production environment data stream, equipment status data stream, and process parameter data stream; Extract the data number, receiving time, generation time, valid data, and data quality markers for each data item; Among them, the valid data for echo waveform data, production environment data and process parameter data are measured values, and the valid data for equipment status data are status values. Among them, the production environment data includes at least temperature, pressure and humidity, and the equipment status data includes at least equipment current, power supply voltage and equipment start-up and shutdown status; The data number, receiving time, generation time, valid data, and data quality markers are combined to form the original dataset.

3. The liquid level gauge production data processing system based on data cleaning according to claim 2, characterized in that, The data normalization module is used for: Read valid data from the original dataset; Retrieve the corresponding baseline mean and scaling factor according to the data source category; The effective data is normalized based on the benchmark mean and scaling factor. The calculation results are combined with the corresponding data number, receiving time, generation time and data quality marker to form a unified format dataset.

4. A liquid level gauge production data processing system based on data cleaning according to any one of claims 1 to 3, characterized in that, The time synchronization module is used for: Read the reception time, generation time, and data source category from a uniform format dataset; Determine time calibration values ​​and communication delay compensation for various data sources; Calculate the unified time of each data based on the received time, generated time, time calibration value, and communication delay compensation; Data pairs with time differences within the set allowable range are filtered based on the preset maximum delay, and data pairs with sequential relationships are filtered based on the production process correspondence table. Generate a time sequence table based on the unified time sequence and the sequence of processes; Based on each echo waveform data, the associated data before and after the generation of the echo are extracted to form the single echo dataset; The data integrity rate is calculated based on the number of data points within a single echo dataset and the expected total number of associated data points.

5. A liquid level gauge production data processing system based on data cleaning according to claim 4, characterized in that, The waveform inspection module is used for: Read echo waveform data from a single echo dataset; The echo waveform data is divided into frequency bands, and the amplitude ratio and echo width of the main echo are calculated based on the energy values ​​of each frequency band. The echo waveform data is compared with a preset reference waveform to calculate the waveform matching degree; Read the device status data from the single echo dataset and calculate the operating deviation based on the degree of deviation from the normal operating state; Read the production environment data from the single echo dataset and calculate the environmental change based on the difference from the standard environmental parameters; Read the process parameter data from the single echo dataset, and calculate the process compliance rate based on the relationship between the degree of deviation of the current value from the reference value and the allowable error range.

6. The liquid level gauge production data processing system based on data cleaning according to claim 5, characterized in that, The data cleaning module is used for: Received main echo amplitude ratio, echo width, waveform matching degree, operating deviation, environmental change, process compliance rate, and data integrity rate; The data deviation score and data effectiveness rate are calculated based on the preset indicator weights. The deletion standard value, the qualification standard value, and the minimum data collection quantity are dynamically set based on the operational deviation, environmental changes, process compliance rate, and data integrity rate. The deletion standard value is less than the qualification standard value, and the minimum data collection quantity is determined based on the expected total number of related data and a preset proportional coefficient. When the data integrity rate is lower than the minimum data collection amount, the corresponding single echo dataset is identified as data to be repaired. When the data integrity rate is not lower than the minimum data collection amount, the data deviation score is not lower than the deletion standard value, and the data effectiveness rate is lower than the qualified standard value, the corresponding single echo dataset will be judged as contaminated data. When the data integrity rate is not lower than the minimum data collection amount, the data effectiveness rate is not lower than the qualified standard value, and the data deviation score is lower than the deletion standard value, the corresponding single echo dataset will be judged as normal test data. In other cases, the corresponding single echo dataset is identified as data to be repaired.

7. A liquid level gauge production data processing system based on data cleaning according to claim 6, characterized in that, The data cleaning module is used to: When processing data to be repaired, it is used for: Extract the original echo waveform data corresponding to the data to be repaired; Based on the waveform distortion points of the original echo waveform data and the filtered data, waveform points are filled in to generate a preliminary repaired waveform; The preliminary repair waveform is tested based on the preset amplitude change limit, smoothness lower limit and matching degree lower limit to obtain the final repaired echo waveform data; The final repaired echo waveform data replaces the original data to be repaired, and the cleaning result containing the repair instructions is output.

8. A liquid level gauge production data processing system based on data cleaning according to claim 7, characterized in that, The result output and update module is used for: Receive the cleaning results for retention, deletion, and repair; The cleaned echo signal sequence, timestamp, single echo dataset number, data deviation score, data effectiveness rate, data integrity rate, and processing mark are saved accordingly and output to the subsequent calibration test and quality inspection stages. Receive subsequent calibration test results, factory inspection results, and manual sampling records; Update the normalization parameters and reference waveforms based on the product data that has been confirmed as qualified by the factory inspection. Update time synchronization parameters based on delay statistics from various data sources; Based on the erroneous deleted records, erroneous retained records, corrected misjudged records, and matching error records in the manual sampling records, update the indicator weights, standard values, expected total number of related data, and minimum data collection volume; The updated parameters are sent to the data normalization module, time synchronization module, waveform inspection module, and data cleaning module, respectively.

9. A liquid level gauge production data processing system based on data cleaning according to claim 8, characterized in that, The result output and update module is used when updating parameters to: Update the reference waveform based on the echo waveform data that passed the factory inspection in the most recent production batch and the historical template; The total number of misjudgments and the total number of repairs were calculated based on manual sampling records. The parameter modification amount for the current batch is calculated based on the total number of misjudgments and the total number of repairs. The parameter modification amount for different parameters is either a numerical increase or decrease or an adjustment ratio coefficient. The parameter modification amount of the current batch is added to the historical parameters according to their weights to obtain the updated standard value, time calibration value, communication delay compensation, expected total number of associated data and minimum data collection amount.

Images