Flow data clutter removal method
By combining the traffic data clutter cause determination network and sensor data correction model of CNN and LSTM, the impurity removal, correction and quality evaluation of traffic data is solved, and more accurate and stable traffic data processing is achieved, and the on-board supply of rail transit is improved The detection capability of the wind unit.
Patent Information
- Application Number
- CN202411505072.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art has failed to effectively solve the problems of impurity removal, flow data correction, quality evaluation and flow data fitting of flow data, resulting in unstable compressed air quality detection of rail transit air supply units, affecting vehicle operation safety and efficiency.
The network uses the clutter type of traffic data combined with CNN and LSTM to determine the clutter type, establish a correction model through sensor data to correct the traffic data, and use filtering algorithms and curve fitting technology to perform data smoothing processing and quality evaluation to ensure the accuracy of the data. Sex and reliability.
It significantly improves the accuracy and efficiency of traffic data identification, reduces the impact of environmental factors on measurement results, provides a more accurate and stable traffic data foundation, and enhances the generalization ability and adaptability of the model.
Smart Images

Figure CN119025842A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of signal processing, and more particularly to a method for removing clutter from flow data. Background Art
[0002] In the daily operation of rail transit vehicles, the on-board air supply unit plays an indispensable and important role. Its core function is to provide a continuous and stable supply of compressed air to the interior of the car, which is essential for the normal operation of multiple key systems, including but not limited to the braking system, door control mechanism, and air suspension. These systems are highly dependent on compressed air, and any instability in the compressed air supply may directly affect the safety and operating efficiency of rail transit vehicles. Therefore, ensuring that the performance of the air supply unit is optimized and has extremely high operational reliability has become one of the core tasks to ensure the safe and smooth operation of rail transit.
[0003] Due to the complex and changeable working environment of rail transit, the air supply unit and the quality of the compressed air inside it face extremely severe tests. This includes not only extreme changes in temperature and high and low fluctuations in humidity, but also potential threats to air quality such as dust and pollutants. These factors may cause the performance of the air supply unit to decline, which in turn affects the operating status of the entire vehicle. Therefore, it is particularly important to conduct comprehensive and accurate testing on rail transit onboard air supply units to ensure that they can maintain optimal performance in various environments and conditions. In order to achieve this, the detection of gas flow data is particularly critical.
[0004] The invention patent with announcement number CN107860430B discloses a time difference measurement method for an ultrasonic gas flow meter based on the time difference method, including: 1) collecting ultrasonic echo signals X1(k) in the downstream direction and X2(k) in the upstream direction; 2) searching for effective maximum points of ultrasonic echo signals in the downstream and upstream directions, and selecting the time corresponding to the third maximum point of the effective maximum points of ultrasonic echo signals in the downstream and upstream directions as the propagation time of ultrasonic waves in the downstream and upstream directions from sending to receiving; 3) calculating the ultrasonic time difference in the downstream and upstream directions; 4) calculating the flow velocity vt of the fluid in the pipeline; 5) calculating the volume flow rate qv of the fluid in the pipeline.
[0005] Although the prior art can accurately obtain the propagation time difference of ultrasonic waves in the forward and reverse flow, has high stability, strong anti-noise interference, and can measure the gas flow to obtain accurate and stable gas flow measurement values, it still fails to solve the problems of impurity removal, flow data correction, quality assessment, and flow data fitting of flow data. Therefore, in order to overcome these limitations, the present invention proposes a flow data clutter removal method. Summary of the invention
[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method for removing clutter from flow data, which solves the problems of impurity removal, flow data correction, quality assessment and flow data fitting of flow data. The quality and flow data of compressed air are collected through the air supply unit and the sensor, and the type of clutter is identified by using the flow data clutter cause determination network combining CNN and LSTM, and the de-cluttering processing method is selected according to the clutter type for preliminary de-cluttering. Further, a correction model is established based on the sensor data, and the flow data is corrected to compensate for the measurement deviation caused by non-ideal environmental conditions. The flow data quality assessment checks whether the corrected flow data still contains impurities, and the flow data is smoothed and curve-fitted through a filtering algorithm.
[0007] To achieve the above object, the present invention provides the following technical solutions: A method for removing clutter from traffic data, comprising: Step S1: Start the air supply unit to allow compressed air to enter through the air inlet of the test equipment, obtain the mass of the compressed air discharged by the air supply unit through the sensor installed on the upper test equipment, and obtain flow data through the gas flow meter; Step S2: pre-process the collected traffic data, introduce a traffic data noise cause determination mechanism, identify the noise type of the traffic data, and perform preliminary noise removal; Step S3: for the initially cleaned flow data, a correction model is further established for the flow data according to the sensor data, and the flow data is corrected to compensate for the measurement deviation caused by non-ideal environmental conditions; Step S4: Perform flow data quality assessment to check whether the corrected flow data still contains impurities that have not been removed. If there are impurities that have not been removed, return to step S2; otherwise, execute step S5; Step S5: Smoothing the flow data through a filtering algorithm and performing curve fitting based on a detection and adjustment mechanism.
[0008] Specifically, the flow data noise cause determination mechanism includes: A traffic data clutter cause determination network is constructed by combining CNN and LSTM, including an input layer, a CNN layer, an LSTM layer, a fully connected layer, and an output layer. The input layer is used to receive traffic data, the CNN layer is used to extract local features and spatial dependencies in traffic data, the LSTM layer is used to extract the dependencies of the traffic data time series, and a fully connected layer is added after the LSTM layer. The fully connected layer is used to convert the output of the LSTM layer into a format suitable for the final classification task, and the output layer is used to identify clutter using an activation function.
[0009] Specifically, the process of establishing a correction model for flow data according to sensor data to correct the flow data includes: Collect sensor data and flow data during test equipment testing in real time, and synchronize the sensor data and flow data; The flow data is adjusted by the pressure correction coefficient. When the pressure sensor data is P, the flow data is expressed as:
[0010] in, is a normal number, is the raw traffic data, The flow data is adjusted by the pressure correction factor.
[0011] Specifically, the method of establishing a correction model for flow data according to sensor data to correct the flow data also includes: Combined with real-time temperature sensor data and pressure sensor data, continue to correct flow data, gas density The calculation formula is ,in, It's pressure. is the ideal gas constant, is the temperature, and the correction formula is:
[0012] in, is the flow data corrected for gas density using the ideal gas state equation, is the flow data after pressure correction, is the gas density under standard conditions, is the gas density under actual conditions.
[0013] Specifically, the flow data quality assessment is performed to check whether the corrected flow data still contains impurities that have not been removed, including: Compare the original flow data with the corrected flow data, calculate the correction strength, and evaluate the correction level based on the correction strength; For flow data with a weak correction level, spectrum analysis is used to identify whether there are undesired frequency components in the corrected flow data.
[0014] Specifically, the calculating of the correction strength and evaluating the correction level according to the correction strength include: Calculate the base correction intensity, ,in, is the basic calibration range, is the corrected flow data, It is the raw traffic data; The environmental impact factor E is introduced and expressed as a function of temperature and pressure, that is, ,in is the temperature change, is the pressure change, and is the adjustment factor; The basic correction intensity and environmental impact factor are dimensionlessly processed to calculate the correction intensity. The formula is as follows:
[0015] in, is the correction intensity, yes The mean of yes The mean of yes The variance of yes The variance of The correction level is evaluated according to the correction strength, and a correction threshold T1 is configured. If the correction strength is greater than the correction threshold, the correction level is a strong level, otherwise it is a weak level.
[0016] Specifically, the identifying whether the corrected flow data has undesired frequency components through spectrum analysis includes: The corrected flow data is transformed into the frequency domain by fast Fourier transform, that is:
[0017] in, is the nth flow data of the flow data in the time domain, is the kth frequency component in the frequency domain, is the total number of samples of flow data; The power spectrum is calculated based on the results of fast Fourier transform. The calculation method is as follows:
[0018] in, yes The amplitude of is extracted from the fast Fourier transform and normalized, that is, .
[0019] Specifically, the identifying whether the corrected flow data has undesired frequency components through spectrum analysis also includes: Set the amplitude threshold T2. The amplitude threshold is used to filter out normal background noise and interference. Frequency components exceeding the amplitude threshold are considered to be potential undesired frequencies. Screening for potential undesired frequencies; According to the power spectrum, we search for the local maximum value exceeding the threshold, and identify the peak by the difference and threshold comparison method, that is, ,and ,at the same time , count the number of peaks exceeding the threshold, and calculate the ratio of the number of peaks to the total frequency components, configure the critical value threshold T3, if the ratio exceeds T3, it is determined that the corrected flow data has undesired frequency components, otherwise it is determined that the corrected flow data does not have undesired frequency components.
[0020] Specifically, the flow data is smoothed by a filtering algorithm, and curve fitting is performed based on a detection adjustment mechanism, including: The flow data were smoothed using a filter and a preliminary curve fit was performed using a Gaussian curve fitting model; Configure the detection adjustment mechanism, determine the preliminary curve fitting adjustment plan through the adjustment mechanism, and obtain the final fitting curve.
[0021] Specifically, the detection and adjustment mechanism includes: Calculate the residuals of the preliminary curve fitting and draw a residual graph. Through the residual graph, determine whether the residuals conform to the random distribution. If the residuals show a non-random distribution, that is, there is a systematic deviation in the preliminary curve fitting, further perform a normality test on the residuals. If the residuals do not conform to the normal distribution, perform a heteroscedasticity test and an autocorrelation test. If the test shows that there are heteroscedasticity or autocorrelation problems, it is determined that the preliminary curve fitting needs to readjust the model; if the test shows that there are no heteroscedasticity or autocorrelation problems, it is determined that the preliminary curve fitting needs to adjust the parameter fitting method; If the residuals are randomly distributed, further goodness-of-fit tests are performed, including calculation of the coefficient of determination and adjustment of the coefficient of determination, and monitoring of the AIC and BIC information criteria. If the coefficient of determination and adjustment of the coefficient of determination are low, or the AIC and BIC values are high, it is determined that the preliminary curve fit needs to be readjusted; otherwise, the preliminary curve fit passes.
[0022] Beneficial effects of the present invention: 1. By introducing a traffic data clutter cause determination mechanism, combined with CNN and LSTM networks, the types of clutter in traffic data can be effectively identified and classified, including electromagnetic interference, mechanical vibration, etc. Compared with traditional manual or single algorithm processing, the intelligent clutter identification method significantly improves the accuracy and efficiency of clutter identification.
[0023] 2. A correction model is further established for the flow data after preliminary decontamination, and the flow data is corrected using sensor data to compensate for the measurement deviation caused by non-ideal environmental conditions. This not only reduces the impact of environmental factors on the measurement results, but also ensures the timeliness and accuracy of the correction by synchronizing sensor data and flow data in real time.
[0024] 3. Using filtering algorithms to smooth and curve fit the traffic data not only optimizes the visualization of traffic data, but also provides a more accurate and stable traffic data foundation for subsequent traffic data modeling and analysis. The design of the detection and adjustment mechanism, including calculating residuals, drawing residual graphs, performing normality tests, heteroscedasticity tests, and autocorrelation tests, enhances the generalization ability and practicality of the model, enabling the model to better adapt to different application scenarios and traffic data characteristics. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 A flow chart of a method for removing clutter from traffic data; Figure 2 Flowchart for flow data quality assessment; Figure 3 A flowchart for correction level determination; Figure 4 Flowchart for detection of undesired frequency components; Figure 5 Flow chart for curve fitting of flow data; Figure 6 Schematic diagram of Gaussian curve fitting; Figure 7 Schematic diagram for comparing parameter estimation methods for Gaussian curve fitting. DETAILED DESCRIPTION
[0026] This embodiment introduces a method for removing clutter from flow data. The specific scheme is as follows: Figure 1 As shown, a method for removing clutter from flow data, the method for removing clutter from flow data comprises: Step S1: Start the air supply unit to allow compressed air to enter through the air inlet of the test equipment, obtain the mass of the compressed air discharged by the air supply unit through the sensor installed on the test equipment, and obtain flow data through the gas flow meter; Step S2: pre-process the collected traffic data, introduce a traffic data noise cause determination mechanism, identify the noise type of the traffic data, and perform preliminary noise removal; Step S3: For the initially cleaned flow data, a correction model is further established for the flow data according to the sensor data to correct the flow data and compensate for the measurement deviation caused by non-ideal environmental conditions to ensure the consistency and comparability of the flow data under different environmental conditions; Step S4: Perform flow data quality assessment to check whether the corrected flow data still contains impurities that have not been removed. If there are impurities that have not been removed, return to step S2; otherwise, execute step S5; Step S5: Smooth the flow data through a filtering algorithm, and perform curve fitting based on a detection and adjustment mechanism to identify the trend of flow data changes and provide clear visual model support for analysis.
[0027] In this embodiment, the test equipment refers to an air supply unit test bench, and the test is carried out using a double tank flow method, wherein the double tank includes a buffer tank and a test tank. The compressed air discharged from the air supply unit enters the equipment through the air inlet of the test equipment, and a sensor is installed at the air inlet. The sensor is used to obtain the quality of the compressed air discharged from the air supply unit. The sensor includes a temperature sensor, a dew point sensor, a pressure sensor, etc. The air inlet is connected to a one-way valve, and the one-way valve is used to ensure that the compressed air does not flow back to the air supply unit. The one-way valve is connected to a buffer tank to ensure that the pulse wind generated by the piston air compressor is buffered in the buffer tank so that it does not produce a large amount of peak flow, thereby ensuring the stability and accuracy of the measurement. The buffer tank is connected to a gas flow meter, which is used to obtain flow data, and the gas flow meter can be adjusted according to the gas displacement to ensure that the gas flow meter itself does not become a limiting condition during the displacement test. The gas flow meter is connected to a test tank, and the test tank is used to ensure the pressure during the test process.
[0028] The flow data collected by the sensor and the gas flow meter are preprocessed to improve the quality and accuracy of the flow data. In addition, a flow data noise cause determination mechanism is introduced to identify the types of noise in the flow data, such as electromagnetic interference, signal attenuation, etc. According to the type of noise, the corresponding flow data noise removal processing method is selected. For example, for the noise caused by electromagnetic interference, a filter is used to remove it; for the noise caused by signal attenuation, a signal enhancement algorithm is used for processing. Through these methods, the flow data is preliminarily removed to eliminate noise and interference. By identifying and processing the types of noise, the characteristics of the flow data can be better understood, thereby providing more targeted guidance for subsequent flow data processing and analysis. The preliminary noise removal processing can also improve the efficiency of flow data processing and reduce the calculation amount and time cost of subsequent processing.
[0029] Correct the flow data based on the sensor data, compare the sensor readings with the flow data under known standard or benchmark conditions, and establish a correction model to compensate for the measurement deviation caused by non-ideal environmental conditions, including adjusting the flow data based on real-time pressure flow data to compensate for the impact of pressure changes because the compression of compressed air changes with pressure; calculate the gas density using the ideal gas state equation combined with temperature and pressure flow data, and then correct the flow data to ensure that the flow data reflects the actual gas volume flow; Perform a flow data quality assessment to check whether the corrected flow data still contains impurities that have not been removed, such as outliers, system errors, or non-random noise. If there are impurities that have not been removed, return to step S2 and review the flow data processing process to find out the reason for insufficient filtering, which may be improper selection of filtering parameters or the algorithm itself is not suitable for the current flow data characteristics. Then adjust the strategy and re-perform filtering. Otherwise, execute step S5.
[0030] Apply filtering algorithms, including Kalman filtering, sliding average filtering or median filtering, to smooth the flow data, eliminate random noise and short-term fluctuations in the flow data, and make the flow data show a more realistic and stable long-term trend. Perform curve fitting on the filtered flow data, and select mathematical models to fit the flow data to understand the inherent change law of the flow data from a mathematical and visual level. By building a mathematical model, the original flow data points are matched, and the potential relationship between the flow data can be captured to identify the trend of flow data changes. Such trends may include periodic fluctuations, long-term growth or decline trends, and sudden peaks or valleys. In order to achieve the best visual effect and analysis accuracy, the smoothness of the fitting curve can also be adjusted to ensure that the curve is neither overfitted to lose the response to the real changes, nor underfitted to ignore the important structure in the flow data. Providing clear visual model support for analysis helps decision makers quickly grasp the core information of flow data, such as peak and valley periods of flow, growth or decline trends, and abnormal change points.
[0031] The flow data noise cause determination mechanism includes: Collect historical test flow data of the test equipment, including sensor data and flow data, remove abnormal points and erroneous flow data points, mark the flow data, and distinguish between normal flow data and flow data containing noise. The types of noise flow data include electromagnetic interference, mechanical vibration, environmental changes, air pressure fluctuations, and signal attenuation; Each set of test flow data is divided into n time series segments by sliding window technology, and each time series segment contains m consecutive measurement points; the divided test flow data is divided into training set, validation set and test set; Combine CNN and LSTM to build a traffic data clutter cause determination network, including input layer, CNN layer, LSTM layer, fully connected layer and output layer. The input layer is used to receive traffic data, the CNN layer is used to extract local features and spatial dependencies in traffic data, and the LSTM layer is used to extract the dependencies of the traffic data time series. A fully connected layer is added after the LSTM layer. The fully connected layer is used to convert the output of the LSTM layer into a format suitable for the final classification task. The output layer is used to identify clutter using an activation function. The traffic data clutter cause determination network is trained using the training set. During the training process, the network weights are iteratively adjusted through the back-propagation algorithm and optimizer to minimize the loss function. The traffic data in the training set is used to teach the model to identify different types of clutter and their characteristics. The model is verified using the validation set to evaluate the performance of the model on unseen traffic data. The model's performance indicators, including accuracy, recall, and F1 score, are monitored, and model parameters, including learning rate, batch size, and network architecture, are adjusted based on these indicators. After training and verification, the model is finally evaluated using the test set to ensure the accuracy and reliability of the model in actual scenarios.
[0032] The method of establishing a correction model for flow data according to sensor data to correct the flow data includes: Real-time collection of sensor data and flow data during test equipment testing. Sensor data includes temperature sensors and pressure sensors, and time synchronization of sensor data and flow data is performed. Real-time collection and time synchronization ensure the temporal consistency of sensor data and flow data, thus providing an accurate and reliable basis for subsequent flow data analysis and correction. Real-time flow data collection enables timely response to environmental changes, provides continuous real-time feedback, and enhances timeliness. Time synchronization helps to quickly identify and eliminate faults and delays in flow data collection, and ensures the integrity and accuracy of flow data. The flow data is adjusted by the pressure correction coefficient. The volume flow of compressed air is inversely proportional to the pressure, that is, when the pressure increases, the volume of gas flowing per unit time decreases; when the pressure decreases, the volume of gas flowing per unit time increases. When the pressure sensor data is P, the flow data is expressed as:
[0033] in, It is a positive constant used to adjust the sensitivity of pressure to gas flow. When the value is large, the effect of pressure change on flow is more significant; when the value is small, the effect of pressure change on flow is small. is the raw traffic data, The flow data is adjusted by the pressure correction coefficient. The introduction of the pressure correction coefficient enables the flow data to dynamically reflect the impact of pressure changes, improving the accuracy and adaptability of the measurement. The value can flexibly control the sensitivity of flow data to pressure changes to meet the needs of different experimental conditions and application scenarios. In an environment with large pressure fluctuations, the pressure correction factor can reduce the fluctuation of flow data and enhance the stability and reliability of flow data.
[0034] Combined with real-time temperature sensor data and pressure sensor data, the ideal gas state equation is used to calculate the gas density, and the flow data and gas density are corrected. The calculation formula is ,in, It's pressure. is the ideal gas constant, is the temperature, and the correction formula is:
[0035] in, is the flow data corrected for gas density using the ideal gas state equation, is the flow data after pressure correction, is the gas density under standard conditions, It is the gas density under actual conditions; taking into account the effects of temperature, pressure and gas density, a comprehensive correction of flow data is achieved, which significantly improves the accuracy of flow data. By comparing the gas density under actual conditions with the gas density under standard conditions, the standardized processing of flow data is achieved, ensuring the comparability of flow data under different conditions. The correction process effectively compensates for the measurement errors caused by changes in environmental conditions, ensuring the consistency and reliability of flow data.
[0036] like Figure 2 As shown, the flow data quality assessment is performed to check whether the corrected flow data still contains impurities that have not been removed, including: Step S4.1: Compare the original flow data with the corrected flow data, calculate the correction strength, and evaluate the correction level according to the correction strength; Step S4.2: for the flow data with a weak correction level, identify whether there are undesired frequency components in the corrected flow data through spectrum analysis; In this embodiment, by accurately quantifying the difference before and after correction, it is ensured that any possible flow data deviation is systematically recorded and analyzed, and the transparency and traceability of flow data correction are enhanced. In combination with environmental influencing factors, the calculation formula of correction intensity is updated to ensure that the comprehensive impact of various factors on flow data correction in the actual environment can be more accurately reflected. The correction model of environmental factors improves the adaptability and accuracy of flow data under different operating conditions, and through standardized processing, it ensures that flow data from different sources are compared on a unified scale, which improves the accuracy and consistency of flow data analysis. Through the binary correction level division, abnormal flow data that needs further attention can be quickly and effectively identified, and resource allocation is optimized to make flow data analysis more efficient.
[0037] The flow data with weak correction level is subjected to spectrum analysis to further check whether it contains unexpected frequency components. The flow data in the time domain is converted into frequency domain signals through fast Fourier transform, thereby revealing the energy distribution of various frequency components in the flow data. The power spectrum is calculated to quantitatively describe the amplitude of each frequency component, and the amplitude threshold T2 is set to identify and filter out potential unexpected frequency spikes. Spectral analysis not only significantly improves the monitoring ability of the quality of the corrected flow data, but also provides a systematic evaluation method for the correction effect of flow data. It ensures the accuracy and reliability of flow data and supports more accurate flow data interpretation and decision making.
[0038] like Figure 3 As shown, the calculation of the correction intensity and the evaluation of the correction level according to the correction intensity include: Calculate the base correction intensity, ,in, is the basic calibration range, The corrected flow data is, It is the original flow data, which reflects the basic change range of flow data before and after correction through the basic correction intensity; The environmental impact factor E is introduced, and the basic correction intensity is corrected according to the environmental impact factor. The environmental impact factor is expressed as a function of temperature and pressure, that is, ,in is the temperature change, is the pressure change, and It is the adjustment coefficient, which more accurately reflects the impact of factors in the actual environment on the flow data correction by introducing environmental factor variables; The basic correction intensity and environmental impact factor are dimensionlessly processed to calculate the correction intensity. The formula is as follows:
[0039] in, is the correction intensity, yes The mean of yes The mean of yes The variance of yes The variance of ensures that the correction strength reflects the changes in the flow data itself, takes into account the influence of environmental factors, and maintains dimensional consistency; The correction level is evaluated according to the correction strength, and the correction threshold T1 is configured. If the correction strength is greater than the correction threshold, the correction level is a strong level, otherwise it is a weak level, which improves the accuracy of flow data quality assessment and enhances the reliability and practicality of the assessment results; like Figure 4 As shown, the method of identifying whether the corrected flow data has undesired frequency components through spectrum analysis includes: The corrected flow data is transformed into the frequency domain by fast Fourier transform, that is:
[0040] in, is the nth flow data of the flow data in the time domain, is the kth frequency component in the frequency domain, and N is the total number of samples of flow data; The power spectrum is calculated based on the results of fast Fourier transform to reflect the energy distribution of each frequency component and identify abnormal frequencies. The calculation method is as follows:
[0041] in, yes The amplitude of is extracted from the fast Fourier transform and normalized, that is, , to reflect the real energy distribution and ensure the consistency and comparability of the amplitude; Set the amplitude threshold T2. The amplitude threshold is used to filter out normal background noise and interference in order to highlight the undesired frequency peaks. The frequency components exceeding the amplitude threshold are considered as potential undesired frequencies. Screening for potential undesired frequencies; According to the power spectrum, we search for the local maximum value exceeding the threshold, and identify the peak by the difference and threshold comparison method, that is, ,and ,at the same time , count the number of peaks exceeding the threshold, and calculate the ratio of the number of peaks to the total frequency components, configure the critical value threshold T3, if the ratio exceeds T3, it is determined that the corrected flow data has undesired frequency components, otherwise it is determined that the corrected flow data does not have undesired frequency components; like Figure 5 As shown, the flow data is smoothed by a filtering algorithm, and curve fitting is performed based on a detection adjustment mechanism, including: Step S5.1: smooth the flow data using a filter and perform preliminary curve fitting using a Gaussian curve fitting model; Step S5.2: configuring a detection adjustment mechanism, determining a preliminary curve fitting adjustment scheme through the adjustment mechanism, and obtaining a final fitting curve; In this embodiment, a filter is used to smooth the flow data. A low-pass filter can be used to remove the peak interference signal, and a median filter can be used to regularize the scattered and fluctuating flow. Figure 6 As shown, a preliminary curve fitting is performed using a Gaussian curve fitting model to fit the flow data to a Gaussian curve described by the following equation:
[0042] in, is the input sequence, is the amplitude, is the center, is the standard deviation, is the offset. For Gaussian noise, the residual is minimized according to the least squares method, that is, , where N is the total number of samples of traffic data, It is weights, It is The result of Gaussian fitting is It is The corrected flow data can be used for parameter estimation; the parameter estimation method can also select the minimum absolute residual and Bisquare fitting method, etc. The Bisquare fitting method adopts the strategy of iterative reweighted least squares. The results of the above parameter estimation methods are compared as follows Figure 7 As shown, in most cases, the Bisquare method is not as sensitive to numbers outside the interval as the least absolute residual method.
[0043] Configure the detection and adjustment mechanism, determine the preliminary curve fitting adjustment plan by analyzing the residuals of the model, performing normality tests, heteroscedasticity tests, autocorrelation tests and other diagnostic tools, identify the reasons for insufficient model fitting through the detection and adjustment mechanism, and select a more appropriate model structure or adjust the model parameter estimation method to refit. Through a series of detection and adjustment steps, obtain a final fitting curve with good statistical characteristics that can accurately capture the characteristics of traffic data.
[0044] The detection and adjustment mechanism includes: Calculate the residuals of the preliminary curve fitting and draw a residual graph. Through the residual graph, determine whether the residuals conform to the random distribution. If the residuals show a non-random distribution, that is, there is a systematic deviation in the preliminary curve fitting, further perform a normality test on the residuals. If the residuals do not conform to the normal distribution, perform a heteroscedasticity test and an autocorrelation test. If the test shows that there are heteroscedasticity or autocorrelation problems, it is determined that the preliminary curve fitting needs to readjust the model; if the test shows that there are no heteroscedasticity or autocorrelation problems, it is determined that the preliminary curve fitting needs to adjust the parameter fitting method; If the residuals are randomly distributed, further goodness-of-fit tests are performed, including calculation of the coefficient of determination and adjustment of the coefficient of determination, and monitoring of the AIC and BIC information criteria. If the coefficient of determination and adjustment of the coefficient of determination are low, or the AIC and BIC values are high, it is determined that the preliminary curve fit needs to be readjusted; otherwise, the preliminary curve fit passes.
[0045] In this embodiment, the predicted value is calculated using the preliminary curve fitting model, and compared with the actual observed value to obtain the residual, and the residual graph is drawn, including a scatter plot of the residual and the fitted value and a time series graph of the residual, so as to observe the distribution and pattern of the residual from different angles, observe the residual graph, check whether the residual fluctuates randomly around zero, without obvious trend or periodic pattern, use Durbin-Watson statistic to detect autocorrelation in the residual, and help confirm the randomness of the residual; if the residual is not randomly distributed, perform Shapiro-Wilk or Kolmogorov-Smirnov normality test to determine whether the residual obeys normal distribution, and the normality test result helps to determine whether the flow data conversion or non-parametric method is needed to improve the model, if the residual does not conform to normal distribution, use Breusch-Pagan or Goldfeld-Quandt test to check heteroscedasticity test, and use Ljung-Box Q test to check autocorrelation. Guide whether more complex modeling techniques such as weighted least squares or time series model are needed. If the residuals are randomly distributed, calculate the coefficient of determination and the adjusted coefficient of determination to assess the proportion of variability explained by the model. Monitor the AIC and BIC information criteria to account for model complexity and to compare the goodness of fit of different models. If the coefficient of determination and the adjusted coefficient of determination are low, or the AIC and BIC values are high, it indicates that the model may be overly complex or not fully capturing the characteristics of the flow data, and the model structure or variables need to be readjusted. If the model passes all tests and the goodness of fit indices are satisfactory, the preliminary curve fit is considered valid.
[0046] The adjustment parameter fitting method comprises: According to the results of residual analysis, the optimization algorithm of model parameters is adjusted, including the use of nonlinear least squares method, genetic algorithm or simulated annealing algorithm to find a better combination of parameters; After adjusting the parameters, model diagnostics were performed again, including residual analysis, normality test, heteroskedasticity test, and autocorrelation test, to ensure that the performance of the new model was improved; Recalculate the AIC and BIC information criteria and compare them with the preliminary model to ensure that the adjusted model achieves a better balance between explanatory power and complexity; Iterate the optimization and repeat the above steps to configure the convergence threshold T4 until the variation range of the AIC and BIC values of the fitting model is less than the convergence threshold T4, then output the final fitting curve.
[0047] In this embodiment, according to the results of residual analysis, a suitable parameter optimization algorithm is selected. The preliminary model uses the basic least squares method, and the residual analysis shows that the model has room for improvement. Then, more efficient optimization techniques such as nonlinear least squares, genetic algorithm or simulated annealing algorithm are considered to search the parameter space more comprehensively and find a better combination of parameters; after adjusting the parameters, the model diagnosis is performed again, including residual analysis, normality test, heteroscedasticity test and autocorrelation test. To confirm whether the new model solves the problems existing in the preliminary model, including the residual of random distribution, the satisfaction of normal distribution, heteroscedasticity test and autocorrelation problem; recalculate information criteria such as AIC and BIC, and compare them with the results of the preliminary model. These information criteria help evaluate the balance between the explanatory power and complexity of the model. Lower AIC and BIC values usually indicate that the model has better goodness of fit and a simpler structure; through the iterative optimization process, the model parameters are gradually adjusted. Set the convergence threshold T4, when the range of variation of the AIC and BIC values of the fitting model is less than the convergence threshold T4, it is considered that the model has reached the termination condition of the optimization. At this time, the final fitting curve is output; the final fitting curve will significantly improve the prediction accuracy and reliability of the model. The optimized model will better capture the key features and potential trends in the traffic data while reducing the impact of outliers and noise.
[0048] The recalibration model includes: Select the refit model, which includes nonlinear regression model, generalized additive model, mixed effect model and machine learning algorithm, and perform model estimation and model diagnosis, and continue to execute step S5.2 until the final fitting curve is output.
[0049] Working principle and effect: The quality and flow data of compressed air are obtained through sensors installed on the test equipment. The flow data noise cause determination mechanism is introduced. The network constructed by combining CNN and LSTM can identify the noise type in the flow data. The CNN layer extracts local features and spatial dependencies, while the LSTM layer captures the dependencies of the time series, providing accurate classification for subsequent de-cluttering processing. A correction model is established based on sensor data to correct the flow data and compensate for the measurement deviation caused by non-ideal environmental conditions, including pressure correction and temperature correction, to ensure the authenticity and accuracy of the flow data. The flow data quality is evaluated to check whether the corrected flow data still contains impurities that have not been removed. If so, return to the de-cluttering processing step; otherwise, proceed to the next step of flow data smoothing and curve fitting; use the filtering algorithm to smooth the flow data, remove interference signals, and use the Gaussian curve fitting model to perform curve fitting. This step improves the smoothness and goodness of fit of the flow data. Configure the detection and adjustment mechanism to determine whether the preliminary curve fitting needs to readjust the model or adjust the parameter fitting method by calculating residuals, drawing residual graphs, and performing normality tests to ensure the accuracy of the final fitting curve.
[0050] Through accurate clutter identification and effective clutter removal processing methods, the noise and interference in the flow data are significantly reduced, and the accuracy and reliability of the flow data are improved. The application of the flow data correction model further compensates for the influence of environmental factors, making the corrected flow data closer to the true value and enhancing the quality of the flow data. A closed-loop processing flow is formed, from flow data collection to preprocessing, correction, quality assessment, smoothing and curve fitting, each step is closely connected, improving the efficiency and automation level of flow data processing. The application of the detection and adjustment mechanism ensures the accuracy of curve fitting, avoids duplication of work caused by improper models or parameters, and further improves processing efficiency. The combination of CNN and LSTM enables the flow data clutter cause determination network to have stronger feature extraction and time series analysis capabilities, enhances the model's ability to identify different types of clutter, and improves the generalization ability of the model. The setting of the flow data quality assessment and adjustment mechanism enables the entire method to adapt to variable operating conditions and environmental factors, and improves the reliability and effectiveness of the model in practical applications.
[0051] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. A method for removing clutter from flow data, characterized in that: include: Step S1: Start the air supply unit to allow compressed air to enter through the air inlet of the test equipment, obtain the mass of the compressed air discharged by the air supply unit through the sensor installed on the test equipment, and obtain flow data through the gas flow meter; Step S2: pre-process the collected traffic data, introduce a traffic data noise cause determination mechanism, identify the noise type of the traffic data, and perform preliminary noise removal; Step S3: for the initially cleaned flow data, a correction model is further established for the flow data according to the sensor data, and the flow data is corrected to compensate for the measurement deviation caused by non-ideal environmental conditions; Step S4: Perform flow data quality assessment to check whether the corrected flow data still contains impurities that have not been removed. If there are impurities that have not been removed, return to step S2; otherwise, execute step S5; Step S5: Smoothing the flow data through a filtering algorithm and performing curve fitting based on a detection and adjustment mechanism.
2. A method for removing clutter from flow data as claimed in claim 1, characterized in that: The flow data noise cause determination mechanism includes: A traffic data clutter cause determination network is constructed by combining CNN and LSTM, including an input layer, a CNN layer, an LSTM layer, a fully connected layer, and an output layer. The input layer is used to receive traffic data, the CNN layer is used to extract local features and spatial dependencies in traffic data, the LSTM layer is used to extract the dependencies of the traffic data time series, and a fully connected layer is added after the LSTM layer. The fully connected layer is used to convert the output of the LSTM layer into a format suitable for the final classification task, and the output layer is used to identify clutter using an activation function.
3. A method for removing clutter from flow data as claimed in claim 1, characterized in that: The method of establishing a correction model for flow data according to sensor data to correct the flow data includes: Collect sensor data and flow data during test equipment testing in real time, and synchronize the sensor data and flow data; The flow data is adjusted by the pressure correction coefficient. When the pressure sensor data is P, the flow data is expressed as: ,in, is a normal number, is the raw traffic data, The flow data is adjusted by the pressure correction factor.
4. A method for removing clutter from flow data as claimed in claim 1, characterized in that: The method of establishing a correction model for the flow data according to the sensor data to correct the flow data also includes: Combined with real-time temperature sensor data and pressure sensor data, continue to correct flow data, gas density The calculation formula is ,in, It's pressure. is the ideal gas constant, is the temperature, and the correction formula is: ,in, is the flow data corrected for gas density using the ideal gas state equation, is the flow data after pressure correction, is the gas density under standard conditions, is the gas density under actual conditions.
5. A method for removing clutter from flow data as claimed in claim 1, characterized in that: The flow data quality assessment is performed to check whether the corrected flow data still contains impurities that have not been removed, including: Compare the original flow data with the corrected flow data, calculate the correction strength, and evaluate the correction level based on the correction strength; For flow data with a weak correction level, spectrum analysis is used to identify whether there are undesired frequency components in the corrected flow data.
6. A method for removing clutter from flow data as claimed in claim 5, characterized in that: The calculating of the correction strength and evaluating the correction level according to the correction strength include: Calculate the base correction intensity, ,in, is the basic calibration range, is the corrected flow data, It is the raw traffic data; Introducing environmental impact factors , the environmental impact factor is expressed as a function of temperature and pressure, that is ,in is the temperature change, is the pressure change, and is the adjustment factor; The basic correction intensity and environmental impact factor are dimensionlessly processed to calculate the correction intensity. The formula is as follows: ,in, is the correction intensity, yes The mean of yes The mean of yes The variance of yes The variance of The correction level is evaluated according to the correction strength, and a correction threshold T1 is configured. If the correction strength is greater than the correction threshold, the correction level is a strong level, otherwise it is a weak level.
7. A method for removing clutter from flow data as claimed in claim 5, characterized in that: The method of identifying whether the corrected flow data has undesired frequency components by spectrum analysis includes: The corrected flow data is transformed into the frequency domain by fast Fourier transform, that is: ,in, is the nth flow data of the flow data in the time domain, is the kth frequency component in the frequency domain, and N is the total number of samples of flow data; The power spectrum is calculated based on the results of fast Fourier transform. The calculation method is as follows: ,in, yes The amplitude of is extracted from the fast Fourier transform and normalized, that is, .
8. A method for removing clutter from flow data as claimed in claim 5, characterized in that: The method of identifying whether the corrected flow data has undesired frequency components by spectrum analysis also includes: Set the amplitude threshold T2. The amplitude threshold is used to filter out normal background noise and interference. Frequency components exceeding the amplitude threshold are considered to be potential undesired frequencies. Screening for potential undesired frequencies; According to the power spectrum, we search for the local maximum value exceeding the threshold, and identify the peak by the difference and threshold comparison method, that is, ,and ,at the same time , count the number of peaks exceeding the threshold, and calculate the ratio of the number of peaks to the total frequency components, configure the critical value threshold T3, if the ratio exceeds T3, it is determined that the corrected flow data has undesired frequency components, otherwise it is determined that the corrected flow data does not have undesired frequency components.
9. A method for removing clutter from flow data as claimed in claim 1, characterized in that: The flow data is smoothed by a filtering algorithm and curve fitting is performed based on a detection adjustment mechanism, including: The flow data were smoothed using a filter and a preliminary curve fit was performed using a Gaussian curve fitting model; Configure the detection adjustment mechanism, determine the preliminary curve fitting adjustment plan through the adjustment mechanism, and obtain the final fitting curve.
10. A method for removing clutter from flow data as claimed in claim 1, characterized in that: The detection and adjustment mechanism includes: Calculate the residuals of the preliminary curve fitting and draw a residual graph. Through the residual graph, determine whether the residuals conform to the random distribution. If the residuals show a non-random distribution, that is, there is a systematic deviation in the preliminary curve fitting, further perform a normality test on the residuals. If the residuals do not conform to the normal distribution, perform a heteroscedasticity test and an autocorrelation test. If the test shows that there are heteroscedasticity or autocorrelation problems, it is determined that the preliminary curve fitting needs to readjust the model; if the test shows that there are no heteroscedasticity or autocorrelation problems, it is determined that the preliminary curve fitting needs to adjust the parameter fitting method; If the residuals are randomly distributed, further goodness-of-fit tests are performed, including calculation of the coefficient of determination and adjustment of the coefficient of determination, and monitoring of the AIC and BIC information criteria. If the coefficient of determination and adjustment of the coefficient of determination are low, or the AIC and BIC values are high, it is determined that the preliminary curve fit needs to be readjusted; otherwise, the preliminary curve fit passes.
Citation Information
Patent Citations
A Time Difference Measurement Method for Ultrasonic Gas Flow Meters Based on Time Difference
CN107860430B
Intelligent gas flow monitoring method
CN117606578A
Automobile pressure sensor data acquisition and calibration method
CN117804673A
Automatic pulse wave quality evaluation system in oscillography blood pressure measurement based on multiple indexes
CN117918805A
Data optimization method and system of ultrasonic flowmeter
CN118689866A
