Probability strategy-based AI modeling data cleaning and obtaining method and system

Abstract

The invention discloses an AI modeling data cleaning and obtaining method and system based on a probability strategy, and relates to the technical field of industrial time series data processing and artificial intelligence modeling, and the method comprises the steps: obtaining target time series data; constructing a time sequence statistical model containing a trend term; performing parameter estimation on the time sequence statistical model based on a probability strategy, and calculating a probability weight corresponding to each data point; performing weighted modeling on the time sequence statistical model to obtain cleaned time sequence representation; based on the cleaned time sequence representation, trend characteristic parameters reflecting the system operation state corresponding to the target time sequence data are extracted; according to the trend characteristic parameters, the dynamic state and the steady state of the target time sequence data are judged, cleaned dynamic data and cleaned steady data are obtained, and classified acquisition of the dynamic data and the steady data of the cleaned data is achieved. The precision, efficiency and adaptability of data processing can be improved.

Description

Technical Field

[0001] This application relates to the fields of industrial time series data processing and artificial intelligence modeling technology, and in particular to a method and system for AI modeling data cleaning and acquisition based on probabilistic strategies. Background Technology

[0002] In artificial intelligence modeling, data quality directly determines the model's performance and reliability. Data cleaning and classification of dynamic and steady-state data are crucial preliminary steps in building high-quality AI models. However, current data cleaning and state classification methods mainly include linear regression, moving average, and threshold detection. While these methods can identify some outliers and steady-state segments to a certain extent, they rely heavily on manually set thresholds or prior knowledge, resulting in low data processing efficiency. Furthermore, these methods cannot accurately distinguish between valid and outlier data, nor can they precisely differentiate between dynamic and steady-state data, leading to low data processing accuracy. In addition, these methods lack applicability and are insufficient to meet the processing needs of different fields and types of time series data.

[0003] Therefore, how to improve the accuracy, efficiency, and adaptability of data processing to provide a high-quality data foundation for artificial intelligence modeling has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for AI modeling data cleaning and acquisition based on probabilistic strategies, which can improve the accuracy, efficiency and adaptability of data processing.

[0005] To achieve the above objectives, this application provides the following solution.

[0006] In a first aspect, this application provides a method for cleaning and acquiring AI modeling data based on a probabilistic strategy, which includes the following steps.

[0007] Obtain the target time series data.

[0008] Based on the target time series data, a time series statistical model containing a trend term is constructed; the time series statistical model is used to describe the trend characteristics and random disturbance characteristics of different data points over time.

[0009] The time series statistical model is used to estimate parameters based on a probabilistic strategy, and the probability weights corresponding to each data point are calculated.

[0010] Based on the probability weights corresponding to each data point, the time series statistical model is weighted and modeled so that outlier data points are adaptively weakened during the model parameter estimation process, resulting in a cleaned time series representation.

[0011] Based on the cleaned time series representation, trend feature parameters reflecting the system operating status corresponding to the target time series data are extracted.

[0012] Based on the trend feature parameters, the target time series data is judged to be in dynamic and steady state, and the cleaned dynamic data and cleaned steady state data are obtained, thereby realizing the classification and acquisition of the cleaned dynamic and steady state data.

[0013] Optionally, based on the target time series data, a time series statistical model containing a trend term is constructed, specifically including the following steps.

[0014] The target time series data is segmented using a sliding window method to obtain several time windows.

[0015] Based on each of the aforementioned time windows, a time series statistical model containing a trend term is constructed.

[0016] Optionally, the time series statistical model is parameter estimated based on a probabilistic strategy, and the probability weights corresponding to each data point are calculated, specifically including the following steps.

[0017] Based on the degree to which each data point deviates from the time series statistical model, the anomaly probability or confidence probability corresponding to each data point is calculated respectively.

[0018] Based on the anomaly probability or confidence probability corresponding to each data point, the probability weight of each data point in the model parameter estimation process is determined.

[0019] Optionally, the probability weights satisfy the following characteristics.

[0020] The smaller the deviation of any data point from the time series statistical model, the greater the probability weight of that data point.

[0021] The greater the deviation of any data point from the time series statistical model, the smaller the probability weight of that data point.

[0022] Optionally, the trend characteristic parameters include the trend slope parameter and its corresponding statistical significance index.

[0023] Based on the trend feature parameters, the target time series data is judged to be in dynamic and steady state, and the cleaned dynamic data and cleaned steady state data are obtained. The classification and acquisition of the cleaned dynamic data and steady state data are realized, which specifically includes the following steps.

[0024] Based on the trend slope parameter and its corresponding statistical significance index, it is determined whether the target time series data shows a significant trend within the time window.

[0025] When the target time series data does not show a significant trend of change within the time window, the corresponding data is determined to be steady-state data, and the cleaned steady-state data is obtained.

[0026] When the target time series data shows a significant trend of change within the time window, the corresponding data is determined to be dynamic data, and the cleaned dynamic data is obtained.

[0027] Optionally, the cleaned dynamic data is used to construct a dynamic artificial intelligence model, which includes a recurrent neural network model, a long short-term memory network model, or other artificial intelligence models with dynamic data processing capabilities.

[0028] The cleaned steady-state data is used to construct a steady-state artificial intelligence model, which includes a support vector regression model, a multilayer perceptron model, a Gaussian process model, or other artificial intelligence models with steady-state data processing capabilities.

[0029] Optionally, in the sliding window method, the window size and step size of the sliding window are adaptively adjusted or manually set according to the characteristics of the target time series data and the actual application scenario.

[0030] Optionally, the target time series data includes at least one of the following: data collected during industrial operations, time series data from the financial sector, time series data from environmental monitoring, and time series data from medical monitoring.

[0031] The data collected during the industrial operation includes at least one of temperature, pressure, flow rate, liquid level, concentration, voltage, and current.

[0032] Optionally, when estimating the parameters of the time series statistical model, a likelihood function combined with an iterative optimization algorithm is used, wherein the iterative optimization algorithm includes at least one of the EM algorithm, gradient descent algorithm, and Newton's iteration algorithm.

[0033] Secondly, this application provides a system for cleaning and acquiring AI modeling data based on probabilistic strategies. The system is used to implement the method for cleaning and acquiring AI modeling data based on probabilistic strategies as described in any of the first aspects. The system includes the following modules.

[0034] The data acquisition module is used to acquire target time series data.

[0035] The time series statistical model construction module is used to construct a time series statistical model containing a trend term based on the target time series data; the time series statistical model is used to describe the trend characteristics and random disturbance characteristics of different data points over time.

[0036] The parameter estimation module is used to estimate the parameters of the time series statistical model based on a probability strategy and calculate the probability weights corresponding to each data point.

[0037] The weighted modeling and cleaning module is used to perform weighted modeling on the time series statistical model according to the probability weights corresponding to each data point, so that abnormal data points are adaptively weakened during the model parameter estimation process, and a cleaned time series representation is obtained.

[0038] The trend feature parameter extraction module is used to extract trend feature parameters that reflect the system operating status corresponding to the target time series data based on the cleaned time series representation.

[0039] The dynamic and steady-state determination module is used to determine the dynamic and steady-state states of the target time series data based on the trend feature parameters, so as to obtain the cleaned dynamic data and the cleaned steady-state data, thereby realizing the classification and acquisition of the dynamic and steady-state data of the cleaned data.

[0040] According to the specific embodiments provided in this application, this application has the following technical effects.

[0041] This application provides a method and system for AI modeling data cleaning and acquisition based on probabilistic strategies. By calculating the probability weights of data points through probabilistic strategies, outlier data points are adaptively weakened during model parameter estimation, eliminating the need for manually setting thresholds or labeling outlier locations. This solves the problems of traditional methods relying on prior knowledge and having poor adaptability, enabling it to adapt to complex anomaly scenarios (such as heavy-tailed anomalies, transient disturbances, and spike noise) in time series data from various fields such as industry and finance, thereby improving the adaptability of data processing. Furthermore, unlike the crude method of simply removing outliers, weighted modeling, while suppressing anomaly interference, can completely preserve the true trend of the target time series (such as industrial operating condition changes and financial market trend fluctuations), ensuring both the smoothness and continuity of the cleaned data and the preservation of key dynamic features, thus ensuring the physical rationality and practical value of the data. In addition, the time series statistical model simultaneously characterizes trend features and random disturbance characteristics. Combined with parameter estimation using probabilistic strategies, the model has strong fault tolerance to complex interferences such as sensor noise, acquisition errors, and environmental disturbances. Even in high-noise, multi-anomaly raw data, it can output high-quality cleaned results, improving the accuracy of data processing. Furthermore, this application determines the state based on trend feature parameters extracted from the cleaned data, relying entirely on the statistical characteristics of the data and abandoning the empirical threshold dependence of traditional methods. This fundamentally improves the objectivity and consistency of dynamic and steady-state classification, reduces subjective errors from human judgment, and thus effectively improves the accuracy of data processing. By seamlessly integrating data cleaning with dynamic and steady-state classification, no additional independent processing steps are required. State classification is achieved directly while anomaly suppression is completed, significantly shortening the data preprocessing cycle and adapting to the high-efficiency processing needs of large-scale time series data. This saves time costs for artificial intelligence modeling and improves data processing efficiency. In addition, this application does not require adjustments to the core logic for specific domain data or model types. It is widely applicable to time series data in various fields, enabling efficient cleaning and accurate classification. This provides a unified and reliable data processing solution for multi-domain data-driven artificial intelligence modeling. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is an application environment diagram of a method for AI modeling data cleaning and acquisition based on a probabilistic strategy, provided in one embodiment of this application.

[0044] Figure 2This is a flowchart illustrating a method for cleaning and acquiring AI modeling data based on a probabilistic strategy, provided in one embodiment of this application.

[0045] Figure 3 This is a schematic diagram of a method for cleaning and acquiring AI modeling data based on a probabilistic strategy, provided in an embodiment of this application.

[0046] Figure 4 This is a schematic diagram of the actual logistics flow data of a factory delayed coking process provided in an embodiment of this application.

[0047] Figure 5 A schematic diagram of the data after adding outliers to the logistics flow data of the delayed coking process in a factory according to an embodiment of this application.

[0048] Figure 6 This is a schematic diagram of time series data provided in an embodiment of this application.

[0049] Figure 7 This is a schematic diagram of the data cleaning results of a probabilistic robust model provided in an embodiment of this application.

[0050] Figure 8 This is a schematic diagram of the steady-state classification result of a probabilistic robust model provided in an embodiment of this application.

[0051] Figure 9 This is a schematic diagram of the structure of a system for AI modeling data cleaning and acquisition based on a probabilistic strategy, provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0053] Currently, time series data collected in fields such as industrial production, financial transactions, environmental monitoring, and medical health often suffer from numerous problems, such as sensor noise, environmental disturbances, acquisition equipment errors, and abnormal burst data. These problems result in a large amount of invalid information and interference factors in the raw data. If directly used for AI modeling, it will seriously affect the training effect of the model, leading to a decrease in model prediction accuracy and insufficient generalization ability. Furthermore, the existing data cleaning and state classification methods mainly include linear regression, moving average, and threshold detection. Although the above methods can identify some abnormal data and steady-state segments to a certain extent, they still have the following problems: (1) Insufficient anomaly suppression ability. When faced with complex dynamic working conditions, heavy-tailed abnormal data, or transient disturbances, it is easy to make misjudgments and it is difficult to accurately distinguish between effective data and abnormal data, resulting in poor data processing accuracy; (2) Insufficient classification accuracy. There is a lack of effective mechanisms to accurately distinguish between dynamic data and steady-state data, resulting in inconsistent data quality for AI modeling; (3) Reliance on manually set thresholds or prior knowledge. It lacks adaptability and generalization ability, making it difficult to meet the processing needs of different fields and different types of time series data. Therefore, the processing efficiency is low and the applicability is insufficient.

[0054] The purpose of this application is to provide a method and system for data cleaning and acquisition in AI modeling based on probabilistic strategies. This method is applicable to data preprocessing scenarios in artificial intelligence modeling. Addressing common problems in time-series data collected from industrial sites, such as outliers, noise interference, and frequent changes in operating conditions, the method first acquires the target time-series data and describes its statistical characteristics based on probabilistic strategy modeling. By constructing a mechanism for determining the probability of data point anomalies and state probabilities, it identifies and cleans abnormal data, obtaining reliable continuous time-series data. Furthermore, based on the probabilistic characteristics of the system's operating state, it automatically classifies the cleaned data into dynamic and steady-state data, obtaining training datasets suitable for different artificial intelligence models. This method can effectively clean raw industrial time-series data without relying on complex mechanistic models, and simultaneously achieve accurate classification of dynamic and steady-state data. This provides a high-quality data foundation for the construction of dynamic artificial intelligence models (such as recurrent neural network models and long short-term memory network models) and steady-state artificial intelligence models (such as support vector regression models, multilayer perceptron models, and Gaussian process models), improving the reliability, adaptability, and engineering application value of the artificial intelligence modeling process.

[0055] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] The method for AI modeling data cleaning and acquisition based on probabilistic strategies provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on other servers. Terminal 102 can send target time series data to server 104. After receiving the target time series data, server 104 constructs a time series statistical model containing trend terms; estimates parameters of the time series statistical model based on a probabilistic strategy, calculating the probability weights corresponding to each data point; performs weighted modeling on the time series statistical model to obtain a cleaned time series representation; extracts trend feature parameters reflecting the system operating state corresponding to the target time series data based on the cleaned time series representation; and determines the dynamic and steady-state states of the target time series data according to the trend feature parameters, obtaining cleaned dynamic data and cleaned steady-state data, thus achieving the classification and acquisition of cleaned dynamic and steady-state data. Server 104 can feed back the obtained cleaned dynamic and steady-state data to terminal 102. In addition, in some embodiments, the method for cleaning and acquiring AI modeling data based on probabilistic strategies can also be implemented separately by the server 104 or the terminal 102. For example, the terminal 102 can directly perform data cleaning on the target time series data, or the server 104 can acquire the target time series data from the data storage system and perform data cleaning on the target time series data.

[0057] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, and IoT devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers, or it can be a cloud server.

[0058] In one exemplary embodiment, such as Figure 2 As shown, a method for AI modeling data cleaning and acquisition based on a probabilistic strategy is provided. This method is executed by a computer device, specifically a terminal or server, or both. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S1 to S6.

[0059] S1: Obtain the target time series data.

[0060] S2: Based on the target time series data, construct a time series statistical model containing a trend term; the time series statistical model is used to describe the trend characteristics and random disturbance characteristics of different data points over time.

[0061] S3: Estimate the parameters of the time series statistical model based on a probabilistic strategy and calculate the probability weights corresponding to each data point.

[0062] S4: Based on the probability weights corresponding to each data point, the time series statistical model is weighted and modeled so that outlier data points are adaptively weakened during model parameter estimation, resulting in a cleaned time series representation. Here, outlier data points refer to data points with a high degree of anomaly. The degree of anomaly for each data point is determined based on the relationship between its corresponding anomaly probability or confidence probability and a preset threshold; a high degree of anomaly means that the anomaly probability or confidence probability of the data point is greater than the preset threshold.

[0063] S5: Based on the cleaned time series representation, extract trend feature parameters that reflect the system operating status corresponding to the target time series data.

[0064] S6: Based on the trend feature parameters, determine the dynamic and steady-state states of the target time series data to obtain cleaned dynamic data and cleaned steady-state data, thereby achieving the classification and acquisition of cleaned dynamic and steady-state data.

[0065] Implementing steps S1 to S6 as described above enables adaptive processing of various time-series data, effectively suppresses abnormal interference, and accurately achieves dynamic and steady-state classification. The data cleaning effect is significant, anomaly suppression is more accurate and robust, and dynamic and steady-state classification is precise and efficient, with superior data adaptability. This supports high-quality development of AI modeling, improves model performance and adaptability, enhances the accuracy, efficiency, and adaptability of data processing, and provides a high-quality data foundation for artificial intelligence modeling.

[0066] As an optional implementation, in step S1, the target time series data includes at least one of the following: data collected during industrial operation, time series data from the financial sector, time series data from environmental monitoring, and time series data from medical monitoring. The data collected during industrial operation includes at least one of the following: temperature, pressure, flow rate, liquid level, concentration, voltage, and current.

[0067] As an optional implementation, step S2 involves constructing a time series statistical model containing a trend term based on the target time series data, specifically including the following steps.

[0068] S21: The target time series data is segmented using a sliding window method to obtain several time windows.

[0069] S22: Based on each of the aforementioned time windows, construct a time series statistical model containing a trend term.

[0070] As an optional implementation, in step S21, the window size and step size of the sliding window in the sliding window method are adaptively adjusted or manually set according to the characteristics of the target time series data and the actual application scenario.

[0071] As an optional implementation, in step S3, when estimating the parameters of the time series statistical model, a likelihood function combined with an iterative optimization algorithm is used. The iterative optimization algorithm includes at least one of the EM (Expectation Maximization) algorithm, gradient descent algorithm, and Newton's iteration algorithm.

[0072] As an optional implementation, step S3 involves estimating the parameters of the time series statistical model based on a probabilistic strategy and calculating the probability weights corresponding to each data point, specifically including the following steps.

[0073] S31: Based on the degree to which each data point deviates from the time series statistical model, calculate the anomaly probability or confidence probability corresponding to each data point.

[0074] S32: Determine the probability weight of each data point in the model parameter estimation process based on the anomaly probability or confidence probability corresponding to each data point.

[0075] As an optional implementation, in step S32, the probability weights satisfy the following characteristics.

[0076] (1) The smaller the deviation of any data point from the time series statistical model, the greater the probability weight of the data point.

[0077] (2) The greater the deviation of any data point from the time series statistical model, the smaller the probability weight of the data point, thereby reducing its influence in the model parameter estimation process.

[0078] As an optional implementation, in step S5, the trend characteristic parameters include the trend slope parameter and its corresponding statistical significance index.

[0079] As an optional implementation, in step S6, the dynamic state and steady state of the target time series data are determined according to the trend feature parameters to obtain the cleaned dynamic data and the cleaned steady-state data, thereby realizing the classification and acquisition of the cleaned dynamic data and steady-state data. Specifically, this includes the following steps.

[0080] S61: Based on the trend slope parameter and its corresponding statistical significance index, determine whether the target time series data shows a significant trend within the time window.

[0081] S62: When the target time series data does not have a significant trend of change within the time window, the corresponding data is determined to be steady-state data, and the cleaned steady-state data is obtained.

[0082] S63: When the target time series data has a significant trend of change within the time window, the corresponding data is determined to be dynamic data, and the cleaned dynamic data is obtained.

[0083] As an optional implementation, in step S63, the cleaned dynamic data is used to construct a dynamic artificial intelligence model, which includes a recurrent neural network model, a long short-term memory network model, or other artificial intelligence models with dynamic data processing capabilities.

[0084] As an optional implementation, in step S62, the cleaned steady-state data is used to construct a steady-state artificial intelligence model, which includes a support vector regression model, a multilayer perceptron model, a Gaussian process model, or other artificial intelligence models with steady-state data processing capabilities.

[0085] To make the technical solution of this application clearer, the specific implementation process of the technical solution of this application will be explained in detail below with examples.

[0086] This application proposes a method and system for AI modeling data cleaning and acquisition based on probabilistic strategies. Through probabilistic robust modeling, it achieves integrated processing of anomaly suppression, trend identification, and data classification, providing high-quality steady-state and dynamic datasets for subsequent artificial intelligence model construction. The method for AI modeling data cleaning and acquisition based on probabilistic strategies includes the following steps.

[0087] (1) Obtain the target time series data.

[0088] (2) Based on the target time series data, construct a time series statistical model containing a trend term; the time series statistical model is used to describe the trend characteristics and random disturbance characteristics of different data points over time.

[0089] (3) Based on a probabilistic strategy, the parameters of the time series statistical model are estimated, and the probability weights corresponding to each data point are calculated. The model is run within a sliding window, so the data points here refer to each time sampling point located within the current sliding window.

[0090] (4) The time series statistical model is weighted according to the probability weights so that the abnormal data points with high abnormality are adaptively weakened in the model parameter estimation process, and the cleaned time series representation is obtained.

[0091] (5) Based on the cleaned time series representation, extract trend feature parameters that reflect the system operating status.

[0092] (6) Based on the trend feature parameters, determine the dynamic state and steady state of the target time series data, and realize the classification and acquisition of dynamic and steady state data of the cleaned data.

[0093] In this embodiment, a time series statistical model containing a trend term is constructed, which specifically includes the following steps.

[0094] The target time series data is segmented using a sliding window method to obtain several time windows.

[0095] Based on each of the aforementioned time windows, a time series statistical model containing a trend term is constructed respectively.

[0096] In this embodiment, the probability strategy includes: calculating the anomaly probability or confidence probability corresponding to each data point based on the degree to which the data point deviates from the time series statistical model; and determining the probability weight of the data point in the model parameter estimation process based on the anomaly probability or confidence probability.

[0097] In this embodiment, the probability weights satisfy the following characteristics.

[0098] 1) The smaller the deviation of any data point from the time series statistical model, the greater its probability weight.

[0099] 2) The greater the deviation of any data point from the time series statistical model, the smaller its probability weight, thereby reducing its influence in the model parameter estimation process.

[0100] In this embodiment, the trend characteristic parameters include a trend slope parameter and its corresponding statistical significance index. Based on the trend characteristic parameters, the dynamic and steady-state states of the target time series data are determined. Specifically, this includes: determining whether the target time series data exhibits a significant trend within a time window based on the trend slope parameter and its statistical significance index; wherein, the determination of a significant trend is based on a slope significance test (t-test), with a threshold value set according to the trend slope to determine a significant trend. When the trend slope parameter does not show a significant trend, the corresponding data is determined to be steady-state data; when the trend slope parameter shows a significant trend, the corresponding data is determined to be dynamic data.

[0101] In this embodiment, the cleaned dynamic data is used to construct a dynamic artificial intelligence model, which includes a recurrent neural network model, a long short-term memory network model, or other artificial intelligence models with dynamic data processing capabilities. The cleaned steady-state data is used to construct a steady-state artificial intelligence model, which includes a support vector regression model, a multilayer perceptron model, a Gaussian process model, or other artificial intelligence models with steady-state data processing capabilities.

[0102] Among them, the dynamic artificial intelligence model: Based on the cleaned dynamic process data, a dynamic artificial intelligence model represented by a long short-term memory network model is constructed to learn the temporal evolution law of the system under non-steady-state operating conditions, realize multi-step prediction of key process variables, and provide operators with early warning and decision support. When the system experiences changes in raw material properties or equipment operating conditions switching, the dynamic AI model can identify behavioral characteristics inconsistent with historical normal dynamic patterns, thereby achieving early identification of potential anomalies or faults.

[0103] Steady-state AI model: Based on the cleaned and screened steady-state data, a steady-state AI model is constructed to characterize the stable mapping relationship between process variables and key performance indicators, providing a basis for process performance evaluation and operational status analysis.

[0104] This application employs a probabilistic strategy for modeling, resulting in a time series statistical model that includes a trend term. The modeling process mainly consists of three parts: model structure selection, noise modeling, and iterative solution of model parameters.

[0105] Regarding the choice of model structure, this application focuses on time series statistical models, with trend modeling as the core, and its mathematical expression is as follows.

[0106] (1).

[0107] in, This represents the trend estimate at time t. The intercept parameter is essentially an intercept term used to describe the baseline level of the data. The slope parameter is essentially a trend term used to characterize the overall trend of data changes over time.

[0108] In this model, the slope parameter It is the core basis for dynamic and steady-state determination. When the trend term (i.e., the slope parameter) When the trend term is not statistically significant, the data is considered to be steady-state data; when the trend term is statistically significant, the data is considered to be dynamic data.

[0109] The model-estimated data will contain error terms compared to the actual data, which are represented by the following formula based on actual observations.

[0110] (2).

[0111] in, These are the original observation values ​​at time t, i.e., the actual data values ​​collected. This is a random disturbance term.

[0112] For the noise model, unlike the traditional assumption of Gaussian white noise, this application addresses the common problems of heavy-tailed perturbations and outliers in industrial data, and improves the random perturbation term. The model is based on a probability distribution with heavy-tailed characteristics, and the noise distribution is assumed as follows.

[0113] (3).

[0114] (4).

[0115] in, is a scaling parameter used to control the overall dispersion of the residual distribution. The degree of freedom parameter is a key robustness parameter that controls the tail thickness of the distribution and determines the model's tolerance for outliers. This distribution exhibits significant heavy-tailed characteristics when the degrees of freedom are small, preventing large-biased data points from being penalized in an extreme probabilistic sense, thus effectively suppressing the impact of outliers on model parameter estimation. The term represents the random disturbance, indicating the deviation between the observed data at time t and the model trend value. Indicates that the random disturbance term follows distributed. This represents the probability density function of the random perturbation term. This is the Gamma function.

[0116] Through the aforementioned noise modeling method, this application statistically achieves adaptive weighting of outlier data. Unlike simple removal or fixed threshold correction, this method dynamically reduces the confidence of outliers based on the statistical characteristics of the data, thereby effectively suppressing the excessive influence of outlier disturbances on trend parameter estimation. This mechanism fundamentally improves the robustness of the model in complex industrial data environments and achieves adaptive suppression of outlier interference.

[0117] Within the aforementioned probabilistic modeling framework, this application introduces a probabilistic weighting mechanism to represent the contribution of each data point to model parameter estimation as a corresponding probabilistic weight. The weight is adaptively determined by the degree of deviation of the data point from the current model; data points with greater deviation have smaller weights, thus being automatically weakened during parameter estimation; data points with smaller deviations receive higher weights and play a major role in model fitting. Therefore, this application achieves probabilistic cleaning of outlier data without explicitly deleting or replacing the original data.

[0118] For iterative solution of model parameters, in order to solve the parameters of the above time series statistical model, this application adopts a probability optimization strategy based on iterative updates, selects the likelihood function model, and gradually approaches the optimal solution by alternately estimating the model parameters and data point weights.

[0119] After estimating the model parameters, this application further extracts trend parameters as state determination features to distinguish between the dynamic and steady-state states of the data. Specifically, by analyzing the statistical significance or probabilistic determination results of the trend parameters, when the trend term is not significant in statistical or probabilistic terms, the system is considered to be in a steady-state state within that time window; when the trend term significantly deviates from the steady-state assumption, the system is determined to be in a dynamic state. This determination method avoids the shortcomings of simply relying on empirical thresholds and improves the objectivity and reliability of dynamic and steady-state classification.

[0120] Based on the aforementioned state determination results, this application classifies the cleaned time series data, categorizing data that meets the steady-state conditions as steady-state data and data that does not meet the steady-state conditions as dynamic data, thereby achieving the classification and acquisition of dynamic and steady-state data from the original time series data. The obtained cleaned steady-state data can be used to construct steady-state artificial intelligence models for steady-state prediction or steady-state identification, such as support vector regression models, multilayer perceptron models, and Gaussian process models. Meanwhile, the cleaned dynamic data can be used to construct dynamic artificial intelligence models describing the dynamic evolution characteristics of the system, such as recurrent neural network models or long short-term memory network models.

[0121] The following describes the specific implementation steps of this method, including the following steps.

[0122] (1) Divide time series data using a sliding window.

[0123] A sliding window technique is used to extract local features from time series data. The selection of the window size T needs to balance local pattern capture with computational efficiency. An overlapping window design with a step size of 1 ensures the continuity of data partitioning and maximizes information utilization, effectively avoiding pattern omissions that may occur with non-overlapping windows. The data within the window must meet the minimum sample requirements for trend analysis; it is generally recommended to cover 2-3 feature periods to capture complete fluctuation patterns.

[0124] (2) Construct a time series statistical model.

[0125] Combining formulas (1) and (2), we can obtain the following formula.

[0126] (5).

[0127] A linear trend model is built for each time window, consisting of deterministic and stochastic components. The deterministic component includes the intercept parameter. and slope parameters intercept parameter Characterizing the mean level, slope parameter Describes long-term trends; the random component includes random disturbance terms. This semi-parametric modeling approach is used to characterize white noise processes, reflecting unpredictable short-term fluctuations. It preserves the main characteristics of the trend while absorbing high-frequency interference through the white noise term, thus enhancing the model's adaptability.

[0128] (3) Likelihood function and parameter estimation objective.

[0129] Given parameters The joint likelihood of the observed data is given by the following formula.

[0130] (6).

[0131] in, Let be the likelihood function, representing the likelihood given model parameters. Under these conditions, observe time series data The joint probability density function that appears.

[0132] Taking the logarithm, we get the following formula.

[0133] (7).

[0134] in, The objective function is the log-likelihood function, which is the objective function obtained by taking the logarithm of the likelihood function. . These are constant terms independent of the model parameters, typically including the distribution normalization constant and the constant related to the model parameters. Irrelevant logarithmic terms.

[0135] The objective is to maximize the likelihood estimate, expressed as follows.

[0136] (8).

[0137] Maximizing the likelihood transforms into minimizing the negative log-likelihood, expressed as the following formula.

[0138] (9).

[0139] (10).

[0140] in, For a moment The residual or prediction error represents the deviation between the observed value and the predicted value of the linear trend model, and is used to describe measurement noise, operating condition disturbances and abnormal effects.

[0141] (4) The scale of the mixed representation.

[0142] The distribution can be expressed as follows.

[0143] (11).

[0144] in, Indicates the scale variable Under the condition, random disturbance term It follows the principle of zero mean and variance. It follows a normal distribution. These are latent variables used to characterize the local uncertainty or confidence level of each sample point, and are introduced as latent variables in the EM algorithm. Representing latent variables Follows a Gamma distribution. for The degrees of freedom parameter of the distribution is used to control the robustness of the distribution.

[0145] The final log-likelihood of the complete data is expressed as the following formula.

[0146] (12).

[0147] in, For the complete data log-likelihood, that is, given the current parameter estimates, the log-likelihood of the latent variables. The conditional expectation of the complete data log-likelihood.

[0148] (5) Solve for parameters using the EM algorithm.

[0149] Apply the E-step (compile weights) to compute latent variables. The conditional expectation is calculated using the following formula.

[0150] (13).

[0151] in, This represents the probability weight of the sample point. Indicates that under a given residual Under the condition of latent variables The expected condition.

[0152] Apply M steps (weighted least squares), including parameter updates and variance updates.

[0153] The parameter update is expressed as follows.

[0154] (14).

[0155] in, The updated model parameters represent the trend parameters estimated using the least squares method under probability weighting. The intercept parameter is essentially an intercept term used to describe the baseline level of the data. The slope parameter is essentially a trend term used to characterize the overall trend of data changes over time.

[0156] Variance update is expressed as follows.

[0157] (15).

[0158] (6) Steady state determination: slope significance test.

[0159] When determining steady state, the slope parameter is evaluated using a t-test. The statistical significance of is determined by the test statistic, which is: ,in The standard deviation of the slope estimate. This is the slope parameter.

[0160] Determine whether the target time series data is in a steady state. Less than the critical value When the target time series data is in a steady state (i.e., steady-state data), it is determined that the target time series data is in a steady state. Greater than or equal to the critical value If the target time series data is in a non-stationary state (i.e., dynamic data), then it is determined that the target time series data is in a non-stationary state.

[0161] When determining whether the target time series data is in a steady state, the results of the slope significance test are considered. < When the trend component is not significant, it indicates that the data in the window (i.e., the target time series data within the current window) is in a steady state. ≥ If a significant linear trend is observed, the window data is considered to be in a non-stationary state. This t-test controls for the Type I error probability, where... , The critical value represents a preset threshold for determining whether the target time series data is in a steady state, determined by the degrees of freedom. Obtained by looking up the table. This is the t-test statistic value for the slope parameter. The value represents the absolute value, and α represents the significance level. In this embodiment, the significance level α is 0.05 or 0.01. The steady-state determination provides a basis for the selection of subsequent modeling strategies. Steady-state data can be processed using stationary models, while non-steady-state data requires the introduction of differencing or cointegration techniques.

[0162] This embodiment uses the delayed coking process in an oil refining unit as an example to illustrate the proposed method. During actual operation, delayed coking units are affected by factors such as periodic switching of the coke tower, adjustment of the heater load, fluctuations in feedstock properties, and operational disturbances. As a result, process variables often exhibit significant dynamic fluctuations, making it difficult to maintain an ideal steady-state operation over a long period. Therefore, accurately acquiring high-quality steady-state data and effective dynamic data under complex operating conditions is a key issue in process modeling and operational analysis.

[0163] The overall execution flow of the method used in this embodiment is as follows: Figure 3 As shown, the process mainly includes data input, sliding window modeling, probabilistic robust parameter estimation, data cleaning output, and steady-state classification determination. The principle is as follows: First, input N sets of time series data; then determine the sliding window T and step size S, thereby determining the window data (0+K×S, T+K×S), where K is the number of window iterations; then construct a time series statistical model based on probability weights, and use the likelihood function combined with the EM algorithm to estimate the parameters of the linear trend model; then perform a t-test to determine the data steady state and achieve data classification; determine whether T+K×S is less than N. If so, K=K+1, and return to the step of "determining the window data (0+K×S, T+K×S)". Otherwise, output the steady-state classification result. The specific implementation steps of this method are as follows.

[0164] (a) Data acquisition.

[0165] In this embodiment, high-precision industrial sensors are used to monitor key material flow variables in the delayed coking process in real time, and auxiliary process parameters such as temperature and pressure are collected simultaneously for comprehensive characterization of the multivariate process state. The data acquisition system operates continuously at a sampling frequency of once per minute, accumulating approximately 20 days of historical operating data to construct a high-temporal-resolution industrial time-series database.

[0166] To facilitate the demonstration and verification of the effectiveness of the proposed method, this embodiment selects continuous operational data of approximately one day as the analysis object, and its original logistics flow time series is as follows: Figure 4As shown, this data can effectively reflect the volatility and unsteady-state characteristics of the delayed coking process in actual operation.

[0167] (ii) Data preprocessing.

[0168] Considering that data collected in industrial settings is often affected by factors such as instrument vibration, signal interference, communication anomalies, and operational disturbances in practical applications, this embodiment artificially introduces several abnormal disturbance points into the original logistics flow data to simulate common abnormal data scenarios in industrial processes. The logistics flow time series after adding outliers is as follows: Figure 5 As shown, this data includes normal operation sections, abnormal disturbance sections, and potential steady-state sections, and has strong engineering representativeness.

[0169] (III) The system process and model execution steps are as follows.

[0170] First, the collected logistics flow time series data is input into the system, and the data is standardized and formatted. Then, a sliding window mechanism is used to segment the time series. The number of time series data samples is N, the window size T is set to 20, and the step size S is 1, meaning that a subset of data of length 20 is sequentially extracted from the original time series each time. This subset of data is then used as the modeling object for the current window.

[0171] (iv) Sliding window modeling of probabilistic robust models.

[0172] Within each sliding window, a local linear trend model is constructed based on the modeling framework of time series statistical model, and the idea of ​​probabilistic robust modeling is introduced to iteratively estimate the model parameters through a probability weight update mechanism.

[0173] The probabilistic robust model used in this application belongs to the time series statistical model. It constructs a local linear trend model as a deterministic structure within a sliding window, and iteratively optimizes the model parameters by introducing a robust estimation mechanism based on probability weights, thereby achieving adaptive suppression of outlier data and stable extraction of trend features.

[0174] The objective and parameter estimates are obtained through likelihood function and parameter estimation. The model uses a scale-mixed representation (based on the EM algorithm) to estimate parameters. This model can adaptively adjust the contribution weight of each observation point in parameter estimation according to the degree of deviation between the data points and the trend model, thereby effectively reducing the impact of outlier data on the trend estimation results.

[0175] By progressively moving the window forward and repeating the above modeling and optimization process across the entire time series, segmented probabilistic modeling of long-term series can be achieved. This process does not require pre-labeling of outlier locations; the model can automatically identify and suppress outlier data during the iteration process.

[0176] (v) Data cleaning results generation and splicing strategy.

[0177] In the model output stage, to avoid data redundancy and boundary discontinuities caused by overlapping sliding windows, this embodiment employs a strategic result preservation mechanism to stitch the model output together. The specific strategy is as follows.

[0178] 1) For the first sliding window, retain the model fitting results for the first 10 time points within the window.

[0179] 2) For the last sliding window, retain the model fitting results for the last 10 time points within the window.

[0180] 3) For the middle sliding window, only retain the fitting result of the 10th time point corresponding to the center position of the window.

[0181] By using the above method, the local fitting results within each window are stitched together into a complete time series, such as... Figure 6 As shown, the final data cleaning result is formed, as follows: Figure 7 As shown in the figure, this strategy effectively reduces the redundancy caused by repeated computation while ensuring temporal continuity.

[0182] (vi) Obtaining steady-state classification results.

[0183] Based on probabilistic robust modeling and data cleaning, statistical characteristic analysis is performed on the trend parameters within each sliding window, and the data states corresponding to the windows are classified according to their significance criteria. By concatenating the classification results of all windows, the final steady-state classification result, dividing the entire time series into steady-state and non-steady-state segments, is obtained, such as... Figure 8 As shown.

[0184] (vii) Analysis of data cleaning results.

[0185] Figure 7 The data cleaning results shown include the original data (dashed line) of the delayed coking process logistics flow data after introducing outliers, and the data cleaning results (solid line) processed using the AI ​​modeling data cleaning and acquisition method based on probabilistic strategies proposed in this application. From Figure 7 As can be seen, the original data contains a large number of high-frequency fluctuations and obvious outliers, manifesting as spike-shaped anomalies with sudden increases or decreases. These anomalies mainly originate from factors such as instrument noise, operating condition disturbances, and acquisition errors. Directly using this type of raw data for subsequent modeling can easily cause significant interference to model parameter estimation and state judgment. However, after processing by the AI ​​modeling data cleaning and acquisition method based on probabilistic strategies proposed in this application, the data cleaning results exhibit the following characteristics.

[0186] 1) Outliers are effectively suppressed. For isolated spikes or short-term large offsets in the original data, the cleaned data curves remain continuous and smooth, without fluctuating drastically with outliers. This indicates that the model automatically assigns lower weights to outlier observations during parameter estimation, thereby reducing their impact on the overall trend.

[0187] 2) The trend structure is accurately preserved. In sections where logistics flow changes significantly (such as a sudden drop in flow in the middle and the subsequent recovery process), the cleaned data can accurately follow the overall trend of the original data. There is no trend weakening due to excessive smoothing, nor is any additional lag effect introduced, indicating that the method in this application achieves a balance between anomaly suppression and trend preservation.

[0188] 3) It possesses data continuity and physical rationality. The cleaned time series maintains good continuity and stability throughout the entire time period, without unreasonable jumps or breakpoints, and conforms to the physical characteristics of industrial process variables changing over time. It can be directly used as high-quality input data for subsequent data modeling or state identification.

[0189] In summary, the method proposed in this application can effectively remove abnormal disturbances in industrial time series without relying on manual threshold settings, while maintaining key dynamic change characteristics, which can provide a reliable data foundation for subsequent dynamic and steady-state classification and data-driven modeling.

[0190] (viii) Analysis of steady-state classification results.

[0191] Based on the completed data cleaning, this application further utilizes the statistical properties of the trend parameters of a probabilistic robust model to perform steady-state detection and dynamic / steady-state classification of time series data. Figure 8 The steady-state classification results shown clearly distinguish between steady-state and non-steady-state data segments, achieving the classification of steady-state and dynamic data. Combined with... Figure 8 The following characteristics can be observed.

[0192] 1) The steady-state segment is clearly and continuously identified. Within a time period in which the overall logistics flow remains relatively stable and only small random fluctuations occur, the model can continuously determine that it is in a steady state. The steady-state identifiers show a continuous distribution without frequent jumps, indicating that the method in this application is for normal random noise and will not be misjudged as non-steady state due to small fluctuations.

[0193] 2) Dynamic change segments are accurately identified. During the transition phase where logistics flow significantly increases or decreases, the steady-state classification result is determined to be a non-steady-state state, which is highly consistent with the dynamic change process caused by operation adjustment or process switching in actual working conditions, indicating that the method of this application can accurately capture the significant trend changes of process variables.

[0194] 3) The dynamic and steady-state boundaries are reasonably located. From the perspective of the transition positions from steady state to unsteady state and from unsteady state to steady state, the steady-state classification results correspond to the start and end positions of the data trend changes. There is no obvious phenomenon of advance or lag, which reflects the accuracy of the steady-state identification mechanism based on the combination of sliding window and probability statistical criteria in terms of time positioning.

[0195] 4) Supports simultaneous acquisition of steady-state and dynamic data. Based on the steady-state classification results described above, high-quality steady-state data segments can be directly extracted from the complete time series for steady-state modeling, parameter identification, or operational analysis. Simultaneously, non-steady-state segments, as dynamic data, can also be used to construct dynamic artificial intelligence models (such as long short-term memory network models) describing the dynamic evolution characteristics of the system, enabling the classified acquisition and targeted utilization of different types of data.

[0196] In summary, the AI ​​modeling data cleaning and acquisition method based on probabilistic strategies proposed in this application can automatically suppress abnormal data and accurately distinguish between steady-state and dynamic segments in complex industrial operation contexts, providing high-quality and interpretable data support for subsequent data-driven modeling, process analysis, and intelligent algorithm training.

[0197] This application divides the original time-series data into sliding windows and establishes a probabilistic model within each window. It adaptively weights outlier data and extracts trend features, enabling automatic classification of dynamic and steady-state data after cleaning. This provides high-quality training data for artificial intelligence models. Furthermore, this application provides a reliable data foundation for data-driven modeling, hybrid AI modeling, and dynamic / steady-state prediction. It eliminates the reliance on manual labeling or single threshold rules for cleaning and classifying industrial process data, instead achieving intelligent processing through probabilistic strategies and model adaptation mechanisms. This not only improves the accuracy of data cleaning and the reliability of steady-state classification but also provides high-quality data support for training dynamic AI models such as recurrent neural networks and long short-term memory networks, as well as steady-state AI models such as support vector regression, multilayer perceptrons, and Gaussian processes, demonstrating the promising application prospects of deep integration between artificial intelligence and industrial processes.

[0198] Compared with existing technologies, this application proposes a probabilistic strategy-based AI modeling data cleaning and acquisition method. In terms of complex industrial process data processing and data quality improvement, it constructs a probabilistic time series statistical model and introduces an iterative optimization mechanism based on probability weight updates. This enables adaptive identification and mitigation of outlier data points in the original time series, stably extracting the true evolutionary characteristics of the data even under conditions of noise interference and operating fluctuations. Simultaneously, by analyzing the statistical characteristics of trend parameters, it effectively distinguishes between dynamic and steady-state data, providing a clear and reliable data foundation for subsequent modeling. Furthermore, this method does not rely on fixed thresholds or empirical rules, possessing data-driven characteristics and model generalization capabilities. It is suitable for both steady-state data screening and the retention of high-quality dynamic data with physical meaning. Maintaining stable data cleaning and classification performance even under multi-source disturbances and high noise backgrounds, it provides data support for industrial process modeling, operating status identification, and intelligent analysis.

[0199] Based on the same inventive concept, this application also provides a system for implementing the above-described method for cleaning and acquiring AI modeling data based on probabilistic strategies. The solution provided by this system is similar to the implementation described in the above method. Therefore, the specific limitations of one or more system embodiments for cleaning and acquiring AI modeling data based on probabilistic strategies provided below can be found in the limitations of the method for cleaning and acquiring AI modeling data based on probabilistic strategies described above, and will not be repeated here.

[0200] In one exemplary embodiment, such as Figure 9 As shown, a system for AI modeling data cleaning and acquisition based on a probabilistic strategy is provided, which includes the following modules.

[0201] The data acquisition module is used to acquire target time series data.

[0202] The time series statistical model construction module is used to construct a time series statistical model containing a trend term based on the target time series data; the time series statistical model is used to describe the trend characteristics and random disturbance characteristics of different data points over time.

[0203] The parameter estimation module is used to estimate the parameters of the time series statistical model based on a probability strategy and calculate the probability weights corresponding to each data point.

[0204] The weighted modeling and cleaning module is used to perform weighted modeling on the time series statistical model according to the probability weights corresponding to each data point, so that abnormal data points are adaptively weakened during the model parameter estimation process, and a cleaned time series representation is obtained.

[0205] The trend feature parameter extraction module is used to extract trend feature parameters that reflect the system operating status corresponding to the target time series data based on the cleaned time series representation.

[0206] The dynamic and steady-state determination module is used to determine the dynamic and steady-state states of the target time series data based on the trend feature parameters, so as to obtain the cleaned dynamic data and the cleaned steady-state data, thereby realizing the classification and acquisition of the dynamic and steady-state data of the cleaned data.

[0207] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0208] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for data cleaning and acquisition in AI modeling based on probabilistic strategies, characterized in that, The method for cleaning and acquiring AI modeling data based on probabilistic strategies includes: Obtain the target time series data; Based on the target time series data, a time series statistical model containing a trend term is constructed; the time series statistical model is used to describe the trend characteristics and random disturbance characteristics of different data points over time. The time series statistical model is parameter estimated based on a probabilistic strategy, and the probability weights corresponding to each data point are calculated. Based on the probability weights corresponding to each data point, the time series statistical model is weighted and modeled so that outlier data points are adaptively weakened during the model parameter estimation process, resulting in a cleaned time series representation. Based on the cleaned time series representation, trend feature parameters reflecting the system operating status corresponding to the target time series data are extracted; Based on the trend feature parameters, the target time series data is judged to be in dynamic and steady state, and the cleaned dynamic data and cleaned steady state data are obtained, thereby realizing the classification and acquisition of the cleaned dynamic and steady state data.

2. The method for cleaning and acquiring AI modeling data based on probabilistic strategies according to claim 1, characterized in that, Based on the target time series data, a time series statistical model including a trend term is constructed, specifically including: The target time series data is segmented using a sliding window method to obtain several time windows; Based on each of the aforementioned time windows, a time series statistical model containing a trend term is constructed.

3. The method for cleaning and acquiring AI modeling data based on probabilistic strategies according to claim 1, characterized in that, The time series statistical model is parameter estimated based on a probabilistic strategy, and the probability weights corresponding to each data point are calculated, specifically including: Based on the degree to which each data point deviates from the time series statistical model, the anomaly probability or confidence probability corresponding to each data point is calculated respectively. Based on the anomaly probability or confidence probability corresponding to each data point, the probability weight of each data point in the model parameter estimation process is determined.

4. The method for cleaning and acquiring AI modeling data based on probabilistic strategies according to claim 3, characterized in that, The probability weights satisfy the following properties: The smaller the deviation of any data point from the time series statistical model, the greater the probability weight of that data point. The greater the deviation of any data point from the time series statistical model, the smaller the probability weight of that data point.

5. The method for cleaning and acquiring AI modeling data based on probabilistic strategies according to claim 1, characterized in that, The trend characteristic parameters include the trend slope parameter and its corresponding statistical significance index; Based on the trend characteristic parameters, the target time series data is judged to be in dynamic and steady state, resulting in cleaned dynamic data and cleaned steady-state data. This achieves the classification and acquisition of cleaned dynamic and steady-state data, specifically including: Based on the trend slope parameter and its corresponding statistical significance index, determine whether the target time series data shows a significant trend within the time window; When the target time series data does not show a significant trend of change within the time window, the corresponding data is determined to be steady-state data, and the cleaned steady-state data is obtained. When the target time series data shows a significant trend of change within the time window, the corresponding data is determined to be dynamic data, and the cleaned dynamic data is obtained.

6. The method for cleaning and acquiring AI modeling data based on probabilistic strategies according to claim 1, characterized in that, The cleaned dynamic data is used to construct a dynamic artificial intelligence model, which includes a recurrent neural network model, a long short-term memory network model, or other artificial intelligence models with dynamic data processing capabilities. The cleaned steady-state data is used to construct a steady-state artificial intelligence model, which includes a support vector regression model, a multilayer perceptron model, a Gaussian process model, or other artificial intelligence models with steady-state data processing capabilities.

7. The method for cleaning and acquiring AI modeling data based on probabilistic strategies according to claim 2, characterized in that, In the sliding window method, the window size and step size of the sliding window are adaptively adjusted or manually set according to the characteristics of the target time series data and the actual application scenario.

8. The method for cleaning and acquiring AI modeling data based on probabilistic strategies according to claim 1, characterized in that, The target time series data includes at least one of the following: data collected during industrial operations, time series data from the financial sector, time series data from environmental monitoring, and time series data from medical monitoring. The data collected during the industrial operation includes at least one of temperature, pressure, flow rate, liquid level, concentration, voltage, and current.

9. The method for cleaning and acquiring AI modeling data based on probabilistic strategies according to claim 1, characterized in that, When estimating the parameters of the time series statistical model, a likelihood function combined with an iterative optimization algorithm is used. The iterative optimization algorithm includes at least one of the EM algorithm, gradient descent algorithm, and Newton's iteration algorithm.

10. A system for AI modeling data cleaning and acquisition based on probabilistic strategies, characterized in that, The system for cleaning and acquiring AI modeling data based on probabilistic strategies is used to implement the method for cleaning and acquiring AI modeling data based on probabilistic strategies as described in any one of claims 1-9, wherein the system for cleaning and acquiring AI modeling data based on probabilistic strategies comprises: The data acquisition module is used to acquire target time series data; The time series statistical model construction module is used to construct a time series statistical model containing a trend term based on the target time series data; the time series statistical model is used to describe the trend characteristics and random disturbance characteristics of different data points over time. The parameter estimation module is used to estimate the parameters of the time series statistical model based on a probability strategy and calculate the probability weights corresponding to each data point. The weighted modeling and cleaning module is used to perform weighted modeling on the time series statistical model according to the probability weights corresponding to each data point, so that abnormal data points are adaptively weakened in the process of model parameter estimation, and a cleaned time series representation is obtained. The trend feature parameter extraction module is used to extract trend feature parameters that reflect the system operating status corresponding to the target time series data based on the cleaned time series representation. The dynamic and steady-state determination module is used to determine the dynamic and steady-state states of the target time series data based on the trend feature parameters, so as to obtain the cleaned dynamic data and the cleaned steady-state data, thereby realizing the classification and acquisition of the dynamic and steady-state data of the cleaned data.

Images