Real-time reservoir data stream processing and storage optimization method
Through real-time reservoir data flow processing and abnormal detection, combined with Fourier transform and isolated forest model, the problems of data lag and untimely response in traditional reservoir management technology are solved, real-time data processing and storage optimization are realized, and oil is improved The efficiency and security of hidden management.
Patent Information
- Application Number
- CN202411148549.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-21
AI Technical Summary
Traditional reservoir management technology relies on discrete sampling and periodic data analysis, resulting in data lag, long decision-making cycles, and untimely production responses, which cannot meet the needs of complex oilfield development.
The real-time reservoir data flow processing method is used to calculate the spectrum through Fourier transform, anomaly detection is performed using an isolated forest model, and the storage structure is optimized based on a time series prediction algorithm, and the storage strategy is dynamically adjusted.
Real-time data processing and abnormal detection accuracy is achieved, data storage efficiency and cost benefits are improved, and the overall performance and reliability of reservoir management are improved.
Smart Images

Figure CN118656772B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of real-time data processing and intelligent storage, and in particular to a method for real-time reservoir data stream processing and storage optimization thereof. Background Art
[0002] In oilfield development and production, applying reservoir data stream processing and adaptive storage decisions to reservoirs can realize the digital transformation of oilfields, facilitate data monitoring and management, provide instant and accurate data feedback for oilfield supervision, optimize the utilization of storage resources, and improve data access efficiency, thereby improving the overall performance of the system. Real-time reservoir data processing and storage device optimization are becoming increasingly important in the digital transformation and intelligent management of reservoirs.
[0003] The Chinese invention with publication number CN117708736A applied by State Grid Fujian Electric Power Co., Ltd. discloses a method for discovering abnormal access to terminal data. Its publication date is March 15, 2024. The method focuses on monitoring and identifying abnormal behavior of terminal devices during data access. By collecting terminal data access behavior, preprocessing data, and using machine learning models such as isolation forest models, abnormal behavior is detected and marked, thereby achieving effective discovery of data access anomalies. However, the application uses the isolation forest model for data anomaly detection, but it does not specify whether it supports the processing of real-time data streams. If the processing is delayed, it cannot be applied to application scenarios that require immediate response. The European invention with publication number EP3451232A1 applied by ServiceNow Company discloses a system and method for anomaly detection. Its publication date is March 6, 2019. It mainly involves an anomaly detection system based on time series analysis. The system uses historical data to classify current data and builds a statistical model to identify anomalies. The system can monitor time series data streams and identify and track outliers based on statistical models. However, although the application performs time series analysis, it does not elaborate on its real-time data processing capabilities, which may affect the performance of the system in applications that require rapid response. In addition, the invention mainly uses traditional statistical models and methods for anomaly detection, focusing on statistical analysis of data. Although it is universal, it lacks targeted data analysis for reservoir data. The Chinese invention application number CN115344839A filed by Xidian University discloses "an identity authentication method and device based on user keystroke dynamics", and its publication date is November 15, 2022. The invention proposes a method for identity authentication using the dynamic characteristics of user keystroke behavior. The method analyzes the dynamic characteristics of the user's keystroke speed and rhythm when entering the PIN code, uses fast Fourier transform to obtain time domain and frequency domain features, and performs feature learning and classification through convolutional neural networks to finally determine the user's identity. This technology is mainly used to improve the protection capabilities of security systems, such as identifying legitimate users in banking systems and security access control systems. However, the invention mainly focuses on identity authentication, especially the method based on keystroke dynamics, which makes its application scenarios relatively limited and is mainly used for security authentication. Moreover, this method focuses on processing keystroke dynamics data, and the processing process involves real-time feature extraction, but does not involve broader data processing or data stream management functions, and lacks the ability to process large-scale or multi-type data.
[0004] It can be seen that traditional reservoir management technology often relies on discrete sampling and periodic data analysis. This approach has problems such as data information lag, long decision-making cycle, and untimely production response, and cannot meet the needs of today's complex oilfield development. Summary of the invention
[0005] In order to solve the above problems existing in the prior art, the present invention is implemented by the following technical solutions:
[0006] The present invention provides a method for real-time reservoir data stream processing and storage optimization, the method comprising:
[0007] S10, collecting real-time reservoir data;
[0008] S20, preprocessing the real-time reservoir data to obtain real-time reservoir preprocessing data;
[0009] S30, using Fourier transform to calculate the spectrum of the pre-processed real-time reservoir pre-processed data to obtain real-time spectrum data;
[0010] S40, using the isolation forest model to perform anomaly detection on the real-time spectrum data to obtain an anomaly detection result, and issue an alarm;
[0011] S50, storing the real-time reservoir preprocessing data and the abnormal category data in the abnormality detection result in different storage media according to the access frequency of the real-time reservoir data;
[0012] S60. Utilize a prediction algorithm based on time series to predict the frequency of future access to the real-time reservoir data, and optimize the storage structure according to the frequency of future access to the real-time reservoir data.
[0013] In one embodiment of the present invention, S20, preprocessing the real-time reservoir data to obtain real-time reservoir preprocessed data, includes:
[0014] S201, constructing data fields and structures for the real-time reservoir data to obtain real-time JSON reservoir data;
[0015] S202: define additional metadata information for the real-time JSON reservoir data and perform data preprocessing to obtain real-time reservoir preprocessing data.
[0016] In one embodiment of the present invention, S30, using Fourier transform to calculate the spectrum of the preprocessed real-time reservoir preprocessing data to obtain real-time spectrum data, includes:
[0017] S301, dividing the real-time reservoir preprocessing data into time series data and static data, wherein the static data includes geological data, seismic data and surface data, and the time series data includes production monitoring data and oil well monitoring data;
[0018] S302, using a fast Fourier transform algorithm to calculate a frequency spectrum for continuous time series data;
[0019] S303: For discrete time series data, use a discrete Fourier transform algorithm to calculate the frequency spectrum.
[0020] In one embodiment of the present invention, S302, using a fast Fourier transform algorithm to calculate a spectrum for continuous time series data, includes: calculating the spectrum according to the following formula:
[0021]
[0022] in, is the frequency The spectrum representation of is a continuous time series data function, The frequency is The complex exponential of .
[0023] In one embodiment of the present invention, S303, for discrete time series data, using a discrete Fourier transform algorithm to calculate a spectrum, including:
[0024] The spectrum is calculated according to the following formula:
[0025]
[0026] in, is the frequency index The spectral components of Represents the discrete spectrum frequency components, is the discrete time series data data points, represents the rotation factor, N Indicates the length of the signal.
[0027] In one embodiment of the present invention, S40, using the isolation forest model to perform anomaly detection on the real-time spectrum data to obtain an anomaly detection result, and issuing an alarm, includes:
[0028] S401, randomly selecting a sub-sample data set from the real-time spectrum data;
[0029] S402, for the selected sub-sample data set, randomly select features, and build a decision tree based on the random features;
[0030] S403, calculating an anomaly score of the real-time spectrum data, setting an anomaly threshold according to the anomaly score, and obtaining an anomaly detection result of the real-time spectrum data based on the anomaly threshold, wherein the anomaly score is an average value of the path lengths of the real-time spectrum data in multiple decision trees;
[0031] S404: Issue a warning or alarm based on the abnormal detection result.
[0032] In one embodiment of the present invention, S402, for the selected sub-sample data set, randomly selecting features, and building a decision tree based on the random features, includes:
[0033] Randomly select a feature as the segmentation feature of the current node and randomly select a segmentation threshold;
[0034] The dataset is divided into two subsets according to the segmentation threshold: the left subtree whose segmentation feature value is less than or equal to the segmentation threshold, and the right subtree whose segmentation feature value is greater than the segmentation threshold;
[0035] The above process is repeated recursively on each subset until the preset stopping condition is reached.
[0036] In one embodiment of the present invention, S50, according to the access frequency of the real-time reservoir data, the real-time reservoir preprocessing data and the abnormal category data in the abnormal detection result are stored in different storage media, including dividing the real-time reservoir preprocessing data into frequency access levels of high-frequency access data, medium-frequency access data and low-frequency access data according to the access frequency of the real-time reservoir data, setting a storage strategy according to the frequency access level, storing the real-time reservoir preprocessing data on different storage media, and storing the statistical results of the access frequency of the real-time reservoir data corresponding to the real-time reservoir preprocessing data per unit time, and storing the abnormal category data in the abnormal detection result, as well as the real-time reservoir data and the real-time reservoir preprocessing data related to the abnormal category data as high-frequency access data.
[0037] In one embodiment of the present invention, S60, using a prediction algorithm based on a time series to predict the future access frequency of the real-time reservoir data, and optimizing the storage structure according to the future access frequency of the real-time reservoir data, includes:
[0038] S601, select the ARIMA (p, d, q) model as the prediction model;
[0039] S602, extracting the statistical result of the access frequency of the real-time reservoir data stored in step S50, and making the statistical result of the access frequency of the real-time reservoir data stable through differential and logarithmic transformation;
[0040] S603, based on the stable performance of the access frequency data of the real-time reservoir data, preliminarily determine the parameters d, p and q;
[0041] S604, training the ARIMA (p, d, q) model using the initially determined parameter d, parameter p, and parameter q;
[0042] S605, optimizing the values of parameter d, parameter p and parameter q based on the information criterion;
[0043] S606, predicting the future access frequency of real-time reservoir data based on the ARIMA (p, d, q) model and the optimized values of parameter d, parameter p and parameter q;
[0044] S607: Adaptively adjust the storage strategy according to the predicted future access frequency of the real-time reservoir data.
[0045] In one embodiment of the present invention, S607, the storage strategy is adaptively adjusted according to the predicted future access frequency of real-time reservoir data, including storing high-frequency access data on a high-performance storage device; storing medium-frequency access data on a storage device with moderate performance relative to the high-performance storage device; and storing low-frequency access data on a low-cost storage medium.
[0046] Compared with the prior art, the present invention has the following beneficial effects:
[0047] 1. The method for real-time reservoir data stream processing and storage optimization provided in this embodiment performs data cleaning on the real-time reservoir data and then performs preprocessing through a cyclic redundancy check algorithm to improve the efficiency and accuracy of subsequent data processing. The real-time reservoir preprocessed data is divided into time series data and static data according to data characteristics. The time series data is calculated using Fourier transform for real-time spectrum data. Subsequently, an isolation forest model is used to construct a decision tree based on random features, thereby increasing the diversity and robustness of the model. By using the isolation forest model to perform anomaly detection on the time series data, the efficiency of data processing and the accuracy of anomaly detection are improved, production safety is improved, and the high requirements for data real-time, accuracy and economic benefits in oilfield development are met.
[0048] 2. The method for real-time reservoir data stream processing and storage optimization provided by the present invention also uses the ARIMA model, a prediction algorithm based on time series, to store real-time reservoir preprocessing data in different storage media according to the statistical results of access frequency per unit time, and dynamically adjust the storage strategy according to the statistical results of access frequency at different access times, so that data that may be frequently accessed in the future is stored in a high-performance solid-state hard disk, and data that will not be accessed for a long time is stored in a low-cost tape library or cloud storage service, so as to optimize the utilization of storage resources and data access performance to the greatest extent, and optimize data storage and access efficiency through dynamic prediction of data access frequency. The present invention provides a series of solutions for real-time data processing, anomaly detection and storage optimization for reservoir data management, improves the efficiency and cost-effectiveness of data storage, and improves the overall performance and reliability of reservoir management.
[0049] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 It is a schematic diagram of the overall steps of a method for real-time reservoir data stream processing and storage optimization provided by an embodiment of the present invention;
[0051] Figure 2 The present invention provides a time series-based prediction algorithm, which is a flowchart of the steps for predicting the future access frequency of real-time reservoir data. DETAILED DESCRIPTION
[0052] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the scheme according to the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0053] The above and other technical contents, features and effects of the present invention are clearly presented in the following detailed description of the specific implementation modes in conjunction with the accompanying drawings. Through the description of the specific implementation modes, the technical means and effects adopted by the present invention to achieve the predetermined purpose can be more deeply and specifically understood. However, the attached drawings are only for reference and explanation purposes and are not used to limit the technical solutions of the present invention.
[0054] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants are intended to cover non-exclusive inclusion, so that an article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. In the absence of more restrictions, the elements defined by the statement "including one..." do not exclude the existence of other identical elements in the article or device including the elements.
[0055] Embodiment 1
[0056] Please refer to Figure 1 , Figure 1 The figure is a schematic diagram of the overall steps of a method for real-time reservoir data stream processing and storage optimization provided by an embodiment of the present invention. The method comprises the following steps:
[0057] S10, collecting real-time reservoir data;
[0058] Real-time reservoir data is collected from the reservoir or sensor network. The types of real-time reservoir data include geological data, seismic data, oil well monitoring data, surface data and production monitoring data, and the real-time reservoir data is displayed based on digital twin technology.
[0059] S20, preprocessing the real-time reservoir data to obtain real-time reservoir preprocessing data;
[0060] S201, constructing data fields and structures for real-time reservoir data to obtain real-time JSON reservoir data;
[0061] According to the type of real-time reservoir data, the structure and fields of each type of data are constructed, the field names and corresponding data types to be extracted are determined, the data are extracted and a JSON object is constructed. Specifically, for each data sample or time point, the corresponding data is extracted according to the determined fields and structures, and organized into a JSON object. For example, the following is some exemplary data of real-time JSON reservoir data about oil well monitoring data such as pressure and temperature constructed according to real-time reservoir data in this implementation:
[0062] {
[0063] "timestamp": "2024-04-09T12:34:56Z",
[0064] "sensor_id": "sensor-001",
[0065] "measurements": {
[0066] "pressure": 1500,
[0067] "temperature": 350,
[0068] "flow_rate": 250
[0069] }
[0070] }
[0071] S202, defining additional metadata information for the real-time JSON reservoir data and performing data preprocessing to obtain real-time reservoir preprocessing data;
[0072] Additional metadata information is added to the real-time JSON reservoir data, including data type information, well name, geographic location, data source, etc. The data type information can be, for example, time series data and static data, so as to facilitate diversion during data stream processing. Preprocessing the data after adding the metadata information includes data cleaning, followed by preprocessing using a cyclic redundancy check algorithm to obtain real-time reservoir preprocessed data. Specifically, after adding the additional metadata information, data cleaning is performed, the isnull() function or isna() function in the pandas library is used to detect missing values in the data, useless or erroneous data points are deleted, the forward filling method is used to fill missing values, and the dimensions are unified to ensure that data from different data sources have the same units and standards. Subsequently, in order to ensure the consistency and accuracy of the converted data, the cyclic redundancy check (CRC) algorithm is used to verify the cleaned data, and the format-converted and cleaned data is output as real-time reservoir preprocessed data, which will be provided to the next step for real-time analysis, anomaly detection, and optimized storage decisions.
[0073] S30, using Fourier transform to calculate the spectrum of the pre-processed real-time reservoir pre-processed data to obtain real-time spectrum data;
[0074] Processing and real-time data analysis of preprocessed reservoir data, that is, the real-time reservoir preprocessing data mentioned above, is a key stage. The specific operations are as follows:
[0075] S301, dividing the real-time reservoir preprocessing data into time series data and static data;
[0076] Specifically, the data stream of the real-time reservoir preprocessing data outputted in step S20 is received in real time, and a data stream access operation is performed to divide these data streams into time series data and static data. Static data is non-time series data, including geological data, seismic data, and surface data. The time series data includes production monitoring data and oil well monitoring data. The production monitoring data specifically includes data such as production rate and water injection rate, and the oil well monitoring data specifically includes monitoring indicators such as oil well temperature, pressure, and flow rate.
[0077] S302, using a Fast Fourier Transform (FFT) algorithm to calculate the frequency spectrum of the continuous time series data;
[0078] According to the metadata information added during data format conversion, real-time analysis and anomaly detection are performed on time series data; specifically, real-time Fourier transform is performed on time series data to convert time domain data into frequency domain data for spectrum analysis. Furthermore, for continuous time series data, the fast Fourier transform algorithm is used to process the data points in the real-time data stream, and the spectrum is obtained according to formula (1):
[0079] (1)
[0080] in, is the frequency The spectrum representation of is a continuous time series data function, The frequency is The complex exponential of .
[0081] S303. For discrete time series data, a discrete Fourier transform (DFT) algorithm is used to calculate the spectrum:
[0082] (2)
[0083] in, is the frequency index The spectral components of Represents the discrete spectrum frequency components, is the discrete time series data data points, is the rotation factor, N is the length of the signal.
[0084] Real-time analysis of the real-time spectrum data obtained after Fourier transformation is performed to identify frequency features and abnormal frequency components in the data stream. The frequency components in the spectrum are checked to identify abnormal or important frequencies. The amplitude spectrum or power spectrum of the frequency, such as |X(f)| or |X(f)|^2, is then calculated for analysis and anomaly detection.
[0085] S40, using an isolation forest model to perform abnormality detection and alarm on the real-time spectrum data;
[0086] Anomaly detection is performed using the isolation forest model. Specifically, the isolation forest model constructs a decision tree based on random features, evaluates the degree of abnormality of real-time spectrum data by constructing multiple random decision trees, and then calculates the path length of the real-time spectrum data in the decision tree to evaluate its degree of abnormality. Specifically, the constructed isolation forest model performs anomaly detection on the data points in the real-time spectrum data, calculates the anomaly score of the data point, and marks the real-time spectrum data as normal or abnormal according to the set threshold to obtain real-time anomaly detection results. Furthermore, the use of the isolation forest model to perform anomaly detection and alarm on the real-time spectrum data includes the following steps:
[0087] S401, randomly selecting a sub-sample data set from the real-time spectrum data;
[0088] A subsample set is randomly selected from the real-time spectrum data obtained in step S30, and the subsample set includes some normal data points and some abnormal data points. The abnormal data points here refer to those data points that have significantly different feature values or behavior patterns in the feature space relative to the normal data points. These abnormal data points usually show obvious outlier characteristics and may correspond to abnormal operating conditions, equipment failures or other abnormal activities.
[0089] S402, for the selected sub-sample data set, randomly select features, and build a decision tree based on the random features;
[0090] Specifically, building a decision tree includes the following steps:
[0091] (1) Randomly select a feature as the segmentation feature of the current node and randomly select a segmentation threshold;
[0092] (2) Divide the data set into two subsets according to the segmentation threshold: the left subtree whose segmentation feature value is less than or equal to the segmentation threshold, and the right subtree whose segmentation feature value is greater than the segmentation threshold;
[0093] (3) Recursively repeat the above process on each subset until a preset stopping condition is reached. The preset stopping condition may be, for example, the maximum depth of the tree or the number of samples in each leaf node is lower than a threshold.
[0094] S403, calculating an anomaly score of the real-time spectrum data, setting an anomaly threshold according to the anomaly score, and obtaining an anomaly detection result of the real-time spectrum data based on the anomaly threshold;
[0095] The real-time spectrum data is used as a training sample set to calculate the anomaly score of the real-time spectrum data in the isolation forest model. The anomaly threshold is set according to the anomaly score. The anomaly score is usually the average of the path lengths of the real-time spectrum data in multiple decision trees. If the anomaly score exceeds the anomaly threshold, the real-time spectrum data is marked as abnormal. The real-time spectrum data is divided into normal category data and abnormal category data according to the isolation forest model to obtain the anomaly detection result. The specific formula for calculating the anomaly score includes: Assume h(x) Represents real-time spectrum data x The path length in the decision tree, E(h) Represents the expected value of the path length in the sample set, according to the anomaly score Set an abnormal threshold, and the data points exceeding the abnormal threshold are marked as abnormal. Real-time spectrum data x Anomaly score It can be calculated as: ,in, c(M) is the expected value of the decision tree path length.
[0096] S404: Issue a warning or alarm based on the abnormal detection result.
[0097] Based on the anomaly detection results, feedback measures are taken in real time, such as triggering early warnings or alarms, adjusting equipment operating parameters, or updating data storage strategies, thereby achieving real-time monitoring and efficient data management.
[0098] The method for real-time reservoir data stream processing and storage optimization provided in this embodiment performs data cleaning on real-time reservoir data and then pre-processes it through a cyclic redundancy check algorithm to improve the efficiency and accuracy of subsequent data processing. The real-time reservoir pre-processed data is divided into time series data and static data according to data characteristics. The time series data is used to calculate real-time spectrum data using Fourier transform. Then, an isolation forest model is used to build a decision tree based on random features. The abnormality of the real-time spectrum data is evaluated by building multiple random decision trees. Then, the path length of the real-time spectrum data in the decision tree is calculated to evaluate its abnormality. By randomly selecting features and thresholds, the isolation forest model is used to build a decision tree based on random features. Each tree in the forest is different, which increases the diversity and robustness of the model. Through Fourier transform and the isolation forest model, real-time anomaly detection is performed on time series data. It not only analyzes the spectral data of time series data, but also can instantly reflect changes in the data stream. Anomaly detection is performed before reservoir data is stored, which can promptly detect anomalies of equipment in the reservoir and quickly respond to potential anomalies or problems. It improves the efficiency of data processing and the accuracy of anomaly detection, enhances the real-time monitoring capability of the system, and improves the efficiency and accuracy of reservoir management, thereby improving production safety and meeting the high requirements for data real-time, accuracy and economic benefits in oil field development.
[0099] S50, storing the real-time reservoir preprocessing data and the abnormal category data in the abnormality detection result in different storage media according to the access frequency of the real-time reservoir data;
[0100] Storage optimization is performed through data classification. Specifically, detailed data on the access frequency of raw data is collected by using log records, monitoring tools or statistical information in a database management system. For example, detailed data on the access frequency of raw data is collected by log records. In this article, raw data refers to the real-time reservoir data initially collected. Specifically, the log format is determined. The log format is generally a standard Apache or Nginx log format. After the log format is determined, the log is read and parsed, and then the data is sorted. The number of accesses is summarized according to the date and time. The real-time reservoir preprocessing data is divided into three levels according to the raw data access frequency standard shown in Table 1, as shown in Table 1 below.
[0101] Table 1:
[0102]
[0103] According to the access frequency of the original data, the real-time reservoir preprocessing data corresponding to the original data is divided into frequency access levels of high-frequency access data, medium-frequency access data and low-frequency access data. Storage strategies are set according to the frequency access levels, and the real-time reservoir preprocessing data is stored on different storage media. The statistical results of the access frequency of the original data corresponding to the real-time reservoir preprocessing data are stored per unit time, so as to facilitate the subsequent dynamic prediction of the access frequency of the original data. The storage strategy is dynamically adjusted according to the statistical results of the access frequency of the original data per unit time within the preset time. By dynamically predicting the statistical results of the access frequency of the original data per unit time, the storage structure of the real-time reservoir preprocessing data is optimized, thereby improving the access efficiency, and improving the effect and cost-effectiveness of reservoir data management.
[0104] In particular, for abnormal category data, since abnormal category data is suitable for scenarios that require rapid response and processing of abnormal events, such as real-time monitoring, early warning or alarm, it is necessary to classify the abnormal category data in the abnormal detection results, the original data related to the abnormal category data (that is, real-time reservoir data) and the real-time reservoir preprocessing data into high-frequency access data, store them according to high-frequency access data, and will not be changed later.
[0105] The method for real-time reservoir data stream processing and storage optimization provided in this embodiment receives real-time reservoir preprocessing data after format conversion, performs anomaly detection by applying Fourier transform method and isolation forest model, and classifies the access frequency of original data per unit time according to classification standard and stores access frequency statistics, stores real-time reservoir preprocessing data corresponding to original data in different storage media according to access frequency statistics per unit time, stores abnormal category data and its related original data and real-time reservoir preprocessing data as high-frequency access data, thereby realizing efficient processing and analysis of reservoir data stream. In this embodiment, the unit time is day, that is, the access frequency is counted in day.
[0106] S60. Utilize a prediction algorithm based on time series to predict the frequency of future access to the real-time reservoir data, and optimize the storage structure according to the frequency of future access to the real-time reservoir data.
[0107] Please refer to Figure 2 , Figure 2 The present invention provides a time series-based prediction algorithm, which is a flowchart of the steps for predicting the future access frequency of real-time reservoir data. The time series-based prediction algorithm is used to predict the future access frequency, access time and access mode of the original data. According to the real-time data access situation and the continuous prediction results of the Auto-Regressive Moving Average (ARIMA) model, the storage strategy is continuously optimized. Specifically, the following steps are included:
[0108] S601, select the ARIMA (p, d, q) model as the prediction model;
[0109] Reservoir data mainly involves changes in geology and production, and is affected by factors such as geological structure, reservoir development stage, and production operations. These factors usually do not show obvious periodic or seasonal changes. Therefore, reservoir data usually does not have obvious seasonality or periodicity. The ARIMA (p, d, q) model can effectively adapt to non-periodic data and improve prediction accuracy, thereby optimizing resource allocation, dynamically adjusting production strategies, reducing risks and improving economic benefits, and realizing data-driven scientific decision-making. In view of this, this embodiment selects the ARIMA (p, d, q) model for reservoir data prediction. Specifically, p is the order of the autoregressive term, which represents the correlation between the current value and the past p time points, d is the order of the difference, which represents the number of differences required to make the time series data stable, and q is the order of the sliding average term, which represents the noise correlation between the current value and the past q time points.
[0110] S602, extracting the statistical result of the access frequency of the real-time reservoir data stored in step S50, and making the statistical result of the access frequency of the real-time reservoir data stable through differential and logarithmic transformation;
[0111] Extract the statistical results of the access frequency of the real-time reservoir data stored in the preset time according to the unit time stored in step S50, which statistics the access frequency of the original data according to the unit time in the preset time. The ARIMA (p, d, q) model requires that the data is stable, and stability means that the statistical characteristics of the time series (such as mean and variance) remain unchanged over time. Therefore, the statsmodels library in Python is first used to perform the Augmented Dickey-Fuller (ADF) test method to test the stability of the statistical results of the access frequency of the original data. If the p value in the ADF test is less than 0.05, the sequence is considered to be stable. If the access frequency data of the real-time reservoir data is not stable, the access frequency of the real-time reservoir data is differentially and logarithmically transformed to check whether the access frequency data of the real-time reservoir data is stable. If it is still not stable, the access frequency of the real-time reservoir data after differential and logarithmic transformation is continued to be differentially and logarithmically transformed, and this cycle is repeated until the access frequency data of the original data is stable.
[0112] S603, based on the stable performance of the access frequency data of the original data, preliminarily determine the parameter d, parameter p and parameter q;
[0113] The value of parameter d is determined based on the minimum number of differences required for the access frequency data of real-time reservoir data to be stable, and the values of parameter p and parameter q are determined based on the value of parameter d. Specifically, the value of parameter d is determined: d is the minimum number of differences required to make the data stable, that is, the minimum number of differences performed in step S602 to make the access data stable. Determine the value of parameter p: Use the stable access frequency data after differentiation to draw a partial autocorrelation function (PACF) diagram. When the lag coefficient first drops to near zero and remains unchanged, the number of lags before this point is the p value. Determine the value of parameter q: Use the stable access frequency data after differentiation to draw an autocorrelation function (ACF) diagram. The number of lags at which the ACF value first falls to near zero and continues to be close to zero.
[0114] S604, training the ARIMA (p, d, q) model using the initially determined parameter d, parameter p, and parameter q;
[0115] The ARIMA (p, d, q) model is trained using the determined parameters d, p, q and the post-stationary access frequency data output in step S602.
[0116] S605, optimizing the values of parameter d, parameter p and parameter q based on the information criterion;
[0117] Model diagnostics, check whether the difference between the actual value and the model predicted value is close to white noise. The actual value refers to the real data point in the time series data, that is, check whether the residual is close to white noise. Ideally, all autocorrelation coefficients should be close to zero within the significance boundary, which indicates that there is no autocorrelation between the residuals, that is, the residuals are white noise. If the residuals perform poorly, it is necessary to redetermine the d, p, and q parameters of the model and refit. Specifically, if the residuals show autocorrelation, increase the value of q if it is positively correlated, and increase the value of p if it is negatively correlated. If the model is too complex or overfitted, try to reduce the values of p and q and check whether the model is still good enough. Select the values of parameters d, p, and q based on the information criterion. Specifically, compare the AIC or BIC values of the models under different parameters, and select the d, p, and q parameters of the models with smaller values of these criteria as the d, p, and q parameters of the ARIMA (p, d, q) model.
[0118] S606, predicting the future access frequency of real-time reservoir data based on the ARIMA (p, d, q) model and the optimized values of parameter d, parameter p and parameter q;
[0119] Based on the values of parameters d, p, and q after the information criterion optimization in step S605, an ARIMA (p, d, q) model is fitted to predict future values.
[0120] S607: Adaptively adjust the storage strategy according to the predicted future access frequency of the real-time reservoir data.
[0121] Based on the model's predicted future access frequency data for real-time reservoir data, the storage strategy and storage configuration are adaptively adjusted to maximize the utilization of storage resources and data access performance.
[0122] According to the predicted data level design and real-time analysis results, the real-time reservoir preprocessing data is stored on different storage media. When selecting the storage strategy, storage is based on the following principles: Store high-frequency access data on high-performance storage devices, that is, store frequently accessed data on high-performance storage devices, such as solid-state drives, or SSDs, so that fast data read and write operations can be achieved, improving the response speed and performance of data access; Store medium-frequency access data on storage devices with moderate performance relative to high-performance storage devices, that is, store medium-frequency access data on storage devices with moderate performance relative to high-performance storage devices, such as traditional hard disk drives, or HDDs. These storage devices provide moderate performance and storage capacity, suitable for storing relatively common data that does not require extremely high performance; Store low-frequency access data on low-cost storage media, that is, use cost-effective storage solutions for less frequently accessed data, and store low-frequency access data on low-cost storage media, such as cold storage options for tape libraries or low-cost cloud storage services. These storage media provide lower access speeds but low costs, and are more suitable for long-term storage and archiving of data.
[0123] The method for real-time reservoir data stream processing and storage optimization provided in this embodiment uses the ARIMA model, a prediction algorithm based on time series, to store real-time reservoir preprocessing data in different storage media according to the statistical results of access frequency per unit time, and dynamically adjusts the storage strategy according to the statistical results of access frequency at different access times, so that data that may be frequently accessed in the future is stored in a solid-state hard disk with higher performance, and data that will not be accessed for a long time is stored in a tape library or cloud storage service with lower cost, so as to optimize the utilization of storage resources and data access performance to the greatest extent, and stores real-time reservoir preprocessing data through dynamic prediction of the access frequency of original data, and optimizes the storage strategy and access efficiency of data. This embodiment provides a series of solutions for real-time data processing, anomaly detection and storage optimization for reservoir data management, which improves the efficiency of data storage, optimizes cost control, and improves the overall performance and reliability of reservoir management. It is particularly suitable for real-time monitoring and efficient data management in big data environments such as oil fields.
[0124] Embodiment 2
[0125] The present embodiment provides a device for real-time reservoir data stream processing and storage optimization thereof, the device comprising a data format conversion module, a data stream processing and real-time analysis module and an adaptive storage module connected in sequence, specifically, the data format conversion module can be used to execute steps S10 and S20 in the method for real-time reservoir data stream processing and storage optimization thereof in embodiment 1 to generate real-time reservoir preprocessing data; the data stream processing and real-time analysis module can receive the real-time reservoir preprocessing data transmitted by the data format conversion module, and perform anomaly detection and classified storage on the real-time reservoir preprocessing data by executing steps S30, S40 and S50; the adaptive storage module can predict the future access frequency of the original data by executing step S60 with a prediction algorithm based on a time series, and optimize the storage structure of the real-time reservoir preprocessing data based on the future access frequency of the original data.
[0126] The device for real-time reservoir data stream processing and storage optimization provided in this embodiment is capable of analyzing the time series real-time data stream from the reservoir in real time for reservoir data management, and dynamically adjusting the data storage strategy to adapt to the changing data access requirements. By efficiently processing and storing abnormal category data and real-time reservoir preprocessing data, an optimization device for a series of solutions for real-time data processing, anomaly detection and storage optimization is provided, which not only improves the efficiency of data storage and the response speed and decision-making quality of reservoir management, but also helps to achieve cost-effectiveness optimization.
[0127] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.
Claims
1. A method for real-time reservoir data stream processing and storage optimization, characterized in that: include: S10, collecting real-time reservoir data; S20, preprocessing the real-time reservoir data to obtain real-time reservoir preprocessing data; S301, dividing the real-time reservoir preprocessing data into time series data and static data, wherein the static data includes geological data, seismic data and surface data, and the time series data includes production monitoring data and oil well monitoring data; S302, using a fast Fourier transform algorithm to calculate the spectrum of the continuous time series data to obtain real-time spectrum data; S303, for discrete time series data, using a discrete Fourier transform algorithm to calculate the spectrum to obtain real-time spectrum data; S40, using the isolation forest model to perform anomaly detection on the real-time spectrum data to obtain an anomaly detection result, and issue an alarm; S50, storing the real-time reservoir preprocessing data and the abnormal category data in the abnormality detection result in different storage media according to the access frequency of the real-time reservoir data; S60. Utilize a prediction algorithm based on time series to predict the frequency of future access to the real-time reservoir data, and optimize the storage structure according to the frequency of future access to the real-time reservoir data.
2. The method for real-time reservoir data stream processing and storage optimization according to claim 1, characterized in that: S20, preprocessing the real-time reservoir data to obtain real-time reservoir preprocessed data, including: S201, constructing data fields and structures for the real-time reservoir data to obtain real-time JSON reservoir data; S202: define additional metadata information for the real-time JSON reservoir data and perform data preprocessing to obtain real-time reservoir preprocessing data.
3. The method for real-time reservoir data stream processing and storage optimization according to claim 1, characterized in that: S302, for the continuous time series data, using a fast Fourier transform algorithm to calculate the spectrum, including calculating the spectrum according to the following formula: in, is the frequency The spectrum representation of is a continuous time series data function, The frequency is The complex exponential of .
4. The method for real-time reservoir data stream processing and storage optimization according to claim 1, characterized in that: S303. For discrete time series data, a discrete Fourier transform algorithm is used to calculate the spectrum, including: The spectrum is calculated according to the following formula: in, is the frequency index The spectral components of Represents the discrete spectrum frequency components, It is the discrete time series data. data points, represents the rotation factor, N Indicates the length of the signal.
5. The method for real-time reservoir data stream processing and storage optimization according to claim 1, characterized in that: S40, using the isolation forest model to perform anomaly detection on the real-time spectrum data to obtain an anomaly detection result, and issue an alarm, including: S401, randomly selecting a sub-sample data set from the real-time spectrum data; S402, for the selected sub-sample data set, randomly select features, and build a decision tree based on the random features; S403, calculating an anomaly score of the real-time spectrum data, setting an anomaly threshold according to the anomaly score, and obtaining an anomaly detection result of the real-time spectrum data based on the anomaly threshold, wherein the anomaly score is an average value of the path lengths of the real-time spectrum data in multiple decision trees; S404: Issue a warning or alarm based on the abnormal detection result.
6. The method for real-time reservoir data stream processing and storage optimization according to claim 5, characterized in that: S402: For the selected sub-sample data set, randomly select features and build a decision tree based on the random features, including: Randomly select a feature as the segmentation feature of the current node and randomly select a segmentation threshold; The dataset is divided into two subsets according to the segmentation threshold: the left subtree whose segmentation feature value is less than or equal to the segmentation threshold, and the right subtree whose segmentation feature value is greater than the segmentation threshold; The above process is repeated recursively on each subset until the preset stopping condition is reached.
7. The method for real-time reservoir data stream processing and storage optimization according to claim 6 is characterized in that: S50, storing the real-time reservoir preprocessing data and the abnormal category data in the abnormality detection result in different storage media according to the access frequency of the real-time reservoir data, including: According to the access frequency of the real-time reservoir data, the real-time reservoir preprocessing data is divided into frequency access levels of high-frequency access data, medium-frequency access data and low-frequency access data, a storage strategy is set according to the frequency access level, the real-time reservoir preprocessing data is stored on different storage media, and the statistical results of the access frequency of the real-time reservoir data corresponding to the real-time reservoir preprocessing data are stored per unit time, and the abnormal category data in the abnormal detection result, as well as the real-time reservoir data and the real-time reservoir preprocessing data related to the abnormal category data are stored as high-frequency access data.
8. The method for real-time reservoir data stream processing and storage optimization according to claim 7, characterized in that: S60, using a prediction algorithm based on a time series to predict the frequency of future access to the real-time reservoir data, and optimizing the storage structure according to the frequency of future access to the real-time reservoir data, including: S601, select the ARIMA (p, d, q) model as the prediction model; S602, extracting the statistical result of the access frequency of the real-time reservoir data stored in step S50, and making the statistical result of the access frequency of the real-time reservoir data stable through differential and logarithmic transformation; S603, based on the stable performance of the access frequency data of the real-time reservoir data, preliminarily determine the parameters d, p and q; S604, training the ARIMA (p, d, q) model using the initially determined parameter d, parameter p, and parameter q; S605, optimizing the values of parameter d, parameter p and parameter q based on the information criterion; S606, predicting the future access frequency of real-time reservoir data based on the ARIMA (p, d, q) model and the optimized values of parameter d, parameter p and parameter q; S607: Adaptively adjust the storage strategy according to the predicted future access frequency of the real-time reservoir data.
9. The method for real-time reservoir data stream processing and storage optimization according to claim 8, characterized in that: S607, adaptively adjusting the storage strategy according to the predicted future access frequency of the real-time reservoir data, including: Store high-frequency access data on high-performance storage devices; store medium-frequency access data on storage devices with moderate performance relative to high-performance storage devices; store low-frequency access data on low-cost storage media.
Citation Information
Patent Citations
Identity authentication method and device based on user keystroke dynamics
CN115344839A
Terminal data access abnormity discovery method
CN117708736A
Methods for the preparation of organotin compounds
EP0002019A1
Systems and methods for anomaly detection
EP3451232A1
Earthquake multi-scale reservoir prediction method for gas reservoir development of huge thick reservoir
CN113671565A
