Abnormal data detection and model training method and device, equipment and storage medium

Abstract

Embodiments of the present application relate to an abnormal data detection and model training method, device and equipment, and a storage medium. The method comprises: obtaining basic sample data of a natural gas value chain optimization system, and generating normal sample data and abnormal sample data according to the basic sample data; training a decision tree model for abnormal data detection according to the normal sample data and the abnormal sample data; obtaining to-be-detected data of the natural gas value chain optimization system; inputting the to-be-detected data into the pre-trained decision tree model, and outputting a result of whether the to-be-detected data is abnormal. Embodiments of the present application realize rapid checking of the optimization result of the natural gas value chain optimization system through the decision tree model, without wasting manpower, and improve the checking efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a method, apparatus, device, and storage medium for anomaly detection and model training. Background Technology

[0002] The natural gas value chain optimization system is used to optimize natural gas production and operation plans, improve the level of natural gas resource optimization and allocation, and enhance production and operation management capabilities. When performing optimization calculations using the natural gas value chain optimization system, due to the large amount of data that needs to be entered, data errors or omissions may occur. Therefore, it is necessary to promptly verify the optimization results.

[0003] Existing methods for analyzing anomalies in natural gas value chain optimization results mainly rely on experienced business personnel to find abnormal data by step-by-step checking of calculation results. This requires a lot of manual time, has low verification efficiency, and also carries the risk that abnormal results cannot be detected manually, resulting in low detection accuracy. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and storage medium for anomaly data detection and model training, in order to solve the technical problems of low efficiency and low accuracy in the existing technology of manually verifying the optimization results of natural gas value chain optimization systems.

[0005] In a first aspect, the present invention provides a method for training an anomaly data detection model, characterized in that it includes:

[0006] Acquire basic sample data for the natural gas value chain optimization system, and generate normal sample data and abnormal sample data based on the basic sample data;

[0007] A decision tree model for anomaly detection is trained based on the normal sample data and the abnormal sample data.

[0008] As an optional embodiment of the present invention, the step of obtaining basic sample data of the natural gas value chain optimization system and generating normal sample data and abnormal sample data based on the basic sample data includes: obtaining a natural gas value chain optimization scheme for a preset time period as the basic sample data, wherein the optimization scheme includes input indicators and output indicators of the natural gas value chain optimization system; adjusting at least one input indicator in the basic sample data within a first preset range, and inputting the adjusted input indicator into the natural gas value chain optimization system to obtain the corresponding output indicator, thereby constructing normal sample data; and performing missing processing or adjustment on at least one input indicator in the basic sample data to exceed a second preset range, and inputting the missing or adjusted input indicator into the natural gas value chain optimization system to obtain the corresponding output indicator, thereby constructing abnormal sample data.

[0009] As an optional embodiment of the present invention, after generating normal sample data and abnormal sample data based on the basic sample data, the method further includes: splitting the normal sample data and abnormal sample data to obtain a training set, a validation set, and a test set; training a decision tree model based on the normal sample data and abnormal sample data includes: training a decision tree model based on the training set; the method further includes: adjusting the parameters of the decision tree model based on the validation set, and / or determining the generalization ability of the decision tree model based on the test set.

[0010] As an optional embodiment of the present invention, the step of training a decision tree model for anomaly detection based on the normal sample data and the anomaly sample data includes: starting from the root node of the decision tree model, recursively executing the following steps until a preset condition is met: determining the Gini coefficient of each indicator in the sample data corresponding to the current node; selecting indicators whose Gini coefficients meet the preset condition and the corresponding split points as optimal features and optimal split points; generating the next-level child node from the current node based on the optimal features and optimal split points, and assigning the sample data corresponding to the current node to the next-level child node.

[0011] As an optional embodiment of the present invention, the preset conditions are determined according to relevant parameters, wherein the relevant parameters include at least one of the following: the maximum depth of the leaf nodes of the decision tree, the minimum number of samples required for node re-division, the minimum number of leaf node samples, the minimum sample weight of the leaf node, the maximum number of leaf nodes, the minimum impurity of node division, and the information gain threshold.

[0012] As an optional embodiment of the present invention, the method further includes: training a sub-decision tree model corresponding to each node in the decision tree model; if the output result of the data to be detected is abnormal, then calling the corresponding sub-decision model according to the path of the data to be detected in the decision tree model, and determining and correcting the abnormal data according to the sub-decision model.

[0013] Secondly, the present invention provides an abnormal data detection method, comprising:

[0014] Acquire the data to be monitored by the natural gas value chain optimization system;

[0015] The data to be detected is input into a pre-trained decision tree model, which outputs a result indicating whether the data to be detected is abnormal.

[0016] As an optional embodiment of the present invention, the training decision tree model is trained using the above-described abnormal data detection model training method.

[0017] Thirdly, the present invention provides an abnormal data detection model training device, comprising:

[0018] The sample acquisition module is used to acquire basic sample data of the natural gas value chain optimization system, and generate normal sample data and abnormal sample data based on the basic sample data.

[0019] The model training module is used to train a decision tree model for anomaly detection based on the normal sample data and the abnormal sample data.

[0020] Fourthly, the present invention provides an abnormal data detection device, comprising:

[0021] The data acquisition module is used to acquire the data to be monitored by the natural gas value chain optimization system;

[0022] The anomaly detection module is used to input the data to be detected into a pre-trained decision tree model and output the result of whether the data to be detected is abnormal.

[0023] Fifthly, the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor, when executing the program stored in the memory, implements the steps of the abnormal data detection model training method described in the first aspect and / or the abnormal data detection method described in the second aspect.

[0024] In a sixth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the abnormal data detection model training method described in the first aspect and / or the abnormal data detection method described in the second aspect.

[0025] The abnormal data detection model training method, apparatus, electronic device, and storage medium provided in this invention are based on the basic sample data of the natural gas value chain optimization system. They construct normal sample data and abnormal sample data, and train a decision tree model based on the normal sample data and abnormal sample data. This model is used for the verification of the optimization results of the natural gas value chain optimization system to detect abnormal data. The model is trained with existing data, and the obtained decision tree model can accurately detect abnormal data. Using this model for abnormal data detection can also solve the problems of long time consumption and low efficiency of manual verification, and improve the speed and efficiency of verification.

[0026] The abnormal data detection method, apparatus, electronic device, and storage medium provided in this invention acquire the data to be detected from the natural gas value chain optimization system; input the data to be detected into a pre-trained decision tree model; and output the result of whether the data to be detected is abnormal. That is, the embodiments of this invention realize the rapid verification of the optimization results of the natural gas value chain optimization system through the decision tree model, without wasting manpower, improving the verification speed and efficiency, and using a pre-trained decision tree model for verification results in high accuracy. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart illustrating the training method for an anomaly data detection model provided in an embodiment of the present invention;

[0030] Figure 2 This is a flowchart illustrating the abnormal data detection method provided in an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of positive and negative sample splitting provided in an embodiment of the present invention;

[0032] Figure 4 A schematic diagram of a decision tree model provided in an embodiment of the present invention;

[0033] Figure 5 A flowchart illustrating another abnormal data detection method provided in an embodiment of the present invention;

[0034] Figure 6 This is a schematic diagram of the structure of an abnormal data detection model training device provided in an embodiment of the present invention;

[0035] Figure 7 This is a schematic diagram of the structure of an abnormal data detection device provided in an embodiment of the present invention;

[0036] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] First, the terms used in this invention will be explained:

[0039] Natural Gas Value Chain Optimization System: The main objective is to establish a natural gas value chain optimization model to maximize the overall benefits of China National Petroleum Corporation (CNPC), support the optimization of natural gas production and operation plans of various energy companies, achieve overall optimization of the natural gas value chain, and improve the level of optimal allocation of oil and gas resources and production and operation management capabilities.

[0040] Decision trees are a form of supervised learning. Based on the type of target variable, decision trees can be divided into discrete decision trees and continuous decision trees. Discrete decision trees have discrete target variables, such as gender (male or female); continuous decision trees have continuous target variables, such as salary, price, and age. Purity is used as a metric when constructing decision trees. Three classic metrics are information gain (ID3 algorithm), information gain ratio (C4.5 algorithm), and the Gini coefficient (Cart algorithm).

[0041] When using a natural gas value chain optimization system for calculations, the large amount of data required can lead to errors or omissions in data entry. Therefore, timely checks on the optimization results are necessary. Current methods for analyzing anomalies in natural gas value chain optimization results primarily rely on experienced business personnel who systematically verify the calculation results to identify abnormal data. This process is labor-intensive and carries the risk of anomalies going undetected by humans.

[0042] To address the aforementioned technical problems, the technical concept of this invention lies in the fact that the optimization result data of the natural gas value chain optimization system have strict logical relationships. The connection relationship between the data can be found by using the top-down recursive principle of the decision tree algorithm, thereby judging the anomalies of the optimization results through the decision tree model.

[0043] Figure 1 This is a flowchart illustrating an abnormal data detection method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the anomaly detection method includes:

[0044] Step S101: Obtain the data to be tested from the natural gas value chain optimization system.

[0045] It should be noted that the executing entity in this embodiment can be an abnormal data detection device, or an electronic device equipped with an abnormal data detection device. In this embodiment, when it is necessary to analyze whether the optimization results of the natural gas value chain optimization system are abnormal, the main data indicators involved in the optimization calculation process of the natural gas value chain optimization system can be obtained, including input indicators and output indicators.

[0046] Step S102: Input the data to be detected into the pre-trained decision tree model and output the result of whether the data to be detected is abnormal.

[0047] In this step, the pre-trained decision tree model is a binary classification model. By inputting the data to be detected into the trained decision tree model, it can be determined whether the data to be detected is normal or abnormal.

[0048] The abnormal data detection method provided in this invention obtains the data to be detected from the natural gas value chain optimization system; inputs the data to be detected into a pre-trained decision tree model; and outputs the result of whether the data to be detected is abnormal. That is, the embodiments of this invention realize the rapid verification of the optimization results of the natural gas value chain optimization system through the decision tree model, without wasting manpower, improving the verification speed and efficiency. Moreover, compared with the black box of other machine learning algorithms, the decision tree algorithm displays the characteristics of the data in a graphical and intuitive way, which is easy to understand and interpret.

[0049] This invention also provides a method for training an anomaly data detection model, used to train a decision tree model. Figure 2 This is a flowchart illustrating the training method for an anomaly data detection model provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the method includes:

[0050] Step S201: Obtain basic sample data of the natural gas value chain optimization system, and generate normal sample data and abnormal sample data based on the basic sample data.

[0051] Step S202: Train a decision tree model for anomaly detection based on normal sample data and abnormal sample data.

[0052] Based on the above embodiments, this invention also provides an abnormal data detection method, firstly by... Figure 2 The method shown trains a decision tree model, and then uses the trained decision tree model to detect abnormal data. The process includes steps S201, S202, S101, and S102.

[0053] The difference between this embodiment and the abnormal data detection method described above lies in that the training process of the decision tree model is further defined. In this embodiment, basic sample data of the natural gas value chain optimization system is acquired, and normal sample data and abnormal sample data are generated based on the basic sample data; a decision tree model is then trained based on the normal sample data and the abnormal sample data.

[0054] Since the number of optimization scheme samples in the natural gas value chain optimization system is relatively small, in this embodiment, by acquiring the basic sample data of the natural gas value chain optimization system, a data adjustment strategy is executed on the basic sample data to generate a sufficient number of normal sample data (positive samples) and abnormal sample data (negative samples). Finally, the decision tree model trained with sufficient positive and negative samples is more accurate.

[0055] As an optional embodiment, step S201 includes: obtaining a natural gas value chain optimization scheme for a preset time period as basic sample data, the optimization scheme including input indicators and output indicators of the natural gas value chain optimization system; adjusting at least one input indicator in the basic sample data within a first preset range, and inputting the adjusted input indicator into the natural gas value chain optimization system to obtain the corresponding output indicator, thereby constructing normal sample data; and handling the missing or exceeding a second preset range adjustment of at least one input indicator in the basic sample data, and inputting the missing or adjusted input indicator into the natural gas value chain optimization system to obtain the corresponding output indicator, thereby constructing abnormal sample data.

[0056] The preset time period can be a preset month, year, week, or 10 days, etc. For example, two scenarios can be obtained for the off-season month (July) and the peak season month (November) of natural gas sales as basic sample data. Each scenario will involve various data indicators of the natural gas value chain optimization system, including oilfield production, gas intake purchase volume, external gas purchase volume, oilfield settlement price, imported gas purchase price, external gas purchase price, sales volume and sales price in each province, gas storage injection and production volume and injection and production costs, LNG gasification volume and gasification costs, natural gas unit pipeline transportation cost, natural gas unit sales price and profit, etc. Among them, price data indicators are input items of the natural gas value chain optimization system, and other data indicators are output items.

[0057] Then, the prices and sales volumes in the basic sample data are adjusted, or both prices and sales volumes are adjusted simultaneously. The adjusted input indicators are then re-input into the natural gas value chain optimization system to obtain the corresponding output indicators. The input and output indicators before and after adjustment constitute normal sample data and are marked. Table 1 shows a positive sample adjustment strategy provided by an embodiment of the present invention. As shown in Table 1, the prices of competing provinces can be adjusted within -5% to 5%, or the prices of all provinces can be adjusted within -5% to 5%, or the sales volumes of competing provinces can be adjusted within -10% to 10%, etc. Alternatively, the sales prices and sales volumes can be adjusted in combination, or the purchase prices and purchase volumes of imported gas can be adjusted in combination, thereby obtaining 4759 positive samples.

[0058] Table 1

[0059]

[0060] Then, analyze the situations where abnormal data occurs and adjust the data for negative samples. For example, if 7 competing provinces need to lower prices, but one province was missed during data adjustment, then the prices of 6 of those provinces can be adjusted within the range of -5% to 0. Or, if 7 competing provinces need to increase sales, but one province was missed during data adjustment, then the sales of 6 of those provinces can be adjusted within the range of 0% to 10%, as shown in Table 2.

[0061] Table 2

[0062]

[0063]

[0064] Optionally, positive and negative samples will include the main data indicators involved in the natural gas value chain optimization system and be labeled as positive and negative samples. The main data indicators include: sales profit per cubic meter of gas, sales price per cubic meter of gas, comprehensive cost per cubic meter of gas, pipeline transportation fee per cubic meter of gas, sales price in a certain province, sales volume in a certain province, total pipeline transportation fee around the oil field, injection and production fee, gasification fee, total gas purchase cost from domestic and foreign sources, long-distance settlement volume of a certain oil field, long-distance pipeline settlement price of a certain oil field, settlement volume around a certain oil field, ex-factory price around a certain oil field, purchase volume of a certain imported gas, and purchase price of a certain imported gas.

[0065] The above adjustments to normal and abnormal sample data can be applied to situations where the number of samples is insufficient. Existing sample data can be adjusted within a reasonable range to construct more normal sample data, or sample data can be processed or adjusted beyond a reasonable range to construct more abnormal sample data, thereby obtaining more sample data and making the trained model more accurate.

[0066] As an optional embodiment, after step S201, the method further includes: splitting the normal sample data and the abnormal sample data to obtain a training set, a validation set, and a test set; the divided training set, validation set, and test set include some previously generated normal sample data and some previously generated abnormal sample data. After dividing the training set, validation set, and test set, when training the decision tree model in step S202, the decision tree model can be trained based on the training set; optionally, the above method further includes: adjusting the parameters of the decision tree model based on the validation set, and / or determining the generalization ability of the decision tree model based on the test set.

[0067] Figure 3 This is a schematic diagram of positive and negative sample splitting provided in an embodiment of the present invention, such as... Figure 3 As shown, the positive and negative samples are divided into three datasets: training set, validation set, and test set. 53% is used as the training set to construct the decision tree model; 27% is used as the validation set to adjust parameters and monitor for overfitting; and 20% is used as the test set to evaluate the generalization ability of the final model. It should be noted that the validation set can be used multiple times, while the test set is used only once.

[0068] As an optional embodiment, step S202 includes: starting from the root node of the decision tree model, recursively executing the following steps until a preset condition is met: determining the Gini coefficient of each indicator in the sample data corresponding to the current node; selecting indicators with Gini coefficients that meet the preset condition and their corresponding split points as optimal features and optimal split points; generating the next-level child node from the current node based on the optimal features and optimal split points, and allocating the sample data corresponding to the current node to the next-level child node. When selecting indicators with Gini coefficients that meet the preset condition and their corresponding split points as optimal features and optimal split points, it is preferable to use the indicator with the smallest Gini coefficient and its corresponding split point as the optimal features and optimal split points; alternatively, the second smallest can be selected, or other preset conditions can be met. The following description uses training the decision tree model using a split training set as an example. In practical applications, the sample data splitting can be omitted, and the previously generated normal sample data and abnormal sample data can be used to train the decision tree model.

[0069] Specifically, the decision tree algorithm can be used to train the decision tree model. Starting from the root node, the following operations are recursively performed on each node:

[0070] The first step is to set the training set of the current node as D and calculate the Gini index of each indicator for that training set. At this point, for each indicator X, for each possible value Y, based on whether the test of the sample point X = Y is "yes" or "no", the training set D is divided into two parts, D1 and D2, and the Gini index when X = Y is calculated using Gini(D, X).

[0071] The second step is to select the indicator with the smallest Gini index and its corresponding possible split point from all possible indicators X and all possible split points Y as the most feature and the optimal split point. Based on the optimal feature and the optimal split point, two child nodes are generated from the current node, and the training set is distributed to the two child nodes according to the indicator.

[0072] Step 3: Determine whether the preset conditions are met. If not, recursively call Step 1 and Step 2 on the two child nodes until the preset conditions are met and a decision tree is generated.

[0073] As an optional embodiment, the preset conditions to be met when training the decision tree model are determined based on relevant parameters, which include at least one of the following: the maximum depth of leaf nodes in the decision tree, the minimum number of samples required for node re-split, the minimum number of samples in a leaf node, the minimum sample weight of a leaf node, the maximum number of leaf nodes, the minimum impurity of node splitting, and the information gain threshold. The corresponding preset conditions can be that the above-mentioned relevant parameters are met, for example: the number of leaf nodes reaches the maximum number of leaf nodes, or the depth of the leaf nodes in the decision tree reaches the maximum depth. Assuming the maximum depth of leaf nodes is 3 layers, then during training, training stops after determining the leaf nodes of the 3rd layer, resulting in a trained decision tree model. The above-mentioned relevant parameters can be preset as needed.

[0074] Figure 4 A schematic diagram of a decision tree model provided in an embodiment of the present invention, such as... Figure 4 As shown, the training set corresponding to the root node is the initial sample set. The Gini coefficient for each indicator in the training set is calculated, and the indicator with the smallest Gini coefficient is selected. For example, if the indicator with the smallest Gini coefficient is the gasification rate, then the gasification cost is taken as the optimal feature and the optimal split point. Figure 4As shown, the root node is identified as having the gasification number as its corresponding indicator. Two child nodes are generated from the root node, and the initial sample set is allocated to these two child nodes according to the indicators. For the left child node, the Gini coefficient for each indicator is calculated based on the allocated sample data. The indicator with the smallest Gini coefficient is identified as the imported gas purchase cost, and the imported gas purchase cost is then used as the corresponding indicator for this child node. Two more child nodes are generated from this child node, and the sample set for this child node is allocated to the two child nodes at the next level according to the indicators. For the right child node, the Gini coefficient for each indicator is calculated based on the allocated sample data. The indicator with the smallest Gini coefficient is identified as the purchased gas purchase price, and the purchased gas purchase price is then used as the corresponding indicator for this child node. Two more child nodes are generated from this child node, and the sample set for this child node is allocated to the two child nodes at the next level according to the indicators. This process is repeated layer by layer to determine the leaf nodes and corresponding indicators for each layer, thus generating the decision tree model. In the decision tree model, the first row of data in each node box is the data range of the indicator, the second row is the number of samples, and the third row is the number of positive samples and negative samples. For example, the data range of the root node is less than or equal to 2.023, the number of samples is 5062, the number of positive samples is 2504, and the number of negative samples is 2558.

[0075] The anomaly detection method provided in this embodiment of the invention obtains basic sample data of a natural gas value chain optimization system, and generates normal sample data and abnormal sample data based on the basic sample data; a decision tree model is trained based on the normal sample data and abnormal sample data; that is, this embodiment of the invention generates a sufficient number of positive and negative samples through basic sample data, and then constructs a more accurate decision tree through the sufficient number of positive and negative samples.

[0076] Based on the above embodiments, Figure 5 This is a flowchart illustrating another anomaly detection method provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the anomaly detection method includes:

[0077] Step S301: Train the sub-decision tree model corresponding to each node in the decision tree model.

[0078] Step S302: Obtain the data to be tested from the natural gas value chain optimization system.

[0079] Step S303: Input the data to be detected into the pre-trained decision tree model and output the result of whether the data to be detected is abnormal.

[0080] Step S304: If the output result of the data to be detected is abnormal, the corresponding sub-decision model is called according to the path of the data to be detected in the decision tree model, and the abnormal data is determined and corrected according to the sub-decision model.

[0081] The implementation of steps S302 and S303 in this embodiment is similar to that of steps S101 and S102 in the above embodiment, and will not be repeated here.

[0082] The difference from the above embodiments is that, in order to further trace the cause of the anomaly and make corrections, in this embodiment, a sub-decision tree model corresponding to each node in the decision tree model is trained; if the output result of the data to be detected is abnormal, the corresponding sub-decision model is called according to the path of the data to be detected in the decision tree model, and the abnormal data is determined and corrected according to the sub-decision model.

[0083] The sub-decision tree is essentially a smaller decision tree trained with the corresponding leaf node as its root node. If the granularity of the decision tree trained initially is not fine enough, for example, using... Figure 4 The decision tree can only locate the anomaly in the imported gas procurement cost, which may be caused by other indicators, but it cannot pinpoint which indicator is causing it. Therefore, based on the imported gas procurement cost, we select other indicators that have a relatively large impact on the imported gas procurement cost and create a small decision tree. These indicators have the same granularity as the input data, so that the abnormal input data can be accurately located.

[0084] In this embodiment, a new decision tree is constructed for each node in the decision tree. For example, for cases where constraints such as "transmission volume by substation" and "sales volume by province and industry" are not recorded, a new decision tree is constructed using more granular variables. For instance, for cases where transmission volume by substation is not recorded, indicators such as gasification fee, unit pipeline transportation fee, imported LNG volume at each receiving station, LNG loading volume at each receiving station, and LNG loading rate at each receiving station are selected. When an anomaly is determined in the data to be detected, the cause is investigated and corrected one by one according to the sub-decision tree corresponding to the selected indicator. Table 3 shows the analysis effect of the decision tree for different anomaly situations. Figure 3 As shown, in the negative sample adjustment strategy, if six provinces in the competing provinces are selected to reduce prices (-5% to 0%), or six provinces in the competing provinces are selected to increase sales (0% to 10%), or long pipeline sections are selected to delete mileage, the decision tree analysis will be weak when these data changes are adjusted. If the unit cost of each oil field is multiplied by 2, or the price of each imported gas is multiplied by 2, or the combination constraint of the distribution volume of each station is deleted, or the combination constraint of the sales volume of each province and industry is deleted, the decision tree analysis will be strong when these data changes are adjusted. However, if these anomalies are adjusted at the same time, the anomalies with larger data changes may be ignored because the anomalies with smaller data changes are ignored. Therefore, the embodiments of the present invention analyze different anomalies separately, that is, establish a refined decision tree corresponding to each node.

[0085] Table 3

[0086]

[0087] The anomaly detection method provided in this embodiment of the invention trains a sub-decision tree model corresponding to each node in the decision tree model. If the output result of the data to be detected is abnormal, the corresponding sub-decision tree model is called according to the path of the data to be detected in the decision tree model, and the abnormal data is detected and corrected according to the sub-decision tree model. For example, when the imported gas procurement cost is abnormal, if it is impossible to determine which indicator(s) caused it, the data to be detected is input into the sub-decision tree model corresponding to the imported gas procurement cost to determine which indicator(s) caused the abnormality in the imported gas procurement cost. That is, by constructing a sub-decision tree corresponding to each node in the decision tree model, this embodiment of the invention makes it easier for users to trace the cause of data anomalies and makes it easier to modify.

[0088] To further understand the present invention, an anomaly detection method is also provided, which includes the following steps:

[0089] First, a sample dataset is constructed to obtain various data indicators involved in the optimization calculation of the natural gas industry chain, including oilfield production, gas intake and procurement volume, external gas procurement volume, oilfield settlement price, imported gas procurement price, external gas procurement price, sales volume and sales price in each province, gas storage injection and production volume and injection and production costs, LNG gasification volume and gasification costs, natural gas unit pipeline transportation cost, natural gas unit sales price and profit, etc.

[0090] Secondly, based on the sample dataset, a data adjustment strategy is used to construct normal samples and negative samples where abnormal data appears, and the samples are labeled.

[0091] Next, the positive and negative samples are split into three datasets: 53% is used as the training set to construct the decision tree model; 27% is used as the validation set to adjust parameters and monitor whether the model is overfitting; and 20% is used as the test set to evaluate the generalization ability of the final model.

[0092] Next, a decision tree model is constructed based on positive and negative samples. The labeling result of each group of data, i.e., positive or negative sample, is used as the target variable Y, and the specific parameter index of each group of data is used as the variable X, thereby constructing an anomaly diagnosis decision tree model.

[0093] Finally, the data to be detected is input into the decision tree model. The decision tree model diagnoses and locates abnormal data by determining the root node and leaf nodes, and updates and iterates the decision tree model based on the abnormal results.

[0094] In summary, the embodiments of the present invention take into account the strict logical relationships between the data of the natural gas value chain optimization results. The decision tree algorithm, which is based on the principle of top-down recursion, can be used to find the connection relationship between the data. Moreover, compared with the black box of other machine learning algorithms, the decision tree algorithm can intuitively display the characteristics of the data through graphics, which is easy to understand and interpret.

[0095] Figure 6 A schematic diagram of an abnormal data detection model training device provided in an embodiment of the present invention includes a sample acquisition module 10 and a model detection module 20; wherein:

[0096] The sample acquisition module 10 is used to acquire basic sample data of the natural gas value chain optimization system and generate normal sample data and abnormal sample data based on the basic sample data.

[0097] The model training module 20 is used to train a decision tree model for anomaly detection based on normal sample data and abnormal sample data.

[0098] As an optional embodiment of the present invention, the model training module 20 is specifically used for: obtaining a natural gas value chain optimization scheme for a preset month as basic sample data, wherein the optimization scheme includes input indicators and output indicators of the natural gas value chain optimization system; adjusting at least one input indicator in the basic sample data within a first preset range, and inputting the adjusted input indicator into the natural gas value chain optimization system to obtain the corresponding output indicator, thereby constructing normal sample data; and handling the missing or exceeding a second preset range adjustment of at least one input indicator in the basic sample data, and inputting the missing or adjusted input indicator into the natural gas value chain optimization system to obtain the corresponding output indicator, thereby constructing abnormal sample data.

[0099] As an optional embodiment of the present invention, the model training module 20 is further configured to: split normal sample data and abnormal sample data to obtain a training set, a validation set and a test set; train a decision tree model based on the training set; adjust the parameters of the decision tree model based on the validation set, and / or determine the generalization ability of the decision tree model based on the test set.

[0100] As an optional embodiment of the present invention, the model training module 20 is specifically used to: starting from the root node of the decision tree model, recursively execute the following steps until the preset conditions are met: determine the Gini coefficient of each indicator in the training set corresponding to the current node; select the indicator with the smallest Gini coefficient and the corresponding split point as the optimal feature and the optimal split point; generate the next-level child node from the current node according to the optimal feature and the optimal split point, and divide the training set corresponding to the current node into the next-level child nodes.

[0101] As an optional embodiment of the present invention, the preset conditions are determined according to relevant parameters, wherein the relevant parameters include at least one of the following: the maximum depth of the leaf nodes of the decision tree, the minimum number of samples required for node re-division, the minimum number of leaf node samples, the minimum sample weight of the leaf node, the maximum number of leaf nodes, the minimum impurity of node division, and the information gain threshold.

[0102] As an optional embodiment of the present invention, the model training module 20 is further configured to: train the sub-decision tree model corresponding to each node in the decision tree model; the detection module 10 is further configured to: if the output of the data to be detected is abnormal, call the corresponding sub-decision model according to the path of the data to be detected in the decision tree model, and determine the abnormal data and correct it according to the sub-decision model.

[0103] Figure 7 This is a schematic diagram of an anomaly detection device provided in an embodiment of the present invention. The anomaly detection device includes a data acquisition module 30 and an anomaly detection module 40; wherein:

[0104] Data acquisition module 30 is used to acquire the data to be detected in the natural gas value chain optimization system;

[0105] The anomaly detection module 40 is used to input the data to be detected into a pre-trained decision tree model and output the result of whether the data to be detected is abnormal.

[0106] Based on the above embodiments, this invention also provides an abnormal data detection device, including... Figure 6 The sample acquisition module 10 and model detection module 20 in the device shown also include Figure 7 The data acquisition module 30 and the anomaly detection module 40 in the device shown first train the model, and then use the trained model to detect abnormal data.

[0107] like Figure 8 As shown, this embodiment of the invention provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0108] Memory 113 is used to store computer programs;

[0109] In one embodiment of the present invention, when the processor 111 executes the program stored in the memory 113, it implements the steps of the abnormal data detection model training method and / or abnormal data detection method provided in any of the foregoing method embodiments.

[0110] The electronic device provided in this embodiment of the invention has a similar implementation principle and technical effect to the above embodiments, and will not be described again here.

[0111] The aforementioned memory 113 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Memory 113 has storage space for program code used to perform any of the method steps described above. For example, the storage space for program code may include individual program codes for implementing the various steps in the methods described above. This program code can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, optical discs (CDs), memory cards, or floppy disks. Such computer program products are typically portable or fixed storage units. The storage unit may have storage segments or storage spaces arranged similarly to memory 113 in the aforementioned electronic device. The program code may be compressed, for example, in a suitable form. Typically, the storage unit includes programs for performing the method steps according to embodiments of the invention, i.e., code that can be read by a processor such as 111, which, when run by the electronic device, causes the electronic device to perform the various steps in the methods described above.

[0112] Embodiments of the present invention also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the anomaly detection model training method and / or anomaly detection method as described above.

[0113] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or it may exist independently and not assembled into the device / apparatus. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of the present invention.

[0114] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0115] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0116] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for training an anomaly data detection model, characterized in that, include: Acquiring basic sample data for a natural gas value chain optimization system, and generating normal and abnormal sample data based on the basic sample data; including: acquiring a natural gas value chain optimization scheme for a preset time period as the basic sample data, the optimization scheme including input and output indicators of the natural gas value chain optimization system; adjusting at least one input indicator in the basic sample data within a first preset range, and inputting the adjusted input indicator into the natural gas value chain optimization system to obtain the corresponding output indicator, thus constructing normal sample data; and handling missing or exceeding a second preset range adjustment for at least one input indicator in the basic sample data, and inputting the missing or adjusted input indicator into the natural gas value chain optimization system to obtain the corresponding output indicator, thus constructing abnormal sample data. A decision tree model for anomaly detection is trained based on the normal sample data and the abnormal sample data, including: starting from the root node of the decision tree model, recursively executing the following steps until a preset condition is met: determining the Gini coefficient of each indicator in the sample data corresponding to the current node; selecting indicators whose Gini coefficient meets the preset condition and the corresponding split points as the optimal features and optimal split points; generating the next-level child node from the current node based on the optimal features and optimal split points, and dividing the sample data corresponding to the current node into the next-level child nodes; The system trains sub-decision tree models corresponding to each node in the decision tree model. If the output shows that the data to be detected is abnormal, the system calls the corresponding sub-decision model according to the path of the data to be detected in the decision tree model, and determines and corrects the abnormal data according to the sub-decision model.

2. The method according to claim 1, characterized in that, After generating normal sample data and abnormal sample data based on the basic sample data, the process further includes: The normal sample data and abnormal sample data are split to obtain a training set, a validation set, and a test set; The step of training a decision tree model based on the normal sample data and the abnormal sample data includes: A decision tree model is obtained by training the training set. The method further includes: adjusting the parameters of the decision tree model according to the validation set, and / or determining the generalization ability of the decision tree model according to the test set.

3. The method according to claim 1, characterized in that, The preset conditions are determined based on parameters, wherein the parameters include at least one of the following: The decision tree determines the maximum depth of a leaf node, the minimum number of samples required for node re-partitioning, the minimum number of samples in a leaf node, the minimum sample weight of a leaf node, the maximum number of leaf nodes, the minimum impurity of node partitioning, and the information gain threshold.

4. An abnormal data detection method, characterized in that, include: Acquire the data to be monitored by the natural gas value chain optimization system; The data to be detected is input into a pre-trained decision tree model, and the result of whether the data to be detected is abnormal is output. The decision tree model is trained using the abnormal data detection model training method described in any one of claims 1-3.

5. An anomaly data detection model training device, characterized in that, include: A sample acquisition module is used to acquire basic sample data of a natural gas value chain optimization system and generate normal sample data and abnormal sample data based on the basic sample data. This includes: acquiring a natural gas value chain optimization scheme for a preset time period as the basic sample data, the optimization scheme including input and output indicators of the natural gas value chain optimization system; adjusting at least one input indicator in the basic sample data within a first preset range, and inputting the adjusted input indicator into the natural gas value chain optimization system to obtain the corresponding output indicator, thus constructing normal sample data; and handling missing or out-of-range adjustments for at least one input indicator in the basic sample data, and inputting the missing or adjusted input indicator into the natural gas value chain optimization system to obtain the corresponding output indicator, thus constructing abnormal sample data. The model training module, used to train a decision tree model for anomaly detection based on the normal sample data and the anomaly sample data, includes: starting from the root node of the decision tree model, recursively executing the following steps until a preset condition is met: determining the Gini coefficient of each indicator in the sample data corresponding to the current node; selecting indicators whose Gini coefficients meet the preset condition and their corresponding split points as optimal features and optimal split points; generating the next-level child node from the current node based on the optimal features and optimal split points, and assigning the sample data corresponding to the current node to the next-level child node; The system trains sub-decision tree models corresponding to each node in the decision tree model. If the output shows that the data to be detected is abnormal, the system calls the corresponding sub-decision model according to the path of the data to be detected in the decision tree model, and determines and corrects the abnormal data according to the sub-decision model.

6. An abnormal data detection device, characterized in that, include: The data acquisition module is used to acquire the data to be monitored by the natural gas value chain optimization system; An anomaly detection module is used to input the data to be detected into a pre-trained decision tree model and output the result of whether the data to be detected is abnormal. The decision tree model is trained using the abnormal data detection model training method described in any one of claims 1-3.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor, when executing a program stored in memory, implements the abnormal data detection model training method according to any one of claims 1-3 and / or the abnormal data detection method according to claim 4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the abnormal data detection model training method as described in any one of claims 1-3 and / or the abnormal data detection method as described in claim 4.

Images