Data processing method and device of financial system, equipment and storage medium

Abstract

Embodiments of the present application disclose a kind of financial system data processing method and device, equipment and storage medium, method includes: obtaining the operation and maintenance monitoring data of multiple data categories of financial system and offline prediction value;With operation and maintenance monitoring data and offline prediction value, determine the operation and maintenance monitoring data of each data category is used to reflect whether the first classification result of operation and maintenance monitoring data is abnormal data;If the first classification result indicates that operation and maintenance monitoring data is abnormal data, then determine operation and maintenance monitoring data as target operation and maintenance monitoring data, and using the target data category of target operation and maintenance monitoring data and target operation and maintenance monitoring data, determine the second classification result of target operation and maintenance monitoring data using preset judgment rule, target operation and maintenance monitoring data. Through the above-mentioned mode, secondary classification can be carried out, and the data processing pressure during secondary classification is reduced, and the accuracy of the classification result of the target operation and maintenance monitoring data with abnormality is improved after secondary classification, and the error is reduced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method, apparatus, equipment, and storage medium for a financial system. Background Technology

[0002] Operational monitoring plays a crucial role in the operations and maintenance (O&M) teams of financial systems. Various internet companies in the financial industry have their own O&M development teams, and the types of data they monitor vary greatly as the financial business of the system grows. Common O&M monitoring data can be broadly categorized into the following main types: infrastructure monitoring, application monitoring, and company-specific critical monitoring data. These main categories are further subdivided into many subcategories. For example, infrastructure monitoring includes CPU usage, memory usage, storage capacity, and network status; application monitoring includes databases, Nginx (servers), SLB (Server Load Balancer), and ESB (Enterprise Service Bus); and company-specific critical monitoring data includes login volume, average login time, and user login success rate for a particular system.

[0003] Traditional monitoring solutions typically involve operations and maintenance personnel defining rules for normal data. The collected data is then evaluated based on these rules to determine whether it is normal. However, facing ever-increasing operational demands, traditional monitoring solutions have many shortcomings. For example, as financial businesses evolve, the types and quantities of data are increasing. Large financial companies, for instance, may have hundreds of thousands or even millions of servers. Rules are insufficient to meet the needs of all data, leading to significant errors in the judgment of abnormal or normal data. Therefore, traditional methods of processing operational monitoring data contain substantial errors. Summary of the Invention

[0004] The main objective of this invention is to provide a data processing method, apparatus, device, and storage medium for a financial system, which can solve the problem of large errors in traditional operation and maintenance monitoring data processing methods in the prior art.

[0005] To achieve the above objectives, the first aspect of the present invention provides a data processing method for a financial system, the method comprising:

[0006] Obtain operation and maintenance monitoring data and offline predicted values ​​for multiple data types of the financial system. The offline predicted values ​​are used to reflect the normal baseline values ​​of operation and maintenance monitoring data for each data type obtained by offline prediction of historical operation and maintenance monitoring data for each data type.

[0007] Using the operation and maintenance monitoring data and the offline predicted value, a first classification result of the operation and maintenance monitoring data for each data type is determined. The first classification result is used to reflect whether the operation and maintenance monitoring data is abnormal data.

[0008] If the first classification result indicates that the operation and maintenance monitoring data is abnormal data, then the operation and maintenance monitoring data is determined to be the target operation and maintenance monitoring data. Then, using the preset judgment rules, the target operation and maintenance monitoring data and the target data type of the target operation and maintenance monitoring data, the second classification result of the target operation and maintenance monitoring data is determined. The judgment rules are used to reflect the abnormal judgment conditions of operation and maintenance monitoring data of different data types.

[0009] If the first classification result indicates that the operation and maintenance monitoring data is non-abnormal data, then return to the step of obtaining the operation and maintenance monitoring data and offline predicted values ​​of the financial system for multiple data types.

[0010] In one feasible implementation, the judgment rule at least includes a preset correspondence between data types and thresholds. Then, determining the second classification result of the target operation and maintenance monitoring data using the preset judgment rule, the target operation and maintenance monitoring data, and the target data types of the target operation and maintenance monitoring data includes:

[0011] Based on the target data type, find the target threshold corresponding to the target data type from the correspondence between the data type and the threshold;

[0012] If the operation and maintenance monitoring data is greater than or equal to the target threshold, then the second classification result is determined to be abnormal data;

[0013] If the operation and maintenance monitoring data is less than the target threshold, then the second classification result is determined to be non-abnormal data.

[0014] In one feasible implementation, the judgment rule at least includes a preset correspondence between data types and data change patterns. Then, determining the second classification result of the target operation and maintenance monitoring data using the preset judgment rule, the target operation and maintenance monitoring data, and the target data types of the target operation and maintenance monitoring data includes:

[0015] Based on the target data type, find the target data change pattern corresponding to the target data type from the correspondence between the data type and the data change pattern;

[0016] Using a preset time period and the target operation and maintenance monitoring data, determine the first data change pattern within the preset time period;

[0017] If the error between the first data change pattern and the target data change pattern is greater than or equal to a preset error threshold, then the second classification result is determined to be abnormal data;

[0018] If the error between the first data change pattern and the target data change pattern is less than a preset error threshold, then the second classification result is determined to be non-abnormal data.

[0019] In one feasible implementation, the judgment rule further includes a machine learning model. Then, determining the second classification result of the target operation and maintenance monitoring data using the preset judgment rule, the target operation and maintenance monitoring data, and the target data type of the target operation and maintenance monitoring data includes:

[0020] The target operation and maintenance monitoring data and the target data type are input into the machine learning model to determine the output value of the machine learning model. The output value is used to reflect the probability that the target operation and maintenance monitoring data is abnormal data.

[0021] If the output value is greater than or equal to a preset probability threshold, then the second classification result is determined to be abnormal data;

[0022] If the output value is less than a preset probability threshold, then the second classification result is determined to be non-abnormal data.

[0023] In one feasible implementation, acquiring the operation and maintenance monitoring data of the financial system for various data types includes:

[0024] Raw operation and maintenance monitoring data of various data types are obtained from the pre-set message middleware in the financial system;

[0025] The original operation and maintenance monitoring data is formatted using preset formatting rules to obtain the operation and maintenance monitoring data. The formatting rules include standard data formats corresponding to various data types.

[0026] In one feasible implementation, determining the first classification result of the operation and maintenance monitoring data for each data type using the operation and maintenance monitoring data and the offline predicted value includes:

[0027] Determine whether the operation and maintenance monitoring data falls within the target value range corresponding to the offline predicted value;

[0028] If the operation and maintenance monitoring data falls within the target value range, then the first classification result of the operation and maintenance monitoring data is determined to be non-abnormal data;

[0029] If the operation and maintenance monitoring data is not within the target value range, then the first classification result of the operation and maintenance monitoring data is determined to be abnormal data.

[0030] In one feasible implementation, obtaining the offline predicted value includes:

[0031] Send a data acquisition request to a preset offline financial system, which is used to calculate the offline predicted value of historical operation and maintenance monitoring data of each data type;

[0032] Receive the offline predicted value returned by the offline financial system.

[0033] To achieve the above objectives, a second aspect of the present invention provides a data processing apparatus for a financial system, the apparatus comprising:

[0034] Data acquisition module: used to acquire operation and maintenance monitoring data and offline predicted values ​​of multiple data types of the financial system. The offline predicted values ​​are used to reflect the normal baseline values ​​of operation and maintenance monitoring data of each data type obtained by offline prediction of historical operation and maintenance monitoring data of each data type.

[0035] First processing module: used to determine the first classification result of the operation and maintenance monitoring data for each data type using the operation and maintenance monitoring data and the offline predicted value, the first classification result being used to reflect whether the operation and maintenance monitoring data is abnormal data;

[0036] The second processing module is used to determine the operation and maintenance monitoring data as target operation and maintenance monitoring data if the first classification result indicates that the operation and maintenance monitoring data is abnormal data, and to determine the second classification result of the target operation and maintenance monitoring data by using preset judgment rules, the target operation and maintenance monitoring data and the target data type of the target operation and maintenance monitoring data. The judgment rules are used to reflect the abnormal judgment conditions of operation and maintenance monitoring data of different data types.

[0037] The third processing module is used to return to the step of obtaining the multi-data-type operation and maintenance monitoring data and offline predicted values ​​of the financial system if the first classification result indicates that the operation and maintenance monitoring data is non-abnormal data.

[0038] To achieve the above objectives, a third aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps shown in the first aspect and any feasible implementation.

[0039] To achieve the above objectives, a fourth aspect of the present invention provides a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps shown in the first aspect and any feasible implementation.

[0040] The embodiments of the present invention have the following beneficial effects:

[0041] This invention provides a data processing method for a financial system. The method includes: acquiring operation and maintenance monitoring data and offline predicted values ​​for multiple data types of the financial system, wherein the offline predicted values ​​reflect the normal baseline values ​​of operation and maintenance monitoring data for each data type obtained by offline prediction from historical operation and maintenance monitoring data for each data type; using the operation and maintenance monitoring data and offline predicted values, determining a first classification result for the operation and maintenance monitoring data for each data type, wherein the first classification result reflects whether the operation and maintenance monitoring data is abnormal data; if the first classification result indicates that the operation and maintenance monitoring data is abnormal data, then determining the operation and maintenance monitoring data as target operation and maintenance monitoring data, and using preset judgment rules, the target operation and maintenance monitoring data, and the target data type of the target operation and maintenance monitoring data, determining a second classification result for the target operation and maintenance monitoring data, wherein the judgment rules reflect the abnormal judgment conditions for operation and maintenance monitoring data of different data types; if the first classification result indicates that the operation and maintenance monitoring data is not abnormal data, then returning to the step of acquiring operation and maintenance monitoring data and offline predicted values ​​for multiple data types of the financial system. Using the above method, offline predicted values ​​can be used to classify the operation and maintenance monitoring data first, and then the target operation and maintenance monitoring data that is abnormal in the first classification can be classified a second time. This reduces the data processing pressure during the second classification, and the second classification also improves the accuracy of the classification results of the target operation and maintenance monitoring data with anomalies and reduces errors. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] in:

[0044] Figure 1 This is a flowchart of a data processing method for a financial system according to an embodiment of the present invention;

[0045] Figure 2 This is another flowchart of a data processing method for a financial system according to an embodiment of the present invention;

[0046] Figure 3 This is a structural block diagram of a data processing device for a financial system according to an embodiment of the present invention;

[0047] Figure 4 This is a structural block diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Please see Figure 1 , Figure 1 This is a flowchart of a data processing method for a financial system according to an embodiment of the present invention, such as... Figure 1 The method shown can be applied to both terminals and servers. This embodiment illustrates its application to a server. The terminal can be a desktop terminal or a mobile terminal; a mobile terminal can be at least one of a mobile phone, tablet, or laptop. The server can be a standalone server or a server cluster consisting of multiple servers. Figure 1 The method shown specifically includes the following steps:

[0050] 101. Obtain the operation and maintenance monitoring data and offline predicted values ​​of the financial system for multiple data types. The offline predicted values ​​are used to reflect the normal baseline values ​​of the operation and maintenance monitoring data for each data type obtained by offline prediction of the historical operation and maintenance monitoring data for each data type.

[0051] It should be noted that the financial system can process financial transactions. The financial system can be a terminal or a server; this embodiment uses a financial system as a server as an example. The financial system can not only process financial transactions but also perform its own operation and maintenance monitoring. That is, this application can also be executed by the financial system, which is not limited here. Operation and maintenance monitoring data reflects the operational status data of various data types during server operation. It is obtained through real-time monitoring of the server's operation. For example, operation and maintenance monitoring data includes, but is not limited to, first data of infrastructure monitoring, second data of application monitoring, and third data of some key monitoring categories specific to the company. The first data of infrastructure monitoring can be further divided into data corresponding to CPU usage, memory usage, storage capacity, and network status; the second data of application monitoring can be further divided into data corresponding to database, Nginx, SLB, and ESB; and the third data of some key monitoring categories specific to the company can be further divided into data corresponding to login volume, average login time, and user login success rate of a certain system.

[0052] To process the operation and maintenance monitoring data of the financial system, it is first necessary to acquire operation and maintenance monitoring data and offline predicted values ​​for multiple data types. The offline predicted values ​​reflect the normal baseline values ​​of each data type's operation and maintenance monitoring data, obtained through offline prediction from historical operation and maintenance monitoring data. In other words, offline predicted values ​​can be obtained by offline prediction from historical operation and maintenance monitoring data for each data type. Therefore, when offline predicted values ​​are needed, the corresponding offline predicted values ​​can be obtained to process the operation and maintenance monitoring data. Taking the financial system as an online system as an example, the online system only needs to obtain the offline predicted values ​​from the offline system. Therefore, the online system does not directly use the model of the external offline system, but uses the output results of the offline system. Regardless of the algorithm, as long as the output is a value or classification, the online system directly imports the results for use. Therefore, it does not concern itself too much with the implementation of the external model itself, thus exhibiting good compatibility with the algorithms of external systems.

[0053] 102. Using the operation and maintenance monitoring data and the offline predicted value, determine the first classification result of the operation and maintenance monitoring data for each data type, wherein the first classification result is used to reflect whether the operation and maintenance monitoring data is abnormal data;

[0054] Furthermore, after obtaining the operation and maintenance monitoring data and offline predicted values, the first classification result for each data type of operation and maintenance monitoring data can be determined using these data. This first classification result reflects whether the operation and maintenance monitoring data is abnormal. Since the offline predicted values ​​can reflect normal baseline values, they can be used as a benchmark to determine whether the operation and maintenance monitoring data is normal or not. Different subsequent processing can be applied to the operation and maintenance monitoring data based on whether it is abnormal or not.

[0055] 103. If the first classification result indicates that the operation and maintenance monitoring data is abnormal data, then the operation and maintenance monitoring data is determined to be the target operation and maintenance monitoring data, and the second classification result of the target operation and maintenance monitoring data is determined by using the preset judgment rule, the target operation and maintenance monitoring data and the target data type of the target operation and maintenance monitoring data. The judgment rule is used to reflect the abnormal judgment conditions of operation and maintenance monitoring data of different data types.

[0056] 104. If the first classification result indicates that the operation and maintenance monitoring data is non-abnormal data, then return to the step of obtaining the operation and maintenance monitoring data and offline predicted values ​​of the financial system for multiple data types.

[0057] It should be noted that if the first classification result indicates that the operation and maintenance monitoring data is abnormal, then the operation and maintenance monitoring data is indeed abnormal. Step 103 will then be executed to perform a second judgment on the abnormal operation and maintenance monitoring data to determine whether it truly belongs to abnormal data. Therefore, when the first classification result indicates that the operation and maintenance monitoring data is abnormal, the abnormal operation and maintenance monitoring data will be identified as the target operation and maintenance monitoring data. Then, using preset judgment rules, the target operation and maintenance monitoring data, and the target data type of the target operation and maintenance monitoring data, the second classification result of the target operation and maintenance monitoring data will be determined. Conversely, if the first classification result indicates that the operation and maintenance monitoring data is not abnormal, then step 104 can be executed to continue the operation and maintenance monitoring of the financial system.

[0058] This invention provides a data processing method for a financial system. The method includes: acquiring operation and maintenance monitoring data and offline predicted values ​​for multiple data types of the financial system, wherein the offline predicted values ​​reflect the normal baseline values ​​of operation and maintenance monitoring data for each data type obtained by offline prediction from historical operation and maintenance monitoring data for each data type; using the operation and maintenance monitoring data and offline predicted values, determining a first classification result for the operation and maintenance monitoring data for each data type, wherein the first classification result reflects whether the operation and maintenance monitoring data is abnormal data; if the first classification result indicates that the operation and maintenance monitoring data is abnormal data, then determining the operation and maintenance monitoring data as target operation and maintenance monitoring data, and using preset judgment rules, the target operation and maintenance monitoring data, and the target data type of the target operation and maintenance monitoring data, determining a second classification result for the target operation and maintenance monitoring data, wherein the judgment rules reflect the abnormal judgment conditions for operation and maintenance monitoring data of different data types; if the first classification result indicates that the operation and maintenance monitoring data is not abnormal data, then returning to the step of acquiring operation and maintenance monitoring data and offline predicted values ​​for multiple data types of the financial system. Using the above method, offline predicted values ​​can be used to classify the operation and maintenance monitoring data first, and then the target operation and maintenance monitoring data that is abnormal in the first classification can be classified a second time. This reduces the data processing pressure during the second classification, and the second classification also improves the accuracy of the classification results of the target operation and maintenance monitoring data with anomalies and reduces errors.

[0059] Please see Figure 2 , Figure 2 This is another flowchart of a data processing method for a financial system according to an embodiment of the present invention, as shown below. Figure 2 The method shown includes the following steps:

[0060] 201. Obtain the operation and maintenance monitoring data and offline prediction values ​​of the financial system for multiple data types. The offline prediction values ​​are used to reflect the normal baseline values ​​of the operation and maintenance monitoring data of each data type obtained by offline prediction of the historical operation and maintenance monitoring data of each data type.

[0061] It should be noted that the content shown in step 201 is the same as... Figure 1 The content of step 101 shown is similar, and will not be repeated here to avoid repetition. For details, please refer to [link / reference needed]. Figure 1 The content of step 101 shown.

[0062] In one feasible implementation, the acquisition of various types of operation and maintenance monitoring data from the financial system includes steps A01 to A02:

[0063] A01. Obtain various types of raw operation and maintenance monitoring data from the message middleware preset in the financial system;

[0064] A02. The original operation and maintenance monitoring data is formatted using preset formatting rules to obtain the operation and maintenance monitoring data. The formatting rules include standard data formats corresponding to various data types.

[0065] It should be noted that for the monitoring data collected by the upstream components, Spark is used to consume the data from the message middleware Kafka. The online system can consume different types of data from different Kafka topics. Each topic can contain one or more types of monitoring data. For different types of data, the online system will first perform standard formatting on the data to make the data of different formats formatted into a unified data format.

[0066] Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action streams from consumers on a website. These actions (web browsing, searching, and other user actions) are a key element of many social functions on the modern web. This data is often handled through log processing and log aggregation due to throughput requirements. It's a viable solution for log data and offline analytics systems like Hadoop, but with the constraint of requiring real-time processing. Kafka aims to unify online and offline message processing through Hadoop's parallel loading mechanism, and also to provide real-time messaging through clustering.

[0067] Furthermore, various types of raw operation and maintenance monitoring data can be obtained from pre-set message middleware within the financial system. Message middleware includes, but is not limited to, distributed publish-subscribe messaging systems such as Kafka. The raw operation and maintenance monitoring data can be stored in different Kafka topics. Then, pre-set formatting rules are used to format the raw operation and maintenance monitoring data, resulting in operation and maintenance monitoring data. These formatting rules include standard data formats corresponding to various data types.

[0068] For example, the formatting method can be as follows: For each piece of data that needs to be alerted (i.e., the operation and maintenance monitoring data), format it into a standard data format such as "alert subject --> alert metric". For instance, if we need to monitor "the number of logins per minute and the average login time per minute of Pocket Bank", then our data will be formatted as "Pocket Bank:{login volume:1000, average time:200}". In this example, Pocket Bank is the alert subject, and the login volume and average time are the alert metrics.

[0069] The data source format varies depending on the alarm service. Data for different service scenarios is similar to the format above, requiring only data filtering (removing dirty data). However, for basic host data, each data entry lacks a clear hierarchical relationship; it simply indicates which metric corresponds to which server and the quantity (e.g., {CPU_USED:90, CICODE:SYSOPS00034367} only indicates that the CPU utilization of server SYSOPS00034367 is 90%). Since the alarms are primarily targeted at each server, and each server has many monitoring metrics (the above is CPU utilization, and the Kafka data stream also records the server's memory utilization, etc.), this system aggregates and formats all metrics from the same server into the above format. This further facilitates the formatting of various data types.

[0070] In one feasible implementation, obtaining the offline predicted value includes steps B01 to B02:

[0071] B01. Send a data acquisition request to a preset offline financial system, which is used to calculate the offline predicted value of historical operation and maintenance monitoring data of each data type.

[0072] B02. Receive the offline predicted value returned by the offline financial system.

[0073] The offline predicted value is calculated by an offline system, which can be an offline financial system. This offline system has a communication connection with the server, and the server can send a data acquisition request to the preset offline financial system to request the offline predicted value. The offline financial system can pre-calculate the offline predicted value using historical operation and maintenance monitoring data of various data types, and return the offline predicted value when it receives the data acquisition request. Alternatively, it can calculate the offline predicted value using historical operation and maintenance monitoring data of various data types and return it to the server only when it receives the data acquisition request. This application does not impose any limitations on this.

[0074] The use of offline predicted values ​​enables the integration of online and offline systems. The offline system predicts the data to be monitored and calculates the predicted values. The online system uses these offline predicted values ​​as a preliminary indicator of anomalous data. If the predicted values ​​do not meet the range calculated offline, the data is initially marked as anomalous. For example, the offline system trains a model using historical data and then predicts the data for the next day based on this historical data. The predicted values ​​are then adjusted upwards and downwards by a certain percentage to obtain the maximum and minimum values, thus determining a specific data range (i.e., the target value range). Different data exhibit different waveform characteristics, requiring different algorithms. The offline system selects an appropriate algorithm based on the nature of the data to predict future data. For example, stationary data might use a month-on-month mean model, while data with strong periodicity might use a Fourier transform (prophet) algorithm. This example does not impose specific limitations.

[0075] 202. Determine whether the operation and maintenance monitoring data is within the target value range corresponding to the offline predicted value;

[0076] 203. If the operation and maintenance monitoring data is within the target value range, then the first classification result of the operation and maintenance monitoring data is determined to be non-abnormal data;

[0077] 204. If the operation and maintenance monitoring data is not within the target value range, then the first classification result of the operation and maintenance monitoring data is determined to be abnormal data;

[0078] Furthermore, by comparing the operation and maintenance monitoring data with the target value range corresponding to the offline predicted value, it is determined whether the operation and maintenance monitoring data is within the target value range corresponding to the offline predicted value. If the operation and maintenance monitoring data is within the target value range, step 203 is executed to determine that the first classification result of the operation and maintenance monitoring data is non-abnormal data. If the operation and maintenance monitoring data is not within the target value range, step 204 is executed to determine that the first classification result of the operation and maintenance monitoring data is abnormal data, thereby obtaining the first classification result.

[0079] 205. If the first classification result indicates that the operation and maintenance monitoring data is abnormal data, then the operation and maintenance monitoring data is determined to be the target operation and maintenance monitoring data, and the second classification result of the target operation and maintenance monitoring data is determined by using the preset judgment rule, the target operation and maintenance monitoring data and the target data type of the target operation and maintenance monitoring data. The judgment rule is used to reflect the abnormal judgment conditions of operation and maintenance monitoring data of different data types.

[0080] It should be noted that the content of step 205 is related to... Figure 1The content of step 103 shown is similar, and will not be repeated here to avoid repetition. For details, please refer to [link / reference needed]. Figure 1 The content of step 103 shown.

[0081] For data that does not meet the predictions of the offline algorithm, the online system further identifies it based on specific rules. These rules include basic threshold range rules, historical data pattern comparison rules, and machine learning algorithm rules, which are categorized by type as regression prediction algorithms (linear fitting algorithm, nearest neighbor ratio algorithm, grey prediction algorithm, etc.) and classification algorithms (isolation forest algorithm, etc.). These online algorithms and rules further identify data that are initially identified as anomalous.

[0082] Taking the threshold range rule as an example, the judgment rule at this time includes at least the preset correspondence between data types and thresholds. The step of determining the second classification result of the target operation and maintenance monitoring data using the preset judgment rule, the target operation and maintenance monitoring data, and the target data types of the target operation and maintenance monitoring data includes steps C01, C02, and C03:

[0083] C01. Based on the target data type, find the target threshold corresponding to the target data type from the correspondence between the data type and the threshold;

[0084] Different data types correspond to different thresholds. This application pre-stores the correspondence between data types and thresholds in the database. Then, for target operation and maintenance monitoring data whose first classification result is abnormal data, step C01 can be used to find the target threshold corresponding to the target data type from the correspondence between data types and thresholds based on the target data type of the target operation and maintenance monitoring data. The target threshold is then used as the threshold of the target operation and maintenance monitoring data for comparison to determine the second classification result of the target operation and maintenance monitoring data.

[0085] C02. If the operation and maintenance monitoring data is greater than or equal to the target threshold, then the second classification result is determined to be abnormal data;

[0086] C03. If the operation and maintenance monitoring data is less than the target threshold, then the second classification result is determined to be non-abnormal data.

[0087] If the operation and maintenance monitoring data is greater than or equal to the target threshold, the second classification result is determined to be abnormal data; if the operation and maintenance monitoring data is less than the target threshold, the second classification result is determined to be non-abnormal data. If the second classification result is non-abnormal data, then step 201 can be returned to continue monitoring the financial system for real-time detection. If the second classification result is abnormal data, then an alarm is output to indicate the existence of an anomaly.

[0088] Taking historical data pattern comparison rules as an example, the judgment rules at this time include at least the preset correspondence between data types and data change patterns. Therefore, the step of determining the second classification result of the target operation and maintenance monitoring data using the preset judgment rules, the target operation and maintenance monitoring data, and the target data types of the target operation and maintenance monitoring data includes steps D01, D02, D03, and D04:

[0089] D01. Based on the target data type, find the target data change pattern corresponding to the target data type from the correspondence between the data type and the data change pattern;

[0090] In other words, anomaly detection can also be performed from the perspective of data change patterns. It can be understood that data change patterns are used to reflect the changing trends of operation and maintenance monitoring data. Among them, the data change patterns of operation and maintenance monitoring data are different for different data types. The database can also store the correspondence between data types and data change patterns, and then, based on the target data type, find the target data change pattern corresponding to the target data type from the correspondence between data types and data change patterns.

[0091] D02. Using the preset time length and the target operation and maintenance monitoring data, determine the first data change pattern under the preset time length;

[0092] Furthermore, by utilizing the target operation and maintenance monitoring data and the time duration, the first data change pattern of the target operation and maintenance monitoring data under that time duration is determined. The first data change pattern is compared with the target data change pattern to determine whether there are any anomalies in the target operation and maintenance monitoring data.

[0093] D03. If the error between the first data change pattern and the target data change pattern is greater than or equal to a preset error threshold, then the second classification result is determined to be abnormal data.

[0094] D04. If the error between the first data change pattern and the target data change pattern is less than a preset error threshold, then the second classification result is determined to be non-abnormal data.

[0095] Specifically, the patterns of change in the first data and the target data represent trends. If there is a significant difference between their trends, the likelihood of an anomaly is higher. Therefore, if the error between the first and target data patterns of change is greater than or equal to a preset error threshold, the second classification result is determined to be abnormal data; if the error is less than the preset error threshold, the second classification result is determined to be non-abnormal data. Data patterns of change include, but are not limited to, data change curves determined by algorithms such as fitting.

[0096] When evaluating the current data, the online system also uses data from a recent period and employs a regression algorithm to fit the recent data trend (for example, using the last 100 data points of this indicator as the fitted data). Then, it uses the fitted function to predict the data at the current moment. Based on the prediction, a safety valve is applied. If the deviation between the actual value and the predicted value is too large, the online system will identify the data as abnormal. If it is within the safety valve range, the online system will identify it as normal data.

[0097] Taking machine learning algorithm rules as an example, the judgment rule here also includes a machine learning model. This machine learning model is trained using training samples corresponding to historical operation and maintenance data. The training samples include the correspondence between historical operation and maintenance data and data labels. Data labels include status labels and data category labels. Status labels include abnormal data or non-abnormal data, and data category labels include, but are not limited to, various data categories. The step of determining the second classification result of the target operation and maintenance monitoring data using the preset judgment rule, the target operation and maintenance monitoring data, and the target data category of the target operation and maintenance monitoring data may further include steps E01, E02, and E03:

[0098] E01. Input the target operation and maintenance monitoring data and the target data type into the machine learning model, and determine the output value of the machine learning model. The output value is used to reflect the probability that the target operation and maintenance monitoring data is abnormal data.

[0099] E02. If the output value is greater than or equal to the preset probability threshold, then the second classification result is determined to be abnormal data.

[0100] E03. If the output value is less than the preset probability threshold, then the second classification result is determined to be non-abnormal data.

[0101] The aforementioned machine learning model can be a classification model. The target operation and maintenance monitoring data and the type of target data can be input into the classification model to obtain its output value. This output value reflects the probability that the target operation and maintenance monitoring data is abnormal data. Furthermore, a high probability indicates that the second classification result is abnormal data, while a low probability indicates that the second classification result is not abnormal data.

[0102] For example, classification algorithms such as Isolation Forest can also classify data based on recent data. Based on the assumption that most recent data is normal data, if the current data does not belong to the same category as the recent data, we will identify it as abnormal data, and otherwise as normal data.

[0103] In one feasible implementation, after processing different data types, the online system uniformly formats the data and outputs the detection results, such as the first classification result and the second classification result, along with other necessary information, to downstream components. For example, the formatted results can be returned to Kafka for output.

[0104] For tasks involving massive amounts of data from a single monitoring service, the online system can appropriately split the data based on the actual situation. For example, if the number of services requiring monitoring is large, servers can be divided into regions, and different Spark tasks can consume data from servers in different regions, thereby reducing the data volume pressure on a single task. Simultaneously, to address the issue of large data volumes due to the variety of monitoring services, the online system can also create different Spark tasks to monitor different types of data. Apache Spark is a fast and versatile computing engine designed specifically for large-scale data processing.

[0105] 206. If the first classification result indicates that the operation and maintenance monitoring data is non-abnormal data, then return to the step of obtaining the operation and maintenance monitoring data and offline predicted values ​​of the financial system for multiple data types.

[0106] It should be noted that the content of step 206 is related to... Figure 1 The content of step 104 is similar, and will not be repeated here to avoid repetition. For details, please refer to [link / reference needed]. Figure 1 The content of step 104.

[0107] This invention provides a data processing method for a financial system, which has the following advantages: First, for different types of business, different Spark tasks will consume different Kafka data sources, distributing different types of data across different tasks for processing. Second, for the same business, Spark will distribute the large amount of data consumed from Kafka to different servers, allowing multiple servers to process multiple data entries simultaneously in parallel, rather than a single server processing data entry one by one. Therefore, it solves the problem of large monitoring data volumes. For large data volumes, the online system adopts big data Spark technology and distributed processing technology, improving processing performance for large amounts of data. For different types of data, the online monitoring system has a standardized data formatting mechanism that can format data into a unified standard for processing. Secondly, offline algorithms vary in type, requiring different conditions, configuration inputs, and even different trained model formats and outputs. The online system does not directly use the models of external offline systems; instead, it uses the output results. Regardless of the algorithm, as long as the output is a value or classification, the online system directly imports the results for use. Therefore, it doesn't concern itself much with the implementation of the external model, resulting in excellent compatibility with external system algorithms. The online system does not require specific types of offline algorithms; it only uses the predicted values ​​calculated by offline algorithms. In other words, it is compatible with any type of offline algorithm and the type of offline predicted values ​​(regression values, classification values). Finally, the system supports online rules and algorithms. For monitoring data of different natures, the online system provides a rich set of online rules and algorithms. These machine learning algorithms can be used on monitoring data of various natures, greatly improving the accuracy of anomaly detection and significantly reducing the reliance on manual intervention in traditional rules.

[0108] Please see Figure 3 , Figure 3 This is a structural block diagram of a data processing device for a financial system according to an embodiment of the present invention, such as... Figure 3 The apparatus shown includes:

[0109] Data acquisition module 301: used to acquire operation and maintenance monitoring data and offline predicted values ​​of multiple data types of the financial system. The offline predicted values ​​are used to reflect the normal baseline values ​​of operation and maintenance monitoring data of each data type obtained by offline prediction of historical operation and maintenance monitoring data of each data type.

[0110] First processing module 302: used to determine a first classification result of the operation and maintenance monitoring data for each data type using the operation and maintenance monitoring data and the offline prediction value, wherein the first classification result is used to reflect whether the operation and maintenance monitoring data is abnormal data;

[0111] The second processing module 303 is used to determine the operation and maintenance monitoring data as target operation and maintenance monitoring data if the first classification result indicates that the operation and maintenance monitoring data is abnormal data, and to determine the second classification result of the target operation and maintenance monitoring data by using preset judgment rules, the target operation and maintenance monitoring data and the target data type of the target operation and maintenance monitoring data. The judgment rules are used to reflect the abnormal judgment conditions of operation and maintenance monitoring data of different data types.

[0112] The third processing module 304 is used to return to the step of obtaining the multi-data-type operation and maintenance monitoring data and offline predicted values ​​of the financial system if the first classification result indicates that the operation and maintenance monitoring data is non-abnormal data.

[0113] It should be noted that, Figure 3 The functions of each module in the device shown are as follows: Figure 1 The steps in the method shown are similar, and will not be repeated here to avoid repetition. For details, please refer to [reference needed]. Figure 1 The content of each step in the method shown.

[0114] This invention provides a data processing device for a financial system. The device includes: a data acquisition module for acquiring operation and maintenance monitoring data and offline predicted values ​​of multiple data types from the financial system, wherein the offline predicted values ​​reflect the normal baseline values ​​of operation and maintenance monitoring data of each data type obtained by offline prediction of historical operation and maintenance monitoring data of each data type; a first processing module for determining a first classification result of operation and maintenance monitoring data of each data type using the operation and maintenance monitoring data and offline predicted values, wherein the first classification result reflects whether the operation and maintenance monitoring data is abnormal data; a second processing module for determining the operation and maintenance monitoring data as target operation and maintenance monitoring data if the first classification result indicates that the operation and maintenance monitoring data is abnormal data, and determining a second classification result of the target operation and maintenance monitoring data using preset judgment rules, the target operation and maintenance monitoring data, and the target data type of the target operation and maintenance monitoring data, wherein the judgment rules reflect the abnormal judgment conditions of operation and maintenance monitoring data of different data types; and a third processing module for returning to the step of acquiring operation and maintenance monitoring data and offline predicted values ​​of multiple data types from the financial system if the first classification result indicates that the operation and maintenance monitoring data is not abnormal data. The above-mentioned device can first classify the operation and maintenance monitoring data using offline predicted values, and then perform secondary classification on the target operation and maintenance monitoring data whose first classification result is abnormal data. This reduces the data processing pressure during secondary classification, and the secondary classification also improves the accuracy of the classification results of the target operation and maintenance monitoring data with abnormalities, thus reducing errors.

[0115] Figure 4 An internal structural diagram of a computer device in one embodiment is shown. This computer device can specifically be a terminal or a server. Figure 4As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program, which, when executed by the processor, causes the processor to perform the aforementioned methods. The internal memory may also store a computer program, which, when executed by the processor, causes the processor to perform the aforementioned methods. Those skilled in the art will understand that… Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0116] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform actions such as... Figure 1 or Figure 2 The steps of the method shown.

[0117] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, causes the processor to perform the following actions: Figure 1 or Figure 2 The steps of the method shown.

[0118] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0119] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A data processing method for a financial system, characterized in that, The method includes: The system acquires operation and maintenance monitoring data and offline predicted values ​​for multiple data types from the financial system. The offline predicted values ​​reflect the normal baseline values ​​of the operation and maintenance monitoring data for each data type, obtained through offline prediction of historical operation and maintenance monitoring data for each data type. These offline predicted values ​​are calculated by the offline system and generated using a selected prediction algorithm. For stable data types, a month-on-month mean algorithm is used; for data types with strong periodicity, a Fourier transform algorithm is used. The offline system pre-predicts future data using historical operation and maintenance monitoring data and prediction algorithms for each data type, obtaining offline predicted values ​​for each data type. These offline predicted values ​​are then adjusted upwards and downwards by a certain percentage to obtain their maximum and minimum values, thus determining the target value range for each data type. The target value range is used to determine the first classification result of the operation and maintenance monitoring data for each data type. Using the operation and maintenance monitoring data and the offline predicted value, a first classification result of the operation and maintenance monitoring data for each data type is determined. The first classification result is used to reflect whether the operation and maintenance monitoring data is abnormal data. If the first classification result indicates that the operation and maintenance monitoring data is abnormal data, then the operation and maintenance monitoring data is determined to be the target operation and maintenance monitoring data. Then, using the preset judgment rules, the target operation and maintenance monitoring data and the target data type of the target operation and maintenance monitoring data, the second classification result of the target operation and maintenance monitoring data is determined. The judgment rules are used to reflect the abnormal judgment conditions of operation and maintenance monitoring data of different data types. If the first classification result indicates that the operation and maintenance monitoring data is non-abnormal data, then return to the step of obtaining the operation and maintenance monitoring data and offline prediction values ​​of the financial system for multiple data types; The judgment rule includes at least a preset correspondence between data types and data change patterns. Therefore, determining the second classification result of the target operation and maintenance monitoring data using the preset judgment rule, the target operation and maintenance monitoring data, and the target data types of the target operation and maintenance monitoring data includes: Based on the target data type, find the target data change pattern corresponding to the target data type from the correspondence between the data type and the data change pattern; Using a preset time period and the target operation and maintenance monitoring data, determine the first data change pattern within the preset time period; If the error between the first data change pattern and the target data change pattern is greater than or equal to a preset error threshold, then the second classification result is determined to be abnormal data; If the error between the first data change pattern and the target data change pattern is less than a preset error threshold, then the second classification result is determined to be non-abnormal data.

2. The method according to claim 1, characterized in that, The judgment rule includes at least a preset correspondence between data types and thresholds. Therefore, determining the second classification result of the target operation and maintenance monitoring data using the preset judgment rule, the target operation and maintenance monitoring data, and the target data types of the target operation and maintenance monitoring data includes: Based on the target data type, find the target threshold corresponding to the target data type from the correspondence between the data type and the threshold; If the operation and maintenance monitoring data is greater than or equal to the target threshold, then the second classification result is determined to be abnormal data; If the operation and maintenance monitoring data is less than the target threshold, then the second classification result is determined to be non-abnormal data.

3. The method according to claim 1, characterized in that, The judgment rule also includes a machine learning model. Therefore, determining the second classification result of the target operation and maintenance monitoring data using the preset judgment rule, the target operation and maintenance monitoring data, and the target data type of the target operation and maintenance monitoring data includes: The target operation and maintenance monitoring data and the target data type are input into the machine learning model to determine the output value of the machine learning model. The output value is used to reflect the probability that the target operation and maintenance monitoring data is abnormal data. If the output value is greater than or equal to a preset probability threshold, then the second classification result is determined to be abnormal data; If the output value is less than a preset probability threshold, then the second classification result is determined to be non-abnormal data.

4. The method according to claim 1, characterized in that, The acquisition of various types of operation and maintenance monitoring data from the financial system includes: Original operation and maintenance monitoring data of various data types are obtained from the pre-set message middleware in the financial system; The original operation and maintenance monitoring data is formatted using preset formatting rules to obtain the operation and maintenance monitoring data. The formatting rules include standard data formats corresponding to various data types.

5. The method according to claim 1, characterized in that, The step of determining the first classification result of the operation and maintenance monitoring data for each data type using the operation and maintenance monitoring data and the offline predicted value includes: Determine whether the operation and maintenance monitoring data falls within the target value range corresponding to the offline predicted value; If the operation and maintenance monitoring data falls within the target value range, then the first classification result of the operation and maintenance monitoring data is determined to be non-abnormal data; If the operation and maintenance monitoring data is not within the target value range, then the first classification result of the operation and maintenance monitoring data is determined to be abnormal data.

6. The method according to claim 1, characterized in that, Obtaining the offline predicted value includes: Send a data acquisition request to a preset offline financial system, which is used to calculate the offline predicted value of historical operation and maintenance monitoring data of each data type; Receive the offline predicted value returned by the offline financial system.

7. A data processing device for a financial system, characterized in that, The device includes: Data acquisition module: used to acquire operation and maintenance monitoring data and offline predicted values ​​of multiple data types of the financial system. The offline predicted values ​​are used to reflect the normal baseline values ​​of operation and maintenance monitoring data of each data type obtained by offline prediction of historical operation and maintenance monitoring data of each data type. The offline predicted values ​​are generated by selecting the corresponding prediction algorithm. For stationary data types, the month-on-month mean algorithm is used to generate them, and for data types with strong periodicity, the Fourier transform algorithm is used to generate them. First processing module: used to determine the first classification result of the operation and maintenance monitoring data for each data type using the operation and maintenance monitoring data and the offline prediction value, the first classification result being used to reflect whether the operation and maintenance monitoring data is abnormal data; The second processing module is used to determine the operation and maintenance monitoring data as target operation and maintenance monitoring data if the first classification result indicates that the operation and maintenance monitoring data is abnormal data, and to determine the second classification result of the target operation and maintenance monitoring data by using preset judgment rules, the target operation and maintenance monitoring data and the target data type of the target operation and maintenance monitoring data. The judgment rules are used to reflect the abnormal judgment conditions of operation and maintenance monitoring data of different data types. The third processing module is used to return to the step of obtaining the multi-data-type operation and maintenance monitoring data and offline predicted values ​​of the financial system if the first classification result indicates that the operation and maintenance monitoring data is non-abnormal data. The judgment rule includes at least a preset correspondence between data types and data change patterns. The step of determining the second classification result of the target operation and maintenance monitoring data using the preset judgment rule, the target operation and maintenance monitoring data, and the target data type of the target operation and maintenance monitoring data includes: based on the target data type, searching for the target data change pattern corresponding to the target data type from the correspondence between data types and data change patterns; using a preset time length and the target operation and maintenance monitoring data, determining the first data change pattern within the preset time length; if the error between the first data change pattern and the target data change pattern is greater than or equal to a preset error threshold, then the second classification result is determined to be abnormal data; if the error between the first data change pattern and the target data change pattern is less than the preset error threshold, then the second classification result is determined to be non-abnormal data.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 6.

9. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 6.

Images