Large-scale transaction data exception identification method and device, equipment and storage medium

Abstract

The invention discloses a large-scale transaction data exception identification method, device and equipment and a storage medium, relates to the technical field of transaction exception identification, and discloses a large-scale transaction data exception identification method, which comprises the following steps: acquiring initial transaction data pushed to Kafka from a transaction system; sending the initial transaction data to a Flink preprocessing module, so that the Flink preprocessing module performs aggregation extraction and downsampling preprocessing on the transaction data, and returns preprocessed transaction data; obtaining current time window data according to the preprocessed transaction data, and performing data selection according to a preset anomaly judgment strategy to obtain a target historical data set; and performing anomaly judgment according to the target historical data set and the current time window data to obtain an anomaly recognition result. And high-efficiency, low-delay and accurate identification of abnormal fluctuation of large-scale transaction data is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of transaction anomaly identification, and particularly relates to a large-scale transaction data anomaly identification method and device, equipment and a storage medium. BACKGROUND

[0002] With the continuous expansion of the financial market transaction scale, it becomes increasingly important to monitor the abnormal fluctuations of transaction data in real time. In order to cope with large-scale and rapidly changing transaction data, many industries have begun to use big data technology to improve the efficiency and accuracy of monitoring. However, the abnormal fluctuations of transaction data are often sudden and the data volume is huge.

[0003] At present, the traditional monitoring method mainly relies on batch processing technology or simple threshold alarm mechanism. The batch processing method can comprehensively process data by collecting and analyzing transaction data in a certain time period, but it usually has a certain time delay in processing, and cannot realize real-time abnormal fluctuation detection. The simple threshold alarm mechanism can monitor data in real time, but it can only make judgments according to preset conditions, lacks sufficient flexibility, and cannot cope with complex and variable transaction patterns. The main disadvantage of the traditional technical solution is that it cannot realize real-time monitoring and timely response, which leads to the inability to discover and take effective measures in the first time when abnormal fluctuations occur, affecting the timeliness and accuracy of decision-making.

[0004] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0005] The main purpose of the present application is to provide a large-scale transaction data anomaly identification method, device, equipment and storage medium, which aims to solve the technical problems of low response speed and low accuracy in real-time monitoring of large-scale transaction data abnormal fluctuations.

[0006] To achieve the above purpose, the present application provides a large-scale transaction data anomaly identification method, which comprises:

[0007] Obtaining initial transaction data pushed from a transaction system to Kafka;

[0008] Sending the initial transaction data to a Flink preprocessing module, so that the Flink preprocessing module performs aggregation extraction and downsampling preprocessing on the transaction data, and returns preprocessed transaction data;

[0009] Obtaining current time window data according to the preprocessed transaction data, and obtaining target historical data set according to a preset anomaly judgment strategy;

[0010] According to the target historical data set and the current time window data, an abnormality determination is performed to obtain an abnormality recognition result.

[0011] In an embodiment, the step of performing the abnormality determination according to the target historical data set and the current time window data to obtain the abnormality recognition result comprises:

[0012] According to the target historical data set, target historical data is selected from historical time window transaction volume, historical daily interval transaction average value and historical weekly interval transaction average value in sequence.

[0013] According to the current time window data and the target historical data, an abnormality determination is performed to obtain an abnormality determination result, wherein the abnormality determination comprises mean value determination, predicted mean value determination and similarity determination.

[0014] According to the abnormality determination result, it is determined whether the current time window data has an abnormality.

[0015] When the current time window data has an abnormality, a historical data determination result of the target historical data set is determined.

[0016] When it is determined that the historical data determination result of the target historical data set is that there are abnormal data items greater than or equal to a preset abnormality recognition quantity, the current time window data is recognized as abnormal to obtain an abnormality recognition result.

[0017] In an embodiment, the step of determining whether the current time window data has an abnormality according to the abnormality determination result comprises:

[0018] According to the abnormality determination result, a mean value determination result, a predicted mean value determination result and a similarity determination result are obtained.

[0019] When the determination result of any two of the mean value determination result, the predicted mean value determination result and the similarity determination result is determined to have an abnormality, it is determined that the current time window data is abnormal.

[0020] In an embodiment, the step of performing the abnormality determination according to the current time window data and the target historical data to obtain an abnormality determination result comprises:

[0021] According to the target historical data, a historical transaction average value is calculated.

[0022] According to the current time window data, a current transaction average value is calculated.

[0023] According to the historical transaction average value and a mean value determination coefficient, a mean value determination threshold is determined.

[0024] When the current transaction mean value is less than the mean value determination threshold, a mean value determination result is determined to be abnormal, and an abnormality determination result is obtained according to the mean value determination result.

[0025] In an embodiment, the step of performing abnormality determination according to the current time window data and the target historical data to obtain an abnormality determination result further comprises:

[0026] A current time data vector and a target historical data vector are obtained according to the current time window data and the target historical data, respectively.

[0027] A cosine similarity is calculated according to the current time data vector and the target historical data vector.

[0028] When an absolute value of the cosine similarity is less than a similarity determination coefficient, a similarity determination result is determined to be abnormal, and an abnormality determination result is obtained according to the similarity determination result.

[0029] In an embodiment, the step of performing abnormality determination according to the current time window data and the target historical data to obtain an abnormality determination result further comprises:

[0030] A current transaction mean value is calculated according to the current time window data.

[0031] An estimation coefficient and an offset are calculated according to the target historical data, and a fitting mean value is calculated according to the estimation coefficient, the offset, and a preset first-order difference function.

[0032] A predicted mean value determination threshold is determined according to the fitting mean value and a predicted mean value determination coefficient.

[0033] When the current transaction mean value is less than the predicted mean value determination threshold, a predicted mean value determination result is determined to be abnormal, and an abnormality determination result is obtained according to the predicted mean value determination result.

[0034] In an embodiment, the step of obtaining the current time window data according to the preprocessed transaction data, and obtaining the target historical data set according to a preset abnormality determination strategy comprises:

[0035] The current time window data is obtained according to the preprocessed transaction data and the current time.

[0036] The historical time window transaction volume is selected according to a nearest time monitoring strategy of the preset abnormality determination strategy.

[0037] A first preset determination time point is selected according to a nearest day mean value monitoring strategy of the preset abnormality determination strategy, and a historical day interval transaction mean value is determined according to the first preset determination time point.

[0038] The nearest week same day monitoring strategy according to the preset abnormality determination strategy selects a preset week offset and a second preset determination time point, and determines a historical week interval transaction average according to the preset week offset and the second preset determination time point;

[0039] According to the historical time window transaction volume, the historical day interval transaction average, and the historical week interval transaction average, target historical data set is obtained by data selection in the preprocessed transaction data.

[0040] In addition, to achieve the above-mentioned purpose, the application further provides a large-scale transaction data anomaly identification device, which comprises a data acquisition module for acquiring initial transaction data pushed from a transaction system to Kafka;

[0041] A data preprocessing module is configured to send the initial transaction data to a Flink preprocessing module, so that the Flink preprocessing module performs aggregation extraction and downsampling preprocessing on the transaction data and returns preprocessed transaction data.

[0042] A data selection module is configured to obtain current time window data according to the preprocessed transaction data, and obtain target historical data set by data selection according to a preset abnormality determination strategy.

[0043] An anomaly identification module is configured to perform abnormality determination according to the target historical data set and the current time window data, and obtain an anomaly identification result.

[0044] In addition, to achieve the above-mentioned purpose, the application further provides a large-scale transaction data anomaly identification device, which comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the large-scale transaction data anomaly identification method as described above.

[0045] In addition, to achieve the above-mentioned purpose, the application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, the computer program being executed by a processor to implement the steps of the large-scale transaction data anomaly identification method as described above.

[0046] In addition, to achieve the above-mentioned purpose, the application further provides a computer program product, which comprises a computer program, the computer program being executed by a processor to implement the steps of the large-scale transaction data anomaly identification method as described above.

[0047] The one or more technical solutions provided by the application have at least the following technical effects:

[0048] Due to the use of data access mode based on message queue Kafka, the decoupling between the transaction system and the abnormal identification module is realized, so as to avoid the interference to the transaction main link and ensure the high throughput and high reliability of data transmission. Flink is used as a data processing engine to perform minute-level aggregation extraction and downsampling preprocessing on the obtained initial transaction data, and representative time window data is extracted from the data stream. In the data identification stage, a multi-dimensional target historical data set is constructed, and comparison is made based on the current time window data, combined with the preset abnormal judgment strategy, to realize multi-angle analysis of data fluctuation. The introduction of the strategy avoids the limitation of single rule misjudgment, and improves the robustness of judgment through the majority voting mechanism. Compared with the prior art, not only the problems of unstable large-scale transaction data access, high processing resource consumption and inaccurate abnormal judgment are solved, but also the real-time performance and identification accuracy are significantly improved through the introduction of stream processing and multi-strategy fusion mechanism, meeting the timeliness and accuracy in high-frequency financial transaction scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0051] Figure 1 Flowchart provided for large-scale transaction data abnormal identification method embodiment one of the present application;

[0052] Figure 2 Transaction volume abnormal fluctuation monitoring architecture diagram provided for large-scale transaction data abnormal identification method embodiment one of the present application;

[0053] Figure 3 Transaction volume identification sliding window diagram provided for large-scale transaction data abnormal identification method embodiment one of the present application;

[0054] Figure 4 Abnormal identification strategy execution flowchart provided for large-scale transaction data abnormal identification method embodiment one of the present application;

[0055] Figure 5 Flowchart provided for large-scale transaction data abnormal identification method embodiment two of the present application;

[0056] Figure 6A module structure schematic diagram of the large-scale transaction data anomaly identification device of the embodiment of the present application;

[0057] Figure 7 A device structure schematic diagram of a hardware running environment involved in the large-scale transaction data anomaly identification method in the embodiment of the present application.

[0058] The object implementation, functional features and advantages of the present application will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0059] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0060] In order to better understand the technical solutions of the present application, the following will be described in detail in conjunction with the drawings and specific embodiments of the specification.

[0061] The main solution of the embodiment of the present application is: obtaining initial transaction data pushed from a transaction system to Kafka; sending the initial transaction data to a Flink preprocessing module, so that the Flink preprocessing module performs aggregation extraction and downsampling preprocessing on the transaction data, and returns preprocessed transaction data; obtaining current time window data according to the preprocessed transaction data, and obtaining target historical data set according to a preset anomaly judgment strategy; performing anomaly judgment according to the target historical data set and the current time window data, and obtaining anomaly identification result.

[0062] In the embodiment, for convenience of description, the following takes a large-scale transaction data anomaly identification device as an execution subject for elaboration.

[0063] Since the prior art has low response speed and low accuracy in real-time monitoring of abnormal fluctuation of large-scale transaction data, the present application provides a solution. Since the data access method based on message queue Kafka is adopted, the decoupling between the transaction system and the abnormal identification module is realized, thereby avoiding interference with the transaction main link and ensuring high throughput and high reliability of data transmission. Flink is adopted as the data processing engine to perform minute-level aggregation extraction and downsampling preprocessing on the obtained initial transaction data, and representative time window data is extracted from the data stream. In the data identification stage, a multi-dimensional target historical data set is constructed, and comparison is performed based on the current time window data, combined with the preset abnormal judgment strategy, to realize multi-angle analysis of data fluctuation. The introduction of the strategy avoids the limitation of single rule misjudgment, and improves the robustness of the judgment through the majority voting mechanism. Compared with the prior art, not only the problems of unstable large-scale transaction data access, high processing resource consumption and inaccurate abnormal judgment are solved, but also the real-time performance and identification accuracy are significantly improved through the introduction of the stream processing and multi-strategy fusion mechanism, meeting the timeliness and precision requirements in the high-frequency financial transaction scenario.

[0064] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a large-scale transaction data abnormal identification device, etc. In the following, the large-scale transaction data abnormal identification device is taken as an example to describe the present embodiment and each of the following embodiments.

[0065] Based on this, the present embodiment provides a large-scale transaction data abnormal identification method, which is described with reference to Figure 1 , Figure 1 The flowchart of the first embodiment of the large-scale transaction data abnormal identification method of the present application is shown in FIG. 1.

[0066] In the present embodiment, the large-scale transaction data abnormal identification method includes steps S10-S40:

[0067] Step S10, obtaining initial transaction data pushed from the transaction system to Kafka;

[0068] It should be noted that the transaction system is a platform for processing various financial transaction businesses, responsible for recording detailed information of transactions, such as the time of transaction, the involved merchants, the type of transaction, the transaction amount, etc., and is the source of initial transaction data.

[0069] In addition, Kafka is a high-throughput distributed publish-subscribe message system that can receive and store a large amount of transaction data from the transaction system, acting as a data transfer station, so that the data producers and consumers can efficiently transfer data.

[0070] Further, the initial transaction data is the raw data generated directly by the transaction system in the process of processing transactions, which contains complete information of each transaction, such as a transaction record completed by a certain payment method of a certain merchant at a certain time. These data are not processed and are the basis for all subsequent processing and analysis.

[0071] It can be understood that the transaction system will send the raw information generated by each transaction to Kafka, and then obtain these raw transaction data from Kafka. The separation of monitoring related operations and the transaction system is realized, that is, the running of the monitoring application will not interfere with the normal work of the transaction system. By obtaining the initial transaction data from Kafka, the decoupling of the monitoring related application and the transaction system is realized, the intrusion of the monitoring operation on the transaction system is avoided, the stable operation of the transaction system is ensured, and a reliable data source is provided for subsequent real-time data processing.

[0072] In step S20, the initial transaction data is sent to the Flink preprocessing module to make the Flink preprocessing module perform aggregation extraction and downsampling preprocessing on the transaction data, and return preprocessed transaction data.

[0073] It should be noted that the Flink preprocessing module is a part for processing data based on Flink technology. Flink is a distributed processing engine that can efficiently process real-time data streams. This module is specifically responsible for preprocessing operations on initial transaction data.

[0074] Further, aggregation extraction is a process of collecting and extracting a large number of dispersed initial transaction data according to certain rules, such as collecting transaction data within a certain time according to merchants and transaction types, and extracting key information such as the total amount of transactions of each merchant under each transaction type.

[0075] In addition, downsampling preprocessing is a processing method for reducing data volume. By selecting representative data within a certain time interval to replace all data in that time period, such as processing multiple transaction data generated per second into transaction summary data per minute, the size of the data is reduced.

[0076] Further, the preprocessed transaction data is the data obtained after aggregation extraction and downsampling processing. It is more concise than the initial transaction data and highlights key information, such as transaction data of merchants in the dimension of transaction type per minute, which is more suitable for subsequent anomaly identification.

[0077] It can be understood that the initial transaction data obtained from Kafka is sent to the Flink preprocessing module. The Flink preprocessing module processes these data, and for each payment type of transaction, a separate Flink application is used for processing. In the processing process, first, aggregation extraction is performed, and the data is summarized according to the dimensions of merchants and transaction types, and then down-sampling is performed to process the data into minute-level summary data, and finally, the preprocessed transaction data is obtained and returned. Referring to Figure 2 , Figure 2 FIG. 1 is a schematic diagram of a transaction volume abnormal fluctuation monitoring architecture according to a first embodiment of the large-scale transaction data anomaly identification method of the present application.

[0078] As shown in Figure 2 , the transaction system first sends data to Kafka, and then the data enters the Flink preprocessing stage, where the data is divided into four categories: Alipay, UnionPay, WeChat and digital currency. The data processed by Flink is transmitted to the anomaly monitoring module, which includes an anomaly identification application and a database. The anomaly identification application analyzes the data and outputs the results as an alarm message.

[0079] In step S30, the current time window data is obtained according to the preprocessed transaction data, and the target historical data set is obtained by selecting data according to a preset abnormality determination strategy.

[0080] It should be noted that the current time window data is extracted from the preprocessed transaction data, and is the transaction data in a specific time period with the current time as the reference, and this time period is the current time window.

[0081] In addition, the preset abnormality determination strategy is a rule for determining how to select historical data, including three strategies: near-time monitoring strategy, near-day mean value monitoring strategy and near-week same day monitoring strategy, each of which specifies a different way of selecting historical data.

[0082] It can be understood that the target historical data set is a data set selected from historical preprocessed transaction data according to the preset abnormality determination strategy, which is used for comparison with the current time window data. Different strategies will select different historical data, for example, according to the near-time monitoring strategy, the transaction data in an adjacent time period before the current time window can be selected as the target historical data set.

[0083] It can be understood that the transaction data corresponding to the current time window is extracted from the pre-processed transaction data, that is, the current time window data. Then, the target historical data set is selected according to the rules in the three preset abnormality determination strategies. If the nearest time monitoring strategy is adopted, the transaction data in a historical time window before the current time window is selected as the target historical data set; if the nearest day average monitoring strategy is adopted, the data set composed of the average of the transaction data in the same time period as the current time window in the past N days is selected as the target historical data set; if the nearest week same day monitoring strategy is adopted, the data set composed of the average of the transaction data in the same time period as the current time window in the past N days with a week offset is selected as the target historical data set. Referring to Figure 3 , Figure 3 , it is a transaction volume identification sliding window schematic diagram of the first embodiment of the large-scale transaction data abnormality identification method of the present application.

[0084] As shown in Figure 3 , the horizontal coordinate t represents time, and is marked with τ, historical window T H , nearest time window T W , and current time t0 from left to right; the vertical coordinate R(t) represents transaction volume. The figure contains three curves of different colors, the blue curve is the daily transaction volume fluctuation curve, the orange curve is the average of the transaction volume in the past N days, and the yellow curve is the average of the transaction volume in the past N days with a week offset. The three curves show different fluctuation patterns in the time axis τ to t0 range, showing the transaction volume of the day, the average of the past N days, and the average of the past N days with a week offset changing with time. Through the interval division of the historical window T H and the nearest time window T W , the fluctuation difference of the transaction volume in different time dimensions is compared and analyzed.

[0085] Step S40, performing abnormality determination according to the target historical data set and the current time window data to obtain an abnormality identification result.

[0086] It can be understood that abnormality determination is a process of comparing the data in the target historical data set and the current time window data to determine whether the current transaction data has abnormal fluctuation through a series of rules and calculations, including mean value decision, predicted mean value decision, and similarity decision.

[0087] It should be noted that the current time window data is the transaction data in the current time period, which is the object to be determined for abnormality. It is determined whether it is abnormal by comparing it with the target historical data set.

[0088] The abnormality identification result is a conclusion obtained after abnormality determination, indicating whether the transaction corresponding to the current time window data has abnormal fluctuation. If there is abnormality, the relevant merchant and transaction type information are recorded. When the abnormality identification result is identified, the identified abnormality is sent to the corresponding handler.

[0089] In addition, it can be understood that for each preset abnormality determination strategy, the data in the target historical data set and the current time window data are used for mean value determination, predicted mean value determination and similarity determination. In the mean value determination, the mean values of the two are compared; in the predicted mean value determination, the predicted mean value obtained from the fitting function of the target historical data set is compared with the mean value of the current time window data; in the similarity determination, the cosine similarity of the two is calculated and judged. When at least two of the three determinations are abnormal, the data in the target historical data set is determined as suspected abnormality. Then, the results of the determination of all the data in the target historical data set are integrated, and if two or more of them are determined as suspected abnormality, it is finally determined as abnormality, and the abnormality identification result is obtained.

[0090] In a feasible implementation, step S40 can include steps S41-S45:

[0091] Step S41, selecting target historical data in the historical time window transaction volume, historical daily interval transaction mean value and historical weekly interval transaction mean value in turn according to the target historical data set;

[0092] It should be noted that the historical time window transaction volume refers to the transaction volume data recorded in a certain specific historical time window in the past, and the time window has the same length as the current time window.

[0093] In addition, the historical daily interval transaction mean value refers to the average value of the transaction volume in the same time period as the current time window in the past N days.

[0094] In addition, the historical weekly interval transaction mean value refers to the average value of the transaction volume in the same time period as the current time window in the past N days, with the time interval being a week.

[0095] Step S42, determining abnormality according to the current time window data and the target historical data to obtain abnormality determination result, the abnormality determination including mean value determination, predicted mean value determination and similarity determination;

[0096] It should be noted that the mean value determination is a way of determining whether it is abnormal by calculating the mean value of the current time window data and the mean value of the target historical data, and then comparing the size relationship of the two. If the mean value of the current time window data is less than the product of the mean value of the target historical data and the determination coefficient, it is determined as abnormal.

[0097] In addition, the prediction mean determination is to fit a first-order difference function according to the target historical data, calculate the fitting mean of the function, and compare the mean of the current time window data with the product of the fitting mean and a decision coefficient. If the former is less than the latter, it is determined as abnormal.

[0098] In addition, the similarity determination is to regard the current time window data and the target historical data as two vectors, calculate the cosine similarity between them, and if the absolute value of the similarity is less than a decision coefficient, it is determined as abnormal.

[0099] Further, the abnormality determination result is the conclusion obtained after the mean determination, the prediction mean determination and the similarity determination, which indicates whether the current time window data is determined as abnormal under the three determination manners.

[0100] It can be understood that the mean determination is to calculate the mean of the current time window data and the mean of the target historical data, respectively. If the current mean is less than the product of the target historical mean and a decision coefficient, the mean determination is abnormal. Then the prediction mean determination is to fit a first-order difference function according to the target historical data, calculate the fitting mean, and if the current mean is less than the product of the fitting mean and a decision coefficient, the prediction mean determination is abnormal. Then the similarity determination is to calculate the cosine similarity between the current time window data and the target historical data, and if the absolute value of the similarity is less than a decision coefficient, the similarity determination is abnormal. Finally, the results of the three determinations are combined to obtain the abnormality determination result.

[0101] In a feasible implementation, the step S42 can include steps A421-A424:

[0102] The step A421 is to calculate a historical transaction mean according to the target historical data.

[0103] It should be noted that the historical transaction mean is a value obtained by averaging the transaction volume in the target historical data, which reflects the average transaction level of the corresponding historical time period. The historical transaction mean is calculated as follows:

[0104]

[0105] In the formula, H avg is the historical transaction mean, H[n] is the target historical data, and N is the interval time.

[0106] The step A422 is to calculate a current transaction mean according to the current time window data.

[0107] It should be noted that the current transaction mean can be calculated as L avgIt represents the value obtained by averaging the transaction volume in the current time window data. It reflects the average level of transactions in the current time window and is used to compare with the historical transaction average to determine whether there are any abnormalities in the current transaction. The current transaction average is calculated as follows:

[0108]

[0109] Where, L avg is the current transaction average, L[n] is the current time window data, and N is the interval time.

[0110] Step A423: determining a mean determination threshold value based on the historical transaction mean and the mean determination coefficient;

[0111] It should be noted that the mean determination coefficient is a pre-set coefficient with a value in the range of (0, 1], which is used to adjust the size of the mean determination threshold. The mean determination coefficient can be expressed as r.

[0112] In addition, the mean judgment threshold is a value obtained by multiplying the historical transaction mean by the mean judgment coefficient. It is the critical value for judging whether the current transaction mean is abnormal. If the current transaction mean is less than the threshold, it may be judged as abnormal. The mean judgment threshold can be expressed as r·H avg .

[0113] Step A424: When the current transaction average is less than the average determination threshold, determine that the average determination result is abnormal, and obtain an abnormality determination result based on the average determination result.

[0114] It can be understood that the current transaction average is compared with the average determination threshold. If the current transaction average is less than the average determination threshold, that is, L avg <r·H avg , then the mean judgment result is determined to be abnormal, and this result is included in the abnormal judgment result.

[0115] In a feasible implementation, step S42 may further include steps B421 to B423:

[0116] Step B421, obtaining a current time data vector and a target historical data vector according to the current time window data and the target historical data respectively;

[0117] It should be noted that the current time data vector is a vector formed by arranging the trading volume in the current time window data in chronological order. Each element corresponds to the trading volume at a point in time, which can intuitively reflect the changing trend of the trading volume in the current time window.

[0118] In addition, the target historical data vector is a vector formed by arranging the transaction volume in the target historical data in chronological order, each element corresponding to the transaction volume of a historical time point in the same time interval as the current time window, for embodying the change trend of historical transactions.

[0119] Step B422, calculating a cosine similarity according to the current time data vector and the target historical data vector;

[0120] It should be noted that the cosine similarity is an index for measuring the directional difference between two vectors, and its value range is between [-1, 1]. When the directions of two vectors are exactly the same, the cosine similarity is 1; when the directions are exactly opposite, the cosine similarity is -1; and when the absolute value of the cosine similarity is smaller, it indicates that the directional difference between the two vectors is greater, i.e., the change trend difference between the two groups of data is more obvious. The cosine similarity is calculated as follows:

[0121]

[0122] In the formula, L is the current time data vector, and H is the target historical data vector.

[0123] Step B423, when the absolute value of the cosine similarity is smaller than a similarity determination coefficient, determining that the similarity determination result is abnormal, and obtaining an abnormal determination result according to the similarity determination result.

[0124] It should be noted that the similarity determination coefficient is a threshold value with a value in the range of (0, 1] and is used to determine whether the directional difference between two vectors reaches an abnormal degree. The similarity determination coefficient can be represented as p.

[0125] It can be understood that the absolute value of the cosine similarity is compared with the similarity determination coefficient. If the absolute value of the cosine similarity is smaller than the similarity determination coefficient, i.e., |γ|<p, it is determined that the similarity determination result is abnormal, and this result is included in the abnormal determination result.

[0126] In a feasible implementation, step S42 can further include steps C421-C424:

[0127] Step C421, calculating a current transaction mean value according to the current time window data;

[0128] It can be understood that the current transaction mean value can be calculated according to the current transaction mean value calculation formula.

[0129] Step C422, calculating an estimation coefficient and an offset according to the target historical data, and calculating a fitting mean value according to the estimation coefficient, the offset, and a preset first-order difference function;

[0130] It should be noted that the estimated coefficient is calculated according to the transaction volume sequence of the target historical data, and is a parameter for reflecting the change trend of the sequence. The calculation method is to take the mean value of the difference of the transaction volume sequence of the target historical data. The estimated coefficient is calculated as follows:

[0131]

[0132] In the formula, a is the estimated coefficient, H[n] is the current time window data, N is the interval time, and K is the number of difference items.

[0133] In addition, the offset is a parameter calculated by combining the transaction volume sequence of the target historical data after obtaining the estimated coefficient, and is used to adjust the preset first-order difference function to make the function more fit the trend of the historical data. The calculation method is to subtract the product of the estimated coefficient and the time point sequence number from the transaction volume at each time point, and then take the average of the results. The offset is calculated as follows:

[0134]

[0135] In the formula, b is the offset, a is the estimated coefficient, H[n] is the current time window data, and N is the interval time.

[0136] Further, the preset first-order difference function is a function of the form y[n] = a x[n] + b, where a is the estimated coefficient, b is the offset, x[n] is the time point sequence number, and y[n] is the fitted transaction volume at the time point, which is used to simulate the change trend of the target historical data.

[0137] It can be understood that the fitting mean value is the average value obtained by adding the fitted transaction volume at each time point calculated by the preset first-order difference function and dividing by the number of time points, which reflects the average transaction volume level based on the historical data trend prediction. The fitting mean value is calculated as follows:

[0138]

[0139] In the formula, F avg is the fitting mean value, y[n] is the fitted transaction volume at the time point, and N is the interval time.

[0140] Step C423, determining a prediction mean value determination threshold according to the fitting mean value and a prediction mean value determination coefficient;

[0141] It should be noted that the prediction mean value determination coefficient is a coefficient with a value in the range of (0, 1] that is set in advance, and is used to adjust the size of the prediction mean value determination threshold. The prediction mean value determination coefficient can be represented as r.

[0142] In addition, the predicted mean determination threshold is a value obtained by multiplying the fitting mean by the predicted mean determination coefficient, and is a critical value for determining whether the current transaction mean deviates from the historical trend. If the current transaction mean is less than the threshold, it can be determined as abnormal. The predicted mean determination threshold can be expressed as r·F avg .

[0143] Step C424, when the current transaction mean is less than the predicted mean determination threshold, determining that the predicted mean determination result is abnormal, and obtaining an abnormality determination result according to the predicted mean determination result.

[0144] It can be understood that the current transaction mean is compared with the predicted mean determination threshold. If the current transaction mean is less than the predicted mean determination threshold, i.e. L avg <r·F avg , it is determined that the predicted mean determination result is abnormal, and this result is included in the abnormality determination result.

[0145] Step S43, determining whether the current time window data is abnormal according to the abnormality determination result;

[0146] It can be understood that the abnormality determination result is used to determine whether the current time window data is abnormal. When at least two of the mean determination, the predicted mean determination, and the similarity determination are determined as abnormal, it is determined that the current time window data is abnormal; if less than two are determined as abnormal, it is determined that the current time window data is not abnormal.

[0147] In a possible implementation, step S43 can include: obtaining a mean determination result, a predicted mean determination result, and a similarity determination result according to the abnormality determination result; and when the determination result of any two of the mean determination result, the predicted mean determination result, and the similarity determination result is determined as abnormal, determining that the current time window data is abnormal.

[0148] It can be understood that the conclusions of the mean determination, the predicted mean determination, and the similarity determination are extracted from the abnormality determination result respectively, i.e. the mean determination result, the predicted mean determination result, and the similarity determination result are obtained. The mean determination result, the predicted mean determination result, and the similarity determination result are counted to check the number of results determined as abnormal. When any two of the three results are determined as abnormal, it is determined that the current time window data is abnormal.

[0149] Step S44, when the current time window data is abnormal, determining a historical data determination result of the target historical data set;

[0150] It should be noted that the historical data determination result is obtained by respectively determining the abnormality of each historical data in the target historical data set, and records whether each historical data is abnormal.

[0151] It can be understood that when it has been determined that the current time window data is abnormal, it is necessary to respectively determine the abnormality of each historical data in the target historical data set, i.e., the historical time window transaction volume, the historical daily interval transaction mean, and the historical weekly interval transaction mean, to obtain the result of whether each historical data is abnormal, i.e., the historical data determination result.

[0152] Step S45, when it is determined that the historical data determination result of the target historical data set is that the number of abnormal data items is greater than or equal to the preset abnormality identification number, the current time window data is identified as abnormal to obtain an abnormality identification result.

[0153] It should be noted that the preset abnormality identification number is a threshold value preset for determining whether the current time window data is identified as abnormal. In this embodiment, the preset abnormality identification number is set to 2.

[0154] It can be understood that the number of abnormal data items in the historical data determination result of the target historical data set is counted, and then compared with the preset abnormality identification number. If the number of abnormal data items is greater than or equal to the preset abnormality identification number, the current time window data is finally identified as abnormal to obtain an abnormality identification result; if it is less than the preset abnormality identification number, it is not identified as abnormal. For example, the preset abnormality identification number is 2, and there are 2 abnormal data items in the historical data determination result of the target historical data set. Then, the current time window data is identified as abnormal to obtain an abnormality identification result containing the merchant and the corresponding transaction type.

[0155] Referring to Figure 4 , Figure 4 FIG. 1 is a flowchart of an abnormality identification strategy execution process of a first embodiment of the large-scale transaction data abnormality identification method.

[0156] As shown in Figure 4 , first, the merchant M's recent time window transaction volume sequence L[n] and historical time window transaction volume sequence H[n] are obtained; then, mean value determination, prediction mean value determination after fitting the difference first-order function with H[n], and similarity determination are carried out in parallel; then, it is determined whether the number of hit rules is greater than or equal to 2. If yes, it is determined as a suspected abnormal fluctuation, and if no, the process ends, and finally reaches the end node.

[0157] By setting a rule that at least two determination methods are abnormal to determine that the current data is abnormal, false positives caused by a single determination method are avoided, and the reliability of determining whether the current time window data is abnormal is improved.

[0158] The embodiment provides a large-scale transaction data anomaly identification method, which solves the technical problems of low data processing efficiency, poor real-time performance and insufficient monitoring accuracy in a traditional transaction monitoring system by using Kafka and Flink technologies to realize real-time collection, preprocessing and analysis of transaction data, and has the beneficial effects of improving data processing efficiency, enhancing monitoring real-time performance and accuracy, thereby effectively identifying transaction anomalies and ensuring the safe and stable operation of a financial transaction system.

[0159] Based on the first embodiment of the application, the same or similar contents as the above embodiment one can be referred to the above introduction, and the subsequent will not be described in detail. On this basis, please refer to Figure 5 , the step S30 of the large-scale transaction data anomaly identification method comprises steps S31-S35:

[0160] Step S31, determining current time window data according to the preprocessed transaction data and the current time;

[0161] It should be noted that the current time refers to the time of data processing and monitoring, and the current time window data is the transaction data in a specific time period extracted from the preprocessed transaction data and based on the current time.

[0162] In a feasible implementation manner, when the current time window data is extracted, a dynamic window granularity decision module can be added, based on the merchant portrait label such as the retail type marked as high-frequency fluctuation and the government type marked as stable transaction and the real-time transaction density dynamically calculated through the transaction number in the sliding window, the time window granularity is automatically switched; when the transaction density is greater than or equal to the threshold T, a 30-second fine-grained window is used to capture instantaneous fluctuation, and when the transaction density is less than the threshold T, a 5-minute coarse-grained window is switched to reduce the calculation overhead. At the same time, a cross-window correlation verification mechanism is constructed, the coefficient of variation (CV) of the transaction volume difference value of adjacent windows is calculated, if the CV is greater than the critical value C, the secondary verification of the window overlap area is triggered to identify the transaction mutation point. After the current time window data is extracted, the dynamic window granularity adaptation and cross-window correlation verification are performed, and the current time window data with the verification mark is output. The intelligent dynamic adaptation of the time window can be realized, the details in the high-frequency scene are captured and the calculation efficiency in the low-frequency scene is considered, and the cross-window verification can identify the mutation anomaly easily missed by the traditional fixed window

[0163] Step S32, selecting a historical time window transaction volume according to a nearest time monitoring strategy of a preset anomaly determination strategy;

[0164] It should be noted that the nearest time monitoring strategy is one of the preset anomaly determination strategies, and the core is to compare the current time window data with the transaction data in the adjacent historical time window to determine whether there is an anomaly.

[0165] Further, the historical time window transaction volume refers to transaction data in a historical time window immediately before the current time window.

[0166] It can be understood that according to the nearby time monitoring strategy, a historical time window immediately before the current time window is determined, and transaction data in the historical time window is extracted from the preprocessed transaction data, that is, the historical time window transaction volume is obtained.

[0167] In step S33, a first preset determination time point is selected according to a nearby day mean value monitoring strategy of the preset abnormality determination strategy, and a historical day interval transaction mean value is determined according to the first preset determination time point.

[0168] It should be noted that the nearby day mean value monitoring strategy is one of the preset abnormality determination strategies, and abnormal fluctuations are identified by comparing the current time window data with the transaction volume mean value of the same time period in the past N days. The first preset determination time point refers to the same time period as the current time window in the past N days.

[0169] Further, the historical day interval transaction mean value is an average value obtained by adding transaction volumes of the first preset determination time points in the past N days and dividing by N.

[0170] It can be understood that according to the nearby day mean value monitoring strategy, the same time period as the current time window in the past N days is selected as the first preset determination time point. Then, transaction volumes of these first preset determination time points are extracted from the preprocessed transaction data. Finally, the transaction volumes are added and divided by N to obtain the historical day interval transaction mean value.

[0171] In step S34, a preset week offset and a second preset determination time point are selected according to a nearby week same day monitoring strategy of the preset abnormality determination strategy, and a historical week interval transaction mean value is determined according to the preset week offset and the second preset determination time point.

[0172] It should be noted that the nearby week same day monitoring strategy is one of the preset abnormality determination strategies, and abnormal fluctuations are identified by comparing the current time window data with the transaction volume mean value of the same time period on the same day in the past N weeks. The preset week offset refers to a time interval in weeks.

[0173] Further, the second preset determination time point refers to the same time period on the same day as the current time window in the past N weeks according to the preset week offset. The historical week interval transaction mean value is an average value obtained by adding transaction volumes of the second preset determination time points in the past N weeks and dividing by N.

[0174] It can be understood that, according to the nearest day mean monitoring strategy, a preset week offset and a number of past N weeks to be referred to are first selected. A same time period in the past N weeks as the current time window on the same day is determined according to the preset week offset, as a second preset judgment time point. The transaction volume of the second preset judgment time point is extracted from the preprocessed transaction data. The transaction volumes are added and then divided by N to obtain a historical week interval transaction mean.

[0175] In step S35, data selection is performed in the preprocessed transaction data according to the historical time window transaction volume, the historical day interval transaction mean, and the historical week interval transaction mean to obtain a target historical data set.

[0176] It can be understood that, according to the historical time window transaction volume, the historical day interval transaction mean, and the historical week interval transaction mean, the target historical data set is extracted from the preprocessed transaction data and integrated together. The target historical data set is formed by integrating the historical data under the three strategies, which provides multi-dimensional and multi-period historical reference data for subsequent anomaly judgment, so that the anomaly judgment can be compared from multiple angles such as short-term, recent day mean, and week mean, and the comprehensiveness and accuracy of anomaly recognition are improved.

[0177] The embodiment provides a large-scale transaction data anomaly recognition method, which extracts multi-dimensional and multi-period historical transaction data from preprocessed transaction data as a target historical data set by using a nearest time monitoring strategy, a nearest day mean monitoring strategy, and a nearest week same day monitoring strategy, solves the technical problems of single anomaly monitoring and lack of multi-angle comparative analysis in the prior art, and achieves the beneficial effects of improving anomaly recognition accuracy and comprehensiveness, so that transaction anomalies can be more accurately recognized, and the security and stability of a financial transaction system are enhanced.

[0178] It should be noted that the above examples are only used for understanding the present application and do not constitute a limitation on the large-scale transaction data anomaly recognition method of the present application. Further simple transformations based on the technical concept are within the protection scope of the present application.

[0179] The present application also provides a large-scale transaction data anomaly recognition device, which is described in detail in the following Figure 6 The large-scale transaction data anomaly recognition device comprises:

[0180] A data acquisition module 10 is configured to acquire initial transaction data pushed from a transaction system to Kafka.

[0181] A data preprocessing module 20 is configured to send the initial transaction data to a Flink preprocessing module, so that the Flink preprocessing module performs aggregation extraction and downsampling preprocessing on the transaction data and returns preprocessed transaction data.

[0182] The data selection module 30 is configured to obtain current time window data according to the preprocessed transaction data, and obtain a target historical data set by data selection according to a preset abnormality determination strategy.

[0183] The abnormality identification module 40 is configured to perform abnormality determination according to the target historical data set and the current time window data, and obtain an abnormality identification result.

[0184] The large-scale transaction data abnormality identification device provided by the present application adopts the large-scale transaction data abnormality identification method in the above embodiments, and can solve the technical problems of low response speed and low accuracy in real-time monitoring of large-scale transaction data abnormal fluctuations. Compared with the prior art, the large-scale transaction data abnormality identification device provided by the present application has the same beneficial effects as the large-scale transaction data abnormality identification method provided by the above embodiments, and other technical features in the large-scale transaction data abnormality identification device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0185] In an embodiment, the abnormality identification module 40 is further configured to select target historical data from historical time window transaction volume, historical daily interval transaction mean value and historical weekly interval transaction mean value in sequence according to the target historical data set, perform abnormality determination according to the current time window data and the target historical data, and obtain an abnormality determination result, wherein the abnormality determination includes mean value determination, predicted mean value determination and similarity determination; determine whether the current time window data is abnormal according to the abnormality determination result; when the current time window data is abnormal, determine a historical data determination result of the target historical data set; and when it is determined that the historical data determination result of the target historical data set is that there are abnormal data items greater than or equal to a preset abnormality identification number, identify the current time window data as abnormal to obtain an abnormality identification result.

[0186] In an embodiment, the abnormality identification module 40 is further configured to obtain a mean value determination result, a predicted mean value determination result and a similarity determination result according to the abnormality determination result; and when the determination result of any two of the mean value determination result, the predicted mean value determination result and the similarity determination result is determined to be abnormal, determine that the current time window data is abnormal.

[0187] In an embodiment, the abnormality identification module 40 is further configured to calculate a historical transaction mean value according to the target historical data; calculate a current transaction mean value according to the current time window data; determine a mean value determination threshold according to the historical transaction mean value and a mean value determination coefficient; when the current transaction mean value is less than the mean value determination threshold, determine that the mean value determination result is abnormal, and obtain an abnormality determination result according to the mean value determination result.

[0188] In an embodiment, the anomaly identification module 40 is further configured to obtain a current time data vector and a target historical data vector according to the current time window data and the target historical data respectively, calculate a cosine similarity according to the current time data vector and the target historical data vector, determine a similarity determination result as abnormal when an absolute value of the cosine similarity is less than a similarity determination coefficient, and obtain an anomaly determination result according to the similarity determination result.

[0189] In an embodiment, the anomaly identification module 40 is further configured to calculate a current transaction mean according to the current time window data, calculate an estimation coefficient and an offset according to the target historical data, and calculate a fitting mean according to the estimation coefficient, the offset, and a preset first-order difference function, determine a predicted mean determination threshold according to the fitting mean and a predicted mean determination coefficient, determine a predicted mean determination result as abnormal when the current transaction mean is less than the predicted mean determination threshold, and obtain an anomaly determination result according to the predicted mean determination result.

[0190] In an embodiment, the data selection module 30 is further configured to obtain current time window data according to the preprocessed transaction data and a current time, select a historical time window transaction volume according to a nearest time monitoring strategy of a preset anomaly determination strategy, select a first preset determination time point according to a nearest day mean monitoring strategy of the preset anomaly determination strategy, and determine a historical day interval transaction mean according to the first preset determination time point, select a preset week offset and a second preset determination time point according to a nearest week same day monitoring strategy of the preset anomaly determination strategy, and determine a historical week interval transaction mean according to the preset week offset and the second preset determination time point, and perform data selection in the preprocessed transaction data according to the historical time window transaction volume, the historical day interval transaction mean, and the historical week interval transaction mean to obtain a target historical data set.

[0191] The present application provides a large-scale transaction data anomaly identification device, which comprises at least one processor and a memory connected in communication with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the large-scale transaction data anomaly identification method in the above-mentioned embodiment one.

[0192] Reference will be made to the following description of embodiments Figure 7, which shows a schematic diagram of the structure of a large-scale transaction data anomaly identification device suitable for implementing an embodiment of the present application. The large-scale transaction data anomaly identification device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The large-scale transaction data anomaly identification device shown is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

[0193] like Figure 7 As shown, the large-scale transaction data anomaly identification device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in ROM (Read Only Memory) 1002 or the program loaded from the storage device 1003 into RAM (Random Access Memory) 1004. In RAM 1004, various programs and data required for the operation of the large-scale transaction data anomaly identification device are also stored. The processing device 1001, ROM 1002 and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the large-scale transaction data anomaly identification device to communicate wirelessly or wired with other devices to exchange data. Although the diagram shows a large-scale transaction data anomaly identification device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented or have alternatively.

[0194] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0195] The large-scale transaction data anomaly identification device provided by the present application adopts the large-scale transaction data anomaly identification method in the above-mentioned embodiments, and can solve the technical problems of low response speed and low accuracy in real-time monitoring of large-scale transaction data anomaly fluctuations. Compared with the prior art, the large-scale transaction data anomaly identification device provided by the present application has the same beneficial effects as the large-scale transaction data anomaly identification method provided by the above-mentioned embodiments, and other technical features in the large-scale transaction data anomaly identification device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0196] It should be understood that various parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0197] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0198] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the large-scale transaction data anomaly identification method in the above-mentioned embodiments.

[0199] The computer readable storage medium provided in the application may be, for example, a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of the computer readable storage medium may include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a RAM (Random Access Memory), a ROM (Read Only Memory), an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a CD-ROM (CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the embodiment, the computer readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination thereof.

[0200] The computer readable storage medium described above may be contained in the large-scale transaction data anomaly identification device, or may exist separately without being assembled into the large-scale transaction data anomaly identification device.

[0201] The computer readable storage medium described above carries one or more programs, when the one or more programs are executed by the large-scale transaction data anomaly identification device, the large-scale transaction data anomaly identification device: obtains initial transaction data pushed from a transaction system to Kafka; sends the initial transaction data to a Flink preprocessing module, so that the Flink preprocessing module performs aggregation extraction and downsampling preprocessing on the transaction data, and returns preprocessed transaction data; obtains current time window data according to the preprocessed transaction data, and obtains a target historical data set according to a preset anomaly judgment strategy; performs anomaly judgment according to the target historical data set and the current time window data, and obtains an anomaly identification result.

[0202] The computer program code for performing the operations of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a LAN (Local Area Network) or a WAN (Wide Area Network), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0203] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0204] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0205] The computer-readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned method for identifying anomalies in large-scale transaction data. This computer-readable storage medium can address the technical issues of slow response speed and low accuracy in real-time monitoring of abnormal fluctuations in large-scale transaction data. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the method for identifying anomalies in large-scale transaction data provided in the aforementioned embodiments, and are not further elaborated here.

[0206] The application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the large-scale transaction data anomaly identification method as described above.

[0207] The computer program product provided by the application can solve the technical problems of low response speed and low accuracy in real-time monitoring of large-scale transaction data anomaly fluctuations. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the large-scale transaction data anomaly identification method provided by the above-mentioned embodiments, and are not described here.

[0208] The above only describes some embodiments of the application, and does not limit the patent scope of the application. Any equivalent structural transformation, direct / indirect application in other related technical fields, or direct / indirect application in other related technical fields within the technical concept of the application, including in the patent protection scope of the application.

Claims

1. A method for identifying anomalies in large-scale transaction data, characterized in that: The method comprises: Get the initial transaction data pushed from the transaction system to Kafka; Sending the initial transaction data to a Flink preprocessing module, so that the Flink preprocessing module performs aggregation extraction and downsampling preprocessing on the transaction data and returns the preprocessed transaction data; Obtaining current time window data based on the preprocessed transaction data, and selecting data based on a preset anomaly determination strategy to obtain a target historical data set; Anomaly determination is performed based on the target historical data set and the current time window data to obtain an anomaly recognition result.

2. The method according to claim 1, wherein The step of performing anomaly determination based on the target historical data set and the current time window data to obtain an anomaly recognition result includes: Selecting target historical data from the historical time window trading volume, the historical daily interval trading average, and the historical weekly interval trading average according to the target historical data set; Performing anomaly determination based on the current time window data and the target historical data to obtain an anomaly determination result, wherein the anomaly determination includes mean determination, predicted mean determination, and similarity determination; Determine whether the current time window data has an abnormality according to the abnormality determination result; When there is an anomaly in the current time window data, determining the historical data determination result of the target historical data set; When it is determined that the historical data determination result of the target historical data set is that there are abnormal data items greater than or equal to the preset abnormality identification number, the current time window data is identified as abnormal to obtain an abnormality identification result.

3. The method according to claim 2, wherein The step of determining whether the data in the current time window is abnormal according to the abnormality determination result includes: Obtaining a mean determination result, a predicted mean determination result, and a similarity determination result according to the abnormality determination result; When the determination results of any two of the mean determination result, the predicted mean determination result, and the similarity determination result are that an anomaly exists, it is determined that the current time window data is abnormal.

4. The method according to claim 2, wherein The step of performing anomaly determination based on the current time window data and the target historical data to obtain an anomaly determination result includes: Calculate the historical transaction average based on the target historical data; Calculate the current transaction average based on the current time window data; Determining a mean determination threshold based on the historical transaction mean and the mean determination coefficient; When the current transaction average is less than the average determination threshold, the average determination result is determined to be abnormal, and an abnormality determination result is obtained according to the average determination result.

5. The method according to claim 2, wherein The step of performing anomaly determination based on the current time window data and the target historical data to obtain an anomaly determination result further includes: Obtaining a current time data vector and a target historical data vector according to the current time window data and the target historical data respectively; Calculating cosine similarity based on the current time data vector and the target historical data vector; When the absolute value of the cosine similarity is less than the similarity determination coefficient, it is determined that the similarity determination result is abnormal, and an abnormality determination result is obtained according to the similarity determination result.

6. The method according to claim 2, wherein The step of performing anomaly determination based on the current time window data and the target historical data to obtain an anomaly determination result further includes: Calculate the current transaction average based on the current time window data; Calculating an estimated coefficient and an offset according to the target historical data, and calculating a fitting mean according to the estimated coefficient, the offset, and a preset first-order difference function; Determining a predicted mean determination threshold value based on the fitted mean and the predicted mean determination coefficient; When the current transaction mean is less than the predicted mean determination threshold, the predicted mean determination result is determined to be abnormal, and an abnormality determination result is obtained based on the predicted mean determination result.

7. The method according to claim 1, wherein The steps of obtaining current time window data based on the pre-processed transaction data and selecting data based on a preset anomaly determination strategy to obtain a target historical data set include: Determine current time window data based on the preprocessed transaction data and current time; Select the historical time window trading volume based on the nearest time monitoring strategy of the preset abnormality determination strategy; Selecting a first preset determination time point according to the nearest day average monitoring strategy of the preset abnormality determination strategy, and determining the historical daily interval transaction average based on the first preset determination time point; Selecting a preset weekly offset and a second preset determination time point according to the nearest same-day monitoring strategy of the preset abnormality determination strategy, and determining a historical weekly interval transaction average based on the preset weekly offset and the second preset determination time point; Data selection is performed in the pre-processed transaction data according to the historical time window transaction volume, the historical daily interval transaction average, and the historical weekly interval transaction average to obtain a target historical data set.

8. A device for identifying anomalies in large-scale transaction data, characterized in that: The device comprises: The data acquisition module is used to obtain the initial transaction data pushed from the transaction system to Kafka; A data preprocessing module is configured to send the initial transaction data to a Flink preprocessing module, so that the Flink preprocessing module performs aggregation extraction and downsampling preprocessing on the transaction data and returns the preprocessed transaction data; A data selection module is used to obtain current time window data based on the pre-processed transaction data, and to select data based on a preset anomaly determination strategy to obtain a target historical data set; The anomaly recognition module is used to perform anomaly judgment based on the target historical data set and the current time window data to obtain an anomaly recognition result.

9. A device for identifying anomalies in large-scale transaction data, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the large-scale transaction data anomaly identification method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the large-scale transaction data anomaly identification method according to any one of claims 1 to 7 are implemented.

Images