Detection method for incremental concept drift in privacy protection financial data

Abstract

The invention discloses a detection method and device for incremental concept drift in privacy protection financial data, equipment and a storage medium, and belongs to the field of data analysis, and the method comprises the steps: obtaining a distribution sequence of to-be-detected financial data; in response to delimitation of a reference distribution subsequence and a target distribution subsequence in the distribution sequence, determining a data distribution difference condition of the reference distribution subsequence and the target distribution subsequence; and under the condition that the data distribution difference condition is represented as mismatching, determining a plurality of verification subsequences from the distribution sequence, and determining the incremental concept drift condition of the distribution sequence according to the statistical matching between the target distribution subsequence and the verification subsequences. The incremental concept drift in the financial data can be accurately detected on the premise of ensuring privacy protection, missing detection and misjudgment are effectively avoided, and the problem that an analysis result is inaccurate when the incremental drift of the financial data is confirmed is solved.

Description

Technical Field

[0001] The present application belongs to the field of data analysis, and specifically relates to a detection method, device, equipment and storage medium for incremental concept drift in privacy-preserving financial data. Background Art

[0002] As the market fluctuates and the economic environment changes, the distribution of financial data and user behavior patterns will change, causing the probability distribution of data to change over time, which is also known as the concept drift phenomenon. This will affect the performance of economic models and the accuracy of decision-making. Therefore, timely detection and response to concept drift in financial data is the key to ensuring the business security and compliance of financial institutions.

[0003] Among the existing concept drift detection methods, the concept drift detection method based on data distribution is mainly used, which uses clustering and distribution measurement techniques to detect changes in data distribution through a sliding window method.

[0004] However, when the label data is unavailable or arrives late, the detection method based on label data may miss detection due to the unclear verification effect. In addition, since the financial data itself adds noise data for privacy protection, the verification results may contain false positives, which in turn causes misjudgment drift. Summary of the invention

[0005] The present application aims to provide a method, apparatus, device and storage medium for detecting incremental concept drift in privacy-preserving financial data, at least to solve the problem of inaccurate analysis results when confirming incremental drift of financial data.

[0006] In a first aspect, the present application discloses a method for detecting incremental concept drift in privacy-preserving financial data, including: Acquire a distribution sequence of financial data to be detected; the distribution sequence includes a plurality of distribution data of the financial data; each of the distribution data has a corresponding sampling time; In response to the demarcation of a reference distribution subsequence and a target distribution subsequence in the distribution sequence, determining a data distribution difference between the reference distribution subsequence and the target distribution subsequence; an end time of a reference time interval corresponding to the reference distribution subsequence is earlier than a start time of a target time interval corresponding to the target distribution subsequence; When the data distribution difference is characterized as a mismatch, multiple verification subsequences are determined from the distribution sequence, and the incremental concept drift of the distribution sequence is determined based on the statistical matching between the target distribution subsequence and the verification subsequence; the time length of the generated time interval corresponding to the verification subsequence is greater than the time length of the target time interval.

[0007] In a second aspect, the embodiment of the present application further discloses a detection device for incremental concept drift in privacy-preserving financial data, including: A sequence acquisition module, used to acquire a distribution sequence of financial data to be detected; the distribution sequence includes a plurality of distribution data of the financial data; each of the distribution data has a corresponding sampling time; a drift detection module, configured to determine the data distribution difference between the reference distribution subsequence and the target distribution subsequence in response to the demarcation of the reference distribution subsequence and the target distribution subsequence in the distribution sequence; the end time of the reference time interval corresponding to the reference distribution subsequence is earlier than the start time of the target time interval corresponding to the target distribution subsequence; A drift verification module is used to determine multiple verification subsequences from the distribution sequence when the data distribution difference is characterized as a mismatch, and determine the incremental concept drift of the distribution sequence based on the statistical matching between the target distribution subsequence and the verification subsequence; the time length of the generated time interval corresponding to the verification subsequence is greater than the time length of the target time interval.

[0008] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application further discloses a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In summary, in the embodiments of the present application, by delineating the reference distribution subsequence and the target distribution subsequence, the data distribution difference between the two is determined, and the time interval is delineated to ensure that the analysis can be focused on important time nodes, accurately detect data distribution changes, and timely discover potential concept drifts; and then, when the data distribution difference is characterized as a mismatch, multiple verification subsequences are determined from the distribution sequence, and the changes in data distribution are verified from multiple angles to further improve the accuracy and reliability of detection, avoid the deviation caused by a single verification, improve the credibility of drift detection, and ensure the accuracy of financial data analysis; finally, according to the statistical matching between the target distribution subsequence and the verification subsequence, the incremental concept drift of the distribution sequence is determined, and matching analysis is performed by using statistical methods, fully considering the data distribution characteristics, and efficiently identifying the incremental concept drift. Therefore, based on the method of the embodiment of the present application, by introducing the division of multiple subsequences and statistical matching analysis, the false positive problems caused by unavailable or delayed arrival of label data and noise data in the prior art are solved. This detection method can accurately detect incremental concept drift in financial data while ensuring privacy protection, effectively avoiding missed detection and misjudgment, improving the performance of economic models and the accuracy of decision-making. It also solves the problem of inaccurate analysis results when confirming incremental drift of financial data. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the attached picture: Figure 1 It is a flowchart of a method for detecting incremental concept drift in privacy-preserving financial data provided by an embodiment of the present application; Figure 2 It is a flowchart of another method for detecting incremental concept drift in privacy-preserving financial data provided by an embodiment of the present application; Figure 3 It is a detection model provided according to an embodiment of the present application; Figure 4 It is a drift detection process of the detection model provided in the embodiment of the present application; Figure 5 It is a drift verification process of the detection model provided in the embodiment of the present application; Figure 6 It is a method adaptation process of the detection model provided in the embodiment of the present application; Figure 7 This is a time series diagram of two types of concept drift; Figure 8 It is a block diagram of a detection device for incremental concept drift in privacy-preserving financial data provided by an embodiment of the present application; Fig. 9 is a block diagram of an electronic device according to an embodiment of the present application; Fig.10 It is a block diagram of an electronic device of another embodiment provided by the embodiments of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0013] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0014] Through verification, financial concepts can be reflected according to the probability distribution of data, that is, , so the concept of a specific time t is defined as Then the concept drift is the distribution change between time t and time m, that is, Thus, in order to detect concept drift, the distribution metric D(t,m) can be used to represent the distribution difference between two time points. If the current D(t,m) is not within the expected distribution range, it indicates that concept drift has occurred: Assume that the concept at time t1 is , and then this concept becomes at time t2 , if there is a significant change in the probability distribution of the data between t1 and t2, the value of D(t1, t2) will reveal a significant distribution change, that is, the value of D(t1, t2) exceeds the expected distribution range, and a concept drift occurs between t1 and t2.

[0015] Furthermore, in order to realize the unsupervised concept drift detection method, it is necessary to measure the difference of data from the perspective of distribution, that is, to choose a suitable distribution difference measure to calculate D(t,m). Taking into account the three reasons of complementarity, sensitivity and accuracy, Jensen-Shannon Divergence (JS) is selected from the perspective of relative entropy for measurement:

[0016] Among them, P(i) and Q(i) are the probability distributions of the data.

[0017] From the perspective of geometric differences, the Hellinger distance is used for measurement:

[0018] in, is the average probability distribution of P and Q, is the KL divergence:

[0019] Since these two metrics have only non-negative values ​​and are symmetric, this ensures the consistency and reliability of the evaluation, and using these two indicators together captures the differences in data distribution more comprehensively than using either indicator alone. This combined application is more accurate and effective in detecting concept drift in data streams. The specific analysis is as follows: At the complementary level, a comprehensive description of the data distribution includes not only traditional statistical indicators such as mean, variance, and standard deviation, but also more complex features such as the entropy and shape of the distribution. Entropy reflects the uncertainty of the data, while the shape involves geometric features such as symmetry, skewness, and kurtosis of the distribution. In order to fully quantify these differences, JS divergence and Hellinger distance provide complementary perspectives. JS divergence focuses on evaluating the differences in the entropy and information content of the distribution, while Hellinger distance focuses on capturing the differences in these geometric features. By combining these two indicators, changes in data distribution can be more accurately portrayed and analyzed.

[0020] At the sensitivity level, for data changes that last for a period of time, such as incremental drift, the data distribution gradually evolves throughout the process. The KL divergence component of the JS divergence is particularly sensitive to the central area of ​​the distribution, and even if the mean or variance of P and Q is slightly different, the KL divergence will amplify these differences in the calculation. On the other hand, the Hellinger distance measures the difference by comparing the square roots of the probability distribution and is extremely sensitive to the overall shape of the distribution, such as symmetry, peaks, or flatness. Using a combination of these two indicators, early signs of concept drift can be captured earlier, thereby detecting it when concept drift occurs.

[0021] At the accuracy level, in data streams, noise can lead to false detections of concept drift. The high sensitivity of JS divergence to the central area of ​​the distribution enables it to accurately capture subtle but real drifts, while reducing the risk of false positives caused by noise by reducing the response to meaningless fluctuations. At the same time, the sensitivity of Hellinger distance to the shape of the distribution enables it to identify statistically significant changes, even if these changes are not visually obvious. This prevents false positives caused by accidental or slight fluctuations. By combining these two methods, concept drift in data streams can be more accurately detected, effectively reducing the false positive rate, while ensuring that real drift is not ignored.

[0022] Based on the above analysis, Figure 1 A method for detecting incremental concept drift in privacy-preserving financial data is provided in an embodiment of the present application.

[0023] The method may include the following steps: Step 101: Obtain a distribution sequence of financial data to be detected.

[0024] The distribution sequence includes multiple distribution data of financial data; each distribution data has a corresponding sampling time.

[0025] In some embodiments of the present application, it is first necessary to obtain a distribution sequence of financial data, which includes distribution data at multiple time points and can reflect the change of data over time. Among them, the distribution sequence includes multiple distribution data of financial data, and each distribution data has a corresponding sampling time. The sampling time refers to the specific time point of each distribution data record. In order to provide basic data support for subsequent distribution analysis and drift detection.

[0026] In a specific example, suppose you need to detect the distribution changes of transaction data over a period of time. You can record the daily transaction data and generate a distribution sequence. The system automatically collects daily transaction data and arranges the data in chronological order to form a distribution sequence containing distribution data at multiple time points. Through such operations, you can obtain the distribution changes of transaction data over a period of time, which provides the necessary data basis for subsequent concept drift detection.

[0027] Step 102, in response to the demarcation of the reference distribution subsequence and the target distribution subsequence in the distribution sequence, determining the data distribution difference between the reference distribution subsequence and the target distribution subsequence.

[0028] The end time of the reference time interval corresponding to the reference distribution subsequence is earlier than the start time of the target time interval corresponding to the target distribution subsequence.

[0029] In some embodiments of the present application, in order to determine the data distribution difference between the reference distribution subsequence and the target distribution subsequence and ensure that the data change can be accurately detected, the data distribution difference between the reference distribution subsequence and the target distribution subsequence is determined in response to the delineation of the reference distribution subsequence and the target distribution subsequence in the distribution sequence, wherein the end time of the reference time interval corresponding to the reference distribution subsequence is earlier than the start time of the target time interval corresponding to the target distribution subsequence. The data distribution difference refers to whether the data distribution between the two time intervals is consistent or significantly different. In this way, potential concept drift can be discovered in a timely manner to ensure the accuracy of data analysis.

[0030] In a specific example, it is necessary to analyze the changes in customer transaction behaviors in different time periods. The first two months of data in a year can be selected as the reference distribution subsequence, and the last two months of data can be selected as the target distribution subsequence. The system demarcates the reference distribution subsequence and the target distribution subsequence based on the timestamp and calculates the difference in their data distribution. If a significant difference is found in the data distribution between the two, the system will promptly prompt the possible concept drift problem. Through such operations, changes in customer behavior patterns can be understood in a timely manner, providing support for further risk management and decision-making.

[0031] Step 103, when the data distribution difference is characterized as mismatch, multiple verification subsequences are determined from the distribution sequence, and the incremental concept drift of the distribution sequence is determined based on the statistical matching between the target distribution subsequence and the verification subsequence.

[0032] The length of the generated time interval corresponding to the verification subsequence is greater than the length of the target time interval.

[0033] In some embodiments of the present application, in order to further confirm the authenticity and accuracy of the incremental concept drift by verifying the subsequence when the data distribution difference is characterized as a mismatch, multiple verification subsequences are determined from the distribution sequence when the data distribution difference is characterized as a mismatch, and the incremental concept drift of the distribution sequence is determined based on the statistical matching between the target distribution subsequence and the verification subsequence. The verification subsequence is a subsequence extracted from the distribution sequence, and the length of its time interval is greater than the length of the target time interval. Through multi-angle verification subsequence analysis, the accuracy and reliability of drift detection can be improved, the deviation caused by a single verification can be avoided, and the credibility of the overall detection can be improved.

[0034] In a specific example, it has been found that there is a significant difference in the distribution of transaction data within a certain time period. Then, after detecting the data distribution difference mismatch between the reference distribution subsequence and the target distribution subsequence, the system extracts multiple verification subsequences from the distribution sequence. The system can calculate the statistical match between each verification subsequence and the target distribution subsequence, and the time interval length of the verification subsequence is greater than the time length of the target time interval. If it is found that the match between these verification subsequences and the target distribution subsequence is low, the system will confirm the existence of incremental concept drift. Through such operations, changes in customer trading behavior can be detected and confirmed more accurately, providing support for further risk management and decision-making.

[0035] In summary, in the embodiments of the present application, by delineating the reference distribution subsequence and the target distribution subsequence, the data distribution difference between the two is determined, and the time interval is delineated to ensure that the analysis can be focused on important time nodes, accurately detect data distribution changes, and timely discover potential concept drifts; and then, when the data distribution difference is characterized as a mismatch, multiple verification subsequences are determined from the distribution sequence, and the changes in data distribution are verified from multiple angles to further improve the accuracy and reliability of detection, avoid the deviation caused by a single verification, improve the credibility of drift detection, and ensure the accuracy of financial data analysis; finally, according to the statistical matching between the target distribution subsequence and the verification subsequence, the incremental concept drift of the distribution sequence is determined, and matching analysis is performed by using statistical methods, fully considering the data distribution characteristics, and efficiently identifying the incremental concept drift. Therefore, based on the method of the embodiment of the present application, by introducing the division of multiple subsequences and statistical matching analysis, the false positive problems caused by unavailable or delayed arrival of label data and noise data in the prior art are solved. This detection method can accurately detect incremental concept drift in financial data while ensuring privacy protection, effectively avoiding missed detection and misjudgment, improving the performance of economic models and the accuracy of decision-making. It also solves the problem of inaccurate analysis results when confirming incremental drift of financial data.

[0036] Figure 2 This is another method for detecting incremental concept drift in privacy-preserving financial data provided by an embodiment of the present application.

[0037] The method may include the following steps: Step 201: Obtain the distribution sequence of financial data to be detected.

[0038] The distribution sequence includes multiple distribution data of financial data; each distribution data has a corresponding sampling time.

[0039] The method shown in this step has been explained in step 101 and will not be repeated here.

[0040] Step 202 : in response to the demarcation of the reference distribution subsequence and the target distribution subsequence in the distribution sequence, determining the data distribution difference between the reference distribution subsequence and the target distribution subsequence.

[0041] The end time of the reference time interval corresponding to the reference distribution subsequence is earlier than the start time of the target time interval corresponding to the target distribution subsequence.

[0042] The method shown in this step has been explained in step 102 and will not be repeated here.

[0043] Optionally, step 202 includes the following sub-steps: Sub-step 2021, according to a preset sampling window threshold, divide a plurality of continuous detection time windows from the sampling moments corresponding to the distribution data.

[0044] Each detection time window includes multiple sampling moments; different detection time windows include different sampling moments.

[0045] In some embodiments of the present application, in order to divide the sampling time of the distribution data into multiple continuous detection time windows, so as to better perform data analysis and drift detection, multiple continuous detection time windows are divided from the sampling time corresponding to the distribution data according to the preset sampling window threshold, wherein each detection time window contains multiple sampling moments; different detection time windows contain different sampling moments. The sampling window threshold refers to the preset time interval used to divide the time window. In this way, the data of a long time series can be divided into multiple small time windows, which is convenient for further processing and analysis, and improves the efficiency and accuracy of data processing.

[0046] In a specific example, it is necessary to analyze the changes in transaction data within a year. The transaction data within a year can be sampled daily, and the data can be divided into multiple continuous detection time windows according to the preset sampling window threshold, so that it can show statistical characteristics in a short period of time. For example, the system samples the transaction data every day, and divides the data into weekly or monthly units to form multiple detection time windows. This will facilitate subsequent analysis and detection, and improve the efficiency of data processing and the accuracy of analysis.

[0047] Sub-step 2022, in response to the selection of a reference detection time window and a target detection time window among the multiple detection time windows, determines the distribution data corresponding to the multiple sampling moments contained in the reference detection time window as the distribution data in the reference distribution subsequence, and determines the distribution data corresponding to the multiple sampling moments contained in the target detection time window as the distribution data in the target distribution subsequence.

[0048] In some embodiments of the present application, in order to select a reference detection time window and a target detection time window from a plurality of detection time windows and determine their corresponding distribution data, so as to perform subsequent distribution feature analysis, in response to the selection of the reference detection time window and the target detection time window in the plurality of detection time windows, the distribution data corresponding to the plurality of sampling moments contained in the reference detection time window are determined as the distribution data in the reference distribution subsequence, and the distribution data corresponding to the plurality of sampling moments contained in the target detection time window are determined as the distribution data in the target distribution subsequence. A distribution subsequence refers to a portion of data extracted from the entire distribution sequence. In this way, the data in the reference detection time window and the target detection time window can be effectively extracted, providing a basis for subsequent data distribution difference analysis.

[0049] In a specific example, it is necessary to analyze the transaction behavior of customers within a year. The system divides the distribution of transaction data within a year into multiple detection time windows, and selects January as the reference detection time window and August as the target detection time window. The system extracts the transaction data corresponding to multiple sampling moments in January and determines them as the distribution data in the reference distribution subsequence. At the same time, it extracts the transaction data corresponding to multiple sampling moments in August and determines them as the distribution data in the target distribution subsequence. Through such operations, the system can obtain the distribution data of the reference detection time window and the target detection time window, providing necessary data support for subsequent distribution feature analysis and drift detection.

[0050] Sub-step 2023, determining a reference distribution feature of the reference distribution subsequence according to the multiple distribution data included in the reference distribution subsequence, and determining a target distribution feature of the target distribution subsequence according to the multiple distribution data included in the target distribution subsequence.

[0051] In some embodiments of the present application, in order to clarify the distribution characteristics of the reference distribution subsequence and the target distribution subsequence, so as to perform subsequent distribution difference analysis, the reference distribution characteristics of the reference distribution subsequence are determined based on the multiple distribution data contained in the reference distribution subsequence, and the target distribution characteristics of the target distribution subsequence are determined based on the multiple distribution data contained in the target distribution subsequence. Distribution characteristics refer to the statistical characteristics of data, such as mean, variance, skewness, and kurtosis. In this way, the distribution characteristics of each time window can be accurately extracted, providing specific indicators and references for subsequent distribution difference analysis.

[0052] In a specific example, it is necessary to analyze the consumption habits of customers in different time periods. The specific scenario is that the system uses the data of the first week of a quarter as the reference distribution subsequence and the data of the last week as the target distribution subsequence. The system calculates the statistical characteristics such as mean, variance, skewness and kurtosis based on the consumption data of the first and last weeks. Through such operations, the system can obtain the specific distribution characteristics of the reference distribution subsequence and the target distribution subsequence, which provides a basis for subsequent distribution difference analysis and helps financial institutions understand changes in customer consumption habits.

[0053] Optionally, sub-step 2023 includes the following sub-steps: Sub-step 20231, performing dimensionality reduction weighting on each distribution data contained in the reference distribution subsequence and the target distribution subsequence to obtain a reference reduced dimensionality subsequence of the reference distribution subsequence and a target reduced dimensionality subsequence of the target distribution subsequence.

[0054] The reference reduced dimension subsequence contains reduced dimension data corresponding to each distribution data in the reference distribution subsequence, and the target reduced dimension subsequence contains reduced dimension data corresponding to each distribution data in the target distribution subsequence.

[0055] In some embodiments of the present application, in order to extract and simplify the data features of the reference distribution subsequence and the target distribution subsequence, each distribution data contained in the reference distribution subsequence and the target distribution subsequence is weighted by dimensionality reduction to obtain a reference dimensionality reduction subsequence of the reference distribution subsequence and a target dimensionality reduction subsequence of the target distribution subsequence. Dimensionality reduction weighting refers to reducing the complexity of the data by reducing the dimension of the data and assigning it a weight, while maintaining its important features. This can simplify the data structure, reduce the computational complexity, and improve the efficiency of subsequent analysis.

[0056] In a specific example, it is necessary to analyze the transaction behavior of customers in different time periods. The specific scenario is that the system has divided the customer transaction data into reference distribution subsequences and target distribution subsequences. The system can perform dimensionality reduction and weighting processing on the transaction data in each subsequence to extract the reference dimensionality reduction subsequence and the target dimensionality reduction subsequence. Dimensionality reduction and weighting processing reduces the dimension of the data while retaining important features. Through such operations, the system can simplify the data structure and improve the efficiency of subsequent data analysis and comparison.

[0057] As a specific implementation method, in the process of dimensionality reduction and weighting, it is necessary to divide the subspace and calculate the weight of each subspace. Considering that the method is for unlabeled data streams, the kernel principal component analysis method can be used for weight calculation. This is a method suitable for unlabeled environments. The specific method is to use kernel principal component analysis to calculate eigenvalues ​​and assign weights to each subspace based on these eigenvalues:

[0058] in, is the weight of the ith subspace, is the eigenvalue corresponding to the subspace, and n is the number of subspaces.

[0059] Sub-step 20232, using the weighted product model to merge the dimensionality reduction data in the reference dimensionality reduction subsequence, so as to use the obtained probability distribution of the reference dimensionality reduction subsequence as the reference distribution feature, and using the weighted product model to merge the dimensionality reduction data in the target dimensionality reduction subsequence, so as to use the obtained probability distribution of the target dimensionality reduction subsequence as the target distribution feature.

[0060] In some embodiments of the present application, in order to generate reference and target distribution features, a weighted product model is used to merge the reduced dimension data in the reference reduced dimension subsequence, so as to obtain the probability distribution of the reference reduced dimension subsequence as the reference distribution feature, and a weighted product model is used to merge the reduced dimension data in the target reduced dimension subsequence, so as to obtain the probability distribution of the target reduced dimension subsequence as the target distribution feature. The weighted product model is a mathematical method that performs weighted averaging by assigning different weights to different data points. In this way, more accurate and robust distribution features can be generated, providing a basis for subsequent data comparison and analysis.

[0061] In a specific example, it is necessary to analyze the probability distribution of customer transaction behaviors in different time periods. The system has already reduced the dimension of customer transaction data and generated reference reduced dimension subsequences and target reduced dimension subsequences. Then the system can use the weighted product model to merge these reduced dimension data separately, and generate the probability distribution of the reference reduced dimension subsequence and the probability distribution of the target reduced dimension subsequence as their distribution features. Through such operations, the system can accurately generate probability distribution features that reflect customer transaction behaviors, providing key data support for subsequent distribution comparison and concept drift detection.

[0062] As a specific implementation method, when merging subspace probability distributions in the process of generating distribution features, the intuitive way is to merge them directly according to the weights, that is, the weighted sum model (WSM) method, or use a variant of the weighted product model (WPM) to calculate the distribution merger:

[0063] in, is the weighted product result, is the weight of the ith subspace, is the weight of the j-th subspace, is the probability distribution on the corresponding subspace.

[0064] Sub-step 2024, determining the data distribution difference between the reference distribution subsequence and the target distribution subsequence according to the reference distribution characteristics and the target distribution characteristics.

[0065] In some embodiments of the present application, in order to analyze the data distribution difference between the reference distribution subsequence and the target distribution subsequence and determine whether there is a concept drift, the data distribution difference between the reference distribution subsequence and the target distribution subsequence will be determined based on the reference distribution characteristics and the target distribution characteristics. The data distribution difference refers to whether there is a significant statistical difference in the distribution characteristics of the two time windows, reflecting the change in data distribution. In this way, changes in data distribution can be effectively identified, potential concept drift can be discovered in a timely manner, and important judgment basis can be provided for financial data analysis.

[0066] In a specific example, we want to analyze the changes in customer transaction behavior within a quarter. The specific scenario is that the system has determined the first week of a quarter as the reference detection time window and the last week as the target detection time window, and has calculated the distribution characteristics of the two respectively. Then the system will calculate the data distribution difference between them through statistical methods based on the distribution characteristics of the two time windows. Through this operation, the system can determine whether there is a significant difference between the two, and thus determine whether concept drift has occurred.

[0067] Optionally, sub-step 2024 includes the following sub-steps: Sub-step 20241, determining a reference confidence radius of the reference distribution subsequence according to the reference distribution characteristics, and determining a target confidence radius of the target distribution subsequence according to the target distribution characteristics.

[0068] In some embodiments of the present application, in order to quantify and compare the distribution characteristics of the reference distribution subsequence and the target distribution subsequence, the reference confidence radius of the reference distribution subsequence is determined according to the reference distribution characteristics, and the target confidence radius of the target distribution subsequence is determined according to the target distribution characteristics. The confidence radius is the range of data distribution at a certain confidence level, which can quantify the degree of dispersion of data points around the mean. This can provide specific and quantitative indicators for subsequent distribution difference analysis and ensure the accuracy of distribution feature comparison.

[0069] In a specific example, it is necessary to evaluate the distribution consistency of transaction data in different time periods. The specific scenario is that the system has calculated the distribution characteristics of the reference time window and the target time window in a quarter. The system calculates the reference confidence radius based on the distribution characteristics of the transaction data in the reference time window; at the same time, it calculates the target confidence radius based on the distribution characteristics of the transaction data in the target time window. The confidence radius provides the degree of statistical dispersion of the data. Through such operations, the system obtains the confidence radius of the reference time window and the target time window, providing a reliable numerical basis for the subsequent data distribution difference analysis.

[0070] Sub-step 20242, when the target confidence radius is less than or equal to the reference confidence radius, the distribution difference is determined as a match; when the target confidence radius is greater than the reference confidence radius, the distribution difference is determined as a mismatch.

[0071] In some embodiments of the present application, in order to determine the data distribution difference of the reference distribution subsequence and the target distribution subsequence according to their confidence radius, when the target confidence radius is less than or equal to the reference confidence radius, the distribution difference is determined to be a match, and when the target confidence radius is greater than the reference confidence radius, the distribution difference is determined to be a mismatch. The confidence radius is the discrete range of the data around the mean at a certain confidence level, reflecting the consistency and concentration of the data. In this way, it can be determined whether the distribution characteristics of the reference distribution subsequence and the target distribution subsequence are consistent, thereby improving the accuracy of the data distribution difference analysis.

[0072] In a specific example, it is necessary to evaluate the similarity of customer transaction behaviors in different time periods. The specific scenario is that the system has calculated the confidence radius of the reference time window and the target time window in a certain month. The system compares the confidence radius of the reference time window with the confidence radius of the target time window. If the target confidence radius is less than or equal to the reference confidence radius, the system determines the distribution difference as a match; if the target confidence radius is greater than the reference confidence radius, the system determines the distribution difference as a mismatch.

[0073] like Figure 2 As shown in the figure, as a specific implementation process, two windows can be maintained during the drift detection process: one is a reference window, denoted as , one is the target window sliding backward, denoted as , let the window size be s, and the sliding step size be s. When selected, the probability distribution of the data contained in the two windows will be calculated. After obtaining the probability distribution, it is necessary to use the distribution difference metric to calculate the distribution difference between the two, and then compare it with the average of the historical values ​​to see if it exceeds the range of 3sigma. In order to reduce the interference of noise, the harmonic mean is used here instead of the arithmetic mean H:

[0074] in, is the sampling value. To simplify the description, the stream data is represented as a series of data blocks S, and the size of each data block is equal to s. In practical applications, if the stream data is input as a single data block, the detection process can be started after accumulating enough s data.

[0075] Step 203, when the data distribution difference is characterized as mismatch, multiple verification subsequences are determined from the distribution sequence, and the incremental concept drift of the distribution sequence is determined based on the statistical matching between the target distribution subsequence and the verification subsequence.

[0076] The length of the generated time interval corresponding to the verification subsequence is greater than the length of the target time interval.

[0077] The method shown in this step has been explained in step 102 and will not be repeated here.

[0078] Optionally, the target distribution subsequence has a corresponding target distribution feature, and step 203 includes the following sub-steps: Sub-step 2031, when the data distribution difference is characterized as a mismatch, a plurality of continuous verification time windows are divided from the sampling moments corresponding to the distribution data according to a preset verification window threshold.

[0079] Each verification time window includes multiple sampling moments; different verification time windows include different sampling moments.

[0080] In some embodiments of the present application, in order to divide a plurality of continuous verification time windows from the sampling moments of the distribution data so as to further analyze the changes in the data distribution, when the data distribution difference is characterized as a mismatch, a plurality of continuous verification time windows are divided from the sampling moments corresponding to the distribution data according to a preset verification window threshold, wherein each verification time window contains a plurality of sampling moments; different verification time windows contain different sampling moments. The verification window threshold refers to the preset time interval used to divide the time window. In this way, the division of the time series data can be refined to ensure that the verification analysis can be more targeted and accurate.

[0081] In a specific example, it is necessary to verify whether the transaction data within one month of a certain year has experienced concept drift. When the system detects a mismatch in data distribution differences, it divides the transaction data within the past year into multiple continuous verification time windows according to the preset verification window threshold. The system can divide the annual transaction data into four verification time windows with each quarter as a new time interval. In this way, the system can carefully analyze the transaction data within each verification time window, improve the accuracy and reliability of data analysis, and ensure more accurate verification of concept drift.

[0082] As a specific implementation method, in the process of dividing the verification time window, considering that it is impossible to know the duration of the drift in advance during the detection process, the size of the window is calculated based on the historical information of the drift interval and the drift duration. In this article, the drift interval is based on the number of window slides, and the drift duration is the same. The basic concept is as follows: when the drift duration is long, more information is needed to confirm the occurrence of drift, so the window should be appropriately increased. Similarly, when the drift duration is short, the window should be appropriately reduced to avoid introducing additional delays. For the drift interval, when the interval is large, it means that the stable period after the drift is relatively long, and the tolerance for delay will be relatively high. The window can be appropriately increased to ensure accurate verification. When the drift interval is short, the window size should be reduced to verify the drift as soon as possible. The specific value can be expressed by the following formula:

[0083] Among them, n is the verification window size, N d is the average length of historical drift, N i is the average interval of historical drift. The square root of the product is used here to avoid the introduction of custom parameters and balance N d and N i The effect on n prevents the window size from growing too fast.

[0084] Sub-step 2032, determining the verification distribution feature corresponding to the verification time window according to the distribution data corresponding to the multiple sampling moments contained in each verification time window.

[0085] In some embodiments of the present application, in order to extract the distribution characteristics of each verification time window, and further analyze the changes in data distribution, the verification distribution characteristics corresponding to the verification time window will be determined based on the distribution data corresponding to the multiple sampling moments contained in each verification time window. Distribution characteristics refer to the statistical characteristics of data, such as mean, variance, skewness, and kurtosis. In this way, the specific distribution characteristics of each verification time window can be obtained, providing data support for subsequent extreme probability value calculations and drift judgments.

[0086] In a specific example, the transaction behavior changes within one month of a certain year are required. The specific scenario is that the system has divided the transaction data of the year into multiple new verification time windows. The system calculates the distribution characteristics of these data based on the transaction data at multiple sampling moments in each verification time window. Through such operations, the system can obtain the specific distribution characteristics of each verification time window, which provides a basis for the subsequent calculation of extreme probability values ​​and concept drift judgment.

[0087] As a specific implementation method, in the process of determining the verification distribution characteristics corresponding to the verification time window, a t-test is used to verify whether drift has indeed occurred on the verification window. Specifically, by comparing the data distribution of the target window and the verification window, it is determined whether there is a significant difference between the two. The null hypothesis (H0) is that the data of the target window and the verification window come from the same distribution (that is, there is no drift), and the alternative hypothesis (H1) is that the two come from different distributions (that is, drift has occurred). In this way, based on the divided subspaces, a t-test is performed on each subspace separately:

[0088] in , are the average values ​​of the target window and validation window data, respectively. and are the sample variances of the target window and the validation window, respectively, and are the number of samples in the target window and validation window, respectively.

[0089] Sub-step 2033, determining the verification extreme probability value of each verification distribution feature and the target extreme probability value of the target distribution feature.

[0090] In some embodiments of the present application, in order to calculate the extreme probability value of each verification distribution feature and the target distribution feature, and thus evaluate the change in data distribution, the verification extreme probability value of each verification distribution feature and the target extreme probability value of the target distribution feature will be determined. The extreme probability value, also known as the p-value, refers to the probability of a data feature appearing in an extreme case in statistical analysis, and is used to measure the extreme case of data distribution. In this way, the distribution characteristics of each verification time window and the target time window in extreme cases can be clarified, providing important reference data for subsequent concept drift judgment.

[0091] In a specific example, it is necessary to evaluate the extreme changes in trading behavior in different time periods. The system has divided the verification time windows for the past year and calculated the distribution characteristics of each verification time window and the target time window. The system further calculates the extreme probability values ​​of each verification distribution characteristic and the target distribution characteristic, for example, the system calculates the p-value in each time window. Through such operations, the system can clearly understand the extreme changes in trading behavior in different time periods, and provide accurate data support for subsequent concept drift judgment.

[0092] Sub-step 2034, when the target extreme probability value is greater than a preset number of verification extreme probability values, determining the target time interval corresponding to the target distribution subsequence as the time interval for the occurrence of incremental concept drift.

[0093] In some embodiments of the present application, in order to finally confirm whether the data distribution within a specific time interval has undergone incremental concept drift, when the target extreme probability value is greater than a preset number of verification extreme probability values, the target time interval corresponding to the target distribution subsequence is determined as the time interval in which the incremental concept drift occurs. The extreme probability value refers to the probability of the data distribution feature appearing under extreme circumstances. In this way, it is possible to clarify whether the data distribution has changed significantly within the target time interval, thereby confirming the occurrence of incremental concept drift and improving the accuracy and reliability of detection.

[0094] In a specific example, it is necessary to confirm whether there is incremental concept drift in the change of customer transaction behavior within a certain month. The system calculates the extreme probability value of the transaction amount in the target time window within a certain month, and compares it with the extreme probability values ​​of multiple verification time windows. For example, when the system finds that the extreme probability value of the target time window is greater than one-third of the extreme probability values ​​of the verification time windows, the system determines the target time interval as the time interval where incremental concept drift occurs. In this way, changes in customer transaction behavior can be accurately judged.

[0095] As a specific implementation method, the corresponding p-value is obtained according to the corresponding t-value. After obtaining the corresponding p-value, the p-value will be merged according to the weight of the subspace. Considering that the verification window size is n times the target window size, n t-test calculations will actually be performed. At this time, the problem is transformed into a multiple verification problem.

[0096] In the case of multiple verification, considering the increase of false positive rate (type I error) and false negative rate (type II error), it can be used The results are further processed by correction method. If the number of values ​​is greater than or equal to half, it means that drift has indeed occurred.

[0097] Step 204: Determine an updated value of the verification window threshold according to the adjustment of the verification window threshold.

[0098] The updated verification window threshold is used to determine the time interval length of the verification window as the minimum value of the time interval lengths in which incremental concept drift occurs.

[0099] In some embodiments of the present application, in order to dynamically adjust the verification window threshold to adapt to the time length of the concept drift in the actual detection process, the updated value of the verification window threshold will be determined based on the adjustment of the verification window threshold, wherein the updated verification window threshold is used to determine the time interval length of the verification window as the minimum value of the time interval length in which the incremental concept drift occurs. The time interval length refers to the time range covered by the verification window when performing drift detection. In this way, the accuracy and efficiency of drift detection can be improved, ensuring that the detection results are more accurate and reliable.

[0100] In a specific example, continuous transaction data needs to be drift detected. In order to ensure that the initially set verification window threshold can accurately meet the needs of subsequent detection, it needs to be adjusted. The system will dynamically adjust the verification window threshold based on the previous detection results to update it to the minimum value that can cover the incremental concept drift time length. Through such operations, the system can flexibly adapt to different detection situations, improve the accuracy of drift detection, and ensure that the results of financial data analysis are further reliable and effective.

[0101] Optionally, step 204 includes the following sub-steps: Sub-step 2041, based on the adjustment of the verification window threshold, determines the variance of the silhouette coefficient of the verification window threshold being adjusted.

[0102] In some embodiments of the present application, in order to calculate the variance of the silhouette coefficient during the adjustment process according to the adjustment of the verification window threshold value to evaluate the stability of the data distribution, the variance of the silhouette coefficient of the verification window threshold value in the adjustment will be determined according to the adjustment of the verification window threshold value. The silhouette coefficient measures the clustering effect, and the variance is used to evaluate the discreteness and stability of the data. In this way, the distribution stability of the verification window can be quantified, providing a basis for the subsequent update of the verification window threshold value.

[0103] In a specific example, the verification window threshold needs to be adjusted dynamically to adapt to changes in transaction data in different time periods. The time window is scaled by adjusting the verification window threshold, and the variance of the silhouette coefficient during the adjustment process is determined. Based on the verification window data after each adjustment, its silhouette coefficient is calculated, and the variance of these coefficients is further calculated. In this way, the system can evaluate the stability of data distribution under different verification window thresholds.

[0104] Sub-step 2042, when the variance is less than or equal to the preset stationary distribution coefficient, stop adjusting the verification window threshold, and determine the verification window threshold determined when stopping as the updated value of the verification window threshold.

[0105] In some embodiments of the present application, in order to determine when to stop adjusting the verification window threshold by monitoring the change in variance during the adjustment process of the verification window threshold to ensure the stability of the data distribution, the adjustment of the verification window threshold will be stopped when the variance is less than or equal to the preset stable distribution coefficient, and the verification window threshold determined when stopping will be determined as the updated value of the verification window threshold. The stable distribution coefficient is a preset standard for judging the stability of the data distribution. In this way, it can be ensured that the setting of the verification window can reflect the stable distribution of the data and improve the accuracy and reliability of the detection results.

[0106] In a specific example, the system can calculate the variance of the contour coefficient of the verification data after each adjustment of the verification window threshold, and compare it with the preset stable distribution coefficient. When the system detects that the variance of the contour coefficient is less than or equal to the preset stable distribution coefficient, the adjustment of the verification window threshold is stopped, and the current verification window threshold is determined as the updated value. Through such operations, financial institutions can ensure that the setting of the verification window can accurately reflect the stability of data distribution and improve the accuracy and reliability of transaction data analysis.

[0107] As a specific implementation method, after the verification drift occurs, the width of the sliding verification window is used to continuously monitor the change of data distribution until the data reaches a stable state. The position where the drift is detected is recorded as k, where each sub-window with the same size as the reference window is represented in turn as . The judgment of whether the data distribution is stable is based on a comprehensive analysis of clustering and silhouette coefficient (Silhouette Score). The silhouette coefficient is used to measure the effect of clustering, and its value range is [−1,1]. The larger the value, the better the clustering effect, that is, the data points are more cohesive in the cluster and the separability between clusters is more obvious. When the silhouette coefficient is close to 1, it means that the distance between data points in the cluster is small and the distance between clusters is large, indicating that the data has reached a relatively stable distribution state; when the silhouette coefficient is close to 0 or negative, it means that the distribution of data points between clusters is not clear, and there may be overlap or drift has not yet ended: ; Among them, s(i) is the silhouette coefficient of a single data point i, a(i) is the average distance from point i to other points in its cluster (intra-cluster distance), and b(i) is the average distance from point i to the nearest other cluster (inter-cluster distance). After the distribution is stable, the reference window is updated using the data in the target window.

[0108] In order to be able to relate the silhouette coefficient to whether the data is stationary, we calculate Variance of the Silhouette Coefficient :

[0109] in Indicates the current adaptive verification window The starting position after sliding back i-1 steps, k+i+j represents the starting position of the sub-window in the adaptive verification window, For adaptive verification window The average silhouette coefficient of each sub-window in .

[0110] in, The calculation formula is:

[0111] When obtained After that, Compared with the stationary distribution parameter δ, when the inequality is satisfied When The data distribution in is unstable. Slide backward with a step size of s. Keep sliding back until the inequality Established. At this point, the data reaches a stable distribution, and the reference window is updated using the stable distribution data. At the same time, the duration of the drift can be obtained and recorded based on the number of slides.

[0112] Optionally, the target time interval is a time interval obtained by delaying the reference time interval by a target number of window times. In this case, the target time interval can be regarded as a time window obtained by sliding backward from the reference time interval. Then, after step 204, the following steps may be further included: Step 205: determine a target reconstruction error for a target time interval, and obtain a plurality of historical reconstruction errors.

[0113] Among them, the historical reconstruction error is a reconstruction error determined based on each time interval confirmed as having incremental concept drift before the target time interval; the target reconstruction error has a corresponding verification window update threshold; and each historical reconstruction error has a corresponding verification window threshold.

[0114] In some embodiments of the present application, in order to further confirm and verify the accuracy and reliability of incremental concept drift by analyzing the target reconstruction error and the historical reconstruction error, the target reconstruction error of the target time interval is determined, and multiple historical reconstruction errors are obtained, wherein the historical reconstruction error is a reconstruction error determined based on each time interval confirmed as having incremental concept drift before the target time interval; the target reconstruction error has a corresponding verification window update threshold; and each historical reconstruction error has a corresponding verification window threshold. Reconstruction error refers to the error generated when the actual data is reconstructed by the model. By analyzing the reconstruction error, the consistency of the data distribution can be judged. This can provide more accurate data support for subsequent distribution matching and drift confirmation, ensuring the accuracy and reliability of the detection results.

[0115] In a specific example, the analysis of transaction data changes within a certain period of time has been completed, and it has been determined whether there is incremental concept drift. The transaction data within a period of time is reconstructed through the Variational Autoencoder (VAE) model, and the target reconstruction error and multiple historical reconstruction errors of the target time interval are calculated. The system sets a corresponding verification window threshold for each reconstruction error based on the target reconstruction error determined by the target time interval and the historical reconstruction errors in each time interval where incremental concept drift has been confirmed to have occurred. In this way, it can accurately determine whether the data distribution within the target time interval has changed significantly, providing an important reference for further drift confirmation and model adjustment.

[0116] Step 206 , verify the significance difference between the target reconstruction error and each historical reconstruction error, and when the significance difference is characterized by distribution mismatch, update the threshold according to the target reconstruction error and the corresponding verification window, and update the reference time interval according to the target time interval.

[0117] In some embodiments of the present application, in order to verify whether there is a significant difference between the target reconstruction error and each historical reconstruction error, so as to record different drift situations to further confirm the change in data distribution, the significant difference between the target reconstruction error and each historical reconstruction error will be verified, and when the significant difference is characterized as a distribution mismatch, the threshold is updated according to the target reconstruction error and the corresponding verification window, and the reference time interval is updated according to the target time interval. A significant difference means that the difference between the two sets of data is statistically significant, indicating that they come from different distributions. When the target reconstruction error does not match all historical reconstruction errors, it means that the target time interval is different from all previous incremental concepts. At this time, the incremental concept has drifted, so the reference time interval can be updated according to the target time interval. In this way, changes in data distribution can be accurately identified, ensuring that subsequent incremental concept drift detection is based on the latest incremental concept drift to ensure the accuracy of subsequent detection.

[0118] In a specific example, the changes in customer trading behavior over a certain period of time have been evaluated, and it is determined that there is a concept drift. Then the system can further calculate the target reconstruction error and multiple historical reconstruction errors for the target time interval. For example, through statistical test methods (such as t-tests), verify the significant difference between the target reconstruction error and each historical reconstruction error. If it is found that the significant differences are characterized by distribution mismatch, the system updates the reference time interval according to the target time interval. Through such operations, it is possible to accurately determine whether the customer's subsequent trading behavior has changed significantly relative to the current trading behavior, and adjust the model and strategy in a timely manner to improve the effectiveness and accuracy of risk management.

[0119] As a specific implementation method, after verifying that drift occurs, all historical records between the last drift and the current drift are divided into a group. A variational autoencoder (VAE) is built based on the group of records, and the reconstruction error of the group of data is calculated and recorded as the feature representation of the group.

[0120] VAE consists of an encoder and a decoder In a specific implementation, a long short-term memory network (LSTM) can be used to construct a VAE to capture the time information in the streaming data: For a sequence containing S elements , VAE reconstructs it as follows:

[0121] Train the VAE so that the reconstruction error Minimize, by learning the parameters of the hidden states of the encoding and decoding parts of the VAE to minimize the reconstruction error of the training set. The number of data in the history record is recorded as , after the model training is completed, the reconstruction error of each record is calculated, and these reconstruction errors are combined into the historical reconstruction error:

[0122] After that, the reconstruction error is calculated for the data in the reference window, and the reference window size is recorded as , this set of reconstruction errors is recorded as:

[0123] For these two sets of reconstruction errors, the Kolmogorov-Smirnov Test (KS-Test) is performed as a non-parametric and distribution-free statistical test. Each reconstruction error dimension is analyzed to check whether there is a significant difference between the historical record error and the reference window reconstruction error. Hypothesis , They are respectively and The two cumulative distribution functions estimated empirically are , then reject the null hypothesis, as shown in Formula 16.

[0124]

[0125] in, and Represented and The corresponding number of data, is the significance level, If the null hypothesis is rejected in any dimension, it is considered that the data distribution in the reference window and this set of historical records is inconsistent. Otherwise, it is considered that the two are distributed consistently, and the set of historical records is re-enabled.

[0126] Under the method disclosed in the embodiments of the present application, Figure 3 As shown, a detection model for incremental concept drift in privacy-preserving financial data is designed according to the method provided in an embodiment of the present application. The drift is determined through a drift detection layer and a drift verification layer, and the subsequent detection process is adjusted through a method adaptation layer.

[0127] based on Figure 3 The detection model, Figure 4 This is the specific data flow process in the drift detection layer: A1: Selection of target window: Ensure that subsequent probability distribution calculations are based on the latest data window.

[0128] A2: Probability distribution calculation: In order to obtain the statistical characteristics of the data, probability distribution calculation is performed to provide a basis for subsequent distribution difference calculation.

[0129] A3: Distribution difference calculation: By comparing the distribution characteristics of different time periods, the distribution difference is calculated to identify potential concept drift.

[0130] A4: Concept drift detection: Detect concept drift based on distribution differences to ensure that the model can adapt to data changes in a timely manner.

[0131] A5: Weighted average standard deviation calculation: Calculate the weighted average standard deviation through historical data to evaluate the stability of data distribution and provide a basis for subsequent drift verification and adjustment.

[0132] Finally, based on the detection results, it is decided whether to perform drift verification. If no drift is detected, the drift is ignored and the data stream continues to be monitored.

[0133] based on Figure 3 The detection model, Figure 5 This is the specific data flow process in the drift verification layer: B1: Calculate the validation window size: In order to determine the appropriate size of the time window used for validation so that sufficient data volume can be considered during the calculation process, ensure that the selected validation window can provide enough data volume for reliable validation analysis.

[0134] B2: Verify statistical consistency through t-test: Use statistical methods to perform consistency test on the data within the verification window to determine whether the change in data distribution is significant, providing a basis for drift verification.

[0135] B3: Verify processing results: Verify the results of the t-test to confirm the accuracy and validity of the statistical analysis.

[0136] B4: Drift result verification: Perform a final verification on the detected concept drift to confirm whether it actually exists.

[0137] Based on the results of the verification process, the occurrence of drift is finally confirmed, and the verification results are recorded to improve the accuracy of the detection model.

[0138] based on Figure 3 The detection model, Figure 6 This is the specific data flow process in the method adaptation layer: C1: Determine the stable position of concept drift: After the drift verification layer determines the authenticity of concept drift, locate the stable position of data distribution for further adjustment and optimization to improve the reliability of subsequent drift detection and adaptability.

[0139] C2: Evaluate stationary distribution: Evaluate whether the current data distribution is stationary to determine whether the detection window or other parameters need to be adjusted to ensure the accuracy of the model in subsequent use.

[0140] C3: Matching historical distribution: Compare the current data distribution with the historical data distribution to confirm their consistency and changes, and improve the accuracy of subsequent drift detection.

[0141] As a verification of the technical effect of the embodiment of the present application, the verification results disclosed in the embodiment of the present application under the two conditions of sudden drift and incremental drift are compared with several existing technical solutions, such as Figure 7 -a, sudden drift is a concept drift that occurs in an instant, such as Figure 7 -b, incremental drift is the concept drift that continues to occur over a period of time.

[0142] The detection methods used for comparison include the unsupervised concept drift detection method Discriminative Drift Detector (D3), Kolmogorov-Smirnov Test (IKS), Hellinger Distance Drift Detection Method (HDDDM) and Online Concept Drift Detection (OCDD). The German language dataset (german) and credit card dataset (creditcard) are used for experiments in the comparison process. Both are publicly available financial datasets. Privacy protection is added to the data through the Laplace method. At the same time, 5 drifts are inserted into the data and sudden drift and incremental drift are tested respectively. The test is repeated 20 times for each case and the average value is taken: Table 1 Comparison of experimental results among various methods on the dataset

[0143] Table 2 Comparison of experimental results among various methods on the creditcard dataset

[0144] It can be seen from Tables 1 and 2 that the verification results of the method of the present application for sudden drift are stronger than those for incremental drift. Under the same type of drift conditions, the drift detection accuracy of the method of the present application is also significantly higher than that of other methods, and the false alarm rate is lower than that of other methods.

[0145] Furthermore, Table 3 verifies that the method disclosed in this application helps the model adapt to drift. The test is performed on the payment dataset. The evaluation indicator here only considers the accuracy of the model. The model is retrained when drift is detected. Different from other methods, since only the occurrence of drift can be obtained, retraining is performed at the location where the drift is detected, while the method in this paper can be retrained at the location of the new stable distribution.

[0146] Table 3 Comparison of experimental results among various methods on the payment dataset

[0147] From the results in Table 3, we can see that the proposed method is also better than other methods in helping the model adapt to drift. In contrast, other methods have a high false alarm rate, which will cause the model to be adjusted frequently and increase additional overhead. This not only affects the stability of the model, but also increases the consumption of computing resources. This also reflects that the proposed method has significant advantages in reducing the number of invalid adjustments and reducing overhead, thereby more effectively supporting the model to adapt to drift.

[0148] In summary, in the embodiments of the present application, by delineating the reference distribution subsequence and the target distribution subsequence, the data distribution difference between the two is determined, and the time interval is delineated to ensure that the analysis can be focused on important time nodes, accurately detect data distribution changes, and timely discover potential concept drifts; and then, when the data distribution difference is characterized as a mismatch, multiple verification subsequences are determined from the distribution sequence, and the changes in data distribution are verified from multiple angles to further improve the accuracy and reliability of detection, avoid the deviation caused by a single verification, improve the credibility of drift detection, and ensure the accuracy of financial data analysis; finally, according to the statistical matching between the target distribution subsequence and the verification subsequence, the incremental concept drift of the distribution sequence is determined, and matching analysis is performed by using statistical methods, fully considering the data distribution characteristics, and efficiently identifying the incremental concept drift. Therefore, based on the method of the embodiment of the present application, by introducing the division of multiple subsequences and statistical matching analysis, the false positive problems caused by unavailable or delayed arrival of label data and noise data in the prior art are solved. This detection method can accurately detect incremental concept drift in financial data while ensuring privacy protection, effectively avoiding missed detection and misjudgment, improving the performance of economic models and the accuracy of decision-making. It also solves the problem of inaccurate analysis results when confirming incremental drift of financial data.

[0149] refer to Figure 8 , which shows a detection device 30 for incremental concept drift in privacy-preserving financial data provided by an embodiment of the present application, comprising: The sequence acquisition module 301 is used to acquire the distribution sequence of the financial data to be detected; the distribution sequence includes multiple distribution data of the financial data; each distribution data has a corresponding sampling time; The drift detection module 302 is used to determine the data distribution difference between the reference distribution subsequence and the target distribution subsequence in response to the demarcation of the reference distribution subsequence and the target distribution subsequence in the distribution sequence; the end time of the reference time interval corresponding to the reference distribution subsequence is earlier than the start time of the target time interval corresponding to the target distribution subsequence; The drift verification module 303 is used to determine multiple verification subsequences from the distribution sequence when the data distribution difference is characterized as a mismatch, and determine the incremental concept drift of the distribution sequence based on the statistical matching between the target distribution subsequence and the verification subsequence; the time length of the generated time interval corresponding to the verification subsequence is greater than the time length of the target time interval.

[0150] Optionally, the drift detection module 302 includes: The window division submodule is used to divide a plurality of continuous detection time windows from the sampling moments corresponding to the distribution data according to a preset sampling window threshold; each detection time window includes a plurality of sampling moments; different detection time windows include different sampling moments; a window selection submodule, for determining, in response to the selection of a reference detection time window and a target detection time window in a plurality of detection time windows, distribution data corresponding to a plurality of sampling moments included in the reference detection time window as distribution data in a reference distribution subsequence, and distribution data corresponding to a plurality of sampling moments included in the target detection time window as distribution data in a target distribution subsequence; A distribution generation submodule, used to determine a reference distribution feature of a reference distribution subsequence according to a plurality of distribution data included in the reference distribution subsequence, and to determine a target distribution feature of a target distribution subsequence according to a plurality of distribution data included in the target distribution subsequence; The difference analysis submodule is used to determine the data distribution difference between the reference distribution subsequence and the target distribution subsequence according to the reference distribution characteristics and the target distribution characteristics.

[0151] Optionally, the distribution generation submodule includes: A dimension reduction unit is used to perform dimension reduction weighting on each distribution data contained in the reference distribution subsequence and the target distribution subsequence to obtain a reference dimension reduction subsequence of the reference distribution subsequence and a target dimension reduction subsequence of the target distribution subsequence; the reference dimension reduction subsequence contains the dimension reduction data corresponding to each distribution data in the reference distribution subsequence, and the target dimension reduction subsequence contains the dimension reduction data corresponding to each distribution data in the target distribution subsequence; A merging unit is used to merge the dimensionality reduction data in the reference dimensionality reduction subsequence using a weighted product model, so as to use the obtained probability distribution of the reference dimensionality reduction subsequence as a reference distribution feature, and to merge the dimensionality reduction data in the target dimensionality reduction subsequence using a weighted product model, so as to use the obtained probability distribution of the target dimensionality reduction subsequence as a target distribution feature.

[0152] Optionally, the difference analysis submodule includes: A confidence range unit, used to determine a reference confidence radius of a reference distribution subsequence according to the reference distribution characteristics, and to determine a target confidence radius of a target distribution subsequence according to the target distribution characteristics; The confirmation unit is used to determine the distribution difference as a match when the target confidence radius is less than or equal to the reference confidence radius, and to determine the distribution difference as a mismatch when the target confidence radius is greater than the reference confidence radius.

[0153] Optionally, the target distribution subsequence has a corresponding target distribution feature, and the drift verification module 303 includes: The verification window submodule is used to divide a plurality of continuous verification time windows from the sampling moments corresponding to the distribution data according to a preset verification window threshold when the data distribution difference is characterized as a mismatch; each verification time window includes a plurality of sampling moments; different verification time windows include different sampling moments; A verification feature submodule, used to determine the verification distribution feature corresponding to the verification time window according to the distribution data corresponding to the multiple sampling moments contained in each verification time window; A verification indicator submodule, for determining a verification extreme probability value for each verification distribution feature and a target extreme probability value for the target distribution feature; The verification submodule is used to determine the target time interval corresponding to the target distribution subsequence as the time interval for the occurrence of incremental concept drift when the target extreme probability value is greater than a preset number of verification extreme probability values.

[0154] Optionally, when the target time interval is determined as a time interval at which incremental concept drift occurs, the apparatus 30 further includes: The verification adjustment module is used to determine the updated value of the verification window threshold according to the adjustment of the verification window threshold; the updated verification window threshold is used to determine the time interval length of the verification window as the minimum value of the time interval length in which the incremental concept drift occurs.

[0155] Optionally, the verification adjustment module includes: A threshold adjustment submodule, used for determining the variance of the contour coefficient of the verification window threshold being adjusted according to the adjustment of the verification window threshold; The threshold fixing submodule is used to stop adjusting the verification window threshold when the variance is less than or equal to the preset stationary distribution coefficient, and determine the verification window threshold determined when stopping as the update value of the verification window threshold.

[0156] Optionally, when the target time interval is determined as a time interval at which incremental concept drift occurs, the apparatus 30 further includes: A historical comparison module is used to determine a target reconstruction error for a target time interval and obtain a plurality of historical reconstruction errors; the historical reconstruction error is a reconstruction error determined based on each time interval that is confirmed to have incremental concept drift before the target time interval; the target reconstruction error has a corresponding verification window update threshold; each historical reconstruction error has a corresponding verification window threshold; The history adding module is used to verify the significant difference between the target reconstruction error and each historical reconstruction error, and when the significant difference is characterized by distribution mismatch, the historical reconstruction error and the verification window threshold are updated according to the target reconstruction error and the corresponding verification window update threshold.

[0157] In summary, in the embodiments of the present application, by delineating the reference distribution subsequence and the target distribution subsequence, the data distribution difference between the two is determined, and the time interval is delineated to ensure that the analysis can be focused on important time nodes, accurately detect data distribution changes, and timely discover potential concept drifts; and then, when the data distribution difference is characterized as a mismatch, multiple verification subsequences are determined from the distribution sequence, and the changes in data distribution are verified from multiple angles to further improve the accuracy and reliability of detection, avoid the deviation caused by a single verification, improve the credibility of drift detection, and ensure the accuracy of financial data analysis; finally, according to the statistical matching between the target distribution subsequence and the verification subsequence, the incremental concept drift of the distribution sequence is determined, and matching analysis is performed by using statistical methods, fully considering the data distribution characteristics, and efficiently identifying the incremental concept drift. Therefore, based on the method of the embodiment of the present application, by introducing the division of multiple subsequences and statistical matching analysis, the false positive problems caused by unavailable or delayed arrival of label data and noise data in the prior art are solved. This detection method can accurately detect incremental concept drift in financial data while ensuring privacy protection, effectively avoiding missed detection and misjudgment, improving the performance of economic models and the accuracy of decision-making. It also solves the problem of inaccurate analysis results when confirming incremental drift of financial data.

[0158] Reference Fig. 9, the electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0159] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0160] The memory 504 is used to store various types of data to support the operation of the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, multimedia, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0161] The power supply component 506 provides power to the various components of the electronic device 500. The power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0162] The multimedia component 508 includes an interface that provides an output interface between the electronic device 500 and the user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0163] The audio component 510 is used to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the electronic device 500 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0164] The input / output I / O interface 512 provides an interface between the processing component 502 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0165] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500, and the sensor assembly 514 can also detect the position change of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0166] The communication component 516 is used to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0167] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement the methods provided in the embodiments of the present application.

[0168] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by a processor 520 of an electronic device 500 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0169] Fig.10 FIG. 6 is a block diagram of an electronic device 600 according to another embodiment of the present invention. For example, the electronic device 600 may be provided as a server.

[0170] Reference Fig.10 , the electronic device 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632, for storing instructions that can be executed by the processing component 622, such as an application. The application stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute instructions to perform the method provided in the embodiment of the present application.

[0171] The electronic device 600 may also include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0172] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0173] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for detecting incremental concept drift in privacy-preserving financial data, characterized in that: include: Acquire a distribution sequence of financial data to be detected; the distribution sequence includes a plurality of distribution data of the financial data; each of the distribution data has a corresponding sampling time; In response to the demarcation of a reference distribution subsequence and a target distribution subsequence in the distribution sequence, determining a data distribution difference between the reference distribution subsequence and the target distribution subsequence; an end time of a reference time interval corresponding to the reference distribution subsequence is earlier than a start time of a target time interval corresponding to the target distribution subsequence; When the data distribution difference is characterized as a mismatch, multiple verification subsequences are determined from the distribution sequence, and the incremental concept drift of the distribution sequence is determined based on the statistical matching between the target distribution subsequence and the verification subsequence; the time length of the generated time interval corresponding to the verification subsequence is greater than the time length of the target time interval.

2. The method according to claim 1, characterized in that The step of determining the data distribution difference between the reference distribution subsequence and the target distribution subsequence in response to the demarcation of the reference distribution subsequence and the target distribution subsequence in the distribution sequence includes: According to a preset sampling window threshold, a plurality of continuous detection time windows are divided from the sampling moments corresponding to the distribution data; each of the detection time windows includes a plurality of sampling moments; different detection time windows include different sampling moments; In response to the selection of a reference detection time window and a target detection time window among the plurality of detection time windows, the distribution data corresponding to the plurality of sampling moments included in the reference detection time window are determined as the distribution data in the reference distribution subsequence, and the distribution data corresponding to the plurality of sampling moments included in the target detection time window are determined as the distribution data in the target distribution subsequence; Determining a reference distribution feature of the reference distribution subsequence according to a plurality of distribution data included in the reference distribution subsequence, and determining a target distribution feature of the target distribution subsequence according to a plurality of distribution data included in the target distribution subsequence; According to the reference distribution feature and the target distribution feature, a data distribution difference between the reference distribution subsequence and the target distribution subsequence is determined.

3. The method according to claim 2, characterized in that The step of determining a reference distribution feature of the reference distribution subsequence according to a plurality of distribution data included in the reference distribution subsequence, and determining a target distribution feature of the target distribution subsequence according to a plurality of distribution data included in the target distribution subsequence, includes: Performing dimension reduction weighting on each distribution data contained in the reference distribution subsequence and the target distribution subsequence to obtain a reference dimension reduction subsequence of the reference distribution subsequence and a target dimension reduction subsequence of the target distribution subsequence; the reference dimension reduction subsequence contains dimension reduction data corresponding to each distribution data in the reference distribution subsequence, and the target dimension reduction subsequence contains dimension reduction data corresponding to each distribution data in the target distribution subsequence; The reduced dimension data in the reference reduced dimension subsequence is merged using a weighted product model, so that the obtained probability distribution of the reference reduced dimension subsequence is used as the reference distribution feature, and the reduced dimension data in the target reduced dimension subsequence is merged using a weighted product model, so that the obtained probability distribution of the target reduced dimension subsequence is used as the target distribution feature.

4. The method according to claim 2, characterized in that The determining, according to the reference distribution feature and the target distribution feature, a data distribution difference between the reference distribution subsequence and the target distribution subsequence comprises: Determining a reference confidence radius of the reference distribution subsequence according to the reference distribution characteristics, and determining a target confidence radius of the target distribution subsequence according to the target distribution characteristics; When the target confidence radius is less than or equal to the reference confidence radius, the distribution difference is determined as a match; and when the target confidence radius is greater than the reference confidence radius, the distribution difference is determined as a mismatch.

5. The method according to claim 1, characterized in that The target distribution subsequence has a corresponding target distribution feature, and when the data distribution difference is characterized as mismatch, a plurality of verification subsequences are determined from the distribution sequence, and according to the statistical matching between the target distribution subsequence and the verification subsequence, the incremental concept drift of the distribution sequence is determined, including: In the case where the data distribution difference is characterized as a mismatch, a plurality of continuous verification time windows are divided from the sampling moments corresponding to the distribution data according to a preset verification window threshold; each of the verification time windows includes a plurality of sampling moments; and different verification time windows include different sampling moments; Determine the verification distribution feature corresponding to the verification time window according to the distribution data corresponding to the multiple sampling moments contained in each of the verification time windows; Determining a validation extreme probability value for each of the validation distribution features and a target extreme probability value for the target distribution feature; In a case where the target extreme probability value is greater than a preset number of the verification extreme probability values, a target time interval corresponding to the target distribution subsequence is determined as a time interval in which incremental concept drift occurs.

6. The method according to claim 1, characterized in that In the case where the target time interval is determined to be a time interval at which incremental concept drift occurs, the method further includes: According to the adjustment of the verification window threshold, an updated value of the verification window threshold is determined; the updated verification window threshold is used to determine the time interval length of the verification window as the minimum value among the time interval lengths in which incremental concept drift occurs.

7. The method according to claim 1, characterized in that The target time interval is a time interval obtained by delaying the reference time interval by a target number of window times. When the target time interval is determined as a time interval at which incremental concept drift occurs, the method further includes: Determine a target reconstruction error for the target time interval, and obtain a plurality of historical reconstruction errors; the historical reconstruction error is a reconstruction error determined according to each time interval confirmed to have incremental concept drift before the target time interval; the target reconstruction error has a corresponding verification window update threshold; each of the historical reconstruction errors has a corresponding verification window threshold; A significant difference between the target reconstruction error and each of the historical reconstruction errors is verified, and when the significant difference is characterized by a distribution mismatch, a threshold is updated according to the target reconstruction error and the corresponding verification window, and the reference time interval is updated according to the target time interval.

8. A device for detecting incremental concept drift in privacy-preserving financial data, characterized in that: include: A sequence acquisition module, used to acquire a distribution sequence of financial data to be detected; the distribution sequence includes a plurality of distribution data of the financial data; each of the distribution data has a corresponding sampling time; a drift detection module, configured to determine the data distribution difference between the reference distribution subsequence and the target distribution subsequence in response to the demarcation of the reference distribution subsequence and the target distribution subsequence in the distribution sequence; the end time of the reference time interval corresponding to the reference distribution subsequence is earlier than the start time of the target time interval corresponding to the target distribution subsequence; A drift verification module is used to determine multiple verification subsequences from the distribution sequence when the data distribution difference is characterized as a mismatch, and determine the incremental concept drift of the distribution sequence based on the statistical matching between the target distribution subsequence and the verification subsequence; the time length of the generated time interval corresponding to the verification subsequence is greater than the time length of the target time interval.

9. An electronic device, characterized in that: include: A processor, a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 7.

Images