A method for detecting incremental concept drift in privacy-preserving financial data

Abstract

This application discloses a method, apparatus, device, and storage medium for detecting incremental concept drift in privacy-preserving financial data, belonging to the field of data analysis. The method includes: acquiring the distribution sequence of the financial data to be detected; determining the data distribution differences between the reference and target distribution subsequences in response to the delineation of the reference and target distribution subsequences within the distribution sequence; and, if the data distribution differences are characterized as mismatch, determining multiple verification subsequences from the distribution sequence, and determining the incremental concept drift of the distribution sequence based on the statistical matching between the target and verification subsequences. This application can accurately detect incremental concept drift in financial data while ensuring privacy protection, effectively avoiding missed detections and false positives, and solving the problem of inaccurate analysis results when confirming incremental drift in financial data.

Description

Technical Field

[0001] This application belongs to the field of data analysis, specifically relating to a method, apparatus, device, and storage medium for detecting incremental concept drift in privacy-preserving financial data. Background Technology

[0002] As markets fluctuate and the economic environment changes, the distribution of financial data and user behavior patterns shift, leading to changes in the probability distribution of the data over time—a phenomenon known as concept drift. This can affect the performance of economic models and the accuracy of decision-making. Therefore, timely detection and response to concept drift in financial data are crucial for ensuring the security and compliance of financial institutions' operations.

[0003] Existing concept drift detection methods mainly rely on data distribution-based methods, utilizing techniques such as clustering and distribution metrics, and employing a sliding window approach to detect changes in data distribution.

[0004] However, detection methods based on tag data may miss detections due to ineffective verification when tag data is unavailable or arrives late. Furthermore, financial data often contains noise for privacy protection, leading to false positives and subsequent misjudgment drift. Summary of the Invention

[0005] This application aims to provide a method, apparatus, device, and storage medium for detecting incremental concept drift in privacy-preserving financial data, at least to solve the problem of inaccurate analysis results when confirming incremental drift in financial data.

[0006] Firstly, embodiments of this application disclose a method for detecting incremental concept drift in privacy-preserving financial data, including:

[0007] Obtain the distribution sequence of the financial data to be detected; the distribution sequence contains multiple distribution data of the financial data; each distribution data has a corresponding sampling time;

[0008] In response to the delineation of a reference distribution subsequence and a target distribution subsequence in the distribution sequence, the data distribution differences between the reference distribution subsequence and the target distribution subsequence are determined; the end time of the reference time interval corresponding to the reference distribution subsequence is earlier than the start time of the target time interval corresponding to the target distribution subsequence.

[0009] When the data distribution difference is characterized as a mismatch, multiple validation subsequences are determined from the distribution sequence, and the incremental concept drift of the distribution sequence is determined based on the statistical matching between the target distribution subsequence and the validation subsequence; the generation time interval corresponding to the validation subsequence is longer than the target time interval.

[0010] Secondly, embodiments of this application also disclose a detection device for incremental concept drift in privacy-preserving financial data, comprising:

[0011] A sequence acquisition module is used to acquire the distribution sequence of the financial data to be detected; the distribution sequence contains multiple distribution data of the financial data; each distribution data has a corresponding sampling time.

[0012] A drift detection module is used to determine the data distribution difference between the reference distribution subsequence and the target distribution subsequence in response to the delineation of the reference distribution subsequence and the target distribution subsequence in the distribution sequence; the end time of the reference time interval corresponding to the reference distribution subsequence is earlier than the start time of the target time interval corresponding to the target distribution subsequence.

[0013] A drift verification module is used to determine multiple verification subsequences from the distribution sequence when the data distribution difference is characterized as a mismatch, and to determine the incremental concept drift of the distribution sequence based on the statistical matching between the target distribution subsequence and the verification subsequence; the generation time interval corresponding to the verification subsequence is longer than the target time interval.

[0014] Thirdly, embodiments of this application also disclose an electronic device, including a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0015] Fourthly, embodiments of this application also disclose a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0016] In summary, in this embodiment, by defining a reference distribution subsequence and a target distribution subsequence, the data distribution differences between the two are determined. By defining time intervals, the analysis is ensured to focus on key time points, accurately detecting data distribution changes and promptly identifying potential concept drift. Furthermore, when the data distribution differences are characterized as mismatches, multiple validation subsequences are determined from the distribution sequence. By verifying data distribution changes from multiple perspectives, the accuracy and reliability of detection are further improved, avoiding bias caused by single validation, increasing the credibility of drift detection, and ensuring the accuracy of financial data analysis. Finally, based on the statistical matching between the target distribution subsequence and the validation subsequences, the incremental concept drift of the distribution sequence is determined. Statistical matching analysis is used to fully consider data distribution characteristics and efficiently identify incremental concept drift. Therefore, the method based on this embodiment, by introducing the division of multiple subsequences and statistical matching analysis, solves the problems of unavailable or delayed label data and false positives caused by noisy data in the prior art. This detection method can accurately detect incremental concept drift in financial data while ensuring privacy protection, effectively avoiding missed detections and false positives, and improving the performance of economic models and the accuracy of decision-making. It solves the problem of inaccurate analysis results when confirming incremental drift in financial data. Attached Figure Description

[0017] In the attached diagram:

[0018] Figure 1 This is a flowchart illustrating the steps of a method for detecting incremental concept drift in privacy-preserving financial data, as provided in an embodiment of this application.

[0019] Figure 2 This is a flowchart of another method for detecting incremental concept drift in privacy-preserving financial data provided in this application embodiment;

[0020] Figure 3 This is a detection model provided according to an embodiment of this application;

[0021] Figure 4 This is a drift detection process of the detection model provided in the embodiments of this application;

[0022] Figure 5 This is a drift verification process for the detection model provided in the embodiments of this application;

[0023] Figure 6 This is an adaptive process of a detection model provided in the embodiments of this application;

[0024] Figure 7 This is a time-series diagram illustrating the drift between the two concepts;

[0025] Figure 8 This is a block diagram of a detection device for incremental concept drift in privacy-preserving financial data, provided in an embodiment of this application.

[0026] Figure 9 This is a block diagram of an electronic device provided in one embodiment of this application;

[0027] Figure 10 This is a block diagram of an electronic device according to another embodiment of the present application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0030] Through verification, financial concepts can be represented by the probability distribution of data, that is... Therefore, the concept of a specific time t is defined as Therefore, concept drift is the change in distribution between time t and time m, that is... Therefore, to detect concept drift, a distribution metric D(t,m) can be used to represent the difference in distribution between two time points. If the current D(t,m) is not within the expected distribution range, it indicates that concept drift has occurred: assuming the concept at time t1 is... Then this concept becomes at time t2 If the probability distribution of the data between t1 and t2 changes significantly, the value of D(t1,t2) will reveal a significant change in distribution, that is, the value of D(t1,t2) exceeds the expected distribution range, and a conceptual drift has occurred between t1 and t2.

[0031] Furthermore, to implement an unsupervised concept drift detection method, it is necessary to measure the difference in the data from a distributional perspective, that is, to select an appropriate distributional difference measure to calculate D(t,m). Considering the factors of complementarity, sensitivity, and accuracy, the Jensen-Shannon Divergence (JS) is chosen as the measure from the perspective of relative entropy:

[0032]

[0033] Here, P(i) and Q(i) are both probability distributions of the data.

[0034] From the perspective of geometric difference, the Hellinger distance is used as a metric:

[0035]

[0036] in, It is the average probability distribution of P and Q. It is the KL divergence:

[0037]

[0038] Because both metrics are non-negative and symmetrical, this ensures the consistency and reliability of the evaluation. Using both metrics simultaneously captures differences in data distribution more comprehensively than using either metric alone. This combined application is more accurate and effective in detecting concept drift in data streams. A detailed analysis follows:

[0039] At the level of complementarity, a comprehensive description of data distribution includes not only traditional statistical indicators such as mean, variance, and standard deviation, but also more complex features such as entropy and shape. Entropy reflects the uncertainty of the data, while shape involves geometric characteristics such as symmetry, skewness, and kurtosis. To fully quantify these differences, JS divergence and Hellinger distance offer complementary perspectives. JS divergence focuses on assessing differences in the entropy and information content of the distribution, while Hellinger distance focuses on capturing differences in these geometric characteristics. By combining these two indicators, variations in data distribution can be more accurately depicted and analyzed.

[0040] At the sensitivity level, for data changes that persist over a period of time, such as incremental drift, the data distribution gradually evolves throughout the process. The KL divergence component of the JS divergence is particularly sensitive to the central region of the distribution; even slight differences in the mean or variance of P and Q will be amplified in the calculation by the KL divergence. On the other hand, the Hellinger distance measures differences by comparing the square roots of the probability distributions and is extremely sensitive to the overall shape of the distribution (e.g., symmetry, peaks, or flatness). Using a combination of these two metrics allows for the earlier detection of early signs of concept drift, enabling its detection once it occurs.

[0041] In terms of accuracy, noise in the data stream can lead to false detections of concept drift. JS divergence's high sensitivity to the distribution's central region enables it to accurately capture subtle but real drifts, while reducing the risk of false alarms caused by noise by minimizing responses to meaningless fluctuations. Meanwhile, Hellinger distance's sensitivity to distribution shape allows it to identify statistically significant changes, even those that are not visually apparent. This prevents false alarms caused by chance or minor fluctuations. By combining these two methods, concept drift in the data stream can be detected more accurately, effectively reducing the false alarm rate while ensuring that genuine drifts are not overlooked.

[0042] Based on the above analysis Figure 1 This application provides a method for detecting incremental concept drift in privacy-preserving financial data.

[0043] The method may include the following steps:

[0044] Step 101: Obtain the distribution sequence of the financial data to be tested.

[0045] The distribution sequence contains multiple distribution data for financial data; each distribution data has a corresponding sampling time.

[0046] In some embodiments of this application, it is first necessary to obtain a distribution sequence of financial data. This distribution sequence contains distribution data at multiple time points, reflecting how the data changes over time. Specifically, the distribution sequence includes multiple distribution data points for the financial data, each with a corresponding sampling time. The sampling time refers to the specific point in time when each distribution data point is recorded. This provides foundational data support for subsequent distribution analysis and drift detection.

[0047] In a specific example, suppose we need to detect changes in the distribution of transaction data over a period of time. We can record daily transaction data to generate a distribution sequence. The system automatically collects the daily transaction data and arranges it chronologically, forming a distribution sequence containing data from multiple time points. This process allows us to obtain the distribution changes of transaction data over a period, providing the necessary data foundation for subsequent concept drift detection.

[0048] Step 102: In response to the delineation of the reference distribution subsequence and the target distribution subsequence in the distribution sequence, determine the data distribution differences between the reference distribution subsequence and the target distribution subsequence.

[0049] The end time of the reference time interval corresponding to the reference distribution subsequence is earlier than the start time of the target time interval corresponding to the target distribution subsequence.

[0050] In some embodiments of this application, to determine the data distribution differences between a reference distribution subsequence and a target distribution subsequence and to ensure accurate detection of data changes, the data distribution differences between the reference distribution subsequence and the target distribution subsequence are determined in response to the delineation of the reference distribution subsequence and the target distribution subsequence in the distribution sequence. The end time of the reference time interval corresponding to the reference distribution subsequence is earlier than the start time of the target time interval corresponding to the target distribution subsequence. The data distribution difference refers to whether the data distribution between the two time intervals is consistent or significantly different. This allows for the timely detection of potential concept drift and ensures the accuracy of data analysis.

[0051] In a specific example, it's necessary to analyze changes in customer transaction behavior over different time periods. Data from the first two months of a year can be selected as a reference distribution subsequence, and data from the last two months as a target distribution subsequence. The system defines the reference and target distribution subsequences based on timestamps and calculates the differences in their data distributions. If a significant difference in the data distributions is found, the system will promptly alert the user to potential concept drift issues. This process allows for timely understanding of changes in customer behavior patterns, providing support for further risk management and decision-making.

[0052] Step 103: When the data distribution difference is characterized as mismatch, multiple validation subsequences are determined from the distribution sequence, and the incremental concept drift of the distribution sequence is determined based on the statistical matching between the target distribution subsequence and the validation subsequence.

[0053] Among them, the generation time interval corresponding to the verification subsequence is longer than the target time interval.

[0054] In some embodiments of this application, to further confirm the authenticity and accuracy of incremental concept drift by using verification subsequences when data distribution differences are characterized as mismatches, multiple verification subsequences are determined from the distribution sequence when the data distribution differences are characterized as mismatches. The incremental concept drift of the distribution sequence is determined based on the statistical matching between the target distribution subsequence and the verification subsequences. The verification subsequences are subsequences extracted from the distribution sequence whose time interval is longer than the target time interval. Through multi-angle verification subsequence analysis, the accuracy and reliability of drift detection can be improved, biases caused by single verifications can be avoided, and the overall confidence of the detection can be enhanced.

[0055] In a specific example, a significant difference in the distribution of transaction data over a certain time period has been identified. After detecting a mismatch between the reference and target distribution subsequences, the system extracts multiple validation subsequences from the distribution sequence. The system calculates the statistical match between each validation subsequence and the target distribution subsequence, ensuring the time interval of the validation subsequence is longer than the target time interval. If the match between these validation subsequences and the target distribution subsequence is low, the system confirms the existence of incremental concept drift. This process allows for more accurate detection and confirmation of changes in customer trading behavior, providing support for further risk management and decision-making.

[0056] In summary, in this embodiment, by defining a reference distribution subsequence and a target distribution subsequence, the data distribution differences between the two are determined. By defining time intervals, the analysis is ensured to focus on key time points, accurately detecting data distribution changes and promptly identifying potential concept drift. Furthermore, when the data distribution differences are characterized as mismatches, multiple validation subsequences are determined from the distribution sequence. By verifying data distribution changes from multiple perspectives, the accuracy and reliability of detection are further improved, avoiding bias caused by single validation, increasing the credibility of drift detection, and ensuring the accuracy of financial data analysis. Finally, based on the statistical matching between the target distribution subsequence and the validation subsequences, the incremental concept drift of the distribution sequence is determined. Statistical matching analysis is used to fully consider data distribution characteristics and efficiently identify incremental concept drift. Therefore, the method based on this embodiment, by introducing the division of multiple subsequences and statistical matching analysis, solves the problems of unavailable or delayed label data and false positives caused by noisy data in the prior art. This detection method can accurately detect incremental concept drift in financial data while ensuring privacy protection, effectively avoiding missed detections and false positives, and improving the performance of economic models and the accuracy of decision-making. It solves the problem of inaccurate analysis results when confirming incremental drift in financial data.

[0057] Figure 2 This is another method for detecting incremental concept drift in privacy-preserving financial data provided in the embodiments of this application.

[0058] The method may include the following steps:

[0059] Step 201: Obtain the distribution sequence of the financial data to be tested.

[0060] The distribution sequence contains multiple distribution data for financial data; each distribution data has a corresponding sampling time.

[0061] The method shown in this step has been explained in step 101 and will not be repeated here.

[0062] Step 202: In response to the delineation of the reference distribution subsequence and the target distribution subsequence in the distribution sequence, determine the data distribution differences between the reference distribution subsequence and the target distribution subsequence.

[0063] The end time of the reference time interval corresponding to the reference distribution subsequence is earlier than the start time of the target time interval corresponding to the target distribution subsequence.

[0064] The method shown in this step has been explained in step 102 and will not be repeated here.

[0065] Optionally, step 202 includes the following sub-steps:

[0066] Sub-step 2021: According to the preset sampling window threshold, divide the sampling time corresponding to the distribution data into multiple consecutive detection time windows.

[0067] Each detection time window contains multiple sampling times; different detection time windows contain different sampling times.

[0068] In some embodiments of this application, to divide the sampling times of the distributed data into multiple consecutive detection time windows for better data analysis and drift detection, multiple consecutive detection time windows are divided from the sampling times corresponding to the distributed data according to a preset sampling window threshold. Each detection time window contains multiple sampling times; different detection time windows contain different sampling times. The sampling window threshold refers to the preset time interval used to divide the time windows. This allows long-term data series to be divided into multiple smaller time windows, facilitating further processing and analysis, and improving the efficiency and accuracy of data processing.

[0069] In a specific example, it's necessary to analyze changes in transaction data over a year. This can be achieved by sampling the transaction data daily and dividing it into multiple consecutive detection time windows based on a preset sampling window threshold. This allows the data to exhibit statistical characteristics over a short period. For instance, the system samples transaction data daily and divides this data into multiple detection time windows on a weekly or monthly basis. This facilitates subsequent analysis and detection, improving data processing efficiency and analytical accuracy.

[0070] Sub-step 2022, in response to the selection of a reference detection time window and a target detection time window among multiple detection time windows, determines the distribution data corresponding to multiple sampling times contained in the reference detection time window as the distribution data in the reference distribution subsequence, and determines the distribution data corresponding to multiple sampling times contained in the target detection time window as the distribution data in the target distribution subsequence.

[0071] In some embodiments of this application, in order to select a reference detection time window and a target detection time window from multiple detection time windows and determine their corresponding distribution data for subsequent distribution feature analysis, in response to the selection of the reference detection time window and the target detection time window, the distribution data corresponding to multiple sampling times contained in the reference detection time window is determined as the distribution data in the reference distribution subsequence, and the distribution data corresponding to multiple sampling times contained in the target detection time window is determined as the distribution data in the target distribution subsequence. A distribution subsequence refers to a portion of data extracted from the entire distribution sequence. This effectively extracts the data within the reference detection time window and the target detection time window, providing a basis for subsequent data distribution difference analysis.

[0072] In a specific example, it's necessary to analyze customer transaction behavior over a year. The system divides the year's transaction data distribution into multiple detection time windows, selecting January as the reference detection time window and August as the target detection time window. The system extracts transaction data corresponding to multiple sampling moments within January, defining them as the distribution data in the reference distribution subsequence, and similarly extracts transaction data corresponding to multiple sampling moments within August, defining them as the distribution data in the target distribution subsequence. Through this operation, the system obtains the distribution data for both the reference and target detection time windows, providing necessary data support for subsequent distribution feature analysis and drift detection.

[0073] Sub-step 2023: Based on the multiple distribution data contained in the reference distribution subsequence, determine the reference distribution characteristics of the reference distribution subsequence, and based on the multiple distribution data contained in the target distribution subsequence, determine the target distribution characteristics of the target distribution subsequence.

[0074] In some embodiments of this application, to clarify the distribution characteristics of the reference distribution subsequence and the target distribution subsequence for subsequent distribution difference analysis, the reference distribution characteristics of the reference distribution subsequence are determined based on multiple distribution data contained in the reference distribution subsequence, and the target distribution characteristics of the target distribution subsequence are determined based on multiple distribution data contained in the target distribution subsequence. Distribution characteristics refer to the statistical properties of data, such as mean, variance, skewness, and kurtosis. In this way, the distribution characteristics of each time window can be accurately extracted, providing specific indicators and references for subsequent distribution difference analysis.

[0075] In a specific example, it's necessary to analyze customer spending habits across different time periods. In this scenario, the system uses data from the first week of a quarter as a reference distribution subsequence and data from the last week as a target distribution subsequence. Based on the spending data from the first and last weeks, the system calculates statistical characteristics such as mean, variance, skewness, and kurtosis. Through this process, the system obtains the specific distribution characteristics of the reference and target distribution subsequences, providing a foundation for subsequent distribution difference analysis and helping financial institutions understand changes in customer spending habits.

[0076] Optionally, sub-step 2023 includes the following sub-steps:

[0077] Sub-step 20231 involves performing dimensionality reduction and weighting on each distribution data contained in the reference distribution subsequence and the target distribution subsequence to obtain the reference dimensionality-reduced subsequence of the reference distribution subsequence and the target dimensionality-reduced subsequence of the target distribution subsequence.

[0078] The reference dimensionality reduction subsequence contains dimensionality reduction data corresponding to each distribution data in the reference distribution subsequence, and the target dimensionality reduction subsequence contains dimensionality reduction data corresponding to each distribution data in the target distribution subsequence.

[0079] In some embodiments of this application, in order to extract and simplify the data features of the reference distribution subsequence and the target distribution subsequence, dimensionality reduction and weighting are performed on each distribution data contained in the reference distribution subsequence and the target distribution subsequence to obtain a reference dimensionality-reduced subsequence of the reference distribution subsequence and a target dimensionality-reduced subsequence of the target distribution subsequence. Dimensionality reduction and weighting refers to reducing the dimensionality of the data and assigning weights to it, thereby reducing data complexity while maintaining its important features. This simplifies the data structure, reduces computational complexity, and improves the efficiency of subsequent analysis.

[0080] In a specific example, it's necessary to analyze customer transaction behavior across different time periods. The scenario involves a system that has already segmented customer transaction data into a reference distribution subsequence and a target distribution subsequence. The system can perform dimensionality reduction and weighting on the transaction data within each subsequence, extracting the reference and target dimensionality-reduced subsequences. Dimensionality reduction and weighting reduces the dimensionality of the data while preserving important features. Through this operation, the system can simplify the data structure and improve the efficiency of subsequent data analysis and comparison.

[0081] As a specific implementation method, the process of dimensionality reduction and weighting requires not only subspace partitioning but also the calculation of weights for each subspace. Considering that the method targets unlabeled data streams, kernel principal component analysis (KPCA) can be used for weight calculation. This is a suitable method for unlabeled environments. Specifically, KPCA is used to calculate eigenvalues, and weights are assigned to each subspace based on these eigenvalues.

[0082]

[0083] in, Let be the weight of the i-th subspace. Let be the eigenvalues ​​corresponding to the subspace, and n be the number of subspaces.

[0084] Sub-step 20232: Use a weighted product model to merge the dimensionality-reduced data in the reference dimensionality-reduced subsequence, so as to use the probability distribution of the obtained reference dimensionality-reduced subsequence as the reference distribution feature, and use a weighted product model to merge the dimensionality-reduced data in the target dimensionality-reduced subsequence, so as to use the probability distribution of the obtained target dimensionality-reduced subsequence as the target distribution feature.

[0085] In some embodiments of this application, to generate reference and target distribution features, a weighted product model is used to merge the dimensionality-reduced data in the reference dimensionality-reduced subsequence, using the probability distribution of the obtained reference dimensionality-reduced subsequence as the reference distribution feature. Similarly, a weighted product model is used to merge the dimensionality-reduced data in the target dimensionality-reduced subsequence, using the probability distribution of the obtained target dimensionality-reduced subsequence as the target distribution feature. The weighted product model is a mathematical method that assigns different weights to different data points for weighted averaging. This generates more accurate and robust distribution features, providing a foundation for subsequent data comparison and analysis.

[0086] In a specific example, it's necessary to analyze the probability distribution of customer transaction behavior across different time periods. The system has already performed dimensionality reduction on the customer transaction data, generating a reference dimensionality-reduced subsequence and a target dimensionality-reduced subsequence. The system can then use a weighted product model to merge these dimensionality-reduced data separately, generating the probability distributions of the reference and target dimensionality-reduced subsequences as their distribution features. Through this operation, the system can accurately generate probability distribution features reflecting customer transaction behavior, providing crucial data support for subsequent distribution comparisons and concept drift detection.

[0087] As a specific implementation method, when merging subspace probability distributions during the generation of distribution features, an intuitive approach is to directly merge them based on weights, i.e., using the Weighted Sum Model (WSM), or to use a variant of the Weighted Product Model (WPM) for distribution merging calculations.

[0088]

[0089] in, For the weighted product result, Let be the weight of the i-th subspace. Let J be the weight of the j-th subspace. Let be the probability distribution on the corresponding subspace.

[0090] Sub-step 2024: Based on the characteristics of the reference distribution and the target distribution, determine the differences in data distribution between the reference distribution subsequence and the target distribution subsequence.

[0091] In some embodiments of this application, to analyze the data distribution differences between the reference distribution subsequence and the target distribution subsequence and determine whether concept drift exists, the data distribution differences between the reference distribution subsequence and the target distribution subsequence are determined based on the characteristics of the reference distribution and the target distribution. The data distribution difference refers to whether there is a significant statistical difference in the distribution characteristics of the two time windows, reflecting changes in the data distribution. This effectively identifies changes in data distribution, promptly detects potential concept drift, and provides important judgment criteria for financial data analysis.

[0092] In a specific example, the goal is to analyze changes in customer transaction behavior within a given quarter. The system has identified the first week of the quarter as the reference detection window and the last week as the target detection window, and has calculated the distribution characteristics for both. Based on these distribution characteristics, the system will use statistical methods to calculate the differences in data distribution between the two windows. Through this process, the system can determine whether a significant difference exists, thereby identifying whether concept drift has occurred.

[0093] Optionally, sub-step 2024 includes the following sub-steps:

[0094] Sub-step 20241: Determine the reference confidence radius of the reference distribution subsequence based on the reference distribution characteristics, and determine the target confidence radius of the target distribution subsequence based on the target distribution characteristics.

[0095] In some embodiments of this application, to quantify and compare the distribution characteristics of a reference distribution subsequence and a target distribution subsequence, a reference confidence radius for the reference distribution subsequence is determined based on the reference distribution characteristics, and a target confidence radius for the target distribution subsequence is determined based on the target distribution characteristics. The confidence radius is the range of data distribution at a certain confidence level, quantifying the degree of dispersion of data points around the mean. This provides a specific and quantifiable indicator for subsequent distribution difference analysis, ensuring the accuracy of the distribution characteristic comparison.

[0096] In a specific example, it's necessary to assess the consistency of transaction data distribution across different time periods. The specific scenario involves the system having already calculated the distribution characteristics of a reference time window and a target time window within a given quarter. Based on the distribution characteristics of the transaction data within the reference time window, the system calculates a reference confidence radius; simultaneously, based on the distribution characteristics of the transaction data within the target time window, it calculates a target confidence radius. The confidence radius provides information about the statistical dispersion of the data. Through this process, the system obtains the confidence radii for the reference and target time windows, providing a reliable numerical basis for subsequent analysis of data distribution differences.

[0097] Sub-step 20242: If the target confidence radius is less than or equal to the reference confidence radius, the distribution difference is determined as a match; if the target confidence radius is greater than the reference confidence radius, the distribution difference is determined as a mismatch.

[0098] In some embodiments of this application, to determine the data distribution differences between the reference and target distribution subsequences based on their confidence radii, the distribution difference is defined as a match if the target confidence radius is less than or equal to the reference confidence radius, and as a mismatch if the target confidence radius is greater than the reference confidence radius. The confidence radius is the discrete range of data around the mean at a certain confidence level, reflecting the consistency and central tendency of the data. This allows for the determination of whether the distribution characteristics of the reference and target distribution subsequences are consistent, thereby improving the accuracy of data distribution difference analysis.

[0099] In a specific example, it's necessary to assess the similarity of customer transaction behavior across different time periods. The specific scenario involves the system having already calculated the confidence radii for a reference time window and a target time window within a given month. The system compares the confidence radius of the reference time window with that of the target time window. If the target confidence radius is less than or equal to the reference confidence radius, the system classifies the distribution discrepancy as a match; if the target confidence radius is greater than the reference confidence radius, the system classifies the distribution discrepancy as a mismatch.

[0100] like Figure 2 As shown, as a specific implementation process, two windows can be maintained during drift detection: one is a reference window, denoted as... A target window that slides backward, denoted as... Let the window size be s, and the sliding step size also be s. Whenever When selected, the probability distribution of the data contained in the two windows is calculated. After obtaining the probability distribution, the difference between the two distributions needs to be calculated using a distribution difference measure. Then, it is compared with the average of historical values ​​to see if it exceeds the range of 3 sigma. To reduce noise interference, the harmonic mean is used instead of the arithmetic mean H.

[0101]

[0102] in, These are sampled values. For simplicity, streaming data is represented as a series of data blocks S, each of size s. In practical applications, if streaming data is input as single data blocks, the detection process can be initiated after accumulating a sufficient number of s data blocks.

[0103] Step 203: When the data distribution difference is characterized as mismatch, multiple validation subsequences are determined from the distribution sequence, and the incremental concept drift of the distribution sequence is determined based on the statistical matching between the target distribution subsequence and the validation subsequence.

[0104] Among them, the generation time interval corresponding to the verification subsequence is longer than the target time interval.

[0105] The method shown in this step has been explained in step 102 and will not be repeated here.

[0106] Optionally, the target distribution subsequence has corresponding target distribution features, and step 203 includes the following sub-steps:

[0107] Sub-step 2031: When the data distribution difference is characterized as mismatch, multiple consecutive verification time windows are divided from the sampling time corresponding to the distribution data according to the preset verification window threshold.

[0108] Each verification time window contains multiple sampling times; different verification time windows contain different sampling times.

[0109] In some embodiments of this application, to divide the sampling times of the distributed data into multiple consecutive verification time windows for further analysis of changes in the data distribution, when the data distribution differences are characterized as mismatches, multiple consecutive verification time windows are divided from the sampling times corresponding to the distributed data according to a preset verification window threshold. Each verification time window contains multiple sampling times; different verification time windows contain different sampling times. The verification window threshold refers to the preset time interval used to divide the time windows. This refines the division of time series data, ensuring that the verification analysis is more targeted and accurate.

[0110] In a specific example, it's necessary to verify whether concept drift has occurred in transaction data within a single month of a given year. When the system detects a mismatch in data distribution, it divides the annual transaction data into multiple consecutive verification time windows based on a preset verification window threshold. The system can divide the annual transaction data into four verification time windows, with each quarter serving as a new time interval. This allows the system to meticulously analyze the transaction data within each verification time window, improving the accuracy and reliability of data analysis and ensuring more accurate verification of concept drift.

[0111] As a specific implementation method, in the process of dividing the verification time window, considering that the duration of drift cannot be known in advance during the detection process, the window size is calculated based on historical information of drift occurrence interval and drift duration. In this paper, the drift occurrence interval is based on the number of window slides, and the drift duration is calculated similarly. The basic concept is as follows: when the drift duration is long, more information is needed to confirm the drift, so the window should be appropriately increased. Similarly, when the drift duration is short, the window should be appropriately decreased to avoid introducing additional delays. As for the drift occurrence interval, when the interval is large, it indicates that the stable period after the drift is relatively long, and the tolerance for delay is relatively high. The window can be appropriately increased to ensure verification accuracy. When the drift interval is short, the window size should be reduced to verify the drift as soon as possible. The specific values ​​can be expressed by the following formula:

[0112]

[0113] Where n is the verification window size, N d N represents the average length of the historical drift. iThis represents the average interval of historical drift. The square root of the product is used here to avoid introducing custom parameters and to balance N. d and N i The effect on n is to prevent the window size from growing too quickly.

[0114] Sub-step 2032: Determine the verification distribution characteristics corresponding to the verification time window based on the distribution data corresponding to the multiple sampling times contained in each verification time window.

[0115] In some embodiments of this application, in order to extract the distribution characteristics of each verification time window and further analyze the changes in data distribution, the verification distribution characteristics corresponding to the verification time window are determined based on the distribution data corresponding to the multiple sampling times included in each verification time window. Distribution characteristics refer to the statistical properties of the data, such as mean, variance, skewness, and kurtosis. In this way, the specific distribution characteristics of each verification time window can be obtained, providing data support for subsequent extreme probability value calculation and drift judgment.

[0116] In a specific example, we need to analyze changes in trading behavior over a single month within a given year. The scenario involves dividing the annual trading data into multiple new verification time windows. The system calculates the distribution characteristics of the trading data from multiple sampling points within each verification time window. This process allows the system to obtain the specific distribution characteristics of each verification time window, providing a foundation for subsequent calculations of extreme probability values ​​and assessments of concept drift.

[0117] As a specific implementation method, in the process of determining the validation distribution characteristics corresponding to the validation time window, a t-test is used to verify whether drift has indeed occurred within the validation window. Specifically, the data distributions of the target window and the validation window are compared to determine whether there is a significant difference between them. The null hypothesis (H0) is that the data in the target window and the validation window come from the same distribution (i.e., no drift), and the alternative hypothesis (H1) is that they come from different distributions (i.e., drift has occurred). Thus, based on the divided subspaces, a t-test is performed separately in each subspace:

[0118]

[0119] in , These are the average values ​​of the data in the target window and the validation window, respectively. and These are the sample variances of the target window and the validation window, respectively. and These represent the number of samples in the target window and the validation window, respectively.

[0120] Sub-step 2033: Determine the validation extreme probability value of each validation distribution feature and the target extreme probability value of the target distribution feature.

[0121] In some embodiments of this application, in order to calculate the extreme probability values ​​of each verification distribution feature and the target distribution feature, and thus assess the changes in data distribution, the verification extreme probability value of each verification distribution feature and the target extreme probability value of the target distribution feature are determined. The extreme probability value, also known as the p-value, refers to the probability of a data feature occurring under extreme conditions in statistical analysis, and is used to measure the extreme cases of data distribution. This clarifies the distribution characteristics of each verification time window and the target time window under extreme conditions, providing important reference data for subsequent concept drift judgment.

[0122] In a specific example, it is necessary to assess extreme changes in trading behavior over different time periods. The system has already divided the past year into verification time windows and calculated the distribution characteristics of each verification and target time window. The system further calculates the extreme probability values ​​for each verification and target distribution characteristic; for example, it calculates the p-value within each time window. Through this process, the system can clearly understand the extreme changes in trading behavior over different time periods, providing accurate data support for subsequent concept drift assessments.

[0123] Sub-step 2034: If the target extreme probability value is greater than the preset number of verification extreme probability values, the target time interval corresponding to the target distribution subsequence is determined as the time interval in which incremental concept drift occurs.

[0124] In some embodiments of this application, to definitively confirm whether incremental concept drift has occurred in the data distribution within a specific time interval, the target time interval corresponding to the target distribution subsequence is determined as the time interval in which incremental concept drift occurred if the target extreme probability value is greater than a preset number of verification extreme probability values. The extreme probability value refers to the probability of the data distribution characteristic occurring under extreme conditions. In this way, it can be clearly determined whether the data distribution has changed significantly within the target time interval, thereby confirming the occurrence of incremental concept drift and improving the accuracy and reliability of the detection.

[0125] In a specific example, it's necessary to confirm whether changes in customer transaction behavior within a given month constitute incremental concept drift. The system calculates the extreme probability value of transaction amount within a target time window of the month and compares it with the extreme probability values ​​of multiple validation time windows. For instance, if the system finds that the extreme probability value of the target time window is greater than the extreme probability values ​​of one-third of the validation time windows, the system determines that target time interval as the time interval where incremental concept drift occurred. In this way, changes in customer transaction behavior can be accurately determined.

[0126] As a specific implementation method, the corresponding p-value is obtained based on the corresponding t-value. After obtaining the corresponding p-value, the p-values ​​are merged according to the weight of the subspace. Considering that the verification window size is n times the target window size, n t-tests will actually be calculated. At this point, the problem is transformed into a multiple verification problem.

[0127] In the case of multiple verification, considering the potential for increased false positive rates (Type I errors) and false negative rates (Type II errors), a method can be adopted... The correction method further processes the results. If the null hypothesis is rejected... If the number of values ​​is greater than or equal to half, it indicates that drift has indeed occurred.

[0128] Step 204: Determine the updated value of the verification window threshold based on the adjustment of the verification window threshold.

[0129] The updated verification window threshold is used to determine the verification window time interval length as the minimum among the time interval lengths at which incremental concept drift occurs.

[0130] In some embodiments of this application, in order to dynamically adjust the verification window threshold to adapt to the duration of concept drift during actual detection, an updated value for the verification window threshold is determined based on the adjustment of the verification window threshold. The updated verification window threshold is used to determine the time interval length of the verification window as the minimum value among the time interval lengths in which incremental concept drift occurs. The time interval length refers to the time range covered by the verification window during drift detection. This improves the accuracy and efficiency of drift detection, ensuring more precise and reliable detection results.

[0131] In a specific example, drift detection is required on continuous transaction data. To ensure that the initially set verification window threshold accurately meets the needs of subsequent detection, it needs to be adjusted. The system will dynamically adjust the verification window threshold based on previous detection results, updating it to cover the minimum value within the incremental concept drift time. Through this operation, the system can flexibly adapt to different detection scenarios, improving the accuracy of drift detection and ensuring that the financial data analysis results are further reliable and effective.

[0132] Optionally, step 204 includes the following sub-steps:

[0133] Sub-step 2041: Based on the adjustment of the verification window threshold, determine the variance of the profile coefficient of the verification window threshold in the adjustment.

[0134] In some embodiments of this application, in order to calculate the variance of the silhouette coefficient during the adjustment process based on the adjustment of the validation window threshold, and to evaluate the stability of the data distribution, the variance of the silhouette coefficient of the validation window threshold during the adjustment is determined based on the adjustment of the validation window threshold. The silhouette coefficient measures the clustering effect, while the variance is used to evaluate the dispersion and stability of the data. This quantifies the distribution stability of the validation window and provides a basis for subsequent updates to the validation window threshold.

[0135] In a specific example, the validation window threshold needs to be dynamically adjusted to adapt to changes in transaction data over different time periods. By adjusting the validation window threshold to scale the time window, the variance of the silhouette coefficients during the adjustment process is determined. Based on the validation window data after each adjustment, the silhouette coefficients are calculated, and the variance of these coefficients is further calculated. In this way, the system can evaluate the stability of the data distribution under different validation window thresholds.

[0136] Sub-step 2042: If the variance is less than or equal to the preset stationary distribution coefficient, stop adjusting the verification window threshold and set the verification window threshold determined at the time of stopping as the updated value of the verification window threshold.

[0137] In some embodiments of this application, to monitor changes in variance during the adjustment of the verification window threshold and determine when to stop the adjustment to ensure the stability of the data distribution, the adjustment of the verification window threshold will be stopped when the variance is less than or equal to a preset stationary distribution coefficient. The verification window threshold determined at the time of stopping will then be used as the updated value for the verification window threshold. The stationary distribution coefficient is a preset standard used to judge the stability of the data distribution. This ensures that the setting of the verification window reflects the stable distribution of the data, improving the accuracy and reliability of the detection results.

[0138] In a specific example, the system can calculate the silhouette coefficient variance of the validation data after each adjustment of the validation window threshold and compare it with a preset stationary distribution coefficient. When the system detects that the silhouette coefficient variance is less than or equal to the preset stationary distribution coefficient, it stops adjusting the validation window threshold and sets the current validation window threshold as the updated value. Through this operation, financial institutions can ensure that the validation window setting accurately reflects the stability of the data distribution, improving the accuracy and reliability of transaction data analysis.

[0139] As a specific implementation method, after a verification drift occurs, the width of the verification window is adjusted to continuously monitor changes in the data distribution until the data reaches a stable state. The position where the drift was detected is denoted as k, where each sub-window of the same size as the reference window is sequentially represented as... The assessment of whether the data distribution is stable is based on a comprehensive analysis of clustering and the silhouette score. The silhouette score measures the effectiveness of clustering, with a value ranging from -1 to 1. A higher value indicates better clustering, meaning stronger cohesion within clusters and greater separability between clusters. When the silhouette score is close to 1, it indicates that the distance between data points within a cluster is small, while the distance between clusters is large, suggesting that the data has reached a relatively stable distribution. Conversely, when the silhouette score is close to 0 or negative, it indicates that the data points are not clearly distributed between clusters, and there may be overlap or ongoing drift.

[0140] ;

[0141] Where s(i) is the silhouette coefficient of a single data point i, a(i) is the average distance from point i to other points within its cluster (intra-cluster distance), and b(i) is the average distance from point i to the nearest other cluster (inter-cluster distance). After the distribution stabilizes, the reference window is updated using the data within the target window.

[0142] In order to correlate the silhouette coefficient with whether the data is stationary, calculations are performed. variance of profile coefficient :

[0143]

[0144] in Indicates the current adaptive verification window The starting position after sliding back i-1 steps, k+i+j represents the starting position of the sub-window in the adaptive verification window. For adaptive verification window The average profile coefficient of each sub-window.

[0145] in, The calculation is as follows:

[0146]

[0147] When obtained Afterwards, Compared with the stationary distribution parameter δ, when the inequality is satisfied At that time, it was believed The data distribution in the middle is unstable. Slide backward with a step size of s. Continue sliding backwards until the inequality is found. This is successful. At this point, the data reaches a stationary distribution, and the reference window is updated using this stationary data. Simultaneously, the duration of the drift can be obtained and recorded based on the number of slides.

[0148] Optionally, the target time interval is the time interval obtained by delaying the reference time interval by a target number of window times. In this case, the target time interval can be regarded as obtained by sliding a time window backward from the reference time interval. Then, step 204 may include the following steps:

[0149] Step 205: Determine the target reconstruction error for the target time interval and obtain multiple historical reconstruction errors.

[0150] The historical reconstruction error is a reconstruction error determined based on each time interval before the target time interval that is confirmed to have caused incremental concept drift; the target reconstruction error has a corresponding verification window update threshold; and each historical reconstruction error has a corresponding verification window threshold.

[0151] In some embodiments of this application, to further confirm and verify the accuracy and reliability of incremental concept drift through analysis of target reconstruction errors and historical reconstruction errors, a target reconstruction error for a target time interval is determined, and multiple historical reconstruction errors are obtained. The historical reconstruction errors are determined based on the reconstruction errors of each time interval confirmed as having caused incremental concept drift before the target time interval. The target reconstruction error has a corresponding verification window update threshold; each historical reconstruction error has a corresponding verification window threshold. Reconstruction error refers to the error generated when reconstructing actual data using a model. Analysis of reconstruction errors can determine the consistency of data distribution. This provides more accurate data support for subsequent distribution matching and drift confirmation, ensuring the accuracy and reliability of the detection results.

[0152] In a specific example, the analysis of transaction data changes over a certain time period has been completed, and the existence of incremental concept drift has been determined. The transaction data for a given period is reconstructed using a Variational Autoencoder (VAE) model, calculating the target reconstruction error for the target time interval and multiple historical reconstruction errors. Based on the target reconstruction error determined for the target time interval, and the historical reconstruction errors within each previously confirmed time interval of incremental concept drift, the system sets a corresponding verification window threshold for each reconstruction error. This allows for accurate determination of whether the data distribution has changed significantly within the target time interval, providing crucial reference for further drift confirmation and model adjustment.

[0153] Step 206: Verify the significant difference between the target reconstruction error and each historical reconstruction error. If the significant difference is characterized as a distribution mismatch, update the threshold according to the target reconstruction error and the corresponding verification window, and update the reference time interval according to the target time interval.

[0154] In some embodiments of this application, to verify whether there is a significant difference between the target reconstruction error and each historical reconstruction error, and to record different drift scenarios to further confirm changes in data distribution, the significant difference between the target reconstruction error and each historical reconstruction error is verified. If the significant difference indicates a distribution mismatch, the threshold is updated based on the target reconstruction error and the corresponding verification window, and the reference time interval is updated based on the target time interval. A significant difference means that the difference between two sets of data is statistically significant, indicating that they come from different distributions. If the target reconstruction error does not match any of the historical reconstruction errors, it means that the target time interval is different from all previous incremental concepts. At this point, the incremental concept has drifted, so the reference time interval can be updated based on the target time interval. This accurately identifies changes in data distribution, ensuring that subsequent incremental concept drift detection is based on the latest incremental concept drift, thus ensuring the accuracy of subsequent detection.

[0155] In a specific example, changes in customer trading behavior over a certain period have been assessed, and concept drift has been identified. The system can then further calculate the target reconstruction error for the target time interval and multiple historical reconstruction errors. For instance, statistical tests (such as t-tests) can be used to verify the significant differences between the target reconstruction error and each historical reconstruction error. If significant differences are found, indicating a distribution mismatch, the system updates the reference time interval based on the target time interval. Through this process, it is possible to accurately determine whether subsequent customer trading behavior has significantly changed relative to current behavior, and to adjust models and strategies in a timely manner, thereby improving the effectiveness and accuracy of risk management.

[0156] As a specific implementation method, after verification that a drift has occurred, all historical records between the previous drift and the current drift are grouped together. A variational autoencoder (VAE) is constructed based on this group of records, and the reconstruction error of this group of data is calculated and recorded as a feature representation of this group.

[0157] VAE consists of an encoder and a decoder Composition. In a specific implementation, a Long Short-Term Memory (LSTM) network can be used to construct a Virtual Application Engine (VAE) to capture temporal information in streaming data:

[0158] For a sequence containing S elements VAE refactored it as follows:

[0159]

[0160] Train the VAE to make the reconstruction error Minimize the reconstruction error of the training set by learning the parameters of the hidden states in the encoding and decoding parts of the VAE. Let the number of data points in the historical records be denoted as... After the model training is completed, the reconstruction error for each record is calculated, and these reconstruction errors are combined to form the historical reconstruction error:

[0161]

[0162] Next, the reconstruction error is calculated on the data in the reference window, and the size of the reference window is denoted as... This set of reconstruction errors is denoted as:

[0163]

[0164] For these two sets of reconstruction errors, the Kolmogorov-Smirnov Test (KS-Test) was performed as a nonparametric and distribution-free statistical test. Each dimension of the reconstruction error was analyzed to examine whether there was a significant difference between the historical error and the reference window reconstruction error. Hypothesis... , They are respectively by and The two cumulative distribution functions estimated empirically, if If the null hypothesis is not met, then the null hypothesis is rejected, as shown in Formula 16.

[0165]

[0166] in, and They represent and The corresponding number of data points It is the significance level. If the null hypothesis is rejected in any dimension, the data distribution in the reference window and the historical data set is considered inconsistent. Conversely, if the null hypothesis is rejected in any dimension, the distributions are considered consistent, and the historical data set is reused.

[0167] Under the method disclosed in the embodiments of this application, such as Figure 3 The diagram shows a detection model for incremental concept drift in privacy-preserving financial data, designed according to the method provided in the embodiments of this application. It determines the drift through a drift detection layer and a drift verification layer, and adjusts the subsequent detection process through a method adaptive layer.

[0168] based on Figure 3 The detection model, Figure 4 This refers to the specific data flow process within the drift detection layer:

[0169] A1: Target window selection: Ensure that subsequent probability distribution calculations are based on the latest data window.

[0170] A2: Probability distribution calculation: In order to obtain the statistical characteristics of the data, the probability distribution is calculated to provide a basis for subsequent distribution difference calculation.

[0171] A3: Distribution Difference Calculation: By comparing the distribution characteristics of different time periods, the distribution difference is calculated to identify potential concept drift.

[0172] A4: Concept drift detection: Detects concept drift based on distribution differences to ensure that the model can adapt to data changes in a timely manner.

[0173] A5: Weighted average standard deviation calculation: The weighted average standard deviation is calculated using historical data to assess the stability of the data distribution and provide a basis for subsequent drift verification and adjustment.

[0174] Ultimately, based on the detection results, a decision is made as to whether to perform drift verification. If no drift is detected, the drift is ignored, and the data stream continues to be monitored.

[0175] based on Figure 3 The detection model, Figure 5 This refers to the specific data flow process within the drift verification layer:

[0176] B1: Calculate the validation window size: In order to determine the appropriate size of the time window used for validation so as to take into account a sufficient amount of data during the calculation process, and to ensure that the selected validation window can provide enough data for reliable validation analysis.

[0177] B2: Verify statistical consistency using t-test: Use statistical methods to test the consistency of data within the validation window to determine whether the changes in data distribution are significant, providing a basis for verifying drift.

[0178] B3: Verification of Results: Verify the results of the t-test to confirm the accuracy and validity of the statistical analysis.

[0179] B4: Drift Result Verification: Perform final verification on the detected concept drift to confirm whether it actually exists.

[0180] Based on the results of the verification process, the occurrence of drift is finally confirmed, and the verification results are recorded to improve the accuracy of the detection model.

[0181] based on Figure 3 The detection model, Figure 6 This refers to the specific data flow process within the adaptive layer of the method:

[0182] C1: Determine the stable position of concept drift: After confirming the authenticity of concept drift in the drift verification layer, locate the stable position of data distribution to facilitate further adjustments and optimizations, thereby improving the reliability of subsequent drift detection and adaptation.

[0183] C2: Evaluate stationary distribution: Evaluate whether the current data distribution is stationary to determine whether the detection window or other parameters need to be adjusted to ensure the accuracy of the model in subsequent use.

[0184] C3: Match historical distribution: Compare the current data distribution with the historical data distribution to confirm their consistency and changes, thereby improving the accuracy of subsequent drift detection.

[0185] As a verification of the technical effects of the embodiments of this application, the verification results disclosed in the embodiments of this application under two conditions, sudden drift and incremental drift, were compared with several existing technical solutions, such as... Figure 7 As shown in -a, sudden drift is a conceptual drift that occurs in an instant, such as... Figure 7 As shown in -b, incremental drift is a concept drift that occurs continuously over a period of time.

[0186] The detection methods used for comparison include the unsupervised concept drift detection method (Discriminative DriftDetector, D3), the Kolmogorov-Smirnov Test (IKS), the Hellinger Distance Drift Detection Method (HDDDM), and the Online Concept Drift Detection (OCDD). Two datasets, the German language dataset and the credit card dataset, were used for the experiments. Both are publicly available financial datasets. Privacy was protected using Laplace's algorithm. Five drift parameters were inserted into the datasets, and both sudden and incremental drift scenarios were tested. Each scenario was repeated twenty times, and the average value was taken.

[0187] Table 1. Comparison of experimental results among various methods on the dataset.

[0188]

[0189] Table 2 Comparison of experimental results among various methods on the creditcard dataset.

[0190]

[0191] As can be seen from Tables 1 and 2, the method of this application has stronger verification results for sudden drift than for incremental drift. Under the same type of drift, the drift detection accuracy of the method of this application is also significantly higher than that of other methods, and the false alarm rate is lower than that of other methods.

[0192] Furthermore, Table 3 verifies the assistance of the method disclosed in this application in adapting the model to drift. The test was conducted on the payment dataset, where the evaluation metric only considers the model's accuracy. When drift is detected, the model is retrained. Unlike other methods, which can only detect the occurrence of drift and therefore retrain at the location where drift is detected, the method in this paper can retrain at the location of the new stable distribution.

[0193] Table 3 Comparison of experimental results among various methods on the payment dataset.

[0194]

[0195] As shown in Table 3, the proposed method outperforms other methods in helping the model adapt to drift. In contrast, other methods, due to their higher false alarm rates, lead to frequent model adjustments, increasing overhead. This not only affects model stability but also increases computational resource consumption. This also reflects the significant advantage of the proposed method in reducing the number of invalid adjustments and lowering overhead, thus more effectively supporting model drift adaptation.

[0196] In summary, in this embodiment, by defining a reference distribution subsequence and a target distribution subsequence, the data distribution differences between the two are determined. By defining time intervals, the analysis is ensured to focus on key time points, accurately detecting data distribution changes and promptly identifying potential concept drift. Furthermore, when the data distribution differences are characterized as mismatches, multiple validation subsequences are determined from the distribution sequence. By verifying data distribution changes from multiple perspectives, the accuracy and reliability of detection are further improved, avoiding bias caused by single validation, increasing the credibility of drift detection, and ensuring the accuracy of financial data analysis. Finally, based on the statistical matching between the target distribution subsequence and the validation subsequences, the incremental concept drift of the distribution sequence is determined. Statistical matching analysis is used to fully consider data distribution characteristics and efficiently identify incremental concept drift. Therefore, the method based on this embodiment, by introducing the division of multiple subsequences and statistical matching analysis, solves the problems of unavailable or delayed label data and false positives caused by noisy data in the prior art. This detection method can accurately detect incremental concept drift in financial data while ensuring privacy protection, effectively avoiding missed detections and false positives, and improving the performance of economic models and the accuracy of decision-making. It solves the problem of inaccurate analysis results when confirming incremental drift in financial data.

[0197] refer to Figure 8 This application illustrates a detection device 30 for incremental concept drift in privacy-preserving financial data, as provided in an embodiment of this application, comprising:

[0198] The sequence acquisition module 301 is used to acquire the distribution sequence of the financial data to be detected; the distribution sequence contains multiple distribution data of the financial data; each distribution data has a corresponding sampling time.

[0199] The drift detection module 302 is used to determine the data distribution difference between the reference distribution subsequence and the target distribution subsequence in response to the delineation of the reference distribution subsequence and the target distribution subsequence in the distribution sequence; the end time of the reference time interval corresponding to the reference distribution subsequence is earlier than the start time of the target time interval corresponding to the target distribution subsequence.

[0200] The drift verification module 303 is used to determine multiple verification subsequences from the distribution sequence when the data distribution difference is characterized as a mismatch, and to determine the incremental concept drift of the distribution sequence based on the statistical matching between the target distribution subsequence and the verification subsequence; the generation time interval corresponding to the verification subsequence is longer than the target time interval.

[0201] Optionally, the drift detection module 302 includes:

[0202] The window partitioning submodule is used to divide the sampling time of the distributed data into multiple consecutive detection time windows according to a preset sampling window threshold; each detection time window contains multiple sampling times; different detection time windows contain different sampling times.

[0203] The window selection submodule is used to respond to the selection of a reference detection time window and a target detection time window among multiple detection time windows, to determine the distribution data corresponding to multiple sampling times contained in the reference detection time window as the distribution data in the reference distribution subsequence, and to determine the distribution data corresponding to multiple sampling times contained in the target detection time window as the distribution data in the target distribution subsequence;

[0204] The distribution generation submodule is used to determine the reference distribution characteristics of the reference distribution subsequence based on the multiple distribution data contained in the reference distribution subsequence, and to determine the target distribution characteristics of the target distribution subsequence based on the multiple distribution data contained in the target distribution subsequence.

[0205] The difference analysis submodule is used to determine the differences in data distribution between the reference distribution subsequence and the target distribution subsequence based on the characteristics of the reference distribution and the target distribution.

[0206] Optionally, the distributed generation submodule includes:

[0207] The dimensionality reduction unit is used to perform dimensionality reduction and weighting on each distribution data contained in the reference distribution subsequence and the target distribution subsequence to obtain the reference dimensionality reduction subsequence of the reference distribution subsequence and the target dimensionality reduction subsequence of the target distribution subsequence; the reference dimensionality reduction subsequence contains the dimensionality reduction data corresponding to each distribution data in the reference distribution subsequence, and the target dimensionality reduction subsequence contains the dimensionality reduction data corresponding to each distribution data in the target distribution subsequence;

[0208] The merging unit is used to merge the dimensionality-reduced data in the reference dimensionality-reduced subsequence using a weighted product model, so as to use the probability distribution of the obtained reference dimensionality-reduced subsequence as the reference distribution feature, and to merge the dimensionality-reduced data in the target dimensionality-reduced subsequence using a weighted product model, so as to use the probability distribution of the obtained target dimensionality-reduced subsequence as the target distribution feature.

[0209] Optional, the difference analysis submodule includes:

[0210] The confidence range unit is used to determine the reference confidence radius of the reference distribution subsequence based on the characteristics of the reference distribution, and to determine the target confidence radius of the target distribution subsequence based on the characteristics of the target distribution.

[0211] The confirmation unit is used to determine the distribution difference as a match when the target confidence radius is less than or equal to the reference confidence radius, and to determine the distribution difference as a mismatch when the target confidence radius is greater than the reference confidence radius.

[0212] Optionally, the target distribution subsequence has corresponding target distribution characteristics, and the drift verification module 303 includes:

[0213] The verification window submodule is used to divide the sampling time of the distributed data into multiple consecutive verification time windows according to a preset verification window threshold when the data distribution difference is characterized as a mismatch. Each verification time window contains multiple sampling times, and different verification time windows contain different sampling times.

[0214] The verification feature submodule is used to determine the verification distribution features corresponding to the verification time window based on the distribution data corresponding to the multiple sampling times contained in each verification time window;

[0215] The validation metrics submodule is used to determine the validation extreme probability value of each validation distribution feature and the target extreme probability value of the target distribution feature;

[0216] The verification submodule is used to determine the target time interval corresponding to the target distribution subsequence as the time interval in which incremental concept drift occurs when the target extreme probability value is greater than a preset number of verification extreme probability values.

[0217] Optionally, if the target time interval is determined to be the time interval at which incremental concept drift occurs, the device 30 further includes:

[0218] The verification adjustment module is used to determine the updated value of the verification window threshold based on the adjustment of the verification window threshold; the updated verification window threshold is used to determine the time interval length of the verification window as the minimum value among the time interval lengths in which incremental concept drift occurs.

[0219] Optionally, the verification adjustment module includes:

[0220] The threshold adjustment submodule is used to determine the variance of the profile coefficient of the verification window threshold during the adjustment.

[0221] The threshold fixing submodule is used to stop adjusting the verification window threshold when the variance is less than or equal to the preset stationary distribution coefficient, and to set the verification window threshold determined at the time of stopping as the updated value of the verification window threshold.

[0222] Optionally, if the target time interval is determined to be the time interval at which incremental concept drift occurs, the device 30 further includes:

[0223] The historical comparison module is used to determine the target reconstruction error for the target time interval and acquire multiple historical reconstruction errors. The historical reconstruction error is the reconstruction error determined based on each time interval before the target time interval that was identified as having caused incremental concept drift. The target reconstruction error has a corresponding verification window update threshold. Each historical reconstruction error has a corresponding verification window threshold.

[0224] The historical addition module is used to verify the significant difference between the target reconstruction error and each historical reconstruction error. When the significant difference is characterized as a distribution mismatch, the historical reconstruction error and the verification window threshold are updated according to the target reconstruction error and the corresponding verification window update threshold.

[0225] In summary, in this embodiment, by defining a reference distribution subsequence and a target distribution subsequence, the data distribution differences between the two are determined. By defining time intervals, the analysis is ensured to focus on key time points, accurately detecting data distribution changes and promptly identifying potential concept drift. Furthermore, when the data distribution differences are characterized as mismatches, multiple validation subsequences are determined from the distribution sequence. By verifying data distribution changes from multiple perspectives, the accuracy and reliability of detection are further improved, avoiding bias caused by single validation, increasing the credibility of drift detection, and ensuring the accuracy of financial data analysis. Finally, based on the statistical matching between the target distribution subsequence and the validation subsequences, the incremental concept drift of the distribution sequence is determined. Statistical matching analysis is used to fully consider data distribution characteristics and efficiently identify incremental concept drift. Therefore, the method based on this embodiment, by introducing the division of multiple subsequences and statistical matching analysis, solves the problems of unavailable or delayed label data and false positives caused by noisy data in the prior art. This detection method can accurately detect incremental concept drift in financial data while ensuring privacy protection, effectively avoiding missed detections and false positives, and improving the performance of economic models and the accuracy of decision-making. It solves the problem of inaccurate analysis results when confirming incremental drift in financial data.

[0226] Reference Figure 9 The electronic device 500 may include one or more of the following components: processing component 502, memory 504, power supply component 506, multimedia component 508, audio component 510, input / output (I / O) interface 512, sensor component 514, and communication component 516.

[0227] Processing component 502 typically controls the overall operation of electronic device 500, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 502 may include one or more modules to facilitate interaction between processing component 502 and other components. For example, processing component 502 may include a multimedia module to facilitate interaction between multimedia component 508 and processing component 502.

[0228] Memory 504 is used to store various types of data to support the operation of electronic device 500. Examples of this data include instructions for any application or method operating on electronic device 500, contact data, phonebook data, messages, pictures, multimedia, etc. Memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0229] Power supply component 506 provides power to various components of electronic device 500. Power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 500.

[0230] Multimedia component 508 includes an interface that provides an output interface between electronic device 500 and user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may not only sense the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When electronic device 500 is in an operating mode, such as shooting mode or multimedia mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0231] Audio component 510 is used to output and / or input audio signals. For example, audio component 510 includes a microphone (MIC) used to receive external audio signals when electronic device 500 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 504 or transmitted via communication component 516. In some embodiments, audio component 510 also includes a speaker for outputting audio signals.

[0232] Input / output (I / O) interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0233] Sensor assembly 514 includes one or more sensors for providing state assessments of various aspects of electronic device 500. For example, sensor assembly 514 may detect the on / off state of electronic device 500, the relative positioning of components such as the display and keypad of electronic device 500, changes in position of electronic device 500 or a component of electronic device 500, the presence or absence of user contact with electronic device 500, orientation or acceleration / deceleration of electronic device 500, and temperature changes of electronic device 500. Sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 514 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0234] Communication component 516 facilitates wired or wireless communication between electronic device 500 and other devices. Electronic device 500 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 516 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0235] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement the methods provided in the embodiments of this application.

[0236] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, which can be executed by a processor 520 of an electronic device 500 to perform the above-described method. For example, the non-transitory storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0237] Figure 10 This is a block diagram of an electronic device 600 according to another embodiment of the present invention. For example, the electronic device 600 may be provided as a server.

[0238] Reference Figure 10 The electronic device 600 includes a processing component 622, which further includes one or more processors, and memory resources represented by memory 632 for storing instructions, such as application programs, that can be executed by the processing component 622. The application programs stored in memory 632 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 622 is configured to execute instructions to perform the methods provided in the embodiments of this application.

[0239] Electronic device 600 may also include a power supply component 626 configured to perform power management of electronic device 600, a wired or wireless network interface 650 configured to connect electronic device 600 to a network, and an input / output (I / O) interface 658. Electronic device 600 may operate on an operating system stored in memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0240] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0241] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for detecting incremental concept drift in privacy-preserving financial data, characterized in that, include: Obtain the distribution sequence of the financial data to be detected; the distribution sequence contains multiple distribution data of the financial data; each distribution data has a corresponding sampling time; In response to the delineation of a reference distribution subsequence and a target distribution subsequence in the distribution sequence, the data distribution differences between the reference distribution subsequence and the target distribution subsequence are determined; the end time of the reference time interval corresponding to the reference distribution subsequence is earlier than the start time of the target time interval corresponding to the target distribution subsequence. When the data distribution discrepancy is characterized as a mismatch, multiple validation subsequences are determined from the distribution sequence, and the incremental concept drift of the distribution sequence is determined based on the statistical matching between the target distribution subsequence and the validation subsequences; the generation time interval corresponding to the validation subsequence is longer than the target time interval. The target distribution subsequence has corresponding target distribution characteristics. When the data distribution difference is characterized as a mismatch, multiple validation subsequences are determined from the distribution sequence, and the incremental concept drift of the distribution sequence is determined based on the statistical matching between the target distribution subsequence and the validation subsequences, including: When the data distribution discrepancy is characterized as a mismatch, multiple consecutive verification time windows are divided from the sampling times corresponding to the distributed data according to a preset verification window threshold; each verification time window contains multiple sampling times; different verification time windows contain different sampling times; Based on the distribution data corresponding to the multiple sampling times contained in each verification time window, determine the verification distribution characteristics corresponding to the verification time window; Determine the verification extreme probability value for each verification distribution feature and the target extreme probability value for the target distribution feature; the verification extreme probability value is used to characterize the extreme case of the data distribution of the verification distribution feature; the target extreme probability value is used to characterize the extreme case of the data distribution of the target distribution feature. If the target extreme probability value is greater than a preset number of the verification extreme probability values, the target time interval corresponding to the target distribution subsequence is determined as the time interval at which incremental concept drift occurs.

2. The method as described in claim 1, characterized in that, The step of determining the data distribution differences between the reference distribution subsequence and the target distribution subsequence in response to the delineation of the reference distribution subsequence and the target distribution subsequence in the distribution sequence includes: According to a preset sampling window threshold, multiple consecutive detection time windows are divided from the sampling times corresponding to the distributed data; each detection time window contains multiple sampling times; different detection time windows contain different sampling times. In response to the selection of a reference detection time window and a target detection time window among the plurality of detection time windows, the distribution data corresponding to the plurality of sampling times contained in the reference detection time window is determined as the distribution data in the reference distribution subsequence, and the distribution data corresponding to the plurality of sampling times contained in the target detection time window is determined as the distribution data in the target distribution subsequence; Based on the multiple distribution data contained in the reference distribution subsequence, the reference distribution characteristics of the reference distribution subsequence are determined, and based on the multiple distribution data contained in the target distribution subsequence, the target distribution characteristics of the target distribution subsequence are determined. Based on the reference distribution characteristics and the target distribution characteristics, the data distribution differences between the reference distribution subsequence and the target distribution subsequence are determined.

3. The method as described in claim 2, characterized in that, The step of determining the reference distribution characteristics of the reference distribution subsequence based on multiple distribution data contained in the reference distribution subsequence, and determining the target distribution characteristics of the target distribution subsequence based on multiple distribution data contained in the target distribution subsequence, includes: Dimensionality reduction and weighting are performed on each distribution data contained in the reference distribution subsequence and the target distribution subsequence to obtain a reference dimension-reduced subsequence of the reference distribution subsequence and a target dimension-reduced subsequence of the target distribution subsequence; the reference dimension-reduced subsequence contains dimension-reduced data corresponding to each distribution data in the reference distribution subsequence, and the target dimension-reduced subsequence contains dimension-reduced data corresponding to each distribution data in the target distribution subsequence; The dimensionality-reduced data in the reference dimensionality-reduced subsequence are merged using a weighted product model, and the probability distribution of the obtained reference dimensionality-reduced subsequence is used as the reference distribution feature. The dimensionality-reduced data in the target dimensionality-reduced subsequence are then merged using a weighted product model, and the probability distribution of the obtained target dimensionality-reduced subsequence is used as the target distribution feature.

4. The method as described in claim 2, characterized in that, The step of determining the data distribution differences between the reference distribution subsequence and the target distribution subsequence based on the reference distribution characteristics and the target distribution characteristics includes: The reference confidence radius of the reference distribution subsequence is determined based on the reference distribution characteristics, and the target confidence radius of the target distribution subsequence is determined based on the target distribution characteristics; the reference confidence radius is used to quantify the degree of dispersion of the trading data around the mean within the reference time window; the target confidence radius is used to quantify the degree of dispersion of the trading data around the mean within the target time window. If the target confidence radius is less than or equal to the reference confidence radius, the distribution difference is determined to be a match; if the target confidence radius is greater than the reference confidence radius, the distribution difference is determined to be a mismatch.

5. The method as described in claim 1, characterized in that, When the target time interval is determined to be the time interval at which incremental concept drift occurs, the method further includes: Based on the adjustment of the verification window threshold, an updated value for the verification window threshold is determined; the updated verification window threshold is used to determine the time interval length of the verification window as the minimum value among the time interval lengths in which incremental concept drift occurs.

6. The method as described in claim 1, characterized in that, The target time interval is a time interval obtained by delaying the reference time interval by a target number of window times. When the target time interval is determined to be the time interval at which incremental concept drift occurs, the method further includes: A target reconstruction error is determined for the target time interval, and multiple historical reconstruction errors are obtained; the historical reconstruction errors are reconstruction errors determined based on each time interval before the target time interval that was confirmed to have caused incremental concept drift; the target reconstruction error has a corresponding verification window update threshold; each historical reconstruction error has a corresponding verification window threshold; Verify the significant difference between the target reconstruction error and each of the historical reconstruction errors, and if the significant difference is characterized as a distribution mismatch, update the reference time interval based on the target reconstruction error, the corresponding verification window update threshold, and the target time interval.

7. A detection device for incremental concept drift in privacy-preserving financial data, characterized in that, include: A sequence acquisition module is used to acquire the distribution sequence of the financial data to be detected; the distribution sequence contains multiple distribution data of the financial data; each distribution data has a corresponding sampling time. A drift detection module is used to determine the data distribution difference between the reference distribution subsequence and the target distribution subsequence in response to the delineation of the reference distribution subsequence and the target distribution subsequence in the distribution sequence; the end time of the reference time interval corresponding to the reference distribution subsequence is earlier than the start time of the target time interval corresponding to the target distribution subsequence. A drift verification module is used to determine multiple verification subsequences from the distribution sequence when the data distribution difference is characterized as a mismatch, and to determine the incremental concept drift of the distribution sequence based on the statistical matching between the target distribution subsequence and the verification subsequences; the generation time interval corresponding to the verification subsequence is longer than the target time interval. The target distribution subsequence has corresponding target distribution features, and the drift verification module includes: The verification window submodule is used to divide the sampling time corresponding to the distributed data into multiple consecutive verification time windows according to a preset verification window threshold when the data distribution difference is characterized as a mismatch; each verification time window contains multiple sampling times; different verification time windows contain different sampling times; The verification feature submodule is used to determine the verification distribution features corresponding to the verification time window based on the distribution data corresponding to the multiple sampling times contained in each verification time window; The verification index submodule is used to determine the verification extreme probability value of each verification distribution feature and the target extreme probability value of the target distribution feature; the verification extreme probability value is used to characterize the extreme case of the data distribution of the verification distribution feature; the target extreme probability value is used to characterize the extreme case of the data distribution of the target distribution feature. The verification submodule is used to determine the target time interval corresponding to the target distribution subsequence as the time interval in which incremental concept drift occurs when the target extreme probability value is greater than a preset number of verification extreme probability values.

8. An electronic device, characterized in that, include: Processor; memory for storing processor-executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as described in any one of claims 1 to 6.

Images