Large-scale data mining method based on multi-tuple data optimization

Abstract

The invention relates to a large-scale data mining method based on multi-tuple data optimization. The method comprises the following steps: firstly, carrying out importance analysis and sorting on each dimension feature in multi-tuple data by designing a feature evaluation mechanism based on information gain and mutual information; on a data set after feature selection and extraction, an improved clustering algorithm is applied, the advantages of density clustering and hierarchical clustering are combined, and a cluster of a structure is recognized in high-dimensional multi-tuple data; based on a cluster obtained through clustering analysis, a cross-dimension association rule mining algorithm is designed, and a cross-dimension association relationship is revealed in the multi-tuple data; constructing a hybrid prediction model fusing multiple regression and deep learning technologies; and the prediction accuracy of the target variable is improved by utilizing the multi-dimensional information in the multi-element group data and the mined association rule.

Description

Technical Field

[0001] The invention relates to a large-scale data mining method, in particular to a large-scale data mining method based on multi-group data optimization. Background Art

[0002] Although the current methods for large-scale data mining have made significant progress in some aspects when dealing with complex and diverse big data, there are still some shortcomings and drawbacks. Taking the Chinese invention patent "Patent No. 201510080263.2 A large-scale data mining method to ensure quality monotonicity" as an example, this method attempts to ensure the quality of data mining results under limited time and resource constraints through the improved KNN classification algorithm and information entropy technology. However, although this method performs well in specific situations, it still has many shortcomings in practical applications. First, the method relies too much on a single improved algorithm, including the KNN classification algorithm and information entropy technology, and cannot fully deal with the problems of diversity and complexity in large-scale data mining. In practical applications, big data often has the characteristics of heterogeneity, dynamics and multi-dimensionality, which makes a single algorithm seem incapable of processing diversified data. Although the KNN algorithm performs well on small-scale data sets, its computational complexity and storage requirements will increase sharply when facing massive data, resulting in low efficiency. In addition, the KNN algorithm itself is sensitive to noise and outliers, and is easily affected by uneven data distribution in large-scale data mining, thereby reducing classification accuracy. Secondly, although the patent optimizes the coding part of the algorithm through information entropy technology, trying to ensure the entropy preservation and quality monotonicity of the mining results, the information entropy technology itself has its limitations. Information entropy is mainly used to measure the degree of disorder and the amount of information in the system. Although it can optimize the performance of the algorithm in some cases, its processing ability for high-dimensional data is limited. When faced with high-dimensional and sparse data sets, information entropy cannot effectively capture the inherent structure of the data, thus affecting the accuracy of the mining results.

[0003] Another significant shortcoming is the limitation of this method in dealing with real-time and dynamic data. With the rapid development of data sources such as the Internet of Things, sensor networks, and social media, data is not only massive, but also highly dynamic. Traditional batch processing methods and static algorithms are difficult to adapt to this rapidly changing environment. The method for ensuring quality monotonicity proposed in this patent is mainly aimed at static data sets and lacks support for real-time data. Therefore, when the distribution, characteristics, or environment of the data changes, the performance of the algorithm will decline rapidly and cannot be adjusted in time to adapt to the new data pattern, resulting in a decrease in the accuracy and reliability of the results. Finally, this method also has limitations in dealing with complex data relationships and advanced data mining tasks. Modern data mining not only focuses on simple classification and clustering tasks, but also pays more attention to complex pattern recognition, relationship mining, and deep knowledge discovery. However, this patent mainly focuses on improving the KNN algorithm and information entropy technology, and lacks support for complex data relationships and advanced mining tasks. In the fields of social network analysis, text mining, and graph data processing, the relationships between data are complex and diverse, and the traditional KNN algorithm is difficult to capture these complex relationships, resulting in insufficient effectiveness and depth of the mining results. Summary of the invention

[0004] The purpose of the present invention is to provide a large-scale data mining method based on multi-group data optimization, so as to solve some of the drawbacks and shortcomings pointed out in the background technology.

[0005] The present invention solves the above-mentioned technical problems by adopting the following technical solutions:

[0006] The following steps are involved:

[0007] S1. Multidimensional feature selection and extraction:

[0008] Firstly, by designing a feature evaluation mechanism based on information gain and mutual information, the importance of each dimension feature in the multi-dimensional data is analyzed and ranked;

[0009] S1.1, then use the dynamic feature selection algorithm to select representative feature sets according to the feature weights and their contribution to the data mining task;

[0010] S1.2, based on S1.1, further design the feature extraction algorithm to generate new high-order combined features by combining the selected features;

[0011] S2. Efficient clustering analysis based on multi-group features:

[0012] On the dataset after feature selection and extraction, an improved clustering algorithm is applied, combining the advantages of density clustering and hierarchical clustering, to identify structural clusters in high-dimensional multivariate data;

[0013] S2.1, using a distributed computing framework to split the clustering process into multiple parallel tasks; the clustering algorithm includes an adaptive optimization function to dynamically adjust clustering parameters according to the internal structure and data distribution of the clustering results;

[0014] S3. Cross-dimensional association rule mining:

[0015] Based on the clusters obtained by cluster analysis, a cross-dimensional association rule mining algorithm is designed to reveal cross-dimensional association relationships in multi-tuple data;

[0016] S3.1. Using a parallelized association rule calculation engine, it supports mining interesting and confident association rules in large-scale data sets.

[0017] S3.2. At the same time, a hierarchical association rule screening mechanism is introduced to filter out redundant or low-value rules, and combined with the characteristics of multi-group data, an association pattern with strong explanatory power is output;

[0018] S4. Construction and optimization of hybrid prediction model:

[0019] Based on the results of steps S1-S3, a hybrid prediction model integrating multivariate regression and deep learning technology is constructed; the multidimensional information in the multivariate data and the mined association rules are used to improve the prediction accuracy of the target variable;

[0020] S4.1, dynamically adjust model parameters to respond to changes in data distribution through adaptive model training algorithms;

[0021] S4.2. Use integrated learning methods to further improve the stability and reliability of prediction results through collaborative prediction of multiple models.

[0022] Furthermore, the dynamic feature selection algorithm comprises the following steps:

[0023] S1. Feature weight calculation:

[0024] Based on the goal of data mining tasks, by defining the function Φ(f i , Y) is used to measure the feature f i The correlation between the target variable Y, where Φ(f i , Y) is expressed as follows:

[0025]

[0026] in, is the information gain function, is the mutual information function, α j and β jis the weight coefficient, which indicates the importance weight of each method; the initial weight of each feature is calculated by the formula i ;

[0027] At the same time, through the function Λ(ω i ) implements feature importance evaluation based on the gradient boosting model, and comprehensively calculates Φ(f i , Y) and Λ(ω i ) Calculate the initial weight Ω of the feature i :

[0028] Ω i =γ·Φ(f i , Y)+δ·Λ(ω i )

[0029] Among them, γ and δ are weight adjustment factors;

[0030] S2. Special contribution evaluation:

[0031] Based on the feature weight calculation, the function Ψ(Ω i , Θ) is used to evaluate the feature Ω i The actual contribution to the data mining task Θ, where Ψ(Ω i , θ) is expressed as follows:

[0032]

[0033] in, is the loss function, which indicates the impact of feature selection including accuracy and F1 score on model performance. is the model performance change function, κ is the adjustment coefficient; through the integral formula, the low contribution features are gradually eliminated, and according to Ψ(Ω i ,Θ) results update feature set;

[0034] S3. Dynamic feature selection mechanism:

[0035] Based on the feature weight Ω i and contribution evaluation Ψ(Ω i , Θ), design a dynamic feature selection function Θ dyn (Ω i , t), where t is the time variable, the function is expressed as follows:

[0036]

[0037] Among them, ∈(t) and η(t) are time-dependent weight factors, is the adjustment function for the interaction between features; through the dynamic feature selection function Θ dyn (Ω i,t), adjust the feature set in real time according to the model performance, changes in data distribution, and interactions between features.

[0038] Furthermore, the clustering method for identifying structure in the multi-tuple data is:

[0039] S1. First, the improved density clustering algorithm is applied in the high-dimensional space of multi-tuple data. The distance metric between different dimensions is adjusted through the dynamic neighborhood search strategy, so that the density clustering algorithm can identify the cluster structure in the high-dimensional space. Specifically, the distance metric function is defined Where x and y are data points, It is expressed as follows:

[0040]

[0041] Among them, ρ i (x) and ρ i (y) represents the density distribution of points x and y in the i-th dimension, γ i is an adjustment factor used to dynamically adjust the distance weights in different dimensions;

[0042] S2. Based on the preliminary density clustering results, a hierarchical clustering algorithm based on the density clustering results is applied to perform hierarchical analysis and dynamic adjustment of the identified clusters by constructing a tree structure; define a weight adjustment function It is expressed as follows:

[0043]

[0044] Among them, ρ(c i ) and ρ(s j ) is cluster c i and j The density value of is the inter-cluster similarity function, ζ l To dynamically adjust the coefficients, hierarchical clustering can dynamically adjust the merging or splitting operations according to the density characteristics of the clusters;

[0045] S3. Finally, by designing a multi-level clustering integration framework, the results of density clustering and hierarchical clustering are integrated, and through repeated iterations and optimization, the cluster structure in high-dimensional data is adaptively identified and optimized; the overall optimization function is defined It is expressed as follows:

[0046]

[0047] in, is the intra-cluster similarity evaluation function, is the cluster distance evaluation function, λ and μ are weight coefficients.

[0048] Furthermore, the construction method of revealing cross-dimensional correlation relationships is:

[0049] S1. First, based on the preliminary clustering clusters, the structural information of high-dimensional data is projected into low-dimensional space through multi-dimensional projection and mapping algorithms to generate a feature vector set that integrates multi-dimensional information; define the projection function Where X is a high-dimensional data set, and the projection function is expressed as follows:

[0050]

[0051] Among them, f i (X) represents the feature mapping function of the i-th dimension, α i To map the weight coefficients, the data features of different dimensions are mapped into a unified feature space through a projection function to form a feature vector set that integrates multi-dimensional information;

[0052] S2. Then, in a unified feature space, the features of different dimensions are combined to generate a cross-dimensional feature combination, and a feature screening mechanism based on cluster density features is introduced; the screening function is defined In order to retain representative and potentially relevant feature combinations, the screening function is expressed as follows:

[0053]

[0054] Among them, ρ j (c k ) is the cluster c k The density value of is the distribution function of feature j, β j is the density weight factor;

[0055] S3. Finally, based on the selected cross-dimensional feature combination, design an association rule mining algorithm that combines frequent pattern growth and Apriori algorithm; define the calculation function of cross-dimensional support and confidence It is expressed as follows:

[0056]

[0057] Among them, Z is the cross-dimensional feature combination set, φ i (Z) and θ j (Z) is the measurement function of support and confidence, and are the support and confidence weight functions in their respective dimensions.

[0058] Furthermore, the hierarchical association rule screening mechanism construction mode includes:

[0059] S1. First, the mined association rules are hierarchically represented and the rules are layered according to their coverage, support, confidence and importance in the data. Specifically, the rule hierarchical function is defined as Where R i is a set of rules, and the function is expressed as follows:

[0060]

[0061] Among them, Cov(R i , D j ) represents rule R i In the data dimension D j Coverage on Sup(R i ) is the support of the rule, Conf(R i ) is the confidence of the rule, Representation rule R i For the target variable T j The influence of Depth(R i ) represents the nesting depth of rules, α j , β j , γ j is the weight adjustment factor;

[0062] S2. Through the hierarchical function, all association rules are organized into an orderly hierarchical structure; on the basis of hierarchical representation, the multi-tuple data characteristics are introduced to further screen the rules; through the multi-tuple sensitivity function Measuring Rule R i In the multivariate group G k The performance consistency on the function is as follows:

[0063]

[0064] in, Representation rule R i In tuple G k The fitness function in , ρ l is the tuple feature weight;

[0065] S3. For redundant or low-value rules, introduce a filtering mechanism based on information gain and entropy value changes; define the rule gain contribution function To calculate the contribution value of each rule, the function is expressed as follows:

[0066]

[0067] Among them, Δε(R i ) is the entropy change of the rule, IG(R i , C p ) indicates that the rule is in category Cp The information gain on p and v p is the weight factor.

[0068] Furthermore, the change strategy of the response data distribution includes:

[0069] S1. By designing a real-time data distribution monitoring module, using sliding window technology and statistical analysis including mean, variance, and covariance matrix, the data distribution is continuously tracked;

[0070] S1.1. When a new batch of data enters the model, the distribution difference between the batch of data and the previous data is calculated, and the change is quantified by the distribution deviation indicator; if the change in data distribution exceeds the preset threshold, the adaptive adjustment of the model is triggered;

[0071] S2, when significant changes in data distribution are detected, the adaptive model training algorithm adjusts the key parameters of the model;

[0072] S2.1. Adjust parameter settings based on preset strategies or dynamically optimize through machine learning algorithms, and iteratively optimize based on the performance of the model under the new data distribution;

[0073] S3. Based on the adaptive parameter adjustment mechanism, use online learning algorithms including SGD with warm restarts or Adagrad to gradually adapt the model to the new data distribution without resetting parameters;

[0074] S3.1. Regularization terms are used to control model complexity, and early stopping strategies are used to avoid excessive fluctuations in model performance.

[0075] Furthermore, the process of the adaptive adjustment mechanism of the trigger model:

[0076] S1. First, we represent the multi-tuple data, extract high-dimensional features and construct feature space; dynamically calculate the distribution difference between the new data batch and the previous data batch in the feature space through the sliding window technology; and use the distribution deviation indicator function to To quantify the distribution change, the formula is as follows:

[0077]

[0078] Where x and Y represent the feature representations of the current batch of data and the previous batch of data, respectively, and α i is the weight adjustment factor;

[0079] S2. According to the distribution changes monitored, preset the adaptive threshold τ(t), and update it in real time based on the data distribution change trend and historical fluctuations; the specific threshold adjustment function It is expressed as follows:

[0080]

[0081] Among them, τ0 is the initial threshold, β j and λ j is the adjustment factor, is the distribution deviation trend function;

[0082] When changes in data distribution are detected Exceeding the adaptive threshold , triggers the adaptive adjustment process of the model; specifically, it re-evaluates and adjusts the key parameters of the model, including the learning rate and regularization strength, and dynamically updates the model parameters through the online learning algorithm.

[0083] The beneficial effects of the present invention are as follows: by combining the high-dimensional features of multi-group data and adopting improved dynamic feature selection and clustering algorithms, important features and structures in data can be more accurately identified. This method can effectively reduce the impact of noise data and improve the accuracy of mining results; through multi-level clustering integration and cross-dimensional association rule mining, complex associations and structures in high-dimensional data can be identified. This capability is particularly suitable for processing heterogeneous data sets and complex data relationships, and can more deeply mine potential patterns and trends in data. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 The present invention is a flowchart of a large-scale data mining method based on multi-group data optimization.

[0085] Figure 2 This is a flow chart of the dynamic feature selection algorithm of the present invention.

[0086] Figure 3 A cluster flow chart for identifying structures in multi-tuple data of the present invention.

[0087] Figure 4 This is a flow chart of the strategy for responding to changes in data distribution according to the present invention. DETAILED DESCRIPTION

[0088] The specific implementation modes of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0089] Combined with Figure 1, a large-scale data mining method and risk assessment method based on multivariate data optimization, firstly, multidimensional feature selection and extraction are performed, and the importance of each dimensional feature in the multivariate data is analyzed and ranked by designing a feature evaluation mechanism based on information gain and mutual information. Information gain is used to evaluate the contribution of each feature to the target variable, while mutual information quantifies the interdependence between two variables. Such analysis can help identify the most influential features. Then, a dynamic feature selection algorithm is used to screen out representative feature sets based on the weights of the features and their contribution to the data mining task. The feature weight is determined by the correlation between the feature and the target variable and the performance of the feature in the task. The screened feature set can better adapt to the specific data mining task and ensure the accuracy and efficiency of the analysis. The feature extraction algorithm is further designed to generate new high-order combined features by combining the selected features. These combined features can capture the complex interactive relationship between the original features and provide richer information dimensions for subsequent analysis.

[0090] Based on the extraction results of the above multidimensional features, cluster analysis is performed, and clustering algorithms are applied to combine the advantages of density clustering and hierarchical clustering to identify structural clusters in high-dimensional multivariate data. Density clustering is suitable for discovering clusters of arbitrary shapes and is robust to noise, while hierarchical clustering can generate cluster structures with hierarchical relationships, which is convenient for further analysis and understanding of the internal organization of the data. A distributed computing framework is adopted to split the clustering process into multiple parallel tasks. This distributed computing not only improves the efficiency of processing large-scale data sets, but also ensures the scalability of clustering analysis. In addition, the clustering algorithm also includes an adaptive optimization function, which can dynamically adjust the clustering parameters according to the internal structure and data distribution of the clustering results. This function ensures the stability and accuracy of the clustering results, and can adaptively optimize the clustering process even in the face of complex data distribution.

[0091] The system designs a cross-dimensional association rule mining algorithm and reveals the cross-dimensional association relationships in multi-dimensional group data based on the clusters obtained by cluster analysis. By analyzing the potential relationships between features of different dimensions, the system can better understand the hidden patterns in the data. The parallelized association rule calculation engine supports the mining of high-interest and high-confidence association rules in large-scale data sets. This parallel calculation can significantly accelerate the association rule mining process and ensure that high computing efficiency can be maintained in a large-scale data environment. At the same time, a hierarchical association rule screening mechanism is introduced to filter out redundant or low-value rules. Combined with the characteristics of multi-dimensional group data, the system outputs association patterns with strong explanatory power, which can provide important references for risk assessment and decision support.

[0092] Based on the results of the previous steps, a hybrid prediction model integrating multivariate regression and deep learning technology is constructed. By combining traditional statistical analysis methods with modern machine learning techniques, the system can utilize the multidimensional information in multivariate data and the mined association rules to significantly improve the prediction accuracy of the target variable. Through the adaptive model training algorithm, the model parameters are dynamically adjusted to respond to changes in data distribution. This adaptive training ensures that the model can maintain good prediction performance in a changing data environment. Finally, a method that supports ensemble learning is adopted to further improve the stability and reliability of the prediction results through the collaborative prediction of multiple models. Ensemble learning can significantly improve the performance of the overall model by combining multiple weak prediction models, reduce the bias and variance problems of a single model, and ensure the robustness of the prediction.

[0093] Embodiment 1:

[0094] Combined with Figure 2 In a large-scale data mining project involving financial market risk assessment, a large-scale data mining method based on multivariate data optimization is adopted, and special attention is paid to the dynamic feature selection algorithm to ensure the accuracy and effectiveness of the data mining process. Multivariate data on customer investment behavior is collected, including stock trading records (price, trading volume), macroeconomic indicators (interest rate, inflation rate), market sentiment data (news sentiment analysis score, social media discussion volume), and customer personal information (age, income level, investment preference).

[0095] First, according to the goal of the data mining task, namely risk prediction, we define the function Φ(f i , Y), used to measure each feature f i Correlation with the target variable Y (risk level). The target variable Y represents the volatility of the portfolio over a certain period of time. Function Φ(f i , Y) is expressed as follows:

[0096]

[0097] in, is the information gain function, which means that after introducing feature f i The degree of reduction of uncertainty in the target variable Y; is the mutual information function, which quantifies the feature f i The dependency relationship between the target variable Y and the target variable Y. Weight coefficient α j and β j The values ​​of are 0.7 and 0.3, respectively, which are used to balance the role of information gain and mutual information in the feature selection process.

[0098] Suppose for a certain feature f i(customer's stock trading volume), information gain function At a certain moment, the derivative value is 0.5, and the mutual information function The value of is 0.3. According to the above formula, Φ(f i , the calculation result of Y) is:

[0099]

[0100] If the derivative calculation result is set to 0.2, the final result is:

[0101] Φ(f i , Y)=0.7·0.2+0.3·0.3=0.14+0.09=0.23

[0102] Next, through the function Λ(ω i ) implements feature importance evaluation based on the gradient boosting model (XGBoost). Set feature f i Initial weights ω in the gradient boosting model i is 0.25. Combined with Φ(f i , Y) and Λ(ω i ), calculate the final initial weight Ω of the feature i :

[0103] Ω i =γ·Φ(f i , Y)+δ·Λ(ω i )

[0104] Among them, γ and δ are 0.6 and 0.4 respectively, which are used as weight adjustment factors. Substitute the calculation results into:

[0105] Ω i =0.6·0.23+0.4·0.25=0.138+0.1=0.238

[0106] Get feature f i The final initial weight of is 0.238, indicating that this feature has high importance in the risk prediction model.

[0107] The embodiment further evaluates the contribution of the determined features, which is a key step to ensure that the final model only contains the most valuable features for the target task (risk prediction). The contribution function expression is as follows:

[0108]

[0109] in, is the loss function, measuring the feature Ω i The impact on the target task Θ, including the model’s accuracy and F1 score; is the model performance change function, which is used to capture the impact of adding or removing features on model performance; κ is the adjustment coefficient, which ranges from 0.01 to 0.1 and is used to balance the weights of the loss function and the performance change function in the evaluation.

[0110] FeaturesΩ i The initial weight of (customer's stock trading volume) has been calculated in the previous step. Now evaluate the actual contribution of this feature in the final model. Set the loss function represents the mean square error (MSE) of the model, and the error value is 0.05 when this feature exists, and the error value after removing this feature is 0.07. The corresponding loss function derivative value is:

[0111]

[0112] Next, consider the model performance change function When this feature is included, the F1 score of the model increases from 0.85 to 0.87. The value of can be expressed as 0.02.

[0113] Set the adjustment coefficient K = 0.05 and substitute these values ​​into Ψ(Ω i , Θ) is integrated:

[0114]

[0115] If you set Ω i =0.238 (from the weight calculation result of the previous step), the integral result is:

[0116]

[0117] Financial institutions can obtain the contribution value Ψ(Ω i , Θ), determine whether the feature should be retained in the final model.

[0118] The final embodiment uses a dynamic feature selection function Θ dyn (Ω i , t), is used to dynamically adjust the feature set according to time t. The function is expressed as follows:

[0119]

[0120] Among them, ∈(t) and η(t) are time-related weight factors, ranging from 0.1 to 1.0, It is an adjustment function for the interaction between features, which is used to modify the weight when there is an interaction effect between features.

[0121] At time t = 100 seconds, there is a sharp fluctuation in interest rates in the market, which will affect the investment behavior of customers and further affect the prediction accuracy of the risk assessment model. In the calculations before this embodiment, the contribution evaluation of stock trading volume Ψ(Ω i , Θ) is the derivative with time The value is 0.015, and the interaction function between features It represents the impact of interest rate changes on this feature, and its value is 0.02.

[0122] Set the time-related weight factors ∈(t) and η(t) to 0.8 and 0.5 respectively, and bring them into the dynamic feature selection function:

[0123]

[0124] The calculation results are:

[0125] Θ dvn (Ω i , 100) = 100·(0.012+0.01) = 100·0.022 = 2.2

[0126] Dynamic feature selection function Θ dvn (Ω i , t) is calculated as 2.2, which means that at the 100th second, the importance of stock trading volume has increased significantly, which is caused by the change in market interest rates.

[0127] Embodiment 2:

[0128] Combined with Figure 3 In the multivariate data optimization project, the embodiment adopts a density clustering algorithm and adjusts the distance metric between different dimensions through a dynamic neighborhood search strategy. By analyzing the customer's transaction behavior, macroeconomic indicators and market sentiment data, customer groups (i.e., clusters) with similar risk characteristics are identified. The distance metric function is used:

[0129]

[0130] ρ i (x) and ρ i (y) represents the density distribution of data points x and y in the i-th dimension, respectively. i It is an adjustment factor with a value range of 0.5 to 2.0, which is used to dynamically adjust the distance weights in different dimensions.

[0131] Analyze a set of trading data of 10,000 customers, each of which has data dimensions including stock trading volume, trading frequency, holding time, risk preference score, etc. In order to identify customer groups with similar risks, the institution selected stock trading volume and trading frequency as the main clustering dimensions.

[0132] For data points x and y (representing two customers respectively), in the two dimensions of stock trading volume and trading frequency, the density functions ρ1(x) and ρ2(x) represent the data density of customer x in these two dimensions respectively. In terms of transaction frequency,

[0133] Set the adjustment factors γ1 = 1.5 and γ2 = 1.0, and set the density function derivatives in these two dimensions to be and According to the above formula, the distance measure between two customers in terms of stock trading volume and trading frequency is calculated:

[0134]

[0135] The calculations show that:

[0136]

[0137] By accurately calculating the distance between data points x and y, financial institutions can accurately identify the relationship between customer groups in multidimensional data space.

[0138] The embodiment further applies a hierarchical clustering algorithm based on the density clustering results. By constructing a tree structure, the customer clusters identified in the previous step are further stratified to identify the hierarchical relationship between the clusters, and customer groups with similar risks are further divided into high-risk, medium-risk and low-risk sub-clusters, and some clusters with lower density are merged to form a larger customer group. Using the weight adjustment function To decide whether to cluster c i and j Merge or further split, the function expression is as follows:

[0139]

[0140] Among them, ρ(c i ) and ρ(s j ) is cluster c i and j The density value of is the inter-cluster similarity function, ζ l It is a dynamic adjustment coefficient, and its value range is 0.1 to 1.0.

[0141] Set three clusters c1, c2 and c3, representing high-frequency trading customers, medium-frequency trading customers and low-frequency trading customers respectively. Now analyze the internal structure of these clusters through hierarchical clustering to decide whether to merge or further subdivide some clusters. For clusters c1 and c2, the density value ρ(c1) = 0.8, ρ(c2) = 0.5. Set the similarity function It is expressed as the average transaction frequency difference between the two clusters, with a value of 0.3.

[0142] The dynamic adjustment coefficient ζl is set to 0.7 at the initial time l = 1, and gradually decreases to 0.5 when l = 10. Substitute these values ​​into the weight adjustment function

[0143]

[0144] in And set Keeping it constant at each time step, we can calculate:

[0145]

[0146] Weight adjustment function The result is 0.168, indicating that there is a certain potential for merging between the two clusters, especially when the difference in transaction frequency is small. At this time, c1 and c2 are merged to form a new customer group with a wider coverage, so as to unify management and formulate risk management strategies.

[0147] Next, the embodiment integrates these clustering results, designs a multi-level clustering integration framework, and adaptively identifies and optimizes the cluster structure in high-dimensional data through repeated iterations and optimization. Define the overall optimization function Used to evaluate and optimize clustering results. The function expression is as follows:

[0148]

[0149] in, is the intra-cluster similarity evaluation function, measuring each cluster c i Similarity between internal data points; is the cluster distance evaluation function, which measures the difference between different clusters d j The distance between clusters; λ and μ are weight coefficients, ranging from 0.1 to 1.0 and 0.1 to 2.0 respectively, which are used to balance the role of intra-cluster similarity and cluster distance in the overall optimization.

[0150] Set up five main customer clusters c1, c2, c3, c4 and c5. These clusters represent different transaction behavior characteristics. For each cluster c i (such as c1), intra-cluster similarity evaluation function It is measured by calculating the average Euclidean distance or other similarity indicators between the data points within the cluster. Set the calculated similarity evaluation value of c1 to 0.85. Cluster distance evaluation function The distance between different clusters is measured by the minimum distance or the distance between the centroids of the clusters. The distance evaluation value between the two clusters c1 and c2 is set to 0.65. The weight coefficients λ = 0.8 and μ = 1.2, then the overall optimization function The calculation is:

[0151]

[0152] Set the similarity evaluation value for each cluster are 0.85, 0.75, 0.80, 0.70 and 0.65 respectively, while the cluster distance evaluation value If they are 0.65, 0.70, 0.75 and 0.80 respectively, the calculation results are:

[0153]

[0154]

[0155] Overall optimization function The calculated result is 6.48, indicating that the clustering results after integration and optimization have high internal similarity and appropriate inter-cluster spacing.

[0156] Embodiment 3:

[0157] In the multi-dimensional data optimization project, a multi-dimensional projection and mapping algorithm is used to project the structural information of high-dimensional data into a low-dimensional space to generate a feature vector set that integrates the multi-dimensional information. First, based on the clusters obtained by the previous cluster analysis, the structural information of the high-dimensional data is projected into a low-dimensional space through a multi-dimensional projection and mapping algorithm. The expression of the projection function is as follows:

[0158]

[0159] Among them, f i (X) represents the feature mapping function of the i-th dimension, α i is the mapping weight coefficient, ranging from 0.1 to 2.0.

[0160] We study a set of multivariate data containing 5,000 customers, each of which has data dimensions including stock trading volume (dimension x1), trading frequency (dimension x2), interest rate (dimension x3), inflation rate (dimension x4), and market sentiment score (dimension x5). In order to reveal the potential correlation between these data dimensions, we project these high-dimensional data into a low-dimensional feature space.

[0161] For the stock trading volume dimension x1, the feature mapping function f1(X) is set to represent the rate of change of the customer's trading volume over a period of time. Set α1 = 1.2, and set the mapping function f2(X) of the trading frequency dimension x2 to represent the change amplitude of the trading frequency, and set α2 = 0.8.

[0162] For a customer's data point X, its projection function It can be expressed as:

[0163]

[0164] When t=0.5, Then the calculation is:

[0165]

[0166] By projecting all customer data points X into a low-dimensional feature space, financial institutions are able to generate a set of feature vectors that incorporate multi-dimensional information.

[0167] Embodiment Next, in a unified feature space, features of different dimensions are combined, and a feature screening mechanism based on cluster density features is introduced to screen out the most representative and potentially relevant feature combinations, using the screening function:

[0168]

[0169] Among them, ρ j (c k ) is the cluster c k The density value of is the distribution function of feature j, β j is the density weight factor, ranging from 0.5 to 1.5.

[0170] Assume that a financial institution has identified three main customer clusters, c1, c2, and c3, representing high-frequency trading customers, medium-frequency trading customers, and low-frequency trading customers, respectively. Now, the institution hopes to combine the features in these clusters to select the most representative cross-dimensional feature combination.

[0171] For cluster c1, the density value ρ1(c1) = 0.75, and the distribution function of feature j1 (such as stock trading volume) is set The derivative value of is 0.6. The density weight factor

[0172] Substitute into the filter function middle:

[0173]

[0174] Assuming that in a specific case, the calculation is:

[0175]

[0176] Filter Function The calculation result of is a negative value, indicating that the feature combination is not highly representative or potentially relevant in cluster c1. In this case, financial institutions will consider further optimizing the feature combination or selecting other features for screening.

[0177] The last example combines the frequent pattern growth and Apriori algorithm to design an association rule mining algorithm, and defines the calculation function of cross-dimensional support and confidence To quantify the strength of these associations, the expression is as follows:

[0178]

[0179] Among them, Z is the cross-dimensional feature combination set, φ i (Z) and θ j (Z) is the measurement function of support and confidence, and are the support and confidence weight functions in their respective dimensions, Indicates time or trading cycle.

[0180] Assume that a financial institution is analyzing a set of transaction data of 10,000 customers and has identified the combination of stock trading volume and market sentiment score Z1 and the combination of interest rate change and transaction frequency Z2. The institution hopes to calculate the support and confidence of these two combinations through association rule mining algorithms and analyze their impact on customer risk behavior.

[0181] Set the support measure function φ1(Z1) of combination Z1 to 0.6, the confidence measure function θ1(Z1) to 0.7, and the corresponding weight function and 0.8 and 0.9 respectively.

[0182] Substituting into the calculation function C(Z), the calculation for the combination Z1 is as follows:

[0183]

[0184] Set time period If the customer's trading behavior remains stable for one year, the calculation result is:

[0185]

[0186] The calculated result is 0.76, indicating that the combination of stock trading volume and market sentiment score Z1 has a high correlation in customer behavior patterns.

[0187] Embodiment 4:

[0188] Combined with Figure 4 In this embodiment, in the multi-tuple data optimization project of financial institutions, a hierarchical association rule screening mechanism is constructed to stratify the mined association rules according to their coverage, support, confidence and importance in the data, so as to identify and use the most valuable rules. First, the mined association rules are hierarchically represented. The hierarchical function is used:

[0189]

[0190] Among them, Cov(R i , D j ) represents rule R i In the data dimension D j Coverage on Sup(R i ) is the support of the rule, Conf(R i ) is the confidence of the rule, Representation rule R i For the target variable T j The influence of Depth(R i ) represents the nesting depth of rules, α j , β j , γj are weight adjustment factors, ranging from 0.1 to 1.0.

[0191] Assume that the financial institution has mined the following association rules:

[0192] Rule R1: When market sentiment is high, clients’ stock trading volume increases;

[0193] Rule R2: When interest rates rise, customers trade less frequently.

[0194] Agencies hope to identify and prioritize the impact of these rules on different data dimensions through a hierarchical representation.

[0195] For rule R1, set its coverage Cov(R1, D1) on the market sentiment dimension D1 to 0.7, and support Sup(R1) to 0.6. Set the weight adjustment factor α1 to 0.8.

[0196] Set the confidence level Conf(R1) of rule R1 to 0.75, and its influence on the target variable (increase in trading volume) T1 is 0.85. The nesting depth Depth(R1) is 2. The weight adjustment factors β1=0.9 and γ1=0.3 are set.

[0197] Substitute into the calculation function middle:

[0198]

[0199] The calculations show that:

[0200]

[0201] Rule Hierarchy Function The calculated result is 1.439, which shows that this rule has high importance in the multidimensional data analysis of financial institutions.

[0202] The embodiment further introduces the characteristics of multi-tuple data and uses a sensitivity function to perform a deeper screening of the rules to ensure the adaptability and consistency of the rules in different tuples. The function is:

[0203]

[0204] in, Representation rule R i In tuple G k The fitness function in , ρ l is the tuple feature weight, ranging from 0.1 to 1.0.

[0205] Set up two main customer groups G1 and G2, representing high net worth customers and medium and low risk customers respectively, through the sensitivity function To evaluate the consistency of rule R1 (the impact of market sentiment on high-frequency trading) in these two groups. Set the fitness function of rule R1 in tuple G1 It is expressed as the rate of change of the customer's trading behavior under specific market sentiment conditions, and ρ1 is set to 0.8. For G2, the adaptability function It represents the frequency of portfolio adjustment of clients under low risk preference, and p2 is set to 0.6.

[0206] Substitute into the calculation function middle:

[0207]

[0208] Set within a certain time period. and The calculation result is:

[0209]

[0210] The calculated result is 0.86, indicating that rule R1 has high adaptability and consistency in the high net worth customer group G1.

[0211] Next, the embodiment introduces a filtering mechanism based on information gain and entropy value changes, aiming to remove redundant or low-value rules and ensure the high quality and efficiency of the final rule set. The rule gain contribution function is as follows:

[0212]

[0213] Among them, Δε(R i ) is the change in entropy value of the rule, which indicates the degree of reduction of uncertainty of the rule in the data set; IG(R i , C p ) represents rule R i In Category C p The information gain on λ measures the contribution of the rule to the prediction of a specific category; p and v p is the weight factor, ranging from 0.1 to 1.0.

[0214] Assume that a financial institution is analyzing a set of multidimensional data involving 10,000 customers, in which multiple association rules R1, R2 and R3 have been identified. The institution hopes to evaluate the effectiveness of these rules through the rule gain contribution function and filter out rules with lower contributions.

[0215] For rule R1, its entropy change Δε(R1) is set to 0.15, indicating that the rule can significantly reduce uncertainty in the data set. The weight factor λ1 is set to 0.7.

[0216] The information gain IG(R1, C1) of rule R1 on category C1 is set to 0.25, indicating that the rule contributes more to the prediction of high-risk customer behavior. The weight factor v1 is set to 0.6.

[0217] Substitute into the calculation function middle:

[0218]

[0219] Rule gain contribution function The calculated result is 0.255, indicating that rule R1 has a high contribution to the overall data mining task.

[0220] Embodiment 5:

[0221] In the multi-tuple data optimization project, a mechanism is designed to trigger the adaptive adjustment of the model. By quantifying the feature representation and dynamic distribution changes of the multi-tuple data, the model can respond to the changes in the data in a timely manner. First, each data batch is characterized, high-dimensional features are extracted, and the feature space is constructed. Then, the sliding window technology is used to dynamically calculate the distribution difference between the new data batch and the previous data batch in the feature space. In order to quantify this difference, the distribution deviation indicator function is defined:

[0222]

[0223] Where X and Y represent the feature representations of the current batch of data and the previous batch of data, respectively, and α i It is a weight adjustment factor ranging from 0.5 to 2.0, which is used to dynamically adjust the weights of different feature dimensions.

[0224] Analyze a transaction data set containing 10,000 customers. The feature dimensions in the data set include stock trading volume x1, trading frequency x2, and market sentiment index x3. For the new data batch X and the previous data batch Y, the feature representation derivatives of the stock trading volume are set to and The derivatives of the feature representation of transaction frequency are and

[0225] Set the weight adjustment factors α1 = 1.2, α2 = 0.8 and substitute them into the calculation function middle:

[0226]

[0227] The calculations show that:

[0228]

[0229] Distribution Deviation Indicator Function The calculated result is 0.08, indicating that the distribution change in the feature space between the new data batch X and the previous data batch Y is relatively small.

[0230] Finally, the embodiment determines that when the monitored data distribution changes exceed the set adaptive threshold, the model's adaptive adjustment is triggered. This threshold is not fixed, but is updated in real time according to the change trend of data distribution and historical fluctuations, so that the model is always kept in the best state.

[0231] First, set the initial threshold τ0, and then adjust the threshold function To update it dynamically:

[0232]

[0233] Among them, τ0 is the initial threshold, β j and λ j is the adjustment coefficient, with a value range of 0.1 to 1.0 and 0.01 to 0.1, It is the distribution deviation trend function, which indicates the distribution change trend between data batches X and Y.

[0234] Processing a set of transaction data involving 10,000 customers, the feature dimensions in the data include stock trading volume, trading frequency, and market sentiment index. The institution sets the initial threshold τ0 to 0.05 and hopes to adjust this threshold in real time based on the distribution changes of daily data batches.

[0235] For two consecutive data batches X and Y, set their distribution deviation trend function It is expressed as the rate of change of daily data volatility. When the current time is set to t = 2 days, The adjustment coefficients β1=0.7 and λ1=0.05 are set.

[0236] Substitute into the calculation function middle:

[0237]

[0238] The calculations show that:

[0239]

[0240] Threshold adjustment function The calculated result is 0.0689, which shows that the threshold value increases slightly as the trend of data distribution change increases.

[0241] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A large-scale data mining method based on multi-group data optimization, characterized by The following steps are involved: S1. Multidimensional feature selection and extraction: Firstly, by designing a feature evaluation mechanism based on information gain and mutual information, the importance of each dimension feature in the multi-dimensional data is analyzed and ranked; S1.1, then use the dynamic feature selection algorithm to select representative feature sets according to the feature weights and their contribution to the data mining task; S1.2, based on S1.1, further design the feature extraction algorithm to generate new high-order combined features by combining the selected features; S2. Efficient clustering analysis based on multi-group features: On the dataset after feature selection and extraction, an improved clustering algorithm is applied, combining the advantages of density clustering and hierarchical clustering, to identify structural clusters in high-dimensional multivariate data; S2.1, using a distributed computing framework to split the clustering process into multiple parallel tasks; the clustering algorithm includes an adaptive optimization function to dynamically adjust clustering parameters according to the internal structure and data distribution of the clustering results; S3. Cross-dimensional association rule mining: Based on the clusters obtained by cluster analysis, a cross-dimensional association rule mining algorithm is designed to reveal cross-dimensional association relationships in multi-tuple data; S3.

1. Using a parallelized association rule calculation engine, it supports mining interesting and confident association rules in large-scale data sets. S3.

2. At the same time, a hierarchical association rule screening mechanism is introduced to filter out redundant or low-value rules, and combined with the characteristics of multi-group data, an association pattern with strong explanatory power is output; S4. Construction and optimization of hybrid prediction model: Based on the results of steps S1-S3, a hybrid prediction model integrating multivariate regression and deep learning technology is constructed; the multidimensional information in the multivariate data and the mined association rules are used to improve the prediction accuracy of the target variable; S4.1, dynamically adjust model parameters to respond to changes in data distribution through adaptive model training algorithms; S4.

2. Use integrated learning methods to further improve the stability and reliability of prediction results through collaborative prediction of multiple models.

2. The large-scale data mining method based on multi-tuple data optimization according to claim 1 is characterized in that The dynamic feature selection algorithm comprises the following steps: S1. Feature weight calculation: Based on the goal of data mining tasks, by defining the function Φ(f i ,Y) is used to measure the feature f i The correlation between the target variable Y, where Φ(f i ,Y) is expressed as follows: in, is the information gain function, is the mutual information function, α j and β j is the weight coefficient, which indicates the importance weight of each method; the initial weight of each feature is calculated by the formula i ; At the same time, through the function Λ(ω i ) implements feature importance evaluation based on the gradient boosting model, and comprehensively calculates Φ(f i , Y) and Λ(ω i ) Calculate the initial weight Ω of the feature i : Oh i =γ·Φ(f i ,Y)+δ·Λ(ω i ) Among them, γ and δ are weight adjustment factors; S2. Special contribution evaluation: Based on the feature weight calculation, the function Ψ(Ω i , Θ) is used to evaluate the feature Ω i The actual contribution to the data mining task Θ, where Ψ(Ω i , θ) is expressed as follows: in, is the loss function, which indicates the impact of feature selection including accuracy and F1 score on model performance. is the model performance change function, κ is the adjustment coefficient; through the integral formula, the low contribution features are gradually eliminated, and according to Ψ(Ω i ,Θ) results update feature set; S3. Dynamic feature selection mechanism: Based on the feature weight Ω i and contribution evaluation Ψ(Ω i , Θ), design a dynamic feature selection function Θ dyn (Ω i , t), where t is the time variable, the function is expressed as follows: Among them, ∈(t) and η(t) are time-dependent weight factors, is the adjustment function for the interaction between features; through the dynamic feature selection function Θ dyn (Ω i ,t), adjust the feature set in real time according to the model performance, changes in data distribution, and interactions between features.

3. The large-scale data mining method based on multi-tuple data optimization according to claim 1 is characterized in that Cluster methods for identifying structure in the multi-tuple data: S1. First, an improved density clustering algorithm is applied in the high-dimensional space of multi-tuple data. The distance metric between different dimensions is adjusted through a dynamic neighborhood search strategy, so that the density clustering algorithm can identify the cluster structure in the high-dimensional space. S2. Based on the preliminary density clustering results, a hierarchical clustering algorithm based on the density clustering results is applied to perform hierarchical analysis and dynamic adjustment of the identified clusters by constructing a tree structure; S3. Finally, by designing a multi-level clustering integration framework, the results of density clustering and hierarchical clustering are integrated, and through repeated iterations and optimization, the cluster structure in high-dimensional data is adaptively identified and optimized.

4. The large-scale data mining method based on multi-tuple data optimization according to claim 1 is characterized in that The construction method for revealing cross-dimensional association relationships: S1. First, based on the initially obtained clusters, the structural information of high-dimensional data is projected into low-dimensional space through multi-dimensional projection and mapping algorithms to generate a feature vector set that integrates multi-dimensional information; S2. Then, in a unified feature space, features of different dimensions are combined to generate cross-dimensional feature combinations, and a feature screening mechanism based on cluster density features is introduced; S3. Finally, based on the selected cross-dimensional feature combination, design an association rule mining algorithm that combines frequent pattern growth and Apriori algorithm; define the calculation function of cross-dimensional support and confidence It is expressed as follows: Among them, Z is the cross-dimensional feature combination set, φ i (Z) and θ j (Z) is the measurement function of support and confidence, and are the support and confidence weight functions in their respective dimensions.

5. The large-scale data mining method based on multi-tuple data optimization according to claim 1 is characterized in that The hierarchical association rule screening mechanism construction mode includes: S1. First, the mined association rules are hierarchically represented and the rules are layered according to their coverage, support, confidence and importance in the data. Specifically, the rule hierarchical function is defined as Where R i is a set of rules, and the function is expressed as follows: Among them, Cov(R i , D j ) represents rule R i In the data dimension D j Coverage on Sup(R i ) is the support of the rule, Conf(R i ) is the confidence of the rule, Representation rule R i For the target variable T j The influence of Depth(R i ) represents the nesting depth of rules, α j , β j , γ j is the weight adjustment factor; S2. Organize all association rules into an orderly hierarchical structure through hierarchical functions; introduce multi-tuple data characteristics on the basis of hierarchical representation to further screen the rules; S3. For redundant or low-value rules, a filtering mechanism based on information gain and entropy value changes is introduced.

6. The large-scale data mining method based on multi-tuple data optimization according to claim 1 is characterized in that The change strategy of the response data distribution includes: S1. By designing a real-time data distribution monitoring module, using sliding window technology and statistical analysis including mean, variance, and covariance matrix, the data distribution is continuously tracked; S1.

1. When a new batch of data enters the model, the distribution difference between the batch of data and the previous data is calculated, and the change is quantified by the distribution deviation indicator; if the change in data distribution exceeds the preset threshold, the adaptive adjustment of the model is triggered; S2, when significant changes in data distribution are detected, the adaptive model training algorithm adjusts the key parameters of the model; S2.

1. Adjust parameter settings based on preset strategies or dynamically optimize through machine learning algorithms, and iteratively optimize based on the performance of the model under the new data distribution; S3. Based on the adaptive parameter adjustment mechanism, use online learning algorithms including SGD with warm restarts or Adagrad to gradually adapt the model to the new data distribution without resetting parameters; S3.

1. Regularization terms are used to control model complexity, and early stopping strategies are used to avoid excessive fluctuations in model performance.

7. The large-scale data mining method based on multi-tuple data optimization according to claim 6 is characterized in that The process of triggering the adaptive adjustment mechanism of the model: S1. First, perform feature representation on multi-tuple data, extract high-dimensional features and construct feature space; dynamically calculate the distribution difference between new data batches and previous data batches in feature space through sliding window technology; S2. According to the distribution changes monitored, preset the adaptive threshold τ(t), and update it in real time based on the data distribution change trend and historical fluctuations; the specific threshold adjustment function It is expressed as follows: Among them, τ0 is the initial threshold, β j and λ j is the adjustment factor, is the distribution deviation trend function; When changes in data distribution are detected Exceeding the adaptive threshold , triggers the adaptive adjustment process of the model; specifically, it re-evaluates and adjusts the key parameters of the model, including the learning rate and regularization strength, and dynamically updates the model parameters through the online learning algorithm.

Images