A data intelligent analysis and receiving method based on supervision system

The method improves regulatory system efficiency by preprocessing data, applying information gain and correlation analysis, and optimizing feature selection, addressing inefficiencies in traditional systems.

CN118673415BActive Publication Date: 2024-12-03NANJING CHISCDC SOFTWARE TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411141159.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-12-03
Estimated Expiration
2044-08-20

AI Technical Summary

Technical Problem

Traditional regulatory systems face inefficiencies in data processing, inability to adapt to rapid data growth and complexity, and inflexibility in responding to new regulatory rules, leading to low intelligence and resource-intensive operations.

Method used

A method involving data intelligent analysis and reception, including data preprocessing, information gain and correlation analysis, recursive feature elimination, and performance selection information analysis to optimize feature selection and adjust parsing strategies.

Benefits of technology

Enhances the accuracy and reliability of regulatory decisions by identifying critical features, reducing redundancy, and optimizing system performance and response speed through intelligent feature analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118673415B_ABST
    Figure CN118673415B_ABST
Patent Text Reader

Abstract

The present invention discloses a data intelligent parsing and receiving algorithm based on a regulatory system, relates to the field of data parsing technology, and is used to solve the problem of low efficiency of data intelligent parsing and receiving of a regulatory system. The algorithm comprises the following steps: screening and preprocessing the regulatory system data by setting preliminary screening conditions to ensure data quality, applying information gain analysis and correlation analysis to evaluate the relationship between features, thereby determining the main analysis features that affect regulatory decisions, and using a recursive feature elimination method to further optimize feature selection, ensuring that the selected feature set contains key features that improve model performance, and analyzing the performance selection information generated by the feature set, accurately judging the parsing effect of the existing regulatory system data, and adjusting the parsing strategy based on actual performance, thereby significantly improving the accuracy and reliability of regulatory decisions, optimizing regulatory system performance, and enhancing the response speed and processing capacity of the regulatory system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a method for intelligent analysis and reception of data based on a supervision system. Background Art

[0002] With the development of information technology and data analysis technology, the regulatory system has gradually shifted from traditional manual supervision to automated and intelligent supervision. This transformation involves the collection and processing of large amounts of real-time data so that regulators can quickly identify violations and respond;

[0003] The existing technology has the following shortcomings: the regulatory system relies on efficient and accurate data processing capabilities to ensure regulatory compliance and respond to potential issues in real time. However, traditional data processing methods are often slow and difficult to adapt to the rapid growth of data scale and complexity. In addition, these regulatory systems are usually inflexible and cannot quickly adapt to new regulatory rules or changing data characteristics, resulting in a low level of intelligence in the regulatory system and the need for more resources.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0005] The purpose of the present invention is to provide a method for intelligently parsing and receiving data based on a supervision system, so as to solve the problem of low efficiency of data parsing and receiving of the supervision system in the background technology.

[0006] In order to achieve the above object, the present invention provides the following technical solution: a data intelligent analysis and receiving method based on a supervision system, comprising the following steps:

[0007] Obtain data from the regulatory system, identify the data source, set initial screening conditions to filter the data, and perform data preprocessing on the filtered data;

[0008] Conduct information gain analysis and correlation analysis on the preprocessed data to evaluate the relationship between features in the data and determine the main analytical features of the data;

[0009] Use recursive feature elimination to select features that affect regulatory decisions, and summarize the features as a feature set. Obtain the performance selection information generated during the feature set selection analysis process, analyze the performance selection information, determine the accurate situation of data parsing in the regulatory system, and adjust the parsing strategy based on the results obtained.

[0010] Preferably, obtaining the data from the supervision system, identifying the data source, setting the initial screening conditions to filter the data, and preprocessing the filtered data include:

[0011] Data sources include real-time data streams, databases, and APIs;

[0012] Perform time filtering to exclude data that is not within the regulatory analysis time window, perform geographic location filtering to filter out data within the area based on geographic location restrictions, and perform rule-driven filtering to automatically filter data that does not conform to business rules;

[0013] Perform data preprocessing, fill or delete missing data, replace missing values ​​with field mean or median, and use Z-score to detect and handle outliers.

[0014] Preferably, information gain analysis and correlation analysis are performed on the preprocessed data to evaluate the relationship between the features in the data and determine the main analysis features of the data. The specific steps include:

[0015] Use information gain to select data features and calculate the entropy of the entire data set;

[0016] For each feature, calculate the conditional entropy after segmentation under the feature, calculate the information gain of the feature based on the conditional entropy and the entropy of the data set, and select the feature with the highest information gain as the analysis feature;

[0017] The analysis features selected by data gain are used as feature data for correlation analysis to evaluate the relationship between each feature variable in the feature data;

[0018] Centralize the feature data into the same data framework, including the target feature variables and potential explanatory feature variables;

[0019] For continuous characteristic variables, correlation coefficients were calculated using Pearson's correlation coefficient, and for categorical characteristic variables, correlation coefficients were calculated using Spearman's rank correlation coefficient;

[0020] The main analysis features in the feature data are determined based on the calculated correlation coefficient.

[0021] Preferably, recursive feature elimination is used to select features that affect regulatory decisions, and the features are summarized as a feature set. The specific steps are as follows:

[0022] Select an importance scoring model, including linear regression, logistic regression, or support vector machine models;

[0023] Train the selected model on the main analysis features and use the data set containing all the main analysis features to train the model;

[0024] After the model training is completed, the importance of each main analysis feature is evaluated. In the linear model, the absolute value of the feature coefficient is used as a measure of importance; in the decision tree model, the feature importance is evaluated using a criterion based on splitting to reduce impurity;

[0025] Based on the importance scores of the features, the main analysis features with the lowest scores were removed;

[0026] After removing the features, the remaining main analysis features are used as new features and as a feature set to retrain the model, and the iterative process is repeated. After each iteration, the main analysis feature with the lowest importance score is removed, and the iteration is stopped when the preset number of features is reached.

[0027] Preferably, the performance selection information generated in the feature set selection analysis process is obtained, and the performance selection information is analyzed to determine the accurate situation of data parsing in the supervision system. The specific steps are as follows:

[0028] Acquire performance selection information generated in the process of feature set selection analysis, the performance selection information includes feature contribution information and feature stability information;

[0029] Feature contribution information includes feature association contribution index, and feature stability information includes feature difference stability index:

[0030] The feature correlation contribution index and feature difference stability index are combined to generate analytical evaluation coefficients;

[0031] The feature association contribution index and feature difference stability index are proportional to the analytical evaluation coefficient.

[0032] Preferably, the analysis strategy is adjusted according to the obtained results, and the specific steps include:

[0033] Compare the generated analytical evaluation coefficient with the push control threshold;

[0034] If the analysis evaluation coefficient is greater than the push control threshold, a data analysis stability signal is generated, indicating that the data analysis status is stable and no additional data analysis strategy adjustment of the regulatory system is required;

[0035] If the analysis evaluation coefficient is less than or equal to the push control threshold, a data analysis control signal is generated, and the data analysis strategy of the regulatory system is optimized or adjusted.

[0036] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0037] The present invention screens and preprocesses the regulatory system data by setting initial screening conditions to ensure data quality. Then, information gain analysis and correlation analysis are applied to evaluate the relationship between features, so as to accurately determine the main analytical features that have the greatest impact on regulatory decisions, and recursive feature elimination methods are used to further optimize feature selection to ensure that the selected feature set contains the most critical features for improving model performance. Through in-depth analysis of the performance selection information generated by the feature set, it is not only possible to accurately judge the analytical effect of the existing regulatory system data, but also to adjust the analytical strategy based on actual performance, thereby significantly improving the accuracy and reliability of regulatory decisions. It also effectively reduces data redundancy, optimizes system performance, and enhances the response speed and processing capacity of the regulatory system through sophisticated data processing and intelligent feature analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0039] Figure 1 The present invention is a flow chart of a method for intelligent analysis and reception of data based on a supervision system. DETAILED DESCRIPTION

[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art.

[0041] Embodiment 1: The present invention provides Figure 1 A data intelligent analysis and receiving method based on a supervision system is shown, and the algorithm includes the following steps:

[0042] Obtain data from the regulatory system, identify the data source, set initial screening conditions to filter the data, and perform data preprocessing on the filtered data;

[0043] Conduct information gain analysis and correlation analysis on the preprocessed data to evaluate the relationship between features in the data and determine the main analytical features of the data;

[0044] Use recursive feature elimination to select features that affect regulatory decisions, and summarize the features as a feature set. Obtain the performance selection information generated during the feature set selection analysis process, analyze the performance selection information, determine the accurate situation of data parsing in the regulatory system, and adjust the parsing strategy based on the results obtained.

[0045] Identify data sources, obtain data from regulatory systems, and determine and access multiple specific data sources, including real-time data streams, databases, APIs, etc.

[0046] Establish a secure data receiving channel, such as using Apache Kafka to capture and transmit data streams, and store raw data in a data lake, such as Hadoop HDFS, for further processing;

[0047] Filter the data, set initial screening conditions, filter by timestamp, filter by region, etc., and exclude data that is not related to the regulatory objectives;

[0048] Perform time filtering to exclude data that is not within the regulatory analysis time window, and only analyze transaction data within the last 30 days, ignoring records outside this time window;

[0049] Geolocation filtering is to filter out data within a specific area based on geographic location restrictions, and only analyze user behavior data in a specific range or region. Filtering rules can be set based on geographic information fields (such as country codes) when receiving data.

[0050] Perform rule-driven filtering and apply predefined business rules to automatically filter out data records that do not meet the standards, as follows:

[0051] Business rule application automatically filters data that does not conform to predefined business rules. Any single transaction exceeding the set maximum transaction amount needs to be marked and reviewed separately;

[0052] For example, any record with a transaction amount outside the normal range will be flagged as a potential anomaly and further analyzed.

[0053] Perform data preprocessing to fill or delete missing data, for example, replace missing values ​​with the field mean or median, use statistical methods (such as Z-score) to detect and process outliers, correct them or mark them as outliers for further analysis by experts;

[0054] Apply Min-Max normalization or Z-score standardization to unify data from different sources and ranges to the same scale for easy comparison and analysis; Encode categorical data, such as using One-Hot Encoding or Label Encoding;

[0055] Perform consistency checks on the data to ensure consistency and logical correctness between data fields, for example, the purchase date must be earlier than or equal to the shipping date.

[0056] Use information gain to select the most relevant data features, that is, determine the importance of data features for the analysis task and reduce the data dimension. The specific steps are as follows:

[0057] Evaluate the information contribution of each feature to the output variable, select features with high information gain, that is, calculate the entropy of the entire data set. For each feature, calculate the conditional entropy after segmentation under this feature, and calculate the information gain. The calculation expression is: Where T represents the entire dataset, a is the feature under consideration, Values(a) is all possible values ​​of feature a in the dataset, and T v is the data subset when feature a takes the value v, H(T) is the entropy of data set T, H(T v ) is a subset T v The entropy of |T v | is a subset of T v is the size of, |T| is the size of the dataset T, is the conditional entropy under feature a;

[0058] The calculation formula for entropy is: Among them, p i is the probability of category i appearing in the total data set T, and n is the probability of category i appearing in the total data set T;

[0059] According to the calculated information gain, the feature with the highest information gain is selected as the analysis feature.

[0060] Conduct correlation analysis on the features screened out through data gain, evaluate the relationship between the variables in the feature data, analyze the correlation of the regulatory target feature variables, and thus determine the importance of the feature variables to the regulatory system analysis and determine the feature variables that can be excluded, and then verify the correlation between these features and the target feature variables, helping to confirm the actual influence and correlation direction of the features. The specific steps are as follows:

[0061] All feature data that need to be analyzed are concentrated in a data framework, including target feature variables and potential explanatory feature variables, and missing values ​​and outliers are processed so that the quality of the data will not affect the results of the correlation analysis;

[0062] For continuous characteristic variables, the Pearson correlation coefficient is used to calculate the correlation coefficient, and the calculation expression is: Among them, x i is the explanatory characteristic variable of the i-th observation, y i is the target variable for the ith observation, is the sample mean of the explanatory feature variable, is the sample mean of the target feature variable, and the value range of r is -1 to 1, where 1 indicates a perfect positive correlation, -1 indicates a perfect negative correlation, and 0 indicates no correlation.

[0063] For categorical feature variables, Spearman rank correlation coefficient was used for correlation analysis and the correlation coefficient was calculated:

[0064] The Spearman rank correlation coefficient is to assign ranks to all observed values ​​of each characteristic variable. If there are tied values ​​in the data, all are assigned the average rank. The rank difference of each pair of observed values ​​is then calculated to calculate the Spearman correlation coefficient. The calculation expression is: Among them, d i is the rank difference of the ith pair of observations, and n is the total number of observations.

[0065] The importance of features can be determined based on the correlation coefficient. The number of features in the data can be reduced by setting a correlation threshold. The main analysis features in the feature data can be determined based on the calculated correlation coefficient, which can reduce the computational cost of subsequent analysis and prediction. When processing large-scale data sets, computing resources may be a limiting factor, and reducing the number of features can speed up data processing, especially in systems that require real-time processing.

[0066] Construct new features according to the needs of the regulatory system, and use recursive feature elimination to select the features that have the greatest impact on regulatory decisions. That is, use recursive feature elimination to optimize the performance of the model by gradually eliminating relatively unimportant main analysis features. The specific process is as follows:

[0067] Choose a model that can provide feature importance scores, such as linear regression, logistic regression, or support vector machine (SVM), which assigns a weight or coefficient to each feature during training.

[0068] The selected model is trained with the main analysis features, that is, the model is trained using a data set containing all the main analysis features;

[0069] After the model training is completed, the importance of each main analysis feature is evaluated. In the linear model, the absolute value of the feature coefficient can be directly used as a measure of importance; in the decision tree model, the standard based on splitting to reduce impurity can be used to evaluate the feature importance;

[0070] Based on the feature importance scores, remove the main analysis features with the lowest scores. Typically, this step removes one or more main analysis features with the lowest importance, which are usually the main analysis features that contribute the least to the model performance;

[0071] After removing the features, use the remaining main analysis features as new features and as the feature set to retrain the model, repeating the above process, removing the main analysis features with the lowest importance score after each iteration until the preset number of features is reached or the model performance reaches a stable level;

[0072] Finally, the main analysis features under screening are retained and summarized as the feature set, and the feature set is used in the subsequent model to predict the target variable.

[0073] By verifying the feature set selection process to determine the accuracy of feature selection, and then intelligently analyzing the data, the effectiveness and accuracy of the supervision system can be improved. The specific steps are as follows;

[0074] Acquire the performance selection information generated in the process of feature set selection and analysis, analyze the performance selection information, determine the accuracy of data parsing in the regulatory system, select the feature set that meets the requirements and analyze it in the environment of the regulatory system, so as to improve the overall performance of the regulatory system and enhance the timeliness and effectiveness of regulatory activities. The performance selection information includes feature contribution information and feature stability information;

[0075] The feature contribution information includes the feature association contribution index, and the feature stability information includes the feature difference stability index. After collection, the feature association contribution index and the feature difference stability index are calibrated as FACI and FDII, respectively.

[0076] The feature association contribution index is used to measure the influence and relevance of the selected feature set on the model prediction target variable. The feature association contribution index combines the contribution of the feature to the model performance and the correlation between the feature and the target variable. The specific impact is as follows:

[0077] Optimize and improve model efficiency: By identifying and retaining the features that have the greatest impact on the prediction target, unnecessary features in the model can be reduced, thereby simplifying the model structure and improving computational efficiency. The feature correlation contribution index provides a quantitative basis for feature selection, helping data scientists and analysts decide which features are worth retaining and which can be excluded. This is especially useful when dealing with high-dimensional data to avoid overfitting.

[0078] Enhance model interpretability: By clarifying the features that contribute the most to the model and understanding the relationship between these features and the target variable, it can help explain the model's decision-making process; the feature correlation contribution index provides an objective standard to evaluate the contribution of features to the model, making the decision process more data-driven and reducing subjective bias.

[0079] The logic for obtaining the feature association contribution index is as follows:

[0080] Divide the datasets containing the feature set and excluding the feature set into K subsets of the same size, and calculate the model performance index by k-fold cross validation on the datasets containing the feature set and excluding the feature set to obtain the model performance index Per when the feature set is included with Compared with the model performance index Per without feature set base, calculate the standard value of the feature set, the calculation expression is: Calculate the standard value of non-feature set, the calculation expression is: Among them, Per i represents the performance of the i-th fold;

[0081] Get the number of samples n in the dataset containing the feature set with The number of samples n in the dataset without the feature set base , calculate the characteristic error: Compute the non-characteristic error: Calculate the feature association contribution index, the calculation expression is:

[0082] It should be noted that the model performance index can be obtained by calculating the accuracy. The model performance index can be obtained by selecting k-fold cross-validation to calculate the accuracy of the model, that is, dividing the data set used for verification into k subsets of the same size, and verifying the subsets in turn.

[0083] The feature difference stability index is used to measure the performance differences and selection stability of the selected feature set in different models or data subsets, that is, to measure the selection frequency of the feature in multiple feature selection processes, reflect the consistency of the feature being selected in different models or cross-validation processes, evaluate the variability of the influence or importance score of the feature in different models or environments, and reflect the stability of the impact of the feature on model performance in different situations. It can be measured by measuring the frequency of the feature being selected as an important feature in multiple iterations or different sub-datasets, and measuring the stability of the importance of the feature in different situations. By using the feature difference stability index, organizations can more accurately evaluate and optimize the feature set in their regulatory model, thereby improving the effectiveness and accuracy of the overall system. The feature difference stability index affects the following aspects:

[0084] Improve model generalization ability: By identifying features that are stable under multiple models and conditions, we can build models with better generalization ability and reduce the risk of overfitting the model on a specific data set;

[0085] Optimize feature engineering: The feature difference stability index helps data stakeholders identify those features that have a greater impact on model results but are unstable, and guides them to perform feature engineering, such as feature transformation, normalization, or other preprocessing steps, to improve the stability of these features.

[0086] The logic for obtaining the feature difference stability index is as follows:

[0087] In the feature selection process, record the features selected as important features in each iteration or model training, and calculate the feature selection frequency. The calculation expression is: In the formula, represents the indicator function. If feature f is selected as an important feature in the jth iteration, S j represents the feature set selected in the jth iteration, then the function value is 1, otherwise it is 0, and M is the total number of iterations;

[0088] Obtain the model training result data X after iteration, obtain the sample points in each training result data, and calculate the mean of the training result data And standard deviation s, calculate the skewness of the training result data, the calculation expression is: Calculate the kurtosis of the characteristic distribution, the calculation expression is: Among them, X t represents the t-th sample point, T represents the total number of samples, and the characteristic difference stability index is calculated. The calculation expression is: FDII = e -p(f) *(SKE+KUR).

[0089] It should be noted that the feature selection process is the recursive feature elimination or random forest feature importance evaluation process as mentioned above.

[0090] The obtained feature correlation contribution index and feature difference stability index are normalized and calculated simultaneously to obtain the analytical evaluation coefficient, which is calibrated as K x , based on the formula: In the formula, α and γ are the preset proportional coefficients of the feature association contribution index and the feature difference stability index, respectively, and both α and γ are greater than 0;

[0091] It should be noted that the formula is obtained by technicians in this field by collecting multiple groups of sample data and setting corresponding preset proportional coefficients for each group of sample data. The specific calculation process is: substituting the acquired feature association contribution index and feature difference stability index into the formula to form an equation group, and then screening and averaging the calculated preset proportional coefficients to obtain the preset proportional coefficients. Since each group of sample data is different, the specific values ​​of the preset proportional coefficients under different sample data are also different.

[0092] It can be seen from the formula that the larger the feature association contribution index and the larger the feature difference stability index, that is, the larger the value of the analytical evaluation coefficient, the stronger the improvement of the model performance and the correlation with the target variable of the selected feature set, and the more stable and consistent the performance of the feature in different model configurations or data subsets. The evaluated features not only have a significant positive impact on the predictive performance of the model, but also show a high degree of consistency and reliability under different model settings or data conditions. Therefore, they should be given priority in the decision-making analysis process of the regulatory system.

[0093] By analyzing the analytical evaluation coefficients, it can help select features that both help improve model accuracy and perform consistently across different model configurations, helping to meet the data analysis requirements of the regulatory system. By reducing model errors and improving the consistency of predictions, it can provide more reliable results in practical applications, thereby improving the effectiveness of the entire regulatory system;

[0094] The smaller the feature association contribution index and the feature difference stability index, that is, the smaller the performance value of the analytical evaluation coefficient, indicates that the selected feature set may need to be re-examined and evaluated for its applicability and effectiveness, and further data exploration and feature engineering are needed, or the introduction of new data sources or the replacement of feature extraction methods may be considered to improve the performance of the model;

[0095] A lower feature association contribution index means that the feature has little effect on improving model performance or is not significantly correlated with the target variable, which may indicate that the feature set is not very effective in helping the model make accurate predictions. A lower feature difference stability index indicates that the performance of the screened feature set varies greatly in different model configurations or data subsets, that is, its influence and importance are inconsistent, which affects its overall stability assessment.

[0096] When both the correlation contribution and differential impact of features are low, it may indicate that these features are not suitable for the current model or for solving the current problem because there is no substantial information interaction between these features and the target variable, or the feature set and the variable may exhibit inconsistent behaviors in different contexts or data environments.

[0097] The generated analysis evaluation coefficient is compared with the push control threshold, and different signals are generated according to the comparison result, and the different signals are respectively generating a data analysis control signal and a data analysis stability signal;

[0098] After obtaining the generated parsing evaluation coefficient, the generated parsing evaluation coefficient is compared with the push control threshold. If the parsing evaluation coefficient is greater than the push control threshold, a data parsing stability signal is generated, and no additional parsing strategy adjustment is required, indicating that the feature set not only contributes significantly to the model's predictive performance, but also shows a high degree of stability in different data sets and model configurations. The data parsing stability signal shows that these features are reliable in providing continuous and consistent prediction results. The feature set performs well in analysis and can provide solid data support for decision-making, which means that the model's input data quality is good and the feature extraction and processing methods are appropriate. In a business or operational environment, being able to rely on the stability and accuracy of the model output helps optimize processes, reduce errors, and improve efficiency. For example, in supply chain management, accurate demand forecasts can minimize inventory costs and out-of-stock risks.

[0099] The generated data analysis stable signal means that the current regulatory system meets the requirements in terms of data intelligent analysis and has a high degree of reliability in supporting complex decision-making. The regulatory system can maintain consistency in predictive performance and results while providing continuous services.

[0100] If the analysis evaluation coefficient is less than or equal to the push control threshold, a data analysis control signal is generated and the analysis strategy needs to be adjusted, indicating that the selected feature set has limited contribution to the model's predictive performance or is weakly correlated with the target variable. At the same time, this may also indicate that the feature set exhibits significant inconsistency in different model configurations or data subsets, thereby affecting its overall reliability, indicating that the regulatory system's data analysis strategy needs to be optimized or adjusted, which can include changing model parameters, trying different algorithms, or using more advanced data preprocessing techniques to enhance model performance, thereby improving the effectiveness of the regulatory system's analysis of data:

[0101] When a data analysis control signal is generated, the current feature set may need further review and optimization to improve the model's performance and accuracy by identifying and resolving potential problems, ensuring that after subsequent feature sets are input into the model, the model's performance in actual operations can meet the expected standards, thereby improving the data analysis stability and accuracy of the regulatory system.

[0102] When a data parsing control signal is generated, an in-depth analysis of the currently used feature set is performed to identify which features have failed to contribute effectively or caused performance degradation; based on the results of the feature analysis, model parameters are adjusted or optimized, which may include changing the model structure or trying different algorithms to improve performance; data preprocessing is enhanced, and data cleaning and preprocessing steps are improved, such as better outlier processing, feature normalization, and encoding methods; features are added or replaced. Based on the results of feature evaluation, new features may need to be introduced or features that do not perform well may need to be replaced to enhance the model's predictive ability. After adjustment, a comprehensive model performance test is performed, using cross-validation or other validation methods to ensure the effectiveness of improvement measures, and a continuous model monitoring system is implemented to monitor changes in model performance over time to ensure that expected standards are continuously met and to prevent the regulatory system from failing to meet efficiency standards.

[0103] When a stable signal of data analysis is generated, if the model is stable and the performance is up to standard, the model can be deployed to the production environment for actual decision support or prediction tasks; after the model is deployed, continue to monitor the actual performance of the model to ensure that it remains efficient and accurate in actual applications

[0104] Establish feedback mechanisms to collect feedback from users and business processes for further optimization and adjustment of models; record and analyze the results and learnings of model deployment, which can help future model development and decision making.

[0105] It should be noted that the relevant threshold information in this embodiment is pre-set by professionals and will not be explained in detail here.

[0106] The present invention screens and preprocesses the regulatory system data by setting initial screening conditions to ensure data quality. Then, information gain analysis and correlation analysis are applied to evaluate the relationship between features, so as to accurately determine the main analytical features that have the greatest impact on regulatory decisions, and recursive feature elimination methods are used to further optimize feature selection to ensure that the selected feature set contains the most critical features for improving model performance. Through in-depth analysis of the performance selection information generated by the feature set, it is not only possible to accurately judge the analytical effect of the existing regulatory system data, but also to adjust the analytical strategy based on actual performance, thereby significantly improving the accuracy and reliability of regulatory decisions. It also effectively reduces data redundancy, optimizes system performance, and enhances the response speed and processing capacity of the regulatory system through sophisticated data processing and intelligent feature analysis.

[0107] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0108] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0109] It should be noted that, in this article, if there are relational terms such as first and second, etc., they are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0110] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data intelligent analysis and receiving method based on a supervision system, characterized in that: The steps include: Obtain data from the regulatory system, identify the data source, set initial screening conditions to filter the data, and perform data preprocessing on the filtered data; Conduct information gain analysis and correlation analysis on the preprocessed data to evaluate the relationship between features in the data and determine the main analytical features of the data; Use recursive feature elimination to select features that affect regulatory decisions, and aggregate the features as feature sets, obtain performance selection information generated during the feature set selection analysis process, and analyze the performance selection information to determine the accurate situation of data parsing in the regulatory system, and adjust the parsing strategy based on the results obtained; Perform information gain analysis and correlation analysis on the preprocessed data to evaluate the relationship between the features in the data and determine the main analysis features of the data. The specific steps include: Use information gain to select data features and calculate the entropy of the entire data set; For each feature, calculate the conditional entropy after segmentation under the feature, calculate the information gain of the feature based on the conditional entropy and the entropy of the data set, and select the feature with the highest information gain as the analysis feature; The analysis features selected by data gain are used as feature data for correlation analysis to evaluate the relationship between each feature variable in the feature data; Centralize the feature data into the same data framework, including the target feature variables and potential explanatory feature variables; For continuous characteristic variables, correlation coefficients were calculated using Pearson's correlation coefficient, and for categorical characteristic variables, correlation coefficients were calculated using Spearman's rank correlation coefficient; Determine the main analysis features in the feature data based on the calculated correlation coefficient; Obtain the performance selection information generated during the feature set selection analysis process, and analyze the performance selection information to determine the accurate situation of data parsing in the supervision system. The specific steps are as follows: Acquire performance selection information generated in the process of feature set selection analysis, the performance selection information includes feature contribution information and feature stability information; Feature contribution information includes feature association contribution index, and feature stability information includes feature difference stability index: The feature association contribution index is used to measure the influence and relevance of the selected feature set on the model prediction target variable; The logic for obtaining the feature association contribution index is as follows: The datasets containing the feature set and excluding the feature set are divided into K subsets of the same size. The model performance index is calculated by k-fold cross validation on the datasets containing the feature set and excluding the feature set to obtain the model performance index Per when the feature set is included. with Compared with the model performance index Per without feature set base , calculate the standard value of the feature set, the calculation expression is: Calculate the standard value of non-feature set, the calculation expression is: Among them, Per i represents the performance of the i-th fold; Get the number of samples n in the dataset containing the feature set with The number of samples n in the dataset without the feature set base , calculate the characteristic error: Compute the non-characteristic error: Calculate the feature association contribution index, the calculation expression is: The feature difference stability index is used to measure the performance difference and selection stability of the selected feature set in different models or data subsets, that is, to measure the selection frequency of the feature in multiple feature selection processes; The logic for obtaining the feature difference stability index is as follows: In the feature selection process, record the features selected as important features in each iteration or model training, and calculate the feature selection frequency. The calculation expression is: In the formula, 1 {f∈Sj} represents the indicator function. If feature f is selected as an important feature in the jth iteration, S j represents the feature set selected in the jth iteration, then the function value is 1, otherwise it is 0, and M is the total number of iterations; Obtain the model training result data X after iteration, obtain the sample points in each training result data, and calculate the mean of the training result data And standard deviation s, calculate the skewness of the training result data, the calculation expression is: Calculate the kurtosis of the characteristic distribution, the calculation expression is: Among them, X t represents the t-th sample point, T represents the total number of samples, and the characteristic difference stability index is calculated. The calculation expression is: FDII = e -p(f) *(SKE+KUR); The obtained feature correlation contribution index and feature difference stability index are normalized and calculated simultaneously to obtain the analytical evaluation coefficient, which is calibrated as K x , based on the formula: In the formula, α and γ are the preset proportional coefficients of the feature association contribution index and the feature difference stability index, respectively, and both α and γ are greater than 0; The feature association contribution index and feature difference stability index are proportional to the analytical evaluation coefficient; Adjust the parsing strategy based on the results obtained. The specific steps include: Compare the generated analytical evaluation coefficient with the push control threshold; If the analysis evaluation coefficient is greater than the push control threshold, a data analysis stability signal is generated, indicating that the data analysis status is stable and no additional data analysis strategy adjustment of the regulatory system is required; If the analysis evaluation coefficient is less than or equal to the push control threshold, a data analysis control signal is generated, and the data analysis strategy of the regulatory system is optimized or adjusted.

2. According to claim 1, a data intelligent analysis and receiving method based on a supervision system is characterized in that: Obtain data from the regulatory system, identify the data source, set initial screening conditions to filter the data, and perform data preprocessing on the filtered data, including: Data sources include real-time data streams, databases, and APIs; Perform time filtering to exclude data that is not within the regulatory analysis time window, perform geographic location filtering to filter out data within the area based on geographic location restrictions, and perform rule-driven filtering to automatically filter data that does not conform to business rules; Perform data preprocessing, fill or delete missing data, replace missing values ​​with field mean or median, and use Z-score to detect and handle outliers.

3. According to claim 1, a data intelligent analysis and receiving method based on a supervision system is characterized in that: Use recursive feature elimination to select features that affect regulatory decisions and summarize the features as a feature set. The specific steps are as follows: Select an importance scoring model, including linear regression, logistic regression, or support vector machine models; Train the selected model on the main analysis features and use the data set containing all the main analysis features to train the model; After the model training is completed, the importance of each main analysis feature is evaluated. In the linear model, the absolute value of the feature coefficient is used as a measure of importance; in the decision tree model, the feature importance is evaluated using a criterion based on splitting to reduce impurity; Based on the importance scores of the features, the main analysis features with the lowest scores were removed; After removing the features, the remaining main analysis features are used as new features and as a feature set to retrain the model, and the iterative process is repeated. After each iteration, the main analysis feature with the lowest importance score is removed, and the iteration is stopped when the preset number of features is reached.

Citation Information

Patent Citations

  • A Financial Data Analysis Method Based on Machine Learning Models

    AU2020101475A4

  • Dynamic analysis and monitoring of machine learning processes

    CA3095517A1

  • Intelligent data early warning decision tree analysis method and system

    CN117349782A

  • Financial risk data management method based on machine learning

    CN117726439A

  • Medical abnormity violation big data risk early warning method based on unsupervised machine learning and integrated learning

    CN117764741A