High-missing-rate feature processing method and device

Abstract

The invention provides a high-missing-rate feature processing method and device, and the method comprises the steps: carrying out the preprocessing of original features in original feature data, obtaining the preprocessed transformation features, carrying out the multi-path embedding transformation of the original features based on a plurality of embedding transformation rules, and obtaining the transformation features of the original features; fusing a plurality of embedded vectors obtained by conversion into a target embedded vector as a first derivative feature; obtaining a comprehensive similarity matrix based on comprehensive similarity obtained by fusing multiple similarities between the target embedding vectors of any two original features; for each original feature, selecting the original feature of which the comprehensive similarity with the original feature meets a preset similarity condition, and generating at least one second derivative feature; and splicing the transformation features, the first derivative features and the second derivative features to obtain a feature set, thereby deeply mining the relationship of the high-missing-rate features, improving the quality of the feature set, and enhancing the model training effect.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a method and apparatus for processing features with high missing rates. Background Technology

[0002] In the current risk control field, feature engineering faces challenges such as high data missing rates, uneven feature distribution, and insufficient discriminative power of derived features. Existing feature processing methods typically employ simple methods such as filling in missing information or adjusting the numerical form of features through feature transformations, which fail to effectively preserve the informational value of the original features and improve the predictive ability of the model.

[0003] In terms of practical application, the feature data obtained after completing the missing information operation is difficult to fully reflect the complex relationship between multiple features in the original data, and may even lead to the loss of feature information in the original data.

[0004] By adjusting the shape of feature values ​​through feature transformation, the model may sometimes learn slowly during subsequent model training, and the model's stability when dealing with new data may also be affected. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a method and apparatus for processing high missing rate features, which can deeply explore the relationship between high missing rate features, improve the quality of feature sets, and enhance the effect of model training.

[0006] This application provides a high missing rate feature processing method, including: The original features in the original feature data are preprocessed to obtain the preprocessed transformed features; the original feature data is the data generated and stored by the specified target system when executing business processes. The original feature is subjected to multi-path embedding transformation based on multiple embedding transformation rules, and the multiple embedding vectors obtained from the transformation of the original feature are fused to obtain the target embedding vector corresponding to the original feature as the first derived feature; The comprehensive similarity matrix is ​​obtained by fusing multiple similarities between target embedding vectors based on any two original features. For each original feature, based on the comprehensive similarity matrix, select an original feature whose comprehensive similarity with the original feature meets a preset similarity condition, and based on the original feature and the selected original feature, generate at least one second derived feature; The preprocessed transformed features, the first derived features, and the at least one second derived features are concatenated to obtain a feature set; the feature set is used for model training.

[0007] In some embodiments, the high missing rate feature processing method, after obtaining the target embedding vector corresponding to the original feature as the first derived feature, the method further includes: Statistical feature extraction is performed on the target embedding vector corresponding to the original features to obtain at least one statistical feature as a third derived feature; Accordingly, the feature set obtained by concatenating the preprocessed transformed features, the first derived features, and the at least one second derived features includes: The preprocessed transformed features, the first derived features, the at least one second derived feature, and the at least one third derived feature are concatenated to obtain a feature set; the feature set is used for model training.

[0008] In some embodiments, the statistical features in the high missing rate feature processing method include: vector element summation feature, maximum value feature, minimum value feature, standard deviation feature, mean feature, median feature, skewness feature, and kurtosis feature.

[0009] In some embodiments, the high missing rate feature processing method further includes, after obtaining the feature set: For the derived features generated from the feature set, calculate the performance score of each derived feature in multiple evaluation dimensions; the evaluation dimensions include: the predictive ability evaluation dimension of the derived feature, the stability evaluation dimension of the derived feature, and the correlation evaluation dimension between the derived feature and the original feature; wherein, each evaluation dimension corresponds to a preset screening condition for that dimension; Based on the performance scores of each derived feature across multiple evaluation dimensions, derived features that meet the preset screening criteria for each evaluation dimension are selected, and the feature set is updated.

[0010] In some embodiments, the high missing rate feature processing method, the calculation of the performance score of each derived feature in multiple evaluation dimensions includes: the performance score of the predictive ability evaluation dimension is obtained by calculating the IV value or information gain of the derived feature; and / or, the performance score of the stability evaluation dimension is obtained by calculating the PSI value of the derived feature; and / or, the performance score of the correlation evaluation dimension is obtained by calculating the correlation coefficient between the derived feature and the original feature. In some embodiments, the high missing rate feature processing method, wherein performing multi-path embedding transformation on the original feature based on multiple embedding transformation rules, and fusing the multiple embedding vectors obtained from the original feature transformation to obtain the target embedding vector corresponding to the original feature as the first derived feature, includes: The original features are converted into a first embedding vector and a second embedding vector respectively through the binning ID embedding path and the WoE value embedding path; The first embedding vector and the second embedding vector are weighted and fused to generate the target embedding vector as the first derived feature.

[0011] In some embodiments, the high missing rate feature processing method, the step of converting the original features into a first embedding vector and a second embedding vector through the bin ID embedding path and the WoE value embedding path respectively, includes: performing supervised binning processing on the original features and assigning a unique bin ID to each bin; The target embedding dimension is dynamically determined based on the cardinality of the original features, and the bin ID is mapped to the first embedding vector; the dimension of the first embedding vector is the target embedding dimension. Calculate the WoE value of the original feature in each bin, and then standardize the WoE value; The standardized WoE values ​​are mapped to a second embedding vector with the same dimension as the first embedding vector. In some embodiments, the high missing rate feature processing method includes, for each original feature, selecting an original feature whose comprehensive similarity with the original feature meets a preset similarity condition based on the comprehensive similarity matrix, and generating at least one second derived feature based on the original feature and the selected original feature, including: For each original feature, a predetermined number of other original features with the highest comprehensive similarity to the original feature are selected from the comprehensive similarity matrix; based on the value of the original feature and the values ​​of the selected predetermined number of other original features, at least one statistical aggregate feature is generated as the second derived feature.

[0012] In some embodiments, the high missing rate feature processing method generates at least one statistical aggregated feature as the second derived feature based on the value of the original feature and the values ​​of a preset number of other original features, including: Calculate the arithmetic mean of the values ​​of the original feature and the selected preset number of other original features to generate a second derived feature; wherein the preset number is dynamically determined based on the importance of the original feature. And / or, calculate the square root of the sum of the squares of the values ​​of the original feature and the selected preset number of other original features to generate another second derived feature.

[0013] In some embodiments, a high missing rate feature processing apparatus is also provided, the apparatus comprising: The preprocessing module is used to preprocess the original features in the original feature data to obtain the preprocessed transformed features; the original feature data is the data generated and stored by the specified target system when executing business processes; The multi-path embedding transformation module is used to perform multi-path embedding transformation on the original feature based on multiple embedding transformation rules, and fuse the multiple embedding vectors obtained from the transformation of the original feature to obtain the target embedding vector corresponding to the original feature as the first derived feature; The multi-dimensional similarity calculation module is used to obtain a comprehensive similarity matrix by fusing multiple similarities between target embedding vectors of any two original features. The dynamic feature derivation module is used to select, for each original feature, an original feature whose comprehensive similarity with the original feature meets a preset similarity condition based on the comprehensive similarity matrix, and generate at least one second derived feature based on the original feature and the selected original feature; The feature concatenation module is used to concatenate the preprocessed transformed features, the first derived features, and the at least one second derived features to obtain a feature set; the feature set is used for model training.

[0014] This application provides a high-missing-rate feature processing method and apparatus. The method preprocesses the original features in the original feature data to obtain preprocessed transformed features. The original feature data is data generated and stored by a specified target system during business process execution. Based on multiple embedding transformation rules, the original features undergo multi-path embedding transformation, and the multiple embedding vectors obtained from the original feature transformation are fused to obtain the target embedding vector corresponding to the original feature as the first derived feature. The multi-path embedding transformation and fusion mechanism deeply mines and retains the information of the original features from different perspectives. Even when the feature has a large number of missing values, it can generate information-rich target embedding vectors, which to some extent solves the information loss problem caused by traditional imputation methods. The comprehensive feature is obtained by fusing multiple similarities between the target embedding vectors of any two original features. Similarity is calculated to obtain a comprehensive similarity matrix. For each original feature, based on the comprehensive similarity matrix, original features whose comprehensive similarity with the original feature meets preset similarity conditions are selected. Based on the original features and the selected original features, at least one second derived feature is generated. By fusing multiple similarities to calculate the comprehensive similarity between features, the correlation between features is reflected more comprehensively and accurately from multiple perspectives. Based on this correlation, the second derived feature is dynamically generated, improving the automation level of feature engineering and its adaptability to different business scenarios. The preprocessed transformed features, the first derived feature, and the at least one second derived feature are concatenated to obtain a feature set. The feature set is used for model training. The concatenated feature set has more comprehensive information, higher quality, and stronger discriminative power, thus improving the quality of model training. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart of the high missing rate feature processing method described in an embodiment of this application is shown; Figure 2 A flowchart illustrating the high missing rate feature processing method described in an embodiment of this application is shown; Figure 3 A flowchart of the method for multi-path embedding transformation of the original feature based on multiple embedding transformation rules according to an embodiment of this application is shown; Figure 4 A flowchart of a method for generating at least one second derived feature according to an embodiment of this application is shown; Figure 5 A flowchart of another high-missing-rate feature processing method described in an embodiment of this application is shown; Figure 6 A schematic diagram of the high missing rate feature processing device described in an embodiment of this application is shown. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0018] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0020] In the current risk control field, feature engineering faces challenges such as high data missing rates, uneven feature distribution, and insufficient discriminative power of derived features. Existing feature processing methods typically employ simple methods to fill in missing information or adjust the numerical form of features through feature transformations, which fail to effectively preserve the informational value of the original features and improve the predictive ability of the model.

[0021] From the perspective of practical application, the feature data obtained after completing the missing information is difficult to fully reflect the complex relationships between multiple features in the original data. In fact, due to the bias in the completed information, the feature information in the original data may be lost, which directly affects the accuracy of subsequent feature analysis. This results in the discrepancy between the rules or conclusions drawn from the data and the real business scenario, making the data foundation of the model inherently insufficient and limiting the training of the model.

[0022] Adjusting the numerical shape of features through feature transformations focuses on maintaining a uniform numerical scale, but lacks optimization of the overall distribution of features. This results in suboptimal data distribution characteristics of the feature data input to the model, increasing the difficulty of model learning. During subsequent model training, the learning speed may be slow, and the model's stability when dealing with new data may also be affected.

[0023] Based on this, this application provides a high-missing-rate feature processing method and apparatus. The method preprocesses the original features in the original feature data to obtain preprocessed transformed features. The original feature data is data generated and stored by a specified target system during business process execution. The method performs multi-path embedding transformation on the original features based on multiple embedding transformation rules, and fuses the multiple embedding vectors obtained from the original feature transformation to obtain the target embedding vector corresponding to the original feature as the first derived feature. The multi-path embedding transformation and fusion mechanism deeply mines and retains the information of the original features from different perspectives. Even when the feature has a large number of missing values, it can generate information-rich target embedding vectors, which to some extent solves the information loss problem caused by traditional imputation methods. The method also provides a comprehensive feature processing method based on the fusion of multiple similarities between the target embedding vectors of any two original features. The similarity is combined to obtain a comprehensive similarity matrix. For each original feature, based on the comprehensive similarity matrix, original features whose comprehensive similarity with the original feature meets the preset similarity conditions are selected. Based on the original features and the selected original features, at least one second derived feature is generated. By fusing multiple similarities to calculate the comprehensive similarity between features, the correlation between features is reflected more comprehensively and accurately from multiple perspectives. The second derived feature is dynamically generated based on this correlation, which improves the automation level of feature engineering and its adaptability to different business scenarios. The preprocessed transformed features, the first derived feature, and the at least one second derived feature are concatenated to obtain a feature set. The feature set is used for model training. The concatenated feature set has more comprehensive information, higher quality, and stronger discriminative power, thus improving the quality of model training.

[0024] Please refer to Figure 1 , Figure 1 A flowchart of the high missing rate feature processing method described in an embodiment of this application is shown; as follows: Figure 1 As shown, the method includes the following steps S101-S105: S101. Preprocess the original features in the original feature data to obtain the preprocessed transformed features; the original feature data is the data generated and stored by the specified target system when executing the business process. S102. Perform multi-path embedding transformation on the original feature based on multiple embedding transformation rules, and fuse the multiple embedding vectors obtained from the transformation of the original feature to obtain the target embedding vector corresponding to the original feature as the first derived feature; S103. The comprehensive similarity matrix is ​​obtained by fusing multiple similarities between target embedding vectors based on any two original features. S104. For each original feature, select an original feature whose comprehensive similarity with the original feature meets the preset similarity condition based on the comprehensive similarity matrix, and generate at least one second derived feature based on the original feature and the selected original feature. S105. The preprocessed transformed features, the first derived features, and the at least one second derived features are concatenated to obtain a feature set; the feature set is used for model training.

[0025] In some embodiments, the high missing rate feature processing method is applied to a high missing rate feature processing system; please refer to Figure 2 , Figure 2 A flowchart illustrating the high missing rate feature processing method described in an embodiment of this application is shown; as follows: Figure 2 As shown in the embodiments of this application, the high missing rate feature processing system mainly includes six core modules: data preprocessing module, multi-path embedding conversion module, multi-dimensional similarity calculation module, dynamic feature derivation module, feature splicing module, and feature quality evaluation module; the modules are closely connected through data flow to form a complete feature processing and derivation process.

[0026] In step S101, the original features in the original feature data are preprocessed to obtain the preprocessed transformed features; the original feature data is the data generated and stored by the specified target system when executing the business process.

[0027] The raw feature data is used in the feature engineering stage of financial risk control or credit assessment models. Specifically, the raw feature data is objective data that is naturally generated and accumulated in specific business activities (such as financial lending).

[0028] Because of the missing rate of these naturally generated and accumulated objective data, technical problems such as difficulty in model training and inaccurate predictions arise.

[0029] The original feature data is the data generated and stored by the specified target system when executing business processes. In a financial risk control scenario, the target system can be a credit approval system, etc.

[0030] The original feature data is generated as the target system executes actual business processes, rather than being simulated. For example, when a credit approval system executes the "user qualification review" process, it will generate data such as the user's income verification records and credit inquiry results.

[0031] The raw features refer to feature columns extracted directly from raw data without in-depth processing. For example, in a risk control scenario, these could be a user's "age," "monthly income," or "number of historical overdue payments."

[0032] In step S101, the data preprocessing module preprocesses the original features in the original feature data. It is mainly responsible for cleaning and standardizing the original features, including missing value identification, Sigmoid transformation standardization, and other steps, in order to optimize the feature distribution and provide a basis for subsequent processing.

[0033] The preprocessed transformed features are the features obtained after the original features have undergone preprocessing such as cleaning, followed by a Sigmoid transformation.

[0034] In other words, the Sigmoid transformation module is responsible for performing a Sigmoid transformation on the pre-processed features, improving the feature distribution and standardizing the feature range. The specific implementation steps of this module are as follows: Feature selection: Select the original rating class features (such as credit score, risk score, etc.) that need to be transformed by Sigmoid.

[0035] Z-Score Standardization: Each selected original feature is subjected to Z-Score standardization. The calculation formula is as follows: X std = (X -μ) / σ; where X is the original feature, X std σ represents the Z-Score standardized features; μ is the mean, and σ is the standard deviation.

[0036] For the standardized z-value (i.e., X) std Applying the Sigmoid function, the eigenvalues ​​are mapped to the interval [0,1]. The transformation formula is as follows: ;in, This represents the Sigmoid function.

[0037] The advantage of the Sigmoid transform is that it effectively improves the feature distribution without changing the original feature's discriminative power and ranking ability, maps feature values ​​to a fixed [0,1] interval, avoids the influence of extreme values ​​on subsequent processing, and preserves the relative relationships of the original features.

[0038] In step S102, the original feature is subjected to multi-path embedding transformation based on multiple embedding transformation rules, and the multiple embedding vectors obtained from the original feature transformation are fused to obtain the target embedding vector corresponding to the original feature as the first derived feature.

[0039] Please refer to Figure 3 , Figure 3 A flowchart illustrating the method for multi-path embedding transformation of the original features based on multiple embedding transformation rules according to an embodiment of this application is shown; as follows: Figure 3As shown, the process of performing multi-path embedding transformation on the original features based on multiple embedding transformation rules, and fusing the multiple embedding vectors obtained from the original feature transformation to obtain the target embedding vector corresponding to the original feature as the first derived feature, includes the following steps S301-S302: S301. The original features are converted into a first embedding vector and a second embedding vector respectively through the binning ID embedding path and the WoE value embedding path; S302. The first embedding vector and the second embedding vector are weighted and fused to generate the target embedding vector as the first derived feature.

[0040] Please refer to Figure 2 The multi-path embedding transformation module performs feature embedding transformation through two paths: bin ID and WoE value. The embedding results of the two paths are then weighted and fused to generate the final embedding vector.

[0041] In some embodiments, converting the original features into a first embedding vector and a second embedding vector via the bin ID embedding path and the WoE value embedding path respectively includes: performing supervised binning on the original features and assigning a unique bin ID to each bin; The target embedding dimension is dynamically determined based on the cardinality of the original features, and the bin ID is mapped to the first embedding vector; the dimension of the first embedding vector is the target embedding dimension. Calculate the WoE value of the original feature in each bin, and then standardize the WoE value; The standardized WoE values ​​are mapped to a second embedding vector with the same dimension as the first embedding vector. Specifically, the process of converting the original features into a first embedding vector based on the bin ID embedding path is as follows.

[0042] Optimal binning: Supervised binning methods (such as chi-square binning and decision tree binning) are used to perform optimal binning on the original features to ensure that the samples in each bin have similar distributions of the target variable; the number of bins is dynamically adjusted according to the cardinality and distribution characteristics of the original features, usually between 5 and 20 bins.

[0043] Bin ID Mapping: Assign a unique bin ID to each bin and establish a mapping relationship between the feature values ​​of the original features and the bin IDs.

[0044] Embedding dimension determination: The embedding dimension is dynamically determined based on the cardinality of the original features. The calculation formula is as follows: embedding dim = min(64, max(4,[log2(cardinality)])); Where cardinality is the number of distinct values ​​of the feature, embedding dim For the embedded dimension.

[0045] Embedding layer construction: Construct an embedding layer to convert bin IDs into low-dimensional dense vectors. The weights of the embedding layer are automatically learned during subsequent model training through the backpropagation algorithm.

[0046] Here, the calculated embedding dimension is the dimension of the first embedding vector, which is the target embedding dimension, referring to the dimension of the continuous vector space to which the features are ultimately mapped.

[0047] Specifically, the process of converting the original features into a second embedding vector based on the WoE value embedding path is as follows.

[0048] The WoE value, or weight of evidence, characterizes the predictive power or "strength of evidence" of the bin for the target variable.

[0049] WoE Value Calculation: Calculate the WoE (Weight of Evidence) value for each feature in each bin. The formula is as follows: WoE = ln(positive) rate negative rate ); Among them, positive rate The proportion of positive samples in the bins, negative rate WoE represents the proportion of negative samples in the bin, and WoE is the WoE value.

[0050] WoE value standardization: The calculated WoE value is standardized to ensure that its value range is within a reasonable range.

[0051] Fully connected layer mapping: The WoE values ​​are mapped to a vector space of the same dimension as the bin ID embedding through a fully connected layer; the weights of the fully connected layer are also learned through the backpropagation algorithm.

[0052] After performing multi-path embedding transformation on all samples and all features in the original feature data, from the perspective of the overall data, one or more "embedding matrices" are obtained. The embedding matrix includes the target embedding vector (i.e. the first derived feature) of each feature of each sample.

[0053] Based on this, in some embodiments, after obtaining the target embedding vector corresponding to the original feature as the first derived feature, the method further includes: Statistical features are extracted from the target embedding vector corresponding to the original features to obtain at least one statistical feature as a third derived feature.

[0054] For example, the statistical features include: vector element summation feature, maximum value feature, minimum value feature, standard deviation feature, mean feature, median feature, skewness feature, and kurtosis feature.

[0055] Please refer to Figure 2 In other words, after the multi-path embedding transformation module performs the multi-path embedding transformation operation, the vector statistical feature derivation module further enriches the feature representation space by extracting the statistical features of the embedding matrix.

[0056] Specifically, the statistical feature extraction involves calculating eight key statistical indicators for each sample vector in the feature matrix after multi-path embedding transformation. These indicators include: vector element summation feature (reflecting the overall strength of the feature vector); maximum / minimum value feature (capturing extreme values ​​in the feature vector); standard deviation feature (measuring the dispersion of the feature vector); mean / median feature (indicating the central tendency of the feature vector); skewness feature (describing the asymmetry of the feature vector distribution); kurtosis feature (reflecting the steepness of the feature vector distribution); and feature enhancement value (effectively supplementing the information gaps in the global distribution characteristics of the original embedded features, enabling the model to learn the intrinsic structure of the data from both local feature patterns and global statistical regularities).

[0057] Vector statistical feature derivation processing serves as a bridge connecting multi-path embedding transformation and subsequent feature fusion, providing important statistical information support for the final high-missing-rate feature processing and derivation, and enhancing the model's ability to capture complex data patterns.

[0058] In some embodiments, in steps S103 and S104, a comprehensive similarity matrix is ​​obtained by fusing multiple similarities between the target embedding vectors of any two original features; for each original feature, an original feature whose comprehensive similarity with the original feature meets a preset similarity condition is selected based on the comprehensive similarity matrix, and at least one second derived feature is generated based on the original feature and the selected original feature.

[0059] Please refer to Figure 2 The multi-dimensional similarity calculation module calculates the multi-dimensional similarity between features (i.e., multiple similarities) and merges these similarities to generate a comprehensive similarity matrix.

[0060] In some embodiments, the multiple similarities include cosine similarity, correlation coefficient similarity, mean difference similarity, etc.

[0061] Specifically, the cosine similarity calculation process is as follows: Vector normalization: The embedded feature vector (i.e. the target embedding vector corresponding to the original feature) is subjected to L2 normalization to ensure that the vector's magnitude is 1.

[0062] Cosine similarity calculation: The cosine similarity between two target embedding vectors is calculated using the following formula: cosine sim(a,b) = (a · b) / (||a|| × ||b||); Where a and b are two eigenvectors, · represents the vector dot product, ||·|| represents the L2 norm of the vector, and cosine sim(a,b) Cosine similarity between a and b.

[0063] Missing value handling: For features with a large number of missing values, only the common non-empty values ​​of the two features are used for similarity calculation; when the number of common non-empty values ​​is lower than the set threshold (such as 10% of the sample size), the cosine similarity is set to the default value (such as 0.5).

[0064] In other words, when calculating the cosine similarity between the target embedding vectors of two original features, a common non-empty sample set between the target embedding vectors of the first and second original features is determined; it is determined whether the number of samples in the common non-empty sample set is lower than a preset number threshold; if it is greater than or equal to the preset number threshold, the cosine similarity between the target embedding vectors of the first and second original features is calculated based on the common non-empty sample set; if it is lower than the preset number threshold, the target embedding vectors of the first and second original features are set to a preset similarity value.

[0065] The correlation coefficient similarity calculation process is as follows: Pearson correlation coefficient calculation: The Pearson correlation coefficient between two target embedding vectors is calculated using the following formula: pearson sim(a,b) = cov(a,b) / (σ a × σ b ); Where cov(a,b) is the covariance of a and b, and a and b are two eigenvectors; σ a and σ b Let a and b be the standard deviations, respectively.

[0066] The process of calculating the mean difference similarity is as follows: Calculate the mean of positive and negative samples: Calculate the mean of each feature in the positive and negative samples respectively.

[0067] Mean difference similarity calculation: Similarity is calculated based on the mean difference of features in positive and negative samples. The formula is as follows: mean_diff sim(a,b) = 1 - |μ a - μ b | / max(μ a , μ b ); Where a and b are two feature vectors, μ a , μ b The mean and diff values ​​for features a and b in the positive samples are respectively. sim(a,b) This represents the similarity of the mean difference.

[0068] Similarity fusion: The formula for calculating the overall similarity is as follows: Combined Sim(i,j) = 0.5 × Cosine Sim(i,j) + 0.3 × Corr Sim(i,j) + 0.2 × Mean_Diff Sim(i,j) ; Among them: Cosine Sim(i,j) Cosine similarity between feature i and feature j; Corr Sim(i,j) Mean_Diff is the Pearson correlation coefficient between feature i and feature j. Sim(i,j) The similarity of the mean difference between feature i and feature j; Combined Sim(i,j) For comprehensive similarity.

[0069] The output of the multi-dimensional similarity calculation module is a comprehensive similarity matrix between features, which provides input for subsequent dynamic feature derivation.

[0070] Please refer to Figure 4 For each original feature, the step of selecting an original feature whose comprehensive similarity with the original feature meets a preset similarity condition based on the comprehensive similarity matrix, and generating at least one second derived feature based on the original feature and the selected original feature, includes the following steps S401-S402: S401. For each original feature, select a preset number of other original features with the highest comprehensive similarity to the original feature from the comprehensive similarity matrix; wherein, the preset number is dynamically determined based on the importance of the original feature; S402. Based on the value of the original feature and the values ​​of the selected preset number of other original features, generate at least one statistical aggregated feature as the second derived feature.

[0071] Please refer to Figure 2The dynamic feature derivation module selects the Top-K most similar features for each feature based on the comprehensive similarity matrix, and calculates the arithmetic mean and square root of the sum of squares (L2 norm) of these features to generate the second derived feature.

[0072] In some embodiments, generating at least one statistically aggregated feature as the second derived feature based on the value of the original feature and the values ​​of a predetermined number of other original features includes: Calculate the arithmetic mean of the values ​​of the original feature and the selected preset number of other original features to generate a second derived feature; And / or, calculate the square root of the sum of the squares of the values ​​of the original feature and the selected preset number of other original features to generate another second derived feature.

[0073] Specifically, the process of selecting Top-K similarity features is as follows: Similarity ranking: For each original feature, rank the other original features according to the comprehensive similarity matrix.

[0074] K-value dynamic adjustment: The K-value (i.e., the number of similar features selected) is dynamically adjusted according to the importance of the features (such as IV value, the importance of features in the model); usually, the K-value of the original features with high importance is larger (such as 10-20), and the K-value of the original features with low importance is smaller (such as 3-5).

[0075] Top-K feature selection: Select the top-K most similar features for each feature.

[0076] The process of generating derived features is as follows: Arithmetic mean derived feature generation: For each feature and its Top-K similar features, an arithmetic mean is calculated to generate new derived features. The calculation formula is as follows: feature mean = (1 / (K+1)) × ∑(i=0 to K) feature i ; Among them, feature i The values ​​of the original feature and its Top-K similar features; feature mean The derived feature is generated from the arithmetic mean.

[0077] Sum of squares and root-derived feature generation: For each feature and its Top-K similar features, the sum of squares and root-derived roots are applied to generate new derived features. The calculation formula is as follows: feature sqrt = √((1 / (K+1)) × ∑(i=0 to K) feature i ²); Among them, feature i The values ​​of the original feature and its Top-K similar features; feature sqrt Derivative features are generated from the sum of squares and the square root.

[0078] In step S105, the preprocessed transformed features, the first derived features, and the at least one second derived features are concatenated to obtain a feature set; the feature set is used for model training.

[0079] In some embodiments, after statistical feature extraction is performed on the target embedding vector corresponding to the original feature to obtain at least one statistical feature as a third derived feature, the corresponding concatenation of the preprocessed transformed feature, the first derived feature, and the at least one second derived feature to obtain a feature set includes: The preprocessed transformed features, the first derived features, the at least one second derived feature, and the at least one third derived feature are concatenated to obtain a feature set; the feature set is used for model training.

[0080] Please refer to Figure 2 The feature concatenation module concatenates features from four different sources to form a complete feature set for subsequent model training.

[0081] The four types of derived features are as follows: Original features of the Sigmoid transform (i.e., preprocessed transformed features): obtained through Z-Score standardization and Sigmoid function transformation, focusing on discriminative ability while improving feature distribution and standardizing the value range to the [0,1] interval; Vectorized embedded features after binning the original features (i.e., the first derived feature): generated through a dual-path embedding architecture of bin ID and WOE value, effectively capturing nonlinear relationships between features; Statistical features based on vectorized features (i.e., the third derived feature): including statistics such as summation, maximum, minimum, standard deviation, mean, median, skewness, and kurtosis, further enriching feature information; Derived features based on the similarity algorithm of the original features (i.e., the second derived feature): including the arithmetic mean and square root features of TopK similarity features, capable of capturing the correlation between features.

[0082] Please refer to Figure 5 In some embodiments, after obtaining the feature set, the high missing rate feature processing method further includes the following steps S501-S502: S501. For the derived features generated in the feature set, calculate the performance score of each derived feature in multiple evaluation dimensions; the evaluation dimensions include: the predictive ability evaluation dimension of the derived feature, the stability evaluation dimension of the derived feature, and the correlation evaluation dimension between the derived feature and the original feature; wherein, each evaluation dimension corresponds to a preset screening condition of the dimension. S502. Based on the performance scores of each derived feature in multiple evaluation dimensions, select derived features that meet the preset screening conditions of each evaluation dimension, and update the feature set.

[0083] In some embodiments, calculating the performance score of each derived feature across multiple evaluation dimensions includes: the performance score of the predictive ability evaluation dimension is obtained by calculating the IV value or information gain of the derived feature; and / or, the performance score of the stability evaluation dimension is obtained by calculating the PSI value of the derived feature; and / or, the performance score of the correlation evaluation dimension is obtained by calculating the correlation coefficient between the derived feature and the original feature. In some embodiments, the preset screening criteria corresponding to the predictive ability evaluation dimension of the derived feature are: when the performance score of the predictive ability evaluation dimension of the derived feature is lower than the first preset threshold, it is removed; and derived features with a score higher than the first preset threshold are retained to ensure the effectiveness of the derived features.

[0084] If the performance score of the stability evaluation dimension of the derived feature is higher than the second preset threshold, it will be removed. If it is higher than the second preset threshold, it indicates that the derived feature is unstable.

[0085] If the performance score of the correlation evaluation dimension between a derived feature and any original feature is higher than the third preset threshold, it is discarded. If the absolute value of the correlation coefficient between a new derived feature and an original feature exceeds the third preset threshold, they are considered to be highly collinear, with serious information redundancy, and need to be discarded.

[0086] Please refer to Figure 2 The feature quality assessment module evaluates the quality of the assembled feature set, including missing rate analysis, discrimination test (such as IV value, information gain) and stability test (such as PSI value), to ensure the effectiveness and stability of the features.

[0087] IV value evaluation: Calculate the IV value for each derived feature to assess its predictive ability, and then perform feature selection. Derivative features with IV values ​​higher than a preset IV value threshold are retained to ensure the effectiveness of the derived features.

[0088] Feature correlation check: Check the correlation between derived features and original features to avoid feature redundancy.

[0089] The output of the dynamic feature derivation module is a set of effective derived features after filtering. This set is combined with the original feature set to form the final feature set, which is then used for subsequent model training.

[0090] In the high missing rate feature processing method described in this application, the original features are optimized through Sigmoid transformation preprocessing, which effectively improves the feature distribution and enhances the subsequent processing effect without changing the discriminative and ranking capabilities of the original features. A dual-path embedding architecture using binning ID and WoE values ​​is employed to perform multi-path feature embedding transformation, simultaneously capturing the category and predictive ability information of the features, and enhancing the expressive power of the embedding vector through dynamic weight fusion. Multi-dimensional similarity, including cosine similarity, correlation coefficient similarity, and mean difference similarity, is comprehensively applied to fully reflect the correlation between features, and a special processing strategy is designed for missing values ​​to improve the accuracy of similarity calculation. Based on the multi-dimensional similarity matrix, Top-K similar features are dynamically selected to generate diverse derived features, improving the discriminative and generalization capabilities of the features. By using only common non-empty values ​​to calculate similarity, the low reliability of similarity calculation in high missing rate scenarios is effectively solved, improving the accuracy and stability of similarity calculation. A complete process from data preprocessing to feature derivation is provided, achieving end-to-end feature processing and ensuring the coherence and consistency of feature processing.

[0091] Based on the same inventive concept, this application also provides a high missing rate feature processing device corresponding to the high missing rate feature processing method. Since the principle of the device in this application is similar to the high missing rate feature processing method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0092] Please refer to Figure 6 , Figure 6 This illustration shows a schematic diagram of the high missing rate feature processing device according to an embodiment of this application; the high missing rate feature processing device includes: The preprocessing module is used to preprocess the original features in the original feature data to obtain the preprocessed transformed features; the original feature data is the data generated and stored by the specified target system when executing the business process; The multi-path embedding transformation module is used to perform multi-path embedding transformation on the original feature based on multiple embedding transformation rules, and fuse the multiple embedding vectors obtained from the transformation of the original feature to obtain the target embedding vector corresponding to the original feature as the first derived feature; The multi-dimensional similarity calculation module is used to obtain a comprehensive similarity matrix by fusing multiple similarities between target embedding vectors of any two original features. The dynamic feature derivation module is used to select, for each original feature, an original feature whose comprehensive similarity with the original feature meets a preset similarity condition based on the comprehensive similarity matrix, and generate at least one second derived feature based on the original feature and the selected original feature; The feature concatenation module is used to concatenate the preprocessed transformed features, the first derived features, and the at least one second derived features to obtain a feature set; the feature set is used for model training.

[0093] In some embodiments, the high missing rate feature processing device further includes: The vector statistical feature derivation module is used to extract statistical features from the target embedding vector corresponding to the original feature after obtaining the target embedding vector corresponding to the original feature as the first derived feature, and obtain at least one statistical feature as the third derived feature. Accordingly, the feature splicing module, when splicing the preprocessed transformed features, the first derived features, and the at least one second derived feature to obtain a feature set, is specifically used for: The preprocessed transformed features, the first derived features, the at least one second derived feature, and the at least one third derived feature are concatenated to obtain a feature set; the feature set is used for model training.

[0094] In some embodiments, the statistical features in the high missing rate feature processing device include: vector element summation feature, maximum value feature, minimum value feature, standard deviation feature, mean feature, median feature, skewness feature, and kurtosis feature.

[0095] In some embodiments, the high missing rate feature processing device further includes: The feature quality assessment module is used to calculate the performance score of each derived feature in multiple assessment dimensions after obtaining the feature set. The assessment dimensions include: the predictive ability assessment dimension of the derived feature, the stability assessment dimension of the derived feature, and the correlation assessment dimension between the derived feature and the original feature. Each assessment dimension corresponds to a preset screening condition. Based on the performance scores of each derived feature across multiple evaluation dimensions, derived features that meet the preset screening criteria for each evaluation dimension are selected, and the feature set is updated.

[0096] In some embodiments, in the high missing rate feature processing apparatus, the feature quality assessment module, when calculating the performance score of each derived feature across multiple assessment dimensions, specifically performs the following: the performance score of the predictive ability assessment dimension is obtained by calculating the IV value or information gain of the derived feature; and / or, the performance score of the stability assessment dimension is obtained by calculating the PSI value of the derived feature; and / or, the performance score of the correlation assessment dimension is obtained by calculating the correlation coefficient between the derived feature and the original feature. In some embodiments, in the high missing rate feature processing apparatus, the multi-path embedding transformation module, when performing multi-path embedding transformation on the original feature based on multiple embedding transformation rules and fusing the multiple embedding vectors obtained from the original feature transformation to obtain the target embedding vector corresponding to the original feature as the first derived feature, is specifically used for: The original features are converted into a first embedding vector and a second embedding vector respectively through the binning ID embedding path and the WoE value embedding path; The first embedding vector and the second embedding vector are weighted and fused to generate the target embedding vector as the first derived feature.

[0097] In some embodiments, in the high missing rate feature processing device, the multi-path embedding conversion module, when converting the original features into a first embedding vector and a second embedding vector through the bin ID embedding path and the WoE value embedding path respectively, is specifically used to: perform supervised binning processing on the original features and assign a unique bin ID to each bin; The target embedding dimension is dynamically determined based on the cardinality of the original features, and the bin ID is mapped to the first embedding vector; the dimension of the first embedding vector is the target embedding dimension. Calculate the WoE value of the original feature in each bin, and then standardize the WoE value; The standardized WoE values ​​are mapped to a second embedding vector with the same dimension as the first embedding vector. In some embodiments, in the high missing rate feature processing apparatus, the dynamic feature derivation module, when selecting an original feature whose comprehensive similarity with the original feature meets a preset similarity condition based on the comprehensive similarity matrix for each original feature, and generating at least one second derived feature based on the original feature and the selected original feature, is specifically used for: For each original feature, a predetermined number of other original features with the highest comprehensive similarity to the original feature are selected from the comprehensive similarity matrix; based on the value of the original feature and the values ​​of the selected predetermined number of other original features, at least one statistical aggregate feature is generated as the second derived feature.

[0098] In some embodiments, in the high missing rate feature processing apparatus, the dynamic feature derivation module, when generating at least one statistical aggregated feature as the second derived feature based on the value of the original feature and the values ​​of a preset number of other original features, is specifically used for: Calculate the arithmetic mean of the values ​​of the original feature and the selected preset number of other original features to generate a second derived feature; wherein the preset number is dynamically determined based on the importance of the original feature. And / or, calculate the square root of the sum of the squares of the values ​​of the original feature and the selected preset number of other original features to generate another second derived feature.

[0099] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0100] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0101] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0102] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0103] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing features with high missing rates, characterized in that, The method includes: The original features in the original feature data are preprocessed to obtain the preprocessed transformed features; the original feature data is the data generated and stored by the specified target system when executing business processes. The original feature is subjected to multi-path embedding transformation based on multiple embedding transformation rules, and the multiple embedding vectors obtained from the transformation of the original feature are fused to obtain the target embedding vector corresponding to the original feature as the first derived feature; The comprehensive similarity matrix is ​​obtained by fusing multiple similarities between target embedding vectors based on any two original features. For each original feature, based on the comprehensive similarity matrix, select an original feature whose comprehensive similarity with the original feature meets a preset similarity condition, and based on the original feature and the selected original feature, generate at least one second derived feature; The preprocessed transformed features, the first derived features, and the at least one second derived features are concatenated to obtain a feature set; the feature set is used for model training.

2. The high missing rate feature processing method according to claim 1, characterized in that, After obtaining the target embedding vector corresponding to the original feature as the first derived feature, the method further includes: Statistical feature extraction is performed on the target embedding vector corresponding to the original features to obtain at least one statistical feature as a third derived feature; Accordingly, the feature set obtained by concatenating the preprocessed transformed features, the first derived features, and the at least one second derived features includes: The preprocessed transformed features, the first derived features, the at least one second derived feature, and the at least one third derived feature are concatenated to obtain a feature set; the feature set is used for model training.

3. The high missing rate feature processing method according to claim 2, characterized in that, The statistical features include: vector element summation feature, maximum value feature, minimum value feature, standard deviation feature, mean feature, median feature, skewness feature, and kurtosis feature.

4. The high missing rate feature processing method according to claim 1, characterized in that, After obtaining the feature set, the method further includes: For the derived features generated from the feature set, calculate the performance score of each derived feature in multiple evaluation dimensions; the evaluation dimensions include: the predictive ability evaluation dimension of the derived feature, the stability evaluation dimension of the derived feature, and the correlation evaluation dimension between the derived feature and the original feature; wherein, each evaluation dimension corresponds to a preset screening condition for that dimension; Based on the performance scores of each derived feature across multiple evaluation dimensions, derived features that meet the preset screening criteria for each evaluation dimension are selected, and the feature set is updated.

5. The high missing rate feature processing method according to claim 4, characterized in that, The calculation of the performance score of each derived feature across multiple evaluation dimensions includes: the performance score of the predictive ability evaluation dimension is obtained by calculating the IV value or information gain of the derived feature; and / or, the performance score of the stability evaluation dimension is obtained by calculating the PSI value of the derived feature; and / or, the performance score of the correlation evaluation dimension is obtained by calculating the correlation coefficient between the derived feature and the original feature.

6. The high missing rate feature processing method according to claim 1, characterized in that, The process of performing multi-path embedding transformation on the original features based on multiple embedding transformation rules, and fusing the multiple embedding vectors obtained from the original feature transformation to obtain the target embedding vector corresponding to the original feature as the first derived feature, includes: The original features are converted into a first embedding vector and a second embedding vector respectively through the binning ID embedding path and the WoE value embedding path; The first embedding vector and the second embedding vector are weighted and fused to generate the target embedding vector as the first derived feature.

7. The high missing rate feature processing method according to claim 6, characterized in that, The step of converting the original features into a first embedding vector and a second embedding vector through the bin ID embedding path and the WoE value embedding path, respectively, includes: performing supervised binning processing on the original features and assigning a unique bin ID to each bin; The target embedding dimension is dynamically determined based on the cardinality of the original features, and the bin ID is mapped to the first embedding vector; the dimension of the first embedding vector is the target embedding dimension. Calculate the WoE value of the original feature in each bin, and then standardize the WoE value; The standardized WoE values ​​are mapped to a second embedding vector with the same dimension as the first embedding vector.

8. The high missing rate feature processing method according to claim 1, characterized in that, For each original feature, based on the comprehensive similarity matrix, original features whose comprehensive similarity with the original feature meets a preset similarity condition are selected. Based on these original features and the selected original features, at least one second derived feature is generated, including: For each original feature, a predetermined number of other original features with the highest comprehensive similarity to the original feature are selected from the comprehensive similarity matrix; based on the value of the original feature and the values ​​of the selected predetermined number of other original features, at least one statistical aggregate feature is generated as the second derived feature.

9. The high missing rate feature processing method according to claim 8, characterized in that, Based on the value of the original feature and the values ​​of a predetermined number of other original features, at least one statistically aggregated feature is generated as the second derived feature, including: Calculate the arithmetic mean of the values ​​of the original feature and the selected preset number of other original features to generate a second derived feature; wherein the preset number is dynamically determined based on the importance of the original feature. And / or, calculate the square root of the sum of the squares of the values ​​of the original feature and the selected preset number of other original features to generate another second derived feature.

10. A high-missing-rate feature processing device, characterized in that, The device includes: The preprocessing module is used to preprocess the original features in the original feature data to obtain the preprocessed transformed features; the original feature data is the data generated and stored by the specified target system when executing business processes; The multi-path embedding transformation module is used to perform multi-path embedding transformation on the original feature based on multiple embedding transformation rules, and fuse the multiple embedding vectors obtained from the transformation of the original feature to obtain the target embedding vector corresponding to the original feature as the first derived feature; The multi-dimensional similarity calculation module is used to obtain a comprehensive similarity matrix by fusing multiple similarities between target embedding vectors of any two original features. The dynamic feature derivation module is used to select, for each original feature, an original feature whose comprehensive similarity with the original feature meets a preset similarity condition based on the comprehensive similarity matrix, and generate at least one second derived feature based on the original feature and the selected original feature; The feature concatenation module is used to concatenate the preprocessed transformed features, the first derived features, and the at least one second derived features to obtain a feature set; the feature set is used for model training.

Images