Data processing methods, apparatus, equipment and storage media
By determining the target data category and feature scores from multiple data sources, and then sorting and selecting features, the problem of low intelligence and low efficiency caused by manual selection in multidimensional data processing is solved. This achieves intelligent and flexible feature selection and improves data processing efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WEBANK (CHINA)
- Filing Date
- 2023-12-21
- Publication Date
- 2026-05-26
Smart Images
Figure CN117725395B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology
[0002] With the development of computers, data processing is becoming increasingly common. For example, credit risk assessment requires corresponding data processing, such as using multidimensional data (e.g., various features from multiple data sources) to evaluate the assessed entity (enterprise, individual). The richer the multidimensional data used for assessment, the more comprehensive and objective the evaluation. However, the diverse data sources and high similarity in data dimensions often lead to assessment bias, unstable evaluation results, high data costs, and maintenance difficulties. Therefore, the selection of features from data sources is crucial, but currently, this selection is done manually, resulting in low intelligence and efficiency in data processing. Summary of the Invention
[0003] The main objective of this application is to provide a data processing method, apparatus, device, and storage medium, which aims to solve the technical problem of how to improve the efficiency of data processing.
[0004] To achieve the above objectives, this application provides a data processing method, which includes the following steps:
[0005] Multiple sets of data sources to be tested and evaluated are identified, wherein each set of data sources includes at least one feature, each feature corresponds to a data category, and the same data category includes at least one feature;
[0006] Determine at least one target data category selected from multiple data categories, and determine the minimum number of features corresponding to each target data category;
[0007] A feature score is determined for each feature based on its discriminative power, and each feature is sorted according to its feature score to obtain a feature sorting queue.
[0008] Based on the feature sorting queue, features are selected according to the minimum number of features corresponding to each target data category. A target feature set is constructed based on the selected features, and the target feature set and the target data source corresponding to the features in the target feature set are output.
[0009] Optionally, the step of determining the feature score corresponding to each feature based on the discriminative power of each feature includes:
[0010] Calculate the discrimination score corresponding to each feature, and normalize each discrimination score to obtain a first score;
[0011] The feature score corresponding to each feature is determined based on the first score.
[0012] Optionally, the step of determining the feature score corresponding to each feature based on the first score includes:
[0013] Construct multiple feature groups, configure the priority of each feature group, and determine the feature group score of each feature group based on the priority;
[0014] Each of the features is traversed, and the traversed features are matched with multiple feature groups;
[0015] If there are multiple feature groups that match the traversed features, then the traversed features are grouped into the highest priority feature group among the multiple matching feature groups.
[0016] A second score is determined based on the feature group score corresponding to each feature, and a feature score corresponding to each feature is determined based on the first score and the second score.
[0017] Optionally, the feature group includes at least one of the following:
[0018] Feature group one: Multiple features from the same data source have the same data category;
[0019] Feature group two: multiple features from different data sources have the same data category;
[0020] Feature group three: Features with different data categories from the same data source;
[0021] Feature group four: There is only one feature under the data source and / or data category.
[0022] Optionally, the step of determining the feature score corresponding to each feature based on the first score and the second score includes:
[0023] For each feature group, determine the number of features belonging to the same feature group;
[0024] If the number of features belonging to the same feature group is one, then the third score corresponding to the feature is set to the preset interval maximum value;
[0025] If there are multiple features belonging to the same feature group, then the similarity of the multiple features within the feature group is calculated to obtain the feature similarity, and the third score corresponding to the feature is determined based on the feature similarity.
[0026] The feature score corresponding to each feature is determined based on the first score, the second score, and the third score.
[0027] Optionally, the step of determining the third score corresponding to the feature based on the feature similarity includes:
[0028] If there are two features whose feature similarity is greater than a preset similarity threshold, the two features corresponding to the feature similarity greater than the preset similarity threshold are regarded as associated features;
[0029] The discrimination of the two associated features is compared. The third score corresponding to the associated feature with the higher discrimination is set to a preset maximum value in the interval, and the third score corresponding to the associated feature with the lower discrimination is set to a preset minimum value in the interval. The third scores corresponding to the other features in the feature group other than the associated feature are set to the maximum values in the interval.
[0030] Optionally, the step of determining the feature score corresponding to each feature based on the first score, the second score, and the third score includes:
[0031] The first score, second score, and third score corresponding to each feature are weighted and calculated to obtain the feature score.
[0032] Furthermore, to achieve the above objectives, this application also provides a data processing apparatus, which includes:
[0033] The data source module is used to determine multiple sets of data sources to be tested and evaluated, wherein each set of data sources includes at least one feature, each feature corresponds to a data category, and the same data category includes at least one feature;
[0034] The determination module is used to determine at least one selected target data category among multiple data categories, and to determine the minimum number of features corresponding to each target data category;
[0035] The sorting module is used to determine the feature score corresponding to each feature based on the discriminative power of each feature, and sort each feature according to the feature score to obtain a feature sorting queue;
[0036] The output module is used to select features according to the minimum number of features corresponding to each target data category based on the feature sorting queue, construct a target feature set based on the selected features, and output the target feature set and the target data source corresponding to the features in the target feature set.
[0037] This application also provides a data processing device, which is a physical device. The data processing device includes: a memory, a processor, and a program of the data processing method stored in the memory and executable on the processor. When the program of the data processing method is executed by the processor, it can implement the steps of the data processing method as described above.
[0038] This application also provides a readable storage medium, which is a computer-readable storage medium, on which a program implementing a data processing method is stored, and the program implementing the data processing method is executed by a processor to implement the steps of the data processing method as described above.
[0039] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data processing method described above.
[0040] The technical solution of this application involves determining at least one target data category and the minimum number of features corresponding to each target data category from multiple data sources for testing and evaluation. A feature score is determined based on the discriminative power of each feature, and the features are then sorted according to their scores to obtain a feature ranking queue. Features are then selected based on the minimum number of features corresponding to each target data category according to the ranking queue. A target feature set is constructed based on the selected features, and the target feature set and the target data sources corresponding to the features within the set are output. This approach incorporates data category considerations during feature selection, avoiding algorithmic bias caused by considering only features. Furthermore, by selecting features based on feature discriminative power and target data category, it avoids the low intelligence of data processing resulting from manual feature selection, achieving intelligent feature selection. Since feature extraction and selection are performed from multiple data sources, it also increases the flexibility and adaptability of feature selection. Compared to manual feature selection, this significantly improves data processing efficiency. Attached Figure Description
[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0042] To more clearly illustrate the technical solutions in this embodiment or the prior art, the accompanying drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 This is a flowchart illustrating the first embodiment of the data processing method in this application;
[0044] Figure 2 This is a flowchart illustrating the data processing procedure in the data processing method of this application;
[0045] Figure 3This is a schematic diagram illustrating the correspondence between data source, features, and data categories in the data processing method of this application;
[0046] Figure 4 This is a flowchart illustrating the second embodiment of the data processing method in this application;
[0047] Figure 5 This is a schematic diagram of a module structure of the data processing device of this application;
[0048] Figure 6 This is a schematic diagram of the hardware operating environment involved in the device in this embodiment.
[0049] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0050] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0051] Example 1
[0052] Based on this, please refer to Figure 1 This embodiment provides a data processing method, which includes:
[0053] Step S10: Determine multiple sets of data sources to be tested and evaluated, wherein each set of data sources includes at least one feature, each feature corresponds to a data category, and the same data category includes at least one feature;
[0054] In this embodiment, the data processing method can be applied to a data processing device.
[0055] Optionally, since the selection of data dimensions is based on manual selection, it is not conducive to the automated selection of data dimensions and is inefficient. Therefore, in this embodiment, the data dimensions can be automatically selected to avoid the corresponding defects. For example, such as Figure 2As shown, the process involves selecting a data set, selecting a data category set, configuring feature parameters, automatic feature selection, and viewing the data source and feature evaluation results. Optionally, selecting a data set can involve choosing a dataset to be tested and evaluated, which may include multiple data sources. Selecting a data category set can involve determining the target data category. Configuring feature parameters can involve configuring the priority of each feature group. Automatic feature selection can involve sorting features and then selecting appropriate features. The data source and feature evaluation results can be the output of the selected features and data sources.
[0056] Optionally, a data dimension can refer to a data category in general, classifying data metrics, labels, variables, and features, such as order data or invoice data. It can also refer to specific metric values within invoice data, such as sales invoice data from the past 12 months. Alternatively, a data dimension can be a data field, data metric, or other data category.
[0057] Optionally, you can select the dataset to be tested and evaluated. Multiple data sources within the dataset can be used as the data sources for testing and evaluation. For example, such as... Figure 2 The dataset can be configured with data sources A, B, and C, and these data sources can be expanded. For example, if 10 data sources are needed for testing and evaluation, then 10 data sources can be configured in the dataset. Optionally, a data source can consist of multiple sets of features, meaning each data source must include at least one feature. Optionally, each feature belongs to a specific data category, such as each feature belonging to a single data category. Different features from different data sources can belong to the same data category. Optionally, data dimensions can include data categories, and can also include features formed by combinations based on preset rules. Optionally, the following examples will only use data categories as data dimensions. Optionally, features under each data category can be specific data values, such as features under invoice data including specific indicator values in the invoice data, such as sales invoice data from the past 12 months. Optionally, data categories can be determined based on the data source and its features, such as sales invoice amount, purchase invoice amount, number of sales invoices, number of invoices issued, transaction amount, etc.
[0058] For example, such as Figure 3 As shown, there are data sources A, B, and C. Data source A includes features A1001 and A1002, data source B includes features B1003 and B2004, and data source C includes features C2005 and C3001. Based on the data sources and features, three data categories can be determined: Data Category 1, Data Category 2, and Data Category 3. Data Category 1 includes features A1001, A1002, and B1003. Data Category 2 includes features B2004 and C2005. Data Category 3 includes feature C3001.
[0059] Step S20: Determine at least one selected target data category among multiple data categories, and determine the minimum number of features corresponding to each target data category;
[0060] Optionally, the data category corresponding to each feature in the data source to be evaluated can be determined, and a data category selection can be performed among all data categories, with at least one selected data category being used as the target data category. Optionally, the data category selection method includes at least one of the following: determining the target data category among at least one data category based on preset rules; receiving an input selection instruction, determining the data category corresponding to the instruction among multiple data categories, and using it as the target data category; comparing the number of features contained in each data category, and selecting the data category with the fewest features as the target data category; detecting the features contained in each data category, determining the total number of data sources to which each feature under each data category belongs (i.e., the number of data sources), and selecting the data category with the largest total number as the target data category.
[0061] Optionally, the minimum number of features corresponding to each data category can be configured to determine the minimum number of features corresponding to the target data category. Optionally, when configuring the minimum number of features corresponding to each data category, it can be based on multiple sets of data sources to be tested and evaluated. For example, all features in at least one set of data sources to be tested and evaluated can be determined, and the data category corresponding to each feature can be determined. Features belonging to the same data category can be merged to obtain multiple features under the data category. The number of multiple features under the data category is taken as the minimum number of features corresponding to the data category. For example, if there are features A1001 and A1002 in the data source, and both features A1001 and A1002 belong to data category 1, then the minimum number of features corresponding to data category 1 can be determined to be 2.
[0062] Optionally, the minimum number of features for each data category can be determined based on the number of features input by the user, for example, using the user-input feature number as the minimum number of features for each data category. Alternatively, the minimum number of features for each data category can be set to a default value, such as 1. Furthermore, the minimum number of features for each data category can be set according to preset feature number rules, such as setting it based on the number of data sources corresponding to the features under that data category.
[0063] Optionally, after determining the minimum number of features corresponding to each data category, the minimum number of features corresponding to each target data category can then be determined.
[0064] Step S30: Determine the feature score corresponding to each feature based on the discriminative power of each feature, and sort each feature according to the feature score to obtain a feature sorting queue;
[0065] Optionally, feature scores can be calculated for all features within the multiple data sources to be tested and evaluated. Alternatively, all features within the multiple data sources to be evaluated can be cleaned, filtered, and useless features, such as empty sets, can be removed. Then, feature scores are calculated for the cleaned and filtered features.
[0066] Optionally, the feature score for each feature can be determined based on the importance and / or discriminative power of each feature.
[0067] Optionally, the feature scores of each feature are compared to determine the numerical value of each feature score. Each feature is then sorted according to its numerical value to obtain a feature sorting queue. Optionally, features with larger numerical values are sorted first, and features with smaller numerical values are sorted last. For example, the feature at the first feature position in the feature sorting queue has the largest numerical value. Optionally, multiple features with the same numerical value can be assigned to the same sequence position. Alternatively, the corresponding sequence positions can be determined according to preset rules.
[0068] Step S40: Select features according to the minimum number of features corresponding to each target data category based on the feature sorting queue, construct a target feature set based on the selected features, and output the target feature set and the target data source corresponding to the features in the target feature set.
[0069] Optionally, for each target data category, the feature with the fewest features needs to be selected, and the selection is performed from front to back according to the feature sorting queue.
[0070] Optionally, after selecting the feature with the fewest features under each target data category, all selected features can be aggregated to obtain the target feature set. The data source corresponding to all features in the target feature set can then be determined and used as the target data source. The target data source and target feature set are then output for subsequent business data processing.
[0071] For example, if it's necessary to integrate data from five scenarios to evaluate the subject, and to ensure objectivity, data from each scenario must be included, this embodiment can achieve this. Furthermore, this embodiment can add data categories to ensure that the features of the required scenarios are covered, effectively avoiding algorithmic and business biases introduced by relying solely on features. It can also select features through weighted scoring based on the discriminative power (or importance) of multi-dimensional features, increasing the flexibility and adaptability of feature selection. Moreover, by integrating business experience and feature selection technology to achieve automatic evaluation and selection of data sources and features, data source testing and evaluation can be completed more quickly, better meeting business and data performance requirements.
[0072] In this embodiment, at least one target data category is selected from multiple data sources for testing and evaluation, along with the minimum number of features corresponding to each target data category. A feature score is determined based on the discriminative power of each feature, and the features are then sorted according to their scores to obtain a feature ranking queue. Features are then selected based on the minimum number of features corresponding to each target data category according to the ranking queue. A target feature set is constructed based on the selected features, and the target feature set and the target data sources corresponding to the features within the target feature set are output. This approach incorporates data category considerations during feature selection, avoiding algorithmic bias caused by considering only features. Furthermore, by selecting features based on feature discriminative power and target data category, the low intelligence of data processing resulting from manual feature selection is avoided, achieving intelligent feature selection. Since feature extraction and selection are performed from multiple data sources, the flexibility and adaptability of feature selection are increased. Compared to manual feature selection, this significantly improves data processing efficiency.
[0073] Example 2
[0074] Based on the first embodiment of this application, in another embodiment of this application, the same or similar content as in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 Step S30, the step of determining the feature score corresponding to each feature based on the discriminative power of each feature, includes:
[0075] Step a: Calculate the discrimination score corresponding to each feature, and normalize each discrimination score to obtain the first score;
[0076] Step b: Determine the feature score corresponding to each feature based on the first score.
[0077] In this embodiment, the discriminative power of each feature can be determined using the Information Value (IV) method. For example, the features are first discretized, then the IV value for each feature is calculated, and the discriminative power is determined based on the IV value. Optionally, a higher IV value indicates a higher discriminative power of the distribution. Optionally, the importance of each feature can also be determined using a model-based method, and the discriminative power of each feature can be determined based on its importance. Optionally, discriminative power values corresponding to different levels of importance can be pre-constructed. Optionally, each feature can be input into a pre-defined neural network, the importance of each feature can be output, and the discriminative power of each feature can be determined based on the discriminative power values corresponding to different levels of importance.
[0078] Optionally, the discrimination score corresponding to each feature can be normalized, and the normalized value can be used as the score of the first score. Optionally, the feature score corresponding to each feature can be calculated based on the score of the first score.
[0079] In this embodiment, a first score is obtained by calculating the discriminative power corresponding to each feature and performing normalization processing. The feature score corresponding to each feature is driven by the first score, thereby ensuring the effectiveness of the determined feature score.
[0080] Further, the step of determining the feature score corresponding to each feature based on the first score includes:
[0081] Step c: Construct multiple feature groups, configure the priority of each feature group, and determine the feature group score of each feature group based on the priority;
[0082] In this embodiment, feature parameters can be configured, such as constructing multiple feature groups according to preset rules and configuring a corresponding priority for each feature group. Optionally, the number of feature groups can be one or more, and each feature group is different from the others.
[0083] Furthermore, the feature group includes at least one of the following:
[0084] Feature group one: Multiple features from the same data source have the same data category;
[0085] Feature group two: multiple features from different data sources have the same data category;
[0086] Feature group three: Features with different data categories from the same data source;
[0087] Feature group four: There is only one feature under the data source and / or data category.
[0088] Furthermore, the feature groups can be set not only according to feature groups one through four, but also according to other rules. For example, feature groups can be set according to different data categories. Optionally, only feature groups one through four will be used as examples in the following explanation.
[0089] Optionally, if the constructed feature groups are feature group one, feature group two, feature group three, and feature group four, for example... Figure 3 As shown, features A1001 and A1002 can be classified into feature group one, features B2004 and C2005 can be classified into feature group two, features B1003 and B2004 can be classified into feature group three, and feature C3001 can be classified into feature group four.
[0090] Priority can be configured for feature groups one through four, for example, configuring the priority as feature group one, feature group two, feature group three, and feature group four. The feature group score for each feature group can be set according to its priority. For example, the feature group score for feature group one can be set to 0.25; the feature group score for feature group two to 0.5; the feature group score for feature group three to 0.75; and the feature group score for feature group four to 1.
[0091] Optionally, the feature group scores for each priority level can be pre-set, and the number of feature group scores can be determined based on the number of feature groups. For example, if there are four feature groups, then there are four corresponding feature group scores, and the four feature group scores can be set based on a certain ratio or threshold.
[0092] Step d: Traverse each of the features and match the traversed features with multiple feature groups;
[0093] Optionally, all features within each data source can be traversed, and the traversed features can be matched sequentially with multiple feature groups to determine the matching feature group. Optionally, conditional judgments can be performed on the features, and the feature group matching the features can be determined based on the conditional judgment results. For example, if the traversed features satisfy the condition that multiple features from the same data source have the same data category, then the matching feature group can be determined as feature group one. If the traversed features satisfy the condition that multiple features from different data sources have the same data category, then the matching feature group can be determined as feature group two. If the traversed features satisfy the condition that features from the same data source have different data categories, then the matching feature group can be determined as feature group three. If the traversed features satisfy the condition that there is only one feature under the data source and / or data category, then the matching feature group can be determined as feature group four.
[0094] Step e: If there are multiple feature groups that match the traversed features, then the traversed feature groups are assigned to the highest priority feature group among the multiple matching feature groups.
[0095] Optionally, if there exists a feature group that matches the traversed features, the traversed features can be grouped into the matching feature group.
[0096] Optionally, if there are multiple feature groups that match the traversed features, the priority of each matching feature group can be determined, and the feature group with the highest priority can be selected and the traversed features can be grouped into the feature group with the highest priority. For example, if the matching feature groups include feature group two and feature group four, and feature group two has a higher priority than feature group four, then the traversed features can be grouped into feature group two.
[0097] Step f: Determine the second score based on the feature group score of the feature group corresponding to each feature, and determine the feature score corresponding to each feature based on the first score and the second score.
[0098] Optionally, the feature group corresponding to each feature can be determined in the manner described above, and each feature can be grouped into its corresponding feature group. Then, based on the feature group score corresponding to each feature group, the feature group score corresponding to each feature can be determined, and the feature group score corresponding to each feature can be used as the second score. For example, feature A1001 belongs to feature group one, feature B1003 belongs to feature group three, and the feature group score corresponding to feature group one is 0.25, while the feature group score corresponding to feature group three is 0.75. Then the second score corresponding to feature A1001 is 0.25, and the second score corresponding to feature B1003 is 0.75.
[0099] Optionally, the feature score corresponding to each feature can be calculated based on the first score and the second score. For example, a weighted calculation can be performed.
[0100] In this embodiment, by constructing multiple feature groups, determining the feature group score of each feature group according to the priority of each feature group, traversing each feature, and when the traversed feature matches multiple feature groups, grouping the traversed feature into the feature group with the highest priority, determining the second score according to the feature group score of the feature group corresponding to each feature, and determining the feature score corresponding to each feature according to the first score and the second score, thereby ensuring the effectiveness of the determined feature scores.
[0101] Further, the step of determining the feature score corresponding to each feature based on the first score and the second score includes:
[0102] Step g: For each feature group, determine the number of features belonging to the same feature group;
[0103] In this embodiment, a score range can be set, such as -1 to 1. The maximum value within the score range is set to 1, and the minimum value within the score range is set to -1. Optionally, a third score corresponding to each feature can be determined based on the score range and the feature group.
[0104] Optionally, for each feature group, a screening process can be performed to determine if features exist, thus obtaining feature groups with the same features, and then the number of features belonging to the same feature group can be determined. Alternatively, the operation of determining the number of features belonging to the same feature group can be performed directly for each feature group.
[0105] Step h: If the number of features belonging to the same feature group is one, then the third score corresponding to the feature is set to the preset interval maximum value;
[0106] Optionally, if there is only one feature belonging to the same feature group, that is, there is only one feature in the feature group, then the third score corresponding to the feature can be directly determined as the maximum value of the interval, such as 1.
[0107] Step i: If there are multiple features belonging to the same feature group, then perform similarity calculation on the multiple features within the feature group to obtain feature similarity, and determine the third score corresponding to the feature based on the feature similarity.
[0108] Optionally, if there are multiple features belonging to the same feature group, similarity can be calculated among all features within the feature group to obtain the similarity between features in the same group, i.e., feature similarity. Optionally, when calculating the similarity between multiple features within a feature group, semantic analysis can be performed on the multiple features within the feature group, and the feature similarity can be determined based on the semantic analysis results.
[0109] Optionally, when a feature group contains multiple features, the features to be prioritized for retention can be determined based on feature similarity. The third score corresponding to the feature to be prioritized for retention can be set to a maximum value in the interval, such as 1, and the third score corresponding to the other features in the feature group can be set to a minimum value in the interval, such as -1.
[0110] Step j: Determine the feature score corresponding to each feature based on the first score, the second score, and the third score.
[0111] Optionally, after determining the first score, second score, and third score corresponding to each feature, corresponding mathematical calculations can be performed based on the first score, second score, and third score to obtain the feature score corresponding to each feature.
[0112] In this embodiment, when there is only one feature belonging to the same feature group, the third score corresponding to that feature is set as the maximum value of the interval. When there are multiple features in the same feature group, similarity calculation is performed, and the third score corresponding to the feature is determined based on the feature similarity. The feature score corresponding to each feature is determined based on the first score, the second score, and the third score. This ensures the effectiveness of the determined feature scores.
[0113] Further, the step of determining the third score corresponding to the feature based on the feature similarity includes:
[0114] Step k: If there are two features whose feature similarity is greater than a preset similarity threshold, the two features corresponding to the feature similarity greater than the preset similarity threshold are taken as associated features;
[0115] Step 1: Compare the discriminative power of the two associated features, set the third score corresponding to the associated feature with high discriminative power to a preset maximum value in the interval, set the third score corresponding to the associated feature with low discriminative power to a preset minimum value in the interval, and set the third scores corresponding to the other features in the feature group other than the associated feature to the maximum value in the interval.
[0116] In this embodiment, if multiple features exist within the same feature group, and pairwise similarity is calculated for each feature, at least one feature similarity is obtained for each feature. For example, if the feature group includes feature 1, feature 2, and feature 3, then the feature similarity for feature 1 includes the feature similarity between feature 1 and feature 2, and the feature similarity between feature 1 and feature 3. The feature similarity for feature 2 includes the feature similarity between feature 1 and feature 2, and the feature similarity between feature 2 and feature 3. The feature similarity for feature 3 includes the feature similarity between feature 1 and feature 3, and the feature similarity between feature 2 and feature 3.
[0117] Optionally, if two features within the same feature group have a feature similarity greater than a preset similarity threshold, these two features can be considered as associated features. For example, if the feature similarity between feature 1 and feature 2 is greater than the preset similarity threshold, then feature 1 and feature 2 are considered as associated features. Optionally, the preset similarity threshold can be a threshold set by the user in advance, such as 0.6.
[0118] Optionally, for two related features with high feature similarity, determine the discriminant strength of the two related features, and set the third score of the related feature with higher discriminant strength to the maximum value of the interval, such as 1. Set the third score of the related feature with lower discriminant strength to the minimum value of the interval, such as -1. For other features in the feature group besides the related features, set the corresponding third score to the maximum value of the interval, such as 1.
[0119] For example, if a feature group contains two features and the feature similarity between the two features is greater than 0.6, the third score corresponding to the feature with higher discriminative power can be set to 1, and the third score corresponding to the feature with lower discriminative power can be set to -1. Optionally, if the feature similarity between the two features is less than 0.6, the third score corresponding to both features can be set to 1.
[0120] Optionally, if multiple features have a similarity greater than a preset similarity threshold, such as the feature similarity between feature 1 and feature 2, and the feature similarity between feature 2 and feature 3, both exceeding the preset similarity threshold, then the corresponding third score can be determined based on the discriminative power of these three features. For example, the third score for the feature with the highest discriminative power can be set to the maximum value of the interval, while the third scores for the other two features can be set to the minimum values of the interval.
[0121] In this embodiment, when there are associated features, the third scores corresponding to associated features with high discriminative power and other features are set to the maximum values of the interval, while the third scores corresponding to associated features with low discriminative power are set to the minimum values of the interval, thereby ensuring the effectiveness of the third scores of the determined features.
[0122] Further, the step of determining the feature score corresponding to each feature based on the first score, the second score, and the third score includes:
[0123] Step m involves weighting the first score, second score, and third score corresponding to each feature to obtain the feature score.
[0124] In this embodiment, after determining the first score, second score, and third score corresponding to each feature, a weighted score can be calculated for each feature to obtain the feature score. For example, the first score, second score, and third score can be directly added together to obtain the feature score. Alternatively, a weight can be assigned to the first score, a weight to the second score, and a weight to the third score, with the sum of the weights for the three scores being 1. Then, a weighted average of the three scores is calculated to obtain the feature score.
[0125] In this embodiment, the feature score is obtained by weighting the first score, second score and third score of each feature, thereby ensuring the validity of the determined feature score.
[0126] Example 3
[0127] This application also provides a data processing apparatus, please refer to... Figure 5 The data processing device includes:
[0128] The data source module A10 is used to determine multiple sets of data sources to be tested and evaluated, wherein each set of data sources includes at least one feature, each feature corresponds to a data category, and the same data category includes at least one feature;
[0129] The determination module A20 is used to determine at least one selected target data category among multiple data categories, and to determine the minimum number of features corresponding to each target data category;
[0130] The sorting module A30 is used to determine the feature score corresponding to each feature based on the discriminative power of each feature, and sort each feature according to the feature score to obtain a feature sorting queue;
[0131] The output module A40 is used to select features according to the minimum number of features corresponding to each target data category based on the feature sorting queue, construct a target feature set based on the selected features, and output the target feature set and the target data source corresponding to the features in the target feature set.
[0132] Optionally, the sorting module A30 is used for:
[0133] Calculate the discrimination score corresponding to each feature, and normalize each discrimination score to obtain a first score;
[0134] The feature score corresponding to each feature is determined based on the first score.
[0135] Optionally, the sorting module A30 is used for:
[0136] Construct multiple feature groups, configure the priority of each feature group, and determine the feature group score of each feature group based on the priority;
[0137] Each of the features is traversed, and the traversed features are matched with multiple feature groups;
[0138] If there are multiple feature groups that match the traversed features, then the traversed features are grouped into the highest priority feature group among the multiple matching feature groups.
[0139] A second score is determined based on the feature group score corresponding to each feature, and a feature score corresponding to each feature is determined based on the first score and the second score.
[0140] Optionally, the feature group includes at least one of the following:
[0141] Feature group one: Multiple features from the same data source have the same data category;
[0142] Feature group two: multiple features from different data sources have the same data category;
[0143] Feature group three: Features with different data categories from the same data source;
[0144] Feature group four: There is only one feature under the data source and / or data category.
[0145] Optionally, the sorting module A30 is used for:
[0146] For each feature group, determine the number of features belonging to the same feature group;
[0147] If the number of features belonging to the same feature group is one, then the third score corresponding to the feature is set to the preset interval maximum value;
[0148] If there are multiple features belonging to the same feature group, then the similarity of the multiple features within the feature group is calculated to obtain the feature similarity, and the third score corresponding to the feature is determined based on the feature similarity.
[0149] The feature score corresponding to each feature is determined based on the first score, the second score, and the third score.
[0150] Optionally, the sorting module A30 is used for:
[0151] If there are two features whose feature similarity is greater than a preset similarity threshold, the two features corresponding to the feature similarity greater than the preset similarity threshold are regarded as associated features;
[0152] The discrimination of the two associated features is compared. The third score corresponding to the associated feature with the higher discrimination is set to a preset maximum value in the interval, and the third score corresponding to the associated feature with the lower discrimination is set to a preset minimum value in the interval. The third scores corresponding to the other features in the feature group other than the associated feature are set to the maximum values in the interval.
[0153] Optionally, the sorting module A30 is used for:
[0154] The first score, second score, and third score corresponding to each feature are weighted and calculated to obtain the feature score.
[0155] The data processing apparatus provided in this application, employing the data processing method of any one of the embodiments 1 to 2 above, can improve the efficiency of data processing. Compared with the prior art, the beneficial effects of the data processing apparatus provided in this application are the same as those of the data processing methods provided in the above embodiments, and other technical features in the data processing apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.
[0156] Example 4
[0157] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the data processing method in the first embodiment described above.
[0158] The following is for reference. Figure 6 It shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0159] like Figure 6 As shown, an electronic device may include a processing unit (such as a central processing unit, graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) or a program loaded from a storage device into random access memory (RAM). The RAM also stores various programs and data required for the operation of the electronic device. The processing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.
[0160] Typically, the following systems can be connected to the I / O interface: input devices including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication devices. Communication devices allow electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although electronic devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.
[0161] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, it performs the functions defined above in the methods of embodiments of this disclosure.
[0162] The electronic device provided in this application, employing the data processing method described in the above embodiments, can improve data processing efficiency. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the data processing method described in the above embodiments, and other technical features of the electronic device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.
[0163] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.
[0164] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0165] Example 5
[0166] This application provides a computer-readable storage medium having computer-readable program instructions stored thereon, which are used to execute the data processing method described above.
[0167] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.
[0168] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.
[0169] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by an electronic device, enable the electronic device to perform the steps in the above embodiments.
[0170] Computer program code for performing the operations of this disclosure can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0171] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0172] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0173] The computer-readable storage medium provided in this application stores computer-readable program instructions for performing the above-described data processing method, thereby improving the efficiency of data processing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data processing method provided in Embodiment 1 or Embodiment 2, and will not be repeated here.
[0174] Example 6
[0175] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data processing method described above.
[0176] The computer program product provided in this application can improve the efficiency of data processing. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiments of this application are the same as the beneficial effects of the data processing methods provided in any of the above embodiments, and will not be repeated here.
[0177] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.
Claims
1. A data processing method, characterized in that, The data processing method includes the following steps: Multiple sets of data sources to be tested and evaluated are identified, wherein each set of data sources includes at least one feature, each feature corresponds to a data category, and the same data category includes at least one feature; Determine at least one target data category selected from multiple data categories, and determine the minimum number of features corresponding to each target data category; A feature score is determined for each feature based on its discriminative power, and each feature is sorted according to its feature score to obtain a feature sorting queue. Based on the feature sorting queue, feature selection is performed according to the minimum number of features corresponding to each target data category. A target feature set is constructed based on the selected features, and the target feature set and the target data source corresponding to the features in the target feature set are output. The step of determining the feature score corresponding to each feature based on the discriminative power of each feature includes: Calculate the discrimination score corresponding to each feature, and normalize each discrimination score to obtain a first score; Construct multiple feature groups, configure the priority of each feature group, and determine the feature group score of each feature group based on the priority; Each of the features is traversed, and the traversed features are matched with multiple feature groups; If there are multiple feature groups that match the traversed features, then the traversed features are grouped into the highest priority feature group among the multiple matching feature groups. The second score is determined based on the feature group score corresponding to each of the aforementioned features; For each feature group, determine the number of features belonging to the same feature group; If the number of features belonging to the same feature group is one, then the third score corresponding to the feature is set to the preset interval maximum value; The first score, second score, and third score corresponding to each feature are weighted and calculated to obtain the feature score.
2. The data processing method as described in claim 1, characterized in that, The feature set includes at least one of the following: Feature group one: Multiple features from the same data source have the same data category; Feature group two: multiple features from different data sources have the same data category; Feature group three: Features with different data categories from the same data source; Feature group four: There is only one feature under the data source and / or data category.
3. The data processing method as described in claim 1, characterized in that, After determining the number of features belonging to the same feature group for each feature group, the method further includes: If there are multiple features belonging to the same feature group, then the similarity of the multiple features within the feature group is calculated to obtain the feature similarity, and the third score corresponding to the feature is determined based on the feature similarity.
4. The data processing method as described in claim 3, characterized in that, The step of determining the third score corresponding to the feature based on the feature similarity includes: If there are two features whose feature similarity is greater than a preset similarity threshold, the two features corresponding to the feature similarity greater than the preset similarity threshold are regarded as associated features; The discrimination of the two associated features is compared. The third score corresponding to the associated feature with the higher discrimination is set to a preset maximum value in the interval, and the third score corresponding to the associated feature with the lower discrimination is set to a preset minimum value in the interval. The third scores corresponding to the other features in the feature group other than the associated feature are set to the maximum values in the interval.
5. A data processing apparatus, characterized in that, The data processing device includes: The data source module is used to determine multiple sets of data sources to be tested and evaluated, wherein each set of data sources includes at least one feature, each feature corresponds to a data category, and the same data category includes at least one feature; The determination module is used to determine at least one selected target data category among multiple data categories, and to determine the minimum number of features corresponding to each target data category; A sorting module is used to determine the feature score corresponding to each feature based on the discriminative power of each feature, and sort each feature according to the feature score to obtain a feature sorting queue. The determination of the feature score corresponding to each feature based on the discriminative power of each feature includes: calculating the discriminative power of each feature and normalizing each discriminative power to obtain a first score; constructing multiple feature groups and configuring the priority of each feature group, and determining the feature group score of each feature group according to the priority; traversing each feature and matching the traversed feature with multiple feature groups; if there are multiple feature groups that match the traversed feature, grouping the traversed feature into the highest priority feature group among the determined matching feature groups; determining a second score based on the feature group score of each feature group; for each feature group, determining the number of features belonging to the same feature group; if the number of features belonging to the same feature group is one, setting the third score corresponding to the feature to a preset interval maximum value; and performing a weighted calculation on the first score, second score, and third score corresponding to each feature to obtain a feature score. The output module is used to select features according to the minimum number of features corresponding to each target data category based on the feature sorting queue, construct a target feature set based on the selected features, and output the target feature set and the target data source corresponding to the features in the target feature set.
6. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the data processing method according to any one of claims 1 to 4.
7. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, on which a program implementing the data processing method is stored, and the program implementing the data processing method is executed by a processor to implement the steps of the data processing method as described in any one of claims 1 to 4.