Data processing method and related apparatus

Abstract

The application provides a data processing method and related device, which can be applied to big data scenarios such as the map field and the traffic field. The method comprises the following steps: receiving target training data sets of a target field sent by a management node, wherein the target training data sets comprise multiple original features, and each training data set comprises original features with the same feature identifier; calling a specific distribution function corresponding to the target field to generate first random features of the target training data sets; performing feature contribution evaluation processing on each original feature in the target training data sets according to the first random features to obtain the feature type of each original feature; screening target features from the multiple original features according to the feature type of each original feature; and sending the screened target features to the management node, so that the management node determines screening features according to the target features in k time periods. The application can improve the accuracy and efficiency of feature screening.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, a data processing apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In big data scenarios such as finance, mapping, and transportation, feature selection has become a crucial part of data analysis. Currently, feature selection methods primarily involve manually filtering multiple raw features from the training dataset to obtain the target features. However, this manual method has relatively low accuracy; therefore, improving the accuracy of feature selection is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] This application provides a data processing method, apparatus, system, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy and efficiency of feature selection.

[0004] On one hand, embodiments of this application provide a data processing method, the method comprising:

[0005] The target training dataset for the target domain is sent by the management node. The target training dataset includes multiple original features. The target training dataset is one of k training datasets. Each training dataset corresponds to a time period. The original features contained in each training dataset have the same feature labels. k is a positive integer.

[0006] Call the specific distribution function corresponding to the target domain to generate the first random feature of the target training dataset;

[0007] Based on the first random feature, feature contribution evaluation is performed on each original feature in the target training dataset to obtain the feature type of each original feature.

[0008] Target features are selected from multiple original features based on the feature type of each original feature;

[0009] The selected target features are sent to the management node so that the management node can determine the selected features based on the target features over k time periods.

[0010] On one hand, embodiments of this application provide a data processing method, the method comprising:

[0011] Obtain a sample dataset belonging to the target domain within the target time period. The target time period includes k time periods. The sample dataset includes training datasets corresponding to the k time periods. Each training dataset includes multiple original features. The feature labels of the original features contained in each training dataset are the same. k is a positive integer.

[0012] K training datasets are sent to k worker nodes respectively, so that the target worker node can select the target feature from multiple original features according to the feature type of each original feature in the target training dataset. The feature type of each original feature is determined by the target worker node after performing feature contribution evaluation processing on the original feature according to the first random feature. The first random feature is a feature generated by calling a specific distribution function corresponding to the target domain. The target worker node is any one of the k worker nodes, and the target training dataset is the training dataset received by the target worker node.

[0013] Receive target features sent by k working nodes, summarize the received target features to obtain filtering features, and output the filtering features.

[0014] On one hand, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0015] The receiving unit is used to receive the target training dataset of the target domain sent by the management node. The target training dataset includes multiple original features. The target training dataset is one of k training datasets. Each training dataset corresponds to a time period. The original features contained in each training dataset have the same feature labels. k is a positive integer.

[0016] The processing unit is used to call a specific distribution function corresponding to the target domain to generate the first random feature of the target training dataset;

[0017] The processing unit is also used to perform feature contribution evaluation processing on each original feature in the target training dataset according to the first random feature, so as to obtain the feature type of each original feature;

[0018] The processing unit is also used to filter target features from multiple raw features based on the feature type of each raw feature;

[0019] The sending unit is used to send the selected target features to the management node, so that the management node can determine the selected features based on the target features over k time periods.

[0020] In one possible implementation, the feature types include accept feature types, reject feature types, and pending feature types; the processing unit filters target features from the plurality of original features according to the feature type of each original feature, for the following operations:

[0021] Based on the feature type of each original feature, adjust the list of accepted features, the list of rejected features, and the list of pending features, with each feature list corresponding to one feature type;

[0022] When the iteration stopping condition is met, the original features recorded in the accept feature list and the undetermined feature list will be used as the target features.

[0023] When the iteration stopping condition is not met, the original features recorded in the accepted feature list and the pending feature list will be combined into a new target training dataset.

[0024] In one possible implementation, the iteration stopping condition is determined to be met when the cumulative number of iterations of the target training dataset reaches a preset number of iterations; or,

[0025] The iteration stopping condition is met when the accept feature list, reject feature list, and pending feature list remain unchanged.

[0026] In one possible implementation, the processing unit performs feature contribution evaluation on each original feature in the target training dataset based on a first random feature, obtaining the feature type of each original feature, which is then used to perform the following operations:

[0027] Randomly rearrange the feature values ​​of multiple original features in the target training dataset to determine multiple second random features;

[0028] The feature scoring model is invoked to evaluate the feature contribution of the first random feature, multiple second random features, and each original feature in the target training dataset, and the evaluation scores of each original feature, the first random feature, and each second random feature are obtained.

[0029] Based on the evaluation scores of each original feature, the evaluation score of the first random feature, and the evaluation scores of each second random feature, feature correction processing is performed on each original feature to obtain the feature type of each original feature.

[0030] In one possible implementation, the processing unit performs feature correction processing on each original feature based on the evaluation scores of each original feature, the evaluation scores of the first random feature, and the evaluation scores of each second random feature, to obtain the feature type of each original feature, which is then used to perform the following operations:

[0031] Based on the evaluation scores of each original feature, each first random feature, and each second random feature, the feature importance of each original feature is evaluated to obtain the number of successful trials for each original feature.

[0032] Based on the number of successful trials for each original feature, feature correction processing is performed on each original feature to obtain the correction result for each original feature;

[0033] Based on the correction results of each original feature, the feature type of each original feature is obtained.

[0034] In one possible implementation, the processing unit evaluates the feature importance of each original feature based on the evaluation scores of each original feature, the evaluation scores of the first random feature, and the evaluation scores of each second random feature, to obtain the number of successful trials for each original feature, which is then used to perform the following operations:

[0035] The highest target evaluation score is determined based on the evaluation scores of the first random feature and the evaluation scores of each of the second random features.

[0036] If the evaluation score of any original feature is higher than the target evaluation score, then the number of successful trials for that original feature is adjusted.

[0037] In one possible implementation, the processing unit performs feature correction processing on each original feature based on the experimental results of each original feature, obtaining a correction result for each original feature, which is then used to perform the following operations:

[0038] Obtain the cumulative number of iterations for the target training dataset;

[0039] Based on the number of successful trials and the cumulative number of iterations for any original feature, determine the significance level value to be corrected for any original feature;

[0040] The significance level value of any original feature to be corrected is corrected to obtain the corrected result of any original feature.

[0041] In one possible implementation, the processing unit performs correction processing on the saliency level value to be corrected of any original feature to obtain the correction result of any original feature, which is then used to perform the following operations:

[0042] The significance level value of any original feature to be corrected is corrected according to the first correction method to obtain the corrected significance level value of any original feature.

[0043] The significance level value of any original feature is corrected again according to the second correction method to obtain the correction result of any original feature.

[0044] In one possible implementation, the correction result for any original feature includes a reference corrected cumulative distribution value and a reference corrected survival function value;

[0045] Based on the correction result of each original feature, the processing unit obtains the feature type of each original feature, which is then used to perform the following operations:

[0046] If the reference correction cumulative distribution value is less than the preset threshold, then the feature type of any original feature will be determined as the accepted feature type;

[0047] If the reference corrected survival function value is less than the preset threshold, then the feature type of any original feature will be determined as the rejection feature type;

[0048] If the reference corrected cumulative distribution value and the reference corrected survival function value of any original feature are both greater than the preset threshold, then the feature type of any original feature is determined as a feature type to be determined.

[0049] In one possible implementation, there are multiple specific distribution functions, any one of which includes any of the following: log-normal distribution function, binomial distribution function, Bernoulli distribution function, exponential distribution function, uniform distribution function, and normal distribution function.

[0050] On one hand, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0051] The acquisition unit is used to acquire a sample dataset belonging to the target domain within a target time period. The target time period includes k time periods, and the sample dataset includes training datasets corresponding to the k time periods. Each training dataset includes multiple original features, and the feature labels of the original features contained in each training dataset are the same. k is a positive integer.

[0052] The sending unit is used to send k training datasets to k worker nodes respectively, so that the target worker node can select the target feature from multiple original features according to the feature type of each original feature in the target training dataset. The feature type of each original feature is determined by the target worker node after performing feature contribution evaluation processing on the original feature according to the first random feature. The first random feature is a feature generated by calling a specific distribution function corresponding to the target domain. The target worker node is any one of the k worker nodes, and the target training dataset is the training dataset received by the target worker node.

[0053] The processing unit is used to receive target features sent by k working nodes, summarize the received target features, obtain the filtering features, and output the filtering features.

[0054] On one hand, embodiments of this application propose a data processing system, which includes a management node and worker nodes, comprising the following steps:

[0055] The management node obtains a sample dataset belonging to the target domain within the target time period. The target time period includes k time periods, and the sample dataset includes training datasets corresponding to the k time periods. Each training dataset includes multiple original features, and the feature labels of the original features contained in each training dataset are the same. k is a positive integer.

[0056] The management node sends the target training dataset to the worker nodes. The target training dataset is one of the k training datasets.

[0057] The worker node invokes a specific distribution function corresponding to the target domain to generate the first random feature of the target training dataset;

[0058] The worker node performs feature contribution evaluation on each original feature in the target training dataset based on the first random feature, and obtains the feature type of each original feature;

[0059] The worker node selects the target feature from multiple raw features based on the feature type of each raw feature;

[0060] The worker nodes will send the selected target features to the management node;

[0061] The management node will summarize the received target features, obtain the filtering features, and output the filtering features.

[0062] On one hand, embodiments of this application provide a computer device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the data processing method described above.

[0063] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when read and executed by a processor of a computer device, causes the computer device to perform the aforementioned data processing method.

[0064] On one hand, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned data processing method.

[0065] As can be seen, in this application, during the feature selection process for each original feature, feature contribution evaluation can be performed based on the first random feature. Since feature selection references both the original features of the training dataset itself and adds the first random feature related to the target domain of the original features, the sample data for feature selection is enriched, thereby improving the accuracy of feature selection. Furthermore, a worker node can be responsible for processing multiple original features of a training dataset for one period to select target features. It can be understood that k training datasets can be executed synchronously by k worker nodes, and finally, the target features selected by each node are sent to the management result aggregation. Compared to a single worker node processing a training dataset one by one, this embodiment of the application allows for distributed processing, thereby improving the efficiency of feature selection. Attached Figure Description

[0066] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a schematic diagram illustrating the principle of a data processing scheme provided in an embodiment of this application;

[0068] Figure 2 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application;

[0069] Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0070] Figure 4 This is a flowchart illustrating a method for determining a feature type, as provided in an embodiment of this application.

[0071] Figure 5 This is a schematic diagram of a process for generating random features provided in an embodiment of this application;

[0072] Figure 6 This is a flowchart illustrating a feature scoring model provided in an embodiment of this application;

[0073] Figure 7 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0074] Figure 8 This is an interactive schematic diagram of a data processing system provided in an embodiment of this application;

[0075] Figure 9This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0076] Figure 10 This is a schematic diagram of another data processing device provided in an embodiment of this application;

[0077] Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0078] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0079] It should be noted that in the subsequent specific embodiments of this application, data related to user information (such as the user's gender and age) are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0080] This application proposes a data processing scheme that can invoke a specific distribution function corresponding to the target domain to generate the first random feature of the target training dataset, thereby improving the accuracy of feature selection. The selected features obtained based on this scheme can be applied to big data application scenarios such as finance, mapping, and transportation, specifically providing corresponding data support for these big data application scenarios. Please refer to... Figure 1 , Figure 1 This is a schematic diagram of a data processing scheme provided in an embodiment of this application. The following will be combined with... Figure 1 A brief description of the general principle of the data processing scheme provided in the embodiments of this application is as follows:

[0081] First, the worker nodes can receive a target training dataset (e.g., in the financial domain) sent by the management node. This target training dataset includes multiple raw features and is one of k training datasets, each corresponding to a time period. The raw features in each training dataset have the same feature labels, where k is a positive integer. Then, a specific distribution function corresponding to the target domain can be invoked to generate a first random feature for the target training dataset. In one possible implementation, the worker nodes can also generate a second random feature based on the raw features.

[0082] Then, the worker node can evaluate the feature contribution of each original feature in the target training dataset based on the first random feature to obtain the feature type of each original feature. Specifically, the worker node can use the XGBoost (Extreme Gradient Boosting) model to rank the feature importance of the first random feature, the second random feature, and the original features to obtain the number of successful trials for each original feature; and perform feature validity testing and various test correction methods (including but not limited to: FDR (False Discovery Rate) correction, Bonferroni correction, etc.) on each original feature to obtain the feature type of each original feature, where the feature type of the original feature can include any of the following: rejection feature type, acceptance feature type, and undetermined feature type.

[0083] Next, the worker nodes can filter out target features from multiple raw features based on the feature type of each raw feature. Understandably, this process can be repeated iteratively until the worker nodes have obtained the target features from the raw features. Finally, the filtered target features can be sent to the management node, allowing the management node to determine the selected features based on the target features over k time periods.

[0084] Therefore, in this application, during the feature selection process for each original feature, feature contribution evaluation can be performed based on the first random feature. Since feature selection considers both the original features of the training dataset itself and adds the first random feature related to the target domain of the original features, the sample data for feature selection is enriched, thereby improving the accuracy of feature selection. Furthermore, a worker node can be responsible for processing multiple original features of a training dataset for one period to select target features. It can be understood that k training datasets can be executed synchronously by k worker nodes, and finally, the target features selected by each node are sent to the management node for aggregation. Compared to a single worker node processing a training dataset one by one, the embodiments of this application can perform distributed processing, thereby improving the efficiency of feature selection.

[0085] Next, the data processing scheme mentioned above will be described using the technical terminology used in this application:

[0086] I. Artificial Intelligence:

[0087] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0088] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0089] In one possible implementation, the data processing scheme of this application can be combined with machine learning techniques in the field of artificial intelligence. For example, a feature scoring model (e.g., an XGBoost model) can be trained using machine learning techniques, and then, based on a first random feature, the feature scoring model can be used to evaluate the feature contribution of each original feature in the target training dataset to obtain the feature type of each original feature.

[0090] II. Blockchain:

[0091] A blockchain network is a network composed of a peer-to-peer (P2P) network and a blockchain. A blockchain refers to a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, it is a decentralized database, which is a series of data blocks (or blocks) linked together using cryptographic methods.

[0092] In one possible implementation, the data processing scheme of this application can be combined with blockchain technology. For example, the training dataset corresponding to each period (i.e., k training datasets) can be uploaded to the blockchain of a blockchain network for storage to prevent the internal data of computer devices from being tampered with, thereby improving the security and privacy of log data.

[0093] III. Cloud Technology:

[0094] Cloud computing is a computing model that distributes computing tasks across a large pool of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, resources in the "cloud" appear infinitely scalable, readily available, on-demand, and expandable, with payment based on usage.

[0095] In one possible implementation, when executing the data processing scheme of this application, each original feature in the target training dataset is evaluated for feature contribution based on the first random feature to obtain the feature type of each original feature. This process involves large-scale computation and requires a large amount of computing power and storage space. Therefore, in one possible implementation of this application, a computer device can obtain sufficient computing power and storage space through cloud computing technology to execute the determination of the feature type of each original feature involved in this application.

[0096] Based on the above description, please refer to the following: Figure 2 , Figure 2 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application. For example... Figure 2 As shown in the schematic diagram, the data processing system may include a server 240 and a cluster of terminal devices. The cluster of terminal devices may include terminal devices 210, 220, ..., 230, etc. Any terminal device in the cluster can be directly or indirectly connected to the server 240 via wired or wireless communication; this application does not impose any restrictions on this. Furthermore, any two terminal devices in the cluster can be of the same or different types; this application also does not impose any restrictions on this.

[0097] Figure 2The server 240 shown can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc.

[0098] Figure 2 The terminal devices 210, 220, and 230 shown may include, but are not limited to: mobile phones, tablets, laptops, handheld computers, mobile internet devices (MIDs), smart voice interaction devices, in-vehicle terminals, roadside devices, aircraft, wearable devices, smart home appliances, or wearable devices with data processing functions such as smartwatches, smart bracelets, and pedometers, etc.

[0099] In one possible implementation, taking terminal device 210 as an example, when terminal device 210 and server 240 jointly execute the data processing scheme of this application, terminal device 210 can act as a worker node, and server 240 can act as a management node. Specifically, terminal device 210 can receive a target training dataset of the target domain sent by server 240. The target training dataset includes multiple original features. The target training dataset is one of k training datasets, each corresponding to a time period. The original features contained in each training dataset have the same feature identifier, where k is a positive integer. Then, terminal device 210 can call a specific distribution function corresponding to the target domain to generate a first random feature of the target training dataset. Next, terminal device 210 can perform feature contribution evaluation processing on each original feature in the target training dataset based on the first random feature to obtain the feature type of each original feature. Then, terminal device 210 can filter target features from multiple original features based on the feature type of each original feature. Finally, terminal device 210 can send the filtered target features to server 240, so that server 240 can determine the filtered features based on the target features of the k time periods.

[0100] Subsequently, server 240 can send the filtering features to each working node in the terminal device cluster (e.g., terminal device 210, terminal device 220, etc.). It should be understood that the above is only an illustrative description of the steps performed by terminal device 210 and server 240, and the embodiments of this application do not limit this.

[0101] In another possible implementation, let's take a terminal device 210 as the target worker node and a server 240 as the management node as an example. First, the server 240 can obtain a sample dataset belonging to the target domain within a target time period. The target time period includes k time cycles, and the sample dataset includes training datasets corresponding to each of the k time cycles. Each training dataset includes multiple original features, and the feature identifiers of the original features in each training dataset are the same, where k is a positive integer. Then, the server 240 can send the k training datasets to the k worker nodes (e.g., terminal device 210, terminal device 220, etc.) respectively, so that the target worker node (terminal device 210) can filter out target features from the multiple original features based on the feature type of each original feature. The feature type of each original feature is determined by the target worker node after performing feature contribution evaluation processing on the original feature based on a first random feature. The first random feature is a feature generated by calling a specific distribution function corresponding to the target domain. The target worker node is any one of the k worker nodes. Finally, the server 240 can receive the target features sent by the k worker nodes, summarize the received target features, obtain the filtered features, and output the filtered features.

[0102] Furthermore, it can be Figure 1 The provided data processing system is deployed on blockchain nodes. For example, terminal devices 210, 220, and server 240 can all be considered blockchain node devices, collectively forming a blockchain network. Specifically, terminal devices 210 and 220 can act as working nodes in the blockchain, while server 240 can act as a management node. Therefore, the data processing flow involved in this application can be executed on the blockchain, which ensures fairness and impartiality in the data processing flow, while also making the data processing flow traceable, thereby improving the security of the data processing flow.

[0103] It is understood that the system architecture diagrams described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0104] Based on the above description of the data processing scheme and data processing system, this application proposes a data processing method. See also... Figure 3 As shown, Figure 3This is a flowchart illustrating a data processing method provided in an embodiment of this application. This data processing method can be executed by the aforementioned terminal device (i.e., working node). For ease of explanation, the following description will use a computer device executing this data processing method as an example. The data processing method may include the following steps S301 to S305:

[0105] S301: Receive the target training dataset of the target domain sent by the management node. The target training dataset includes multiple original features. The target training dataset is one of the k training datasets. Each training dataset corresponds to a time period. The original features contained in each training dataset have the same feature labels. k is a positive integer.

[0106] In this embodiment, the target domain may include, but is not limited to, big data domains such as finance, the internet, mapping, and transportation. The target training dataset is any one of k training datasets, with each dataset corresponding to a time period. If the time period is one week, the target training dataset may include multiple original features corresponding to one week; if the time period is one month, the target training dataset may include multiple original features corresponding to one month. If the target domain is finance, the original features included in the target training dataset may be, for example, gender, age, credit, etc. Furthermore, any original feature may include a feature value (e.g., a vector or matrix) and a feature identifier (e.g., a unique identifier such as an ID or a hash value of the feature name). It is understood that the feature identifiers of the original features included in each of the k training datasets are the same. This means that both the first and second training datasets may include gender, age, and credit features, but the first training dataset records multiple original features obtained in the first week of January, and the second training dataset records multiple original features obtained in the second week of January.

[0107] S302: Call the specific distribution function corresponding to the target domain to generate the first random feature of the target training dataset.

[0108] In one possible implementation, if the target domain is the financial domain, the specific distribution function corresponding to the financial domain may include, but is not limited to: log-normal distribution function, binomial distribution function, Bernoulli distribution function, exponential distribution function, uniform distribution function, and normal distribution function.

[0109] It is understood that the number of first random features can be one or more, and a specific type of distribution function can generate one or more first random features. For example, a computer device can call the log-normal distribution function corresponding to the target domain to generate multiple first random features. Or, a computer device can call the log-normal distribution function and the binomial distribution function corresponding to the target domain to generate multiple first random features, and so on. The embodiments of this application do not specifically limit the number of specific distribution functions or the number of first random features.

[0110] By generating the first random features of these common feature distributions in the financial field through a specific distribution function, the richness of the basic feature distribution can be guaranteed, and the stability of feature selection can be enhanced.

[0111] S303: Based on the first random feature, perform feature contribution evaluation processing on each original feature in the target training dataset to obtain the feature type of each original feature.

[0112] In one possible implementation, the computer device invokes a feature scoring model to evaluate the feature contribution of each original feature in the first random feature and the target training dataset, obtaining an evaluation score for each original feature and an evaluation score for the first random feature. Then, based on the evaluation scores of each original feature and the evaluation score of the first random feature, the computer device performs feature correction processing on each original feature to obtain the feature type of each original feature.

[0113] Specifically, the computer device calls the feature scoring model to evaluate the feature contribution of the first random feature and each original feature in the target training dataset, and obtains the evaluation score of each original feature and the evaluation score of the first random feature. This can include: concatenating the first random feature and each original feature in the target training dataset to obtain a first concatenated feature; and then inputting the first concatenated feature into the feature scoring model for feature contribution evaluation, so as to output the evaluation score of each original feature and the evaluation score of the first random feature.

[0114] In another possible implementation, the computer device randomly rearranges the feature values ​​of multiple original features in the target training dataset to determine multiple second random features. Then, the computer device invokes a feature scoring model to evaluate the feature contribution of the first random features, the multiple second random features, and each original feature in the target training dataset, obtaining evaluation scores for each original feature, the first random feature, and each second random feature. Finally, based on the evaluation scores of each original feature, the first random feature, and the second random feature, the computer device performs feature correction processing on each original feature to obtain the feature type of each original feature.

[0115] Specifically, the computer device invokes a feature scoring model to evaluate the feature contribution of a first random feature, multiple second random features, and each original feature in the target training dataset, obtaining evaluation scores for each original feature, the first random feature, and each second random feature. This can include: concatenating the first random feature, multiple second random features, and each original feature in the target training dataset to obtain a second concatenated feature; then inputting the second concatenated feature into the feature scoring model for feature contribution evaluation, so as to output the evaluation scores for each original feature, the first random feature, and each second random feature.

[0116] For example, the feature scoring models mentioned above may include, but are not limited to, Random Forest models, XGBoost models, etc. In this embodiment, the XGBoost model can be used as the feature scoring model. The XGBoost model is a classic ensemble boosting algorithm framework widely used in various fields of machine learning. It is understood that in this embodiment, the randomness of the XGBoost model can be achieved through its column sampling function. Since the XGBoost model has been widely used and proven effective in the industry, using this model can ensure relatively stable performance, thereby guaranteeing the accuracy of feature contribution evaluation.

[0117] S304: Select the target feature from multiple original features based on the feature type of each original feature.

[0118] In one possible implementation, the feature types of the original features can include accept feature types, reject feature types, and pending feature types. Acceptable feature types indicate that the original features are of high importance and have high reference value for data analysis in the target domain, and can be accepted. Rejected feature types indicate that the original features are of low importance and have low reference value for data analysis in the target domain, and can be rejected. Pending feature types indicate that the original features are of moderate importance and their reference value for data analysis in the target domain is uncertain; the acceptance or rejection of this type of original feature can be customized according to requirements. In this way, compared to directly rejecting or accepting features, adding pending feature types as a transition can improve the accuracy of feature selection.

[0119] Specifically, the computer device selects target features from multiple original features based on the feature type of each original feature. This can include: first, adjusting the accept feature list, reject feature list, and pending feature list according to the feature type of each original feature, with each feature list corresponding to one feature type; then, when the iteration stopping condition is met, the computer device uses the original features recorded in the accept feature list and the pending feature list as target features; finally, when the iteration stopping condition is not met, the computer device combines the original features recorded in the accept feature list and the pending feature list into a new target training dataset.

[0120] For example, suppose the target training dataset contains m original features, which can be represented as original feature 1, original feature 2, original feature 3, ... original features, where m is a positive integer. For instance, if original feature 1 is an accept feature, the computer can add it to the accept feature list, which records its feature identifier and value. Similarly, if original feature 2 is a reject feature, the computer can add it to the reject feature list, which records its feature identifier and value. Likewise, if original feature 3 is a pending feature, the computer can add it to the pending feature list, which records its feature identifier and value. In this way, the accept feature list, reject feature list, and pending feature list can be adjusted according to the feature type of each original feature. The accept feature list, reject feature list, and pending feature list are shown in Tables 1-3 below.

[0121] Table 1. List of Acceptable Features

[0122] Feature Name Feature identifier Eigenvalues gender id1 [0.1，0.2，0.3] age id2 [0.1，-0.2，0.5] Credit id3 [-0.3，0.1，0.6]

[0123] Table 2. List of Rejection Features

[0124] Feature Name Feature identifier Eigenvalues Feature a id3 [-0.1，0.2，-0.3] ... ... ... Feature b id5 [0.2，-0.1，-0.6]

[0125] Table 3. List of features to be determined

[0126] Feature Name Feature identifier Eigenvalues Feature x id6 [0.4，0.5，0.8] ... ... ... Feature y Id8 [1.2，0.1，-0.6]

[0127] In one possible implementation, the iteration stopping condition is determined to be met when the cumulative number of iterations of the target training dataset reaches a preset number of iterations; or, the iteration stopping condition is determined to be met when the accept feature list, reject feature list, and pending feature list remain unchanged. It is understood that the specific processes involved in S301-S304 above can be executed iteratively until the iteration stopping condition is met, at which point the iteration stops. For example, the preset number of iterations can be set to 100, then the iteration stopping condition is determined to be met when the cumulative number of iterations of the target training dataset reaches 100; or, for example, the iteration stopping condition is determined to be met when the accept feature list, reject feature list, and pending feature list after the nth iteration and after the (n+1)th iteration remain unchanged. This iterative execution method can eliminate the randomness introduced by a single processing step, thereby improving the accuracy of the selected target features.

[0128] In another possible implementation, when the iteration stopping condition is met, the computer device accepts the original features recorded in the feature list as target features and randomly selects a preset number of original features from the pending feature list as target features. For example, if the accepting feature list records 10 original features and the pending feature list records 20 original features, with a preset number of 8, then the computer device can randomly select 8 original features from the 20 original features recorded in the pending feature list and use these 8 original features as target features; and also use the 10 original features recorded in the accepting feature list as target features. Similarly, when the iteration stopping condition is not met, the computer device can randomly select 8 original features from the 20 original features recorded in the pending feature list and use the 10 original features recorded in the accepting feature list as a new target training dataset, meaning the new target training dataset includes 18 original features.

[0129] S305: Send the selected target features to the management node so that the management node can determine the selected features based on the target features over k time periods.

[0130] It is understood that the above process refers to the detailed steps executed by any worker node. In this embodiment, training datasets for k time periods can be broadcast to k worker nodes using a distributed algorithm (such as Spark, an open-source general-purpose big data computing framework and currently the mainstream big data computing engine in the industry). Each worker node then obtains a training dataset for one time period, and the k worker nodes jointly execute the above process, with each worker node obtaining the selected target features. Finally, each worker node can send the selected target features to the management node, which determines the selected features based on the target features for the k time periods. Through this distributed processing method, the training time for k training datasets can be shortened to 1 / k, thereby improving the efficiency of feature selection.

[0131] In this embodiment, firstly, a first random feature representing the common feature distributions within the financial field can be generated using a specific distribution function. This ensures the richness of the basic feature distribution and enhances the stability of feature selection. Then, using Spark distributed processing, the training time for k training datasets can be reduced to 1 / k, thereby improving the efficiency of feature selection. Therefore, in this application, feature contribution evaluation can be performed based on the first random feature during feature selection for each original feature. Since feature selection considers both the original features of the training dataset itself and adds the first random feature related to the target domain of the original features, the sample data for feature selection is enriched, thus improving the accuracy of feature selection. Furthermore, a worker node can be responsible for processing multiple original features of a training dataset for one period to select target features. It can be understood that k training datasets can be executed synchronously by k worker nodes, and the target features selected by each node are ultimately sent to the management result aggregation. Compared to a single worker node processing a training dataset one by one, this embodiment allows for distributed processing, thereby improving the efficiency of feature selection.

[0132] based on Figure 3 The following is a description of the embodiments; please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic flowchart illustrating a method for determining a feature type according to an embodiment of this application. Figure 4 Examples can be Figure 3 This is a specific embodiment of step S303 in the implementation example. The feature type determination method can be executed by the aforementioned terminal device (i.e., the working node). For ease of explanation, the following description will use a computer device executing the feature type determination method as an example. The feature type determination method may include the following steps S401 to S505:

[0133] S401: Randomly rearrange the feature values ​​of multiple original features in the target training dataset to determine multiple second random features.

[0134] In this embodiment of the application, each original feature in the target training dataset includes a feature value and a feature label, and one feature value is associated with a corresponding feature label. If the target training dataset includes m original features in the first iteration, the computer device can randomly rearrange the feature values ​​of the m original features, which may include: keeping the feature labels of the m original features unchanged, and randomly rearranging the feature values ​​of the m original features to obtain m second random features.

[0135] In one possible implementation, at the start of an iteration, a random number seed corresponding to the current iteration can be generated using a random number function, and then a second random characteristic for the current iteration can be generated based on the random number seed. For example, see [link to example]. Figure 5 , Figure 5 This is a schematic diagram of a process for generating random features provided in an embodiment of this application. For example... Figure 5 As shown, assuming each original feature stored in the target training dataset corresponds to a feature label, and during the random rearrangement of the m original features, the label column remains unchanged, i.e., the feature labels remain unchanged. The feature values ​​of the m original features are randomly rearranged according to a random number seed. It can be understood that the number of second random features obtained after random rearrangement is the same as the number of original features. In this way, it can be ensured that the generated second random features maintain the distribution of the original features while eliminating the association between the original features and specific labels.

[0136] S402: Call the feature scoring model to evaluate the feature contribution of the first random feature, multiple second random features, and each original feature in the target training dataset, and obtain the evaluation score of each original feature, the evaluation score of the first random feature, and the evaluation score of each second random feature.

[0137] The first random feature is a random feature generated by the computer device by calling a specific distribution function corresponding to the target domain. If the target domain is the financial domain, the specific distribution function corresponding to the financial domain may include, but is not limited to: log-normal distribution function, binomial distribution function, Bernoulli distribution function, exponential distribution function, uniform distribution function, and normal distribution function. It is understood that the number of first random features can be one or more, and this application embodiment does not specifically limit this.

[0138] In one possible implementation, the computer device can concatenate the original features, the first random feature, and the second random features in the target training dataset to obtain a second concatenated feature. Then, the computer device calls a feature scoring model to evaluate the feature contribution of the second concatenated feature, obtaining evaluation scores for each original feature, the first random feature, and each second random feature. The feature scoring model can include, but is not limited to, Random Forest models, XGBoost models, etc. In this embodiment, to ensure relatively stable performance of the feature scoring model, the XGBoost model can be used.

[0139] For example, see Figure 6 , Figure 6 This is a flowchart illustrating a feature scoring model provided in an embodiment of this application. For example... Figure 6 As shown, assuming there are 3 prediction samples in the target training dataset, each prediction sample includes 3 original features, 2 first random features, and 3 second random features. For the first prediction sample, the feature values ​​of its 3 original features can be represented as x11, x12, and x13, the feature values ​​of its 3 second random features can be represented as y11, y12, and y13, and the feature values ​​of its 2 first random features can be represented as z11 and z12. It can be understood that... Figure 6 In the matrix shown, each row of data represents a predicted sample. Similarly, for the second predicted sample, the feature values ​​of its three original features can be represented as x21, x22, and x23, the feature values ​​of its three second random features can be represented as y21, y22, and y23, and the feature values ​​of its two first random features can be represented as z21 and z22. Similarly, for the second predicted sample, the feature values ​​of its three original features can be represented as x31, x32, and x33, the feature values ​​of its three second random features can be represented as y31, y32, and y33, and the feature values ​​of its two first random features can be represented as z31 and z32. Finally, the original features, the first random feature, and the second random feature of the three predicted samples are concatenated and input into the XGBoost model. After feature contribution evaluation processing by the XGBoost model, the evaluation scores of each original feature (p1, p2, p3), each first random feature (p7, p8), and each second random feature (p4, p5, p6) are output. For original feature 1, p1 refers to the weighted average of the evaluation scores corresponding to each predicted sample. Similarly, for the first random feature 1, p7 also refers to the weighted average of the evaluation scores corresponding to each predicted sample.

[0140] S403: Based on the evaluation scores of each original feature, the evaluation scores of each first random feature, and the evaluation scores of each second random feature, evaluate the feature importance of each original feature to obtain the number of successful trials for each original feature.

[0141] In one possible implementation, the computer device evaluates the feature importance of each original feature based on the evaluation scores of each original feature, the evaluation scores of each first random feature, and the evaluation scores of each second random feature, to obtain the number of successful trials for each original feature. This includes: first, the computer device determines the highest target evaluation score based on the evaluation scores of the first random features and the evaluation scores of each second random feature; if the evaluation score of any original feature is higher than the target evaluation score, the number of successful trials for that original feature is adjusted. The feature importance evaluation in this embodiment can be based on the evaluation score of each feature.

[0142] Specifically, in this embodiment, each original feature can be ranked according to the SHAP (SHapley Additive Explanation) feature importance index (evaluation score) or the total gain feature importance index (evaluation score). For each original feature in the target training dataset, if the importance index (evaluation score) of the original feature is higher than that of the most important random feature (the second random feature generated by random shuffle, or the first random feature generated by a specific distribution function), then the original feature is considered successful in this trial, and the success count of the original feature is incremented by 1. The SHAP feature importance index is a method used to interpret the output of a machine learning model (XGBoost model). Inspired by cooperative game theory, SHAP constructs an additive interpretation model where all features (including the first random feature, the second random feature, and the original feature) are considered "contributors." For each predicted sample, the XGBoost model can generate a predicted value; the SHAP value is the value assigned to each feature in that predicted sample. The Total Gain feature importance index refers to the cumulative value of the information gain brought by the feature across all trees when a node splits in the XGBoost model. Understandably, for each non-random feature (original feature), each model training iteration, obtaining feature importance (evaluation score), and comparing the evaluation score of the original feature with the evaluation scores of the first and second random features constitutes a randomized trial. During model iteration, such trials can statistically reject or accept the original feature, avoiding the uncertainty brought about by a single randomization.

[0143] For example, according to step S402 above, for example, for original feature 1, p1 is compared with p4 to p8 respectively. If p1 is greater than any of p4 to p8, then original feature 1 can be considered successful in this trial, and the success count of original feature 1 is incremented by 1. Similarly, for original feature 2, p1 is compared with p4 to p8 respectively. If there is a p1 less than any of p4 to p8, then original feature 2 can be considered unsuccessful in this trial, and the success count of original feature 2 remains unchanged. And so on, using the above method, the success count of each original feature can be recorded during each model training process.

[0144] S404: Based on the number of successful trials for each original feature, perform feature correction processing on each original feature to obtain the correction result for each original feature.

[0145] In one possible implementation, the computer device performs feature correction processing on each original feature based on the number of successful trials for each original feature, and obtains the correction result for each original feature, including: first, obtaining the cumulative number of iterations of the target training dataset; then, determining the significance level value to be corrected for any original feature based on the number of successful trials and the cumulative number of iterations for any original feature.

[0146] Next, we will provide a detailed explanation of how to perform feature correction processing on each original feature:

[0147] 1) Record the number of successful trials for the original features:

[0148] Understandably, for each original feature in the target training dataset, we can obtain a current value for the number of successful trials. We can denote the number of successful trials for any original feature as k, and the cumulative number of iterations in the target training dataset as n. Since each trial is a binomial distribution test, we can calculate the significance level of the original feature being better than the random feature using the current number of successful trials and the cumulative number of iterations.

[0149] 2) Determine the significance level values ​​(cumulative distribution value and corrected survival function value) of the original features to be corrected:

[0150] In this embodiment, the null hypothesis is that each original feature has a 50% probability of being more important than a random feature in each trial. That is, the null hypothesis is as shown in formula (1):

[0151] H0:π=0.5 (1)

[0152] In formula (1), π represents an event whose feature importance is higher than that of a random feature.

[0153] The binomial distribution probability mass function is given by formula (2):

[0154]

[0155] In formula (2), k∈{0,1,...,n},0≤p≤1.

[0156] From the above, the corresponding cumulative distribution function can be obtained as shown in formula (3):

[0157]

[0158] As per the null hypothesis, p = 0.5. Substituting the current n, k, and p of each original feature into formula (3), we can obtain the current cumulative distribution value of each original feature, denoted as cdf. k .

[0159] Similarly, by substituting the current n,k-1,p for each original feature into the formula, we can obtain another cumulative distribution value, denoted as cdf. k-1 Next, the survival function value sf can be defined as shown in formula (4):

[0160] sf = 1 - cdf k-1 (4)

[0161] In one possible implementation, the computer device performs correction processing on the saliency level value of any original feature to obtain a correction result for any original feature, including: first, correcting the saliency level value of any original feature according to a first correction method to obtain a corrected saliency level value for any original feature; then, correcting the corrected saliency level value of any original feature again according to a second correction method to obtain a correction result for any original feature.

[0162] ① The first correction method may include the FDR correction method. The FDR correction method uses a relatively mild method to correct the p-value (cumulative distribution value and corrected survival function value). The corrected p-value achieves a balance between false positives and false negatives, and controls the false / true positive ratio within a certain range.

[0163] Understandably, since each feature is an independent trial, comparing multiple features (original feature, first random feature, and second random feature) is essentially a multi-validation process. For multi-validation, FDR correction is necessary; otherwise, the false positive rate will be much higher than expected. In this embodiment, the Benjamini / Hochberg FDR correction method can be used, and the expected threshold can be set to 0.05.

[0164] As mentioned above, the significance level value to be corrected for any original feature can include: the cumulative distribution value (cdf) and the survival function value (sf). Therefore, correcting the significance level value to be corrected for any original feature according to the first correction method to obtain the corrected significance level value for any original feature can include: calling the FDR correction function to correct the cumulative distribution value (cdf) of any original feature, obtaining the FDR-corrected cumulative distribution value (FDR(cdf)); and calling the FDR correction function to correct the survival function value (sf) of any original feature, obtaining the FDR-corrected survival function value (FDR(sf)).

[0165] This correction method can be used to predict that, in a single iteration of all feature trials, the proportion of incorrectly accepted original features will be less than 0.05. For example, in selecting from 5000 original features, if 20 original features are accepted, it is expected that no more than one original feature will be incorrectly accepted (false positive). However, without FDR correction, 270 original features might be accepted, with 250 of them being incorrectly accepted. Therefore, FDR correction can improve the accuracy of feature selection.

[0166] ② The second correction method can include the Bonferroni correction method, which can reduce the threshold to an extremely low level to reduce the false positive rate. If n independent hypotheses are tested simultaneously on the same training dataset, then the statistical significance level used for each hypothesis should be 1 / n of the significance level when testing only one hypothesis.

[0167] Understandably, since the features need to be iterated multiple times, each iteration involves multiple checks. To avoid errors in any overall check, a more stringent Bonferroni correction method is used. Let the p-value after Bonferroni correction be denoted as p. adjust Given that the current cumulative iteration count is n and the expected threshold is p, the Bonferroni correction can be expressed as:

[0168]

[0169] As shown in formula (5), this ensures that the false positive rate remains as expected after n iterations. Then, by applying the second correction method to the significance level value of any original feature, the correction result of any original feature (reference corrected cumulative distribution value and reference corrected survival function value) can be obtained. This can include: determining the ratio between the FDR-corrected cumulative distribution value FDR(cdf) and the cumulative iteration number n as the reference corrected cumulative distribution value; and determining the ratio between the FDR-corrected survival function value FDR(sf) and the cumulative iteration number n as the reference corrected survival function value. After the above steps, the final correction result of the original feature is as follows:

[0170]

[0171]

[0172] Wherein, p in formula (6) accept This is expressed as the reference corrected cumulative distribution value; p in formula (7) reject This is represented as the reference corrected survival function value.

[0173] By using the Bonferroni correction method described above, and further correcting the significance level values ​​of the original features based on the FDR correction method, the accuracy of feature selection can be further improved.

[0174] S405: Based on the correction results of each original feature, obtain the feature type of each original feature.

[0175] In one possible implementation, the correction result of any original feature includes a reference corrected cumulative distribution value and a reference corrected survival function value. The computer device, based on the correction result of each original feature, obtains the feature type of each original feature, which may include: if the reference corrected cumulative distribution value is less than a preset threshold, then the feature type of any original feature is determined as an acceptable feature type; if the reference corrected survival function value is less than the preset threshold, then the feature type of any original feature is determined as a rejected feature type; if both the reference corrected cumulative distribution value and the reference corrected survival function value of any original feature are greater than the preset threshold, then the feature type of any original feature is determined as a pending feature type.

[0176] Specifically, the corrected reference corrected cumulative distribution value p for each original feature can be obtained through S404. accept and reference corrected survival function value p reject Then the reference corrected cumulative distribution value p can be used. accept and reference corrected survival function value p reject Each value is compared with a preset threshold (assumed to be 0.05).

[0177] ①If the reference corrected cumulative distribution value p of the original feature accept If the value is below a preset threshold, the feature type of the original feature is determined to be an acceptable feature type, and the original feature can be added to the current list of acceptable features.

[0178] ②If the reference corrected survival function value p of the original feature reject If the value is below a preset threshold, the original feature is determined to be a rejection feature type, and the original feature can be added to the current rejection feature list, and will not participate in the subsequent iterative experiment process.

[0179] ③If the reference corrected cumulative distribution value p of the original feature accept Compared with the reference-corrected survival function value p reject If all values ​​are greater than the preset threshold, the feature type of the original feature is determined to be a pending feature type, and the original feature can be added to the current pending feature list. The original features recorded in the pending feature list can be selected as needed to participate in subsequent iteration experiments.

[0180] Understandably, accepted original features will continue to participate in the iterative calculation of p-values ​​and may even be removed from the accepted feature list. This helps avoid some low-probability events and enhances algorithm stability. The above process can then be iterated continuously, stopping after a preset number of iterations, or after a specific number of iterations when all feature lists (accepted feature list, rejected feature list, and pending feature list) remain unchanged. After stopping iteration, the original features recorded in the current accepted feature list are the target features obtained after feature filtering; these target features are all significantly better than random features. The pending feature list is for user reference, allowing users to decide whether further experimentation or inclusion in the model is necessary. The original features recorded in the rejected feature list are those significantly worse than random features and will be removed from the training dataset, no longer used for model iteration training.

[0181] The above method evaluates the feature contribution of each original feature in the target training dataset based on the first and second random features, and performs feature correction processing on each original feature according to two correction methods, thereby determining the feature type of each original feature based on the correction results. By adding richer and more diverse random features, the accuracy of feature selection can be improved; furthermore, employing multiple correction methods to correct the original features can further improve the accuracy of feature selection.

[0182] Please see Figure 7 , Figure 7This is a flowchart illustrating another data processing method provided in an embodiment of this application. This data processing method can be executed by the aforementioned server (i.e., management node). For ease of explanation, the following description will use a computer device executing this data processing method as an example. The data processing method may include the following steps S701 to S703:

[0183] S701: Obtain a sample dataset belonging to the target domain within the target time period. The target time period includes k time cycles. The sample dataset includes training datasets corresponding to the k time cycles. Each training dataset includes multiple original features. The feature labels of the original features contained in each training dataset are the same. k is a positive integer.

[0184] In this embodiment of the application, the target time period can be one month, and a time cycle can be one week. The target time period can also be one year, and a time cycle can be one month. It is understood that each sample dataset represents multiple original features collected from the same batch of prediction samples at different time cycles. Furthermore, the feature labels of the original features contained in each training dataset are the same.

[0185] S702: Send k training datasets to k worker nodes respectively, so that the target worker node can select target features from multiple original features according to the feature type of each original feature in the target training dataset. The feature type of each original feature is determined by the target worker node after performing feature contribution evaluation processing on the original feature according to the first random feature. The first random feature is a feature generated by calling a specific distribution function corresponding to the target domain. The target worker node is any one of the k worker nodes, and the target training dataset is the training dataset received by the target worker node.

[0186] In practical applications, training datasets with multiple time periods are typically required. This embodiment of the application can broadcast the training datasets for each time period to k distributed worker nodes via Spark, and then each node obtains a training dataset for one time period. After obtaining a corresponding training dataset, each worker node can then proceed according to... Figure 3 The method described in this embodiment performs a process of filtering one or more target features from multiple raw features. Finally, each working node sends the target features filtered by the corresponding node to the management node. In this way, the training time can be shortened to 1 / k, thereby improving the efficiency of feature filtering.

[0187] It should be noted that the detailed execution steps for each working node, "selecting target features from multiple raw features based on the feature type of each raw feature in the target training dataset," can be found in [link to documentation]. Figure 3The processes performed in steps S301-S305 of the embodiments will not be described again in this application embodiment.

[0188] S703: Receives target features sent by k working nodes, summarizes the received target features, obtains filtering features, and outputs the filtering features.

[0189] In one possible implementation, the management node summarizes the received target features to obtain filtering features and outputs the filtering features. This may include: the management node counts the frequency of occurrence of target features in k time periods, determines the target features whose frequency of occurrence is greater than or equal to a preset frequency as filtering features, and then outputs the filtering features.

[0190] For example, suppose there are k worker nodes, including worker node 1, worker node 2, and worker node 3. Worker node 1 sends target features including: original feature 1 and original feature 2; worker node 2 sends target features including: original feature 2 and original feature 3; and worker node 3 sends target features including: original feature 1 and original feature 4. Then, based on the target features sent by each worker node, the management node can statistically determine the frequency of occurrence of the feature identifiers of original feature 1 (2), original feature 2 (2), original feature 3 (1), and original feature 4 (1). Assuming a preset frequency of 2, the management node can use original feature 1 and original feature 2 as filtering features. Finally, the management node can output original feature 1 and original feature 2. The subsequent filtered features can serve as data sources for various financial analysis and modeling scenarios; for example, the output filtered features can be used as training data for a product recommendation model in the big data field.

[0191] In another possible implementation, the management node summarizes the received target features to obtain filtering features and outputs the filtering features. This can also include: the management node performing a union operation on the received target features, using the resulting union as the filtering features, and outputting the filtering features. For example, suppose there are k worker nodes, including worker node 1, worker node 2, and worker node 3. Worker node 1 sends target features including: original feature 1 and original feature 2; worker node 2 sends target features including: original feature 2 and original feature 3; and worker node 3 sends target features including: original feature 1 and original feature 4. Then, the management node performing a union operation on the received target features can use original feature 1, original feature 2, original feature 3, and original feature 4 as filtering features. Finally, the management node can output original feature 1, original feature 2, original feature 3, and original feature 4.

[0192] Furthermore, the embodiments of this application can be applied to the financial field. For example, in multiple credit scenarios of financial payments, the distributed feature automatic screening scheme of this application has been utilized, resulting in a significant improvement in performance. Relevant tests were conducted on a training dataset from actual business operations, and the comparative results obtained after the tests are shown in Table 4 below:

[0193] Table 4 Comparison of Feature Filtering Results

[0194]

[0195] As can be seen from Table 4 above, compared with the most commonly used manual + XGBoost screening results in practical applications, the number of features after distributed automatic screening provided in this application embodiment is 10.1% of the comparison scheme, while the model performance is improved by 2.3%. The number of features is greatly reduced, but the model performance is significantly improved.

[0196] In this embodiment, the management node can send k training datasets for k periods to k worker nodes according to the provided distributed feature automatic filtering method, so that the k worker nodes can execute the feature filtering steps in parallel. Finally, the management node summarizes the target features from the k worker nodes to obtain the filtered features. Because it uses Spark's distributed feature automatic filtering method, compared to a single worker node processing a training dataset one by one, this embodiment can perform distributed processing, thereby improving the efficiency of feature filtering.

[0197] Please see Figure 8 , Figure 8 This is an interactive schematic diagram of a data processing system provided in an embodiment of this application. The data processing system includes a management node and worker nodes. The management node can be the aforementioned server, and the worker nodes can be the aforementioned terminal devices. Specifically, when the worker nodes and management nodes jointly execute the data processing method provided in this application, the process may include the following steps S801-S807:

[0198] S801: The management node obtains a sample dataset belonging to the target domain within the target time period. The target time period includes k time periods, and the sample dataset includes training datasets corresponding to the k time periods. Each training dataset includes multiple original features, and the feature labels of the original features contained in each training dataset are the same. k is a positive integer.

[0199] S802: The management node sends the target training dataset to the worker node. The target training dataset is one of the k training datasets.

[0200] Here, a worker node refers to any worker node (also called a target worker node) that receives a period of the target training dataset. It can be understood that the management node can distribute k training datasets to k worker nodes. For ease of explanation, the following descriptions will use any worker node as an example.

[0201] S803: The worker node calls the specific distribution function corresponding to the target domain to generate the first random feature of the target training dataset.

[0202] S804: The working node performs feature contribution evaluation processing on each original feature in the target training dataset based on the first random feature, and obtains the feature type of each original feature.

[0203] S805: The working node selects the target feature from multiple original features based on the feature type of each original feature.

[0204] It is understood that the specific steps performed by the working node in steps S803-S805 of this embodiment can be found in the detailed description. Figure 3 The specific steps of steps S302-S304 in the embodiments will not be repeated here.

[0205] S806: The worker node sends the selected target features to the management node.

[0206] In one possible implementation, the target feature may include a feature identifier and a feature value, and the worker node may send the feature identifier of the selected target feature to the management node.

[0207] S807: The management node will summarize the received target features, obtain the filtering features, and output the filtering features.

[0208] In one possible implementation, after receiving the target features sent by the worker nodes, the management node can obtain the node identifier of the worker nodes and verify the worker nodes based on the node identifier (e.g., security verification, legality verification). If the management node determines that the verification of the worker node is successful, it triggers the execution of the step of "the management node summarizes the received target features, obtains the filter features, and outputs the filter features".

[0209] It is understandable that the management node will summarize the received target features to obtain the filtering features. The specific execution steps for outputting the filtering features can be found in [link to relevant documentation]. Figure 7 The execution steps corresponding to step S703 in the embodiments will not be repeated here.

[0210] In this embodiment, firstly, the management node can send k training datasets to k worker nodes using Spark distributed processing. Then, upon receiving the target training dataset, each worker node can generate a first random feature representing the common feature distributions within the financial field using a specific distribution function. This ensures the richness of the basic feature distribution and enhances the stability of feature selection. Therefore, in this application, feature contribution evaluation can be performed based on the first random feature during feature selection for each original feature. Since feature selection considers both the original features of the training dataset itself and adds the first random feature related to the target domain of the original features, the sample data for feature selection is enriched, thereby improving the accuracy of feature selection. Furthermore, a worker node can be responsible for processing multiple original features of a training dataset for one period to select the target feature. It can be understood that the k training datasets can be executed synchronously by the k worker nodes, and the target features selected by each node are ultimately sent to the management result aggregation. Compared to a single worker node processing a training dataset one by one, this embodiment allows for distributed processing, thereby improving the efficiency of feature selection.

[0211] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application. The data processing apparatus 900 can be applied to... Figure 2 The corresponding terminal device (work node) in the data processing system. The data processing device 900 can be a computer program (including program code) running on a computer device; for example, the data processing device 900 is application software. The data processing device can be used to execute the corresponding steps in the data processing method provided in the embodiments of this application. The data processing device 900 may include:

[0212] The receiving unit 901 is used to receive the target training dataset of the target domain sent by the management node. The target training dataset includes multiple original features. The target training dataset is one of k training datasets. Each training dataset corresponds to a time period. The original features contained in each training dataset have the same feature labels. k is a positive integer.

[0213] Processing unit 902 is used to call a specific distribution function corresponding to the target domain to generate the first random feature of the target training dataset;

[0214] The processing unit 902 is also configured to perform feature contribution evaluation processing on each original feature in the target training dataset according to the first random feature, so as to obtain the feature type of each original feature;

[0215] Processing unit 902 is also used to filter target features from multiple original features based on the feature type of each original feature;

[0216] The sending unit 903 is used to send the selected target features to the management node so that the management node can determine the selected features based on the target features over k time periods.

[0217] In one possible implementation, the feature types include accept feature types, reject feature types, and pending feature types; the processing unit 902 filters target features from the plurality of original features according to the feature type of each original feature, for performing the following operations:

[0218] Based on the feature type of each original feature, adjust the list of accepted features, the list of rejected features, and the list of pending features, with each feature list corresponding to one feature type;

[0219] When the iteration stopping condition is met, the original features recorded in the accept feature list and the undetermined feature list will be used as the target features.

[0220] When the iteration stopping condition is not met, the original features recorded in the accepted feature list and the pending feature list will be combined into a new target training dataset.

[0221] In one possible implementation, the iteration stopping condition is determined to be met when the cumulative number of iterations of the target training dataset reaches a preset number of iterations; or,

[0222] The iteration stopping condition is met when the accept feature list, reject feature list, and pending feature list remain unchanged.

[0223] In one possible implementation, processing unit 902 performs feature contribution evaluation processing on each original feature in the target training dataset based on the first random feature to obtain the feature type of each original feature, which is then used to perform the following operations:

[0224] Randomly rearrange the feature values ​​of multiple original features in the target training dataset to determine multiple second random features;

[0225] The feature scoring model is invoked to evaluate the feature contribution of the first random feature, multiple second random features, and each original feature in the target training dataset, and the evaluation scores of each original feature, the first random feature, and each second random feature are obtained.

[0226] Based on the evaluation scores of each original feature, the evaluation score of the first random feature, and the evaluation scores of each second random feature, feature correction processing is performed on each original feature to obtain the feature type of each original feature.

[0227] In one possible implementation, the processing unit 902 performs feature correction processing on each original feature based on the evaluation scores of each original feature, the evaluation scores of the first random feature, and the evaluation scores of each second random feature, to obtain the feature type of each original feature, which is then used to perform the following operations:

[0228] Based on the evaluation scores of each original feature, each first random feature, and each second random feature, the feature importance of each original feature is evaluated to obtain the number of successful trials for each original feature.

[0229] Based on the number of successful trials for each original feature, feature correction processing is performed on each original feature to obtain the correction result for each original feature;

[0230] Based on the correction results of each original feature, the feature type of each original feature is obtained.

[0231] In one possible implementation, the processing unit 902 evaluates the feature importance of each original feature based on the evaluation scores of each original feature, the evaluation scores of the first random feature, and the evaluation scores of each second random feature, to obtain the number of successful trials for each original feature, which is then used to perform the following operations:

[0232] The highest target evaluation score is determined based on the evaluation scores of the first random feature and the evaluation scores of each of the second random features.

[0233] If the evaluation score of any original feature is higher than the target evaluation score, then the number of successful trials for that original feature is adjusted.

[0234] In one possible implementation, the processing unit 902 performs feature correction processing on each original feature based on the test results of each original feature, obtaining a correction result for each original feature, which is used to perform the following operations:

[0235] Obtain the cumulative number of iterations for the target training dataset;

[0236] Based on the number of successful trials and the cumulative number of iterations for any original feature, determine the significance level value to be corrected for any original feature;

[0237] The significance level value of any original feature to be corrected is corrected to obtain the corrected result of any original feature.

[0238] In one possible implementation, the processing unit 902 performs correction processing on the saliency level value to be corrected of any original feature to obtain the correction result of any original feature, which is then used to perform the following operations:

[0239] The significance level value of any original feature to be corrected is corrected according to the first correction method to obtain the corrected significance level value of any original feature.

[0240] The significance level value of any original feature is corrected again according to the second correction method to obtain the correction result of any original feature.

[0241] In one possible implementation, the correction result for any original feature includes a reference corrected cumulative distribution value and a reference corrected survival function value;

[0242] Based on the correction result of each original feature, the processing unit 902 obtains the feature type of each original feature, and performs the following operations:

[0243] If the reference correction cumulative distribution value is less than the preset threshold, then the feature type of any original feature will be determined as the accepted feature type;

[0244] If the reference corrected survival function value is less than the preset threshold, then the feature type of any original feature will be determined as the rejection feature type;

[0245] If the reference corrected cumulative distribution value and the reference corrected survival function value of any original feature are both greater than the preset threshold, then the feature type of any original feature is determined as a feature type to be determined.

[0246] In one possible implementation, there are multiple specific distribution functions, any one of which includes any of the following: log-normal distribution function, binomial distribution function, Bernoulli distribution function, exponential distribution function, uniform distribution function, and normal distribution function.

[0247] In this embodiment, firstly, a first random feature representing the common feature distributions within the financial field can be generated using a specific distribution function. This ensures the richness of the basic feature distribution and enhances the stability of feature selection. Then, using Spark distributed processing, the training time for k training datasets can be reduced to 1 / k, thereby improving the efficiency of feature selection. Therefore, in this application, feature contribution evaluation can be performed based on the first random feature during feature selection for each original feature. Since feature selection considers both the original features of the training dataset itself and adds the first random feature related to the target domain of the original features, the sample data for feature selection is enriched, thus improving the accuracy of feature selection. Furthermore, a worker node can be responsible for processing multiple original features of a training dataset for one period to select target features. It can be understood that k training datasets can be executed synchronously by k worker nodes, and the target features selected by each node are ultimately sent to the management result aggregation. Compared to a single worker node processing a training dataset one by one, this embodiment allows for distributed processing, thereby improving the efficiency of feature selection.

[0248] Please see Figure 10 , Figure 10 This is a schematic diagram of another data processing apparatus provided in an embodiment of this application. The data processing apparatus 1000 can be applied to... Figure 2 The corresponding server (management node) in the data processing system. The data processing device 1000 can be a computer program (including program code) running on a computer device; for example, the data processing device 1000 is application software. The data processing device can be used to execute corresponding steps in the data processing method provided in the embodiments of this application. The data processing device 1000 may include:

[0249] The acquisition unit 1001 is used to acquire a sample dataset belonging to the target domain within a target time period. The target time period includes k time periods, and the sample dataset includes training datasets corresponding to the k time periods. Each training dataset includes multiple original features, and the feature labels of the original features contained in each training dataset are the same. k is a positive integer.

[0250] The sending unit 1002 is used to send k training datasets to k worker nodes respectively, so that the target worker node can select target features from multiple original features according to the feature type of each original feature in the target training dataset. The feature type of each original feature is determined by the target worker node after performing feature contribution evaluation processing on the original feature according to the first random feature. The first random feature is a feature generated by calling a specific distribution function corresponding to the target domain. The target worker node is any one of the k worker nodes, and the target training dataset is the training dataset received by the target worker node.

[0251] The processing unit 1003 is used to receive target features sent by k working nodes, summarize the received target features, obtain the filtering features, and output the filtering features.

[0252] In one possible implementation, the processing unit 1003 summarizes the received target features to obtain the filtered features, and outputs the filtered features for performing the following operations:

[0253] The frequency of occurrence of target features over k time periods is statistically analyzed, and target features with a frequency greater than or equal to a preset frequency are identified as screening features.

[0254] Output filtering features.

[0255] In this embodiment, the management node can send k training datasets for k periods to k worker nodes according to the provided distributed feature automatic filtering method, so that the k worker nodes can execute the feature filtering steps in parallel. Finally, the management node summarizes the target features from the k worker nodes to obtain the filtered features. Because it uses Spark's distributed feature automatic filtering method, compared to a single worker node processing a training dataset one by one, this embodiment can perform distributed processing, thereby improving the efficiency of feature filtering.

[0256] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device can be the terminal device (working node) or server (management node) described above. The computer device 1100 is used to execute the steps performed by the computer device in the aforementioned method embodiments. The computer device 1100 includes: one or more processors 1110; one or more input devices 1120; one or more output devices 1130; and a memory 1140. The processors 1110, input devices 1120, output devices 1130, and memory 1140 are connected via a bus 1150. The memory 1140 is used to store a computer program, which includes program instructions. The processor 1110 is used to call the program instructions stored in the memory 1140 to perform the following operations:

[0257] The target training dataset for the target domain is sent by the management node. The target training dataset includes multiple original features. The target training dataset is one of k training datasets. Each training dataset corresponds to a time period. The original features contained in each training dataset have the same feature labels. k is a positive integer.

[0258] Call the specific distribution function corresponding to the target domain to generate the first random feature of the target training dataset;

[0259] Based on the first random feature, feature contribution evaluation is performed on each original feature in the target training dataset to obtain the feature type of each original feature.

[0260] Target features are selected from multiple original features based on the feature type of each original feature;

[0261] The selected target features are sent to the management node so that the management node can determine the selected features based on the target features over k time periods.

[0262] In one possible implementation, the feature types include accept feature types, reject feature types, and pending feature types; the processor 1110 filters target features from the plurality of original features according to the feature type of each original feature, for the following operations:

[0263] Based on the feature type of each original feature, adjust the list of accepted features, the list of rejected features, and the list of pending features, with each feature list corresponding to one feature type;

[0264] When the iteration stopping condition is met, the original features recorded in the accept feature list and the undetermined feature list will be used as the target features.

[0265] When the iteration stopping condition is not met, the original features recorded in the accepted feature list and the pending feature list will be combined into a new target training dataset.

[0266] In one possible implementation, the iteration stopping condition is determined to be met when the cumulative number of iterations of the target training dataset reaches a preset number of iterations; or,

[0267] The iteration stopping condition is met when the accept feature list, reject feature list, and pending feature list remain unchanged.

[0268] In one possible implementation, the processor 1110 performs feature contribution evaluation processing on each original feature in the target training dataset based on a first random feature, to obtain the feature type of each original feature, which is then used to perform the following operations:

[0269] Randomly rearrange the feature values ​​of multiple original features in the target training dataset to determine multiple second random features;

[0270] The feature scoring model is invoked to evaluate the feature contribution of the first random feature, multiple second random features, and each original feature in the target training dataset, and the evaluation scores of each original feature, the first random feature, and each second random feature are obtained.

[0271] Based on the evaluation scores of each original feature, the evaluation score of the first random feature, and the evaluation scores of each second random feature, feature correction processing is performed on each original feature to obtain the feature type of each original feature.

[0272] In one possible implementation, the processor 1110 performs feature correction processing on each original feature based on the evaluation scores of each original feature, the evaluation scores of the first random feature, and the evaluation scores of each second random feature, to obtain the feature type of each original feature, which is used to perform the following operations:

[0273] Based on the evaluation scores of each original feature, each first random feature, and each second random feature, the feature importance of each original feature is evaluated to obtain the number of successful trials for each original feature.

[0274] Based on the number of successful trials for each original feature, feature correction processing is performed on each original feature to obtain the correction result for each original feature;

[0275] Based on the correction results of each original feature, the feature type of each original feature is obtained.

[0276] In one possible implementation, the processor 1110 evaluates the feature importance of each original feature based on the evaluation scores of each original feature, the evaluation scores of the first random feature, and the evaluation scores of each second random feature, to obtain the number of successful trials for each original feature, which is then used to perform the following operations:

[0277] The highest target evaluation score is determined based on the evaluation scores of the first random feature and the evaluation scores of each of the second random features.

[0278] If the evaluation score of any original feature is higher than the target evaluation score, then the number of successful trials for that original feature is adjusted.

[0279] In one possible implementation, the processor 1110 performs feature correction processing on each original feature based on the test results of each original feature, obtaining a correction result for each original feature, which is used to perform the following operations:

[0280] Obtain the cumulative number of iterations for the target training dataset;

[0281] Based on the number of successful trials and the cumulative number of iterations for any original feature, determine the significance level value to be corrected for any original feature;

[0282] The significance level value of any original feature to be corrected is corrected to obtain the corrected result of any original feature.

[0283] In one possible implementation, processor 1110 performs correction processing on the saliency level value to be corrected of any original feature to obtain the correction result of any original feature, which is used to perform the following operations:

[0284] The significance level value of any original feature to be corrected is corrected according to the first correction method to obtain the corrected significance level value of any original feature.

[0285] The significance level value of any original feature is corrected again according to the second correction method to obtain the correction result of any original feature.

[0286] In one possible implementation, the correction result for any original feature includes a reference corrected cumulative distribution value and a reference corrected survival function value;

[0287] Based on the correction result of each original feature, processor 1110 obtains the feature type of each original feature, which is used to perform the following operations:

[0288] If the reference correction cumulative distribution value is less than the preset threshold, then the feature type of any original feature will be determined as the accepted feature type;

[0289] If the reference corrected survival function value is less than the preset threshold, then the feature type of any original feature will be determined as the rejection feature type;

[0290] If the reference corrected cumulative distribution value and the reference corrected survival function value of any original feature are both greater than the preset threshold, then the feature type of any original feature is determined as a feature type to be determined.

[0291] In one possible implementation, there are multiple specific distribution functions, any one of which includes any of the following: log-normal distribution function, binomial distribution function, Bernoulli distribution function, exponential distribution function, uniform distribution function, and normal distribution function.

[0292] Understandably, processor 1110 is used to call program instructions stored in memory 1140, and can also perform the following operations:

[0293] Obtain a sample dataset belonging to the target domain within the target time period. The target time period includes k time periods. The sample dataset includes training datasets corresponding to the k time periods. Each training dataset includes multiple original features. The feature labels of the original features contained in each training dataset are the same. k is a positive integer.

[0294] K training datasets are sent to k worker nodes respectively, so that the target worker node can select the target feature from multiple original features according to the feature type of each original feature in the target training dataset. The feature type of each original feature is determined by the target worker node after performing feature contribution evaluation processing on the original feature according to the first random feature. The first random feature is a feature generated by calling a specific distribution function corresponding to the target domain. The target worker node is any one of the k worker nodes, and the target training dataset is the training dataset received by the target worker node.

[0295] Receive target features sent by k working nodes, summarize the received target features to obtain filtering features, and output the filtering features.

[0296] In one possible implementation, the processor 1110 summarizes the received target features to obtain filter features, outputs the filter features, and performs the following operations:

[0297] The frequency of occurrence of target features over k time periods is statistically analyzed, and target features with a frequency greater than or equal to a preset frequency are identified as screening features.

[0298] Output filtering features.

[0299] In this embodiment, firstly, a first random feature representing the common feature distributions within the financial field can be generated using a specific distribution function. This ensures the richness of the basic feature distribution and enhances the stability of feature selection. Then, using Spark distributed processing, the training time for k training datasets can be reduced to 1 / k, thereby improving the efficiency of feature selection. Therefore, in this application, feature contribution evaluation can be performed based on the first random feature during feature selection for each original feature. Since feature selection considers both the original features of the training dataset itself and adds the first random feature related to the target domain of the original features, the sample data for feature selection is enriched, thus improving the accuracy of feature selection. Furthermore, a worker node can be responsible for processing multiple original features of a training dataset for one period to select target features. It can be understood that k training datasets can be executed synchronously by k worker nodes, and the target features selected by each node are ultimately sent to the management result aggregation. Compared to a single worker node processing a training dataset one by one, this embodiment allows for distributed processing, thereby improving the efficiency of feature selection.

[0300] Furthermore, it should be noted that this application also provides a computer storage medium storing a computer program, which includes program instructions. When the processor executes these program instructions, it can perform the methods described in the preceding embodiments. Therefore, further details will not be provided here. For technical details not disclosed in the embodiments of the computer storage medium involved in this application, please refer to the description of the method embodiments of this application. As an example, the program instructions can be deployed on a computer device, executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network.

[0301] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described in the preceding embodiments; therefore, further details will not be provided here.

[0302] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0303] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, This is applied to a distributed architecture, which includes a management node and k worker nodes, and the k worker nodes execute in parallel, where k is an integer greater than or equal to 2; The method is executed by any one of the worker nodes, including: The system receives a target training dataset for a target domain sent by the management node. The target training dataset includes multiple original features. The target training dataset is one of k training datasets, each training dataset corresponds to a time period, and the feature identifiers of the original features contained in each training dataset are the same. The target domain includes the financial domain, and the original features include gender features, age features, and credit features. Invoke the specific distribution function corresponding to the target domain to generate the first random feature of the target training dataset; Randomly rearrange the feature values ​​of multiple original features in the target training dataset to determine multiple second random features; The feature scoring model is invoked to evaluate the feature contribution of the first random feature, the plurality of second random features, and each original feature in the target training dataset, to obtain the evaluation score of each original feature, the evaluation score of the first random feature, and the evaluation score of each second random feature. Based on the evaluation scores of the first random feature and the evaluation scores of each of the second random features, the highest target evaluation score is determined; if the evaluation score of any original feature is higher than the target evaluation score, the number of successful trials for that original feature is adjusted. Based on the number of successful trials for each original feature, feature correction processing is performed on each original feature to obtain the correction result for each original feature; the correction result for any original feature includes a reference corrected cumulative distribution value and a reference corrected survival function value; If the reference corrected cumulative distribution value is less than a preset threshold, the feature type of any original feature is determined as an accepted feature type; if the reference corrected survival function value is less than a preset threshold, the feature type of any original feature is determined as a rejected feature type; if both the reference corrected cumulative distribution value and the reference corrected survival function value of any original feature are greater than the preset threshold, the feature type of any original feature is determined as a pending feature type. Target features are selected from the plurality of original features based on the feature type of each original feature; The selected target features are sent to the management node so that the management node can determine the selected features based on the target features over k time periods.

2. The method as described in claim 1, characterized in that, The feature types include accept feature types, reject feature types, and pending feature types; The step of filtering target features from the plurality of original features based on the feature type of each original feature includes: Based on the feature type of each original feature, the accept feature list, reject feature list, and pending feature list are adjusted, with each feature list corresponding to one feature type. When the iteration stopping condition is met, the original features recorded in the accepted feature list and the undetermined feature list are used as the target features; When the iteration stopping condition is not met, the original features recorded in the accepted feature list and the pending feature list are combined into a new target training dataset.

3. The method as described in claim 2, characterized in that, When the cumulative number of iterations of the target training dataset reaches the preset number of iterations, the iteration stopping condition is determined to be met; or, When the list of accepted features, the list of rejected features, and the list of pending features all remain unchanged, the iteration stopping condition is determined to be met.

4. The method as described in claim 1, characterized in that, The process of performing feature correction on each original feature based on the number of successful trials for each original feature to obtain the correction result for each original feature includes: Obtain the cumulative number of iterations for the target training dataset; The significance level value to be corrected for any original feature is determined based on the number of successful trials for any original feature and the cumulative number of iterations. The significance level value of any one of the original features to be corrected is corrected to obtain the correction result of the original feature.

5. The method as described in claim 4, characterized in that, The step of correcting the significance level value of any one of the original features to obtain the corrected result of the original feature includes: The significance level value of any one of the original features to be corrected is corrected according to the first correction method to obtain the corrected significance level value of any one of the original features; The significance level value of any one of the original features is corrected again according to the second correction method to obtain the correction result of any one of the original features.

6. The method as described in claim 1, characterized in that, The number of specific distribution functions is multiple, and any one of the following is included: log-normal distribution function, binomial distribution function, Bernoulli distribution function, exponential distribution function, uniform distribution function, and normal distribution function.

7. A data processing method, characterized in that, This is applied to a distributed architecture, which includes a management node and k worker nodes, and the k worker nodes execute in parallel, where k is an integer greater than or equal to 2; The method is executed by the management node and includes: Obtain a sample dataset belonging to the target domain within a target time period. The target time period includes k time periods. The sample dataset includes training datasets corresponding to the k time periods. Each training dataset includes multiple original features, and the feature labels of the original features contained in each training dataset are the same. The target domain includes the financial domain, and the original features include gender features, age features, and credit features. K training datasets are sent to k worker nodes, allowing the target worker nodes to select target features from the multiple original features based on the feature type of each original feature in the target training dataset. The feature type of each original feature is determined based on the correction result of the original feature, which includes a reference corrected cumulative distribution value and a reference corrected survival function value. If the reference corrected cumulative distribution value is less than a preset threshold, the feature type of the original feature is determined as an accept feature type; if the reference corrected survival function value is less than a preset threshold, the feature type of the original feature is determined as a reject feature type; if both the reference corrected cumulative distribution value and the reference corrected survival function value of the original feature are greater than the preset threshold, the feature type of the original feature is determined as a pending feature type. The correction result of each original feature is based on the number of successful trials performed on each original feature. The number of successful trials for each original feature obtained through correction processing is determined by setting the highest target evaluation score based on the evaluation scores of the first random feature and each of the second random features, and adjusted when the evaluation score of the original feature is higher than the target evaluation score. The evaluation scores of the original feature, the first random feature, and each of the second random features are obtained by calling a feature scoring model to evaluate the feature contribution of the first random feature, the multiple second random features, and each original feature in the target training dataset. The second random feature is determined by randomly rearranging the feature values ​​of multiple original features in the target training dataset. The first random feature is a feature generated by calling a specific distribution function corresponding to the target domain. The target working node is any one of the k working nodes, and the target training dataset is the training dataset received by the target working node. Receive the target features sent by the k working nodes, summarize the received target features to obtain the filtering features, and output the filtering features.

8. The method as described in claim 7, characterized in that, The process of summarizing the received target features to obtain filtering features and outputting the filtering features includes: The frequency of occurrence of target features over k time periods is statistically analyzed, and target features with a frequency greater than or equal to a preset frequency are identified as screening features. Output the filtering features.

9. A data processing apparatus, characterized in that, This is applied to a distributed architecture, which includes a management node and k worker nodes, and the k worker nodes execute in parallel, where k is an integer greater than or equal to 2; The device is executed by any one of the working nodes, including: The receiving unit is used to receive a target training dataset of the target domain sent by the management node. The target training dataset includes multiple original features. The target training dataset is one of k training datasets. Each training dataset corresponds to a time period. The feature labels of the original features contained in each training dataset are the same. The target domain includes the financial domain. The original features include gender features, age features, and credit features. The processing unit is used to call a specific distribution function corresponding to the target domain to generate the first random feature of the target training dataset; The processing unit is further configured to randomly rearrange the feature values ​​of multiple original features in the target training dataset to determine multiple second random features; The processing unit is further configured to call a feature scoring model to perform feature contribution evaluation processing on the first random feature, the plurality of second random features and each original feature in the target training dataset, to obtain the evaluation score of each original feature, the evaluation score of the first random feature, and the evaluation score of each second random feature. The processing unit is further configured to determine the highest target evaluation score based on the evaluation score of the first random feature and the evaluation scores of each of the second random features; if the evaluation score of any original feature is higher than the target evaluation score, the number of successful trials of the original feature is adjusted. The processing unit is further configured to perform feature correction processing on each original feature based on the number of successful trials for each original feature, to obtain a correction result for each original feature; the correction result for any original feature includes a reference corrected cumulative distribution value and a reference corrected survival function value; The processing unit is further configured to: if the reference-corrected cumulative distribution value is less than a preset threshold, determine the feature type of any original feature as an accept feature type; if the reference-corrected survival function value is less than a preset threshold, determine the feature type of any original feature as a reject feature type; if both the reference-corrected cumulative distribution value and the reference-corrected survival function value of any original feature are greater than a preset threshold, determine the feature type of any original feature as a pending feature type. The processing unit is further configured to filter out target features from the plurality of original features according to the feature type of each original feature; The sending unit is used to send the selected target features to the management node, so that the management node can determine the selected features based on the target features over k time periods.

10. A data processing apparatus, characterized in that, This is applied to a distributed architecture, which includes a management node and k worker nodes, and the k worker nodes execute in parallel, where k is an integer greater than or equal to 2; The device is executed by the management node and includes: The acquisition unit is used to acquire a sample dataset belonging to the target domain within a target time period. The target time period includes k time periods, and the sample dataset includes training datasets corresponding to the k time periods. Each training dataset includes multiple original features, and the feature labels of the original features contained in each training dataset are the same. The target domain includes the financial domain, and the original features include gender features, age features, and credit features. A sending unit is configured to send k training datasets to k worker nodes respectively, so that the target worker nodes can select target features from the multiple original features according to the feature type of each original feature in the target training dataset. The feature type of each original feature is determined based on the correction result of the original feature, which includes a reference corrected cumulative distribution value and a reference corrected survival function value. If the reference corrected cumulative distribution value is less than a preset threshold, the feature type of the original feature is determined as an accept feature type; if the reference corrected survival function value is less than a preset threshold, the feature type of the original feature is determined as a reject feature type; if both the reference corrected cumulative distribution value and the reference corrected survival function value of the original feature are greater than the preset threshold, the feature type of the original feature is determined as a pending feature type. The correction result of each original feature is based on the number of successful trials for each original feature. The number of successful trials for each original feature is obtained by performing feature correction processing. The highest target evaluation score is determined based on the evaluation scores of the first random feature and the evaluation scores of each second random feature. Adjustments are made when the evaluation score of the original feature is higher than the target evaluation score. The evaluation scores of the original feature, the first random feature, and each second random feature are obtained by calling a feature scoring model to evaluate the feature contribution of the first random feature, the multiple second random features, and each original feature in the target training dataset. The second random feature is determined by randomly rearranging the feature values ​​of multiple original features in the target training dataset. The first random feature is a feature generated by calling a specific distribution function corresponding to the target domain. The target working node is any one of the k working nodes, and the target training dataset is the training dataset received by the target working node. The processing unit is used to receive the target features sent by the k working nodes, summarize the received target features to obtain the filtering features, and output the filtering features.

11. A computer device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, implements the data processing method as described in any one of claims 1-6 or 7-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the data processing method as described in any one of claims 1-6 or 7-8.

13. A computer program product, characterized in that, The computer program product includes a computer program adapted to be loaded by a processor and execute the data processing method as described in any one of claims 1-6 or 7-8.

Images