Data sampling method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a data sampling method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of data processing. The data sampling method comprises the steps that a sample data set is obtained, the sample data set comprises a first data set and a second data set, and the generation time of the second data set is later than that of the first data set; determining a tendency score of each piece of first sample data in the first data set; wherein the tendency score is used for representing the probability that the first sample data falls into the second data set; based on the tendency score of the first sample data, matching the first sample data with second sample data in the second data set; determining a first sampling weight of the first data set and a second sampling weight of each piece of first sample data in the first data set according to a matching result; and sampling target sample data from the sample data set based on the first sampling weight and the second sampling weight. According to the method, adaptive data sampling is realized, and the model index performance is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data sampling method, device, electronic device and computer-readable storage medium. Background Art

[0002] Data sampling refers to the process of extracting a portion of data from a large-scale data set. The purpose of data sampling is to reduce the amount of data to make it easier to process while maintaining the representativeness of the overall data distribution. It has important applications in data mining, statistical analysis, market research, and machine learning. For example, in the field of machine learning, data sampling can be used to obtain training data for model training.

[0003] In the related art, data sampling is usually performed by random sampling, stratified sampling or cluster sampling. However, when the above data sampling methods are applied to the scenario of model training, they are all independently sampled based on the training set samples of the model, without considering the time relationship within the model training set or its relationship with the test set. At the same time, it is also impossible to characterize the weak periodicity of the data. Therefore, the sampled data obtained by the above data sampling method is used for model training, and the model performance obtained is poor. Summary of the invention

[0004] In view of this, the embodiments of the present application propose a data sampling method, device, electronic device and computer-readable storage medium to solve the problem that when the data sampling method in the related technology is applied to the scenario in the model training field, it neither considers the time relationship within the model training set nor its relationship with the test set, and at the same time, it cannot characterize the weak periodicity of the data.

[0005] According to a first aspect of the present application, a data sampling method is provided, comprising: obtaining a sample data set, the sample data set comprising a first data set and a second data set, the second data set being generated later than the first data set; determining a propensity score for each first sample data in the first data set; wherein the propensity score is used to characterize the probability that the first sample data falls into the second data set; matching the first sample data with the second sample data in the second data set based on the propensity score of the first sample data; determining a first sampling weight for the first data set and a second sampling weight for each first sample data in the first data set based on the matching result; sampling target sample data from the sample data set based on the first sampling weight and the second sampling weight, the target sample data being used to train a target model.

[0006] According to a second aspect of the present application, a data sampling device is provided, comprising: an acquisition module, used to acquire a sample data set, the sample data set comprising a first data set and a second data set, the second data set being generated later than the first data set; an operation module, used to determine a propensity score of each first sample data in the first data set; wherein the propensity score is used to characterize the probability that the first sample data falls into the second data set; a matching module, used to match the first sample data and the second sample data in the second data set based on the propensity score of the first sample data; the operation module is further used to determine a first sampling weight of the first data set and a second sampling weight of each first sample data in the first data set according to the matching result; a sampling module, used to sample target sample data from the sample data set based on the first sampling weight and the second sampling weight, the target sample data being used to train a target model.

[0007] According to a third aspect of the present application, an electronic device is provided, comprising: a processor; and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, cause the processor to perform the above-mentioned data sampling method.

[0008] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned data sampling method.

[0009] The exemplary embodiments of the present application may have some or all of the following beneficial effects:

[0010] In the data sampling method provided in the example implementation of the present application, a sample data set is obtained, the sample data set includes a first data set and a second data set, and the generation time of the second data set is later than the generation time of the first data set; the propensity score of each first sample data in the first data set is determined; wherein the propensity score is used to characterize the probability that the first sample data falls into the second data set; based on the propensity score of the first sample data, the first sample data and the second sample data in the second data set are matched; according to the matching result, the first sampling weight of the first data set and the second sampling weight of each first sample data in the first data set are determined; based on the first sampling weight and the second sampling weight, the target sample data is sampled from the sample data set, and the target sample data is used to train the target model, and the target model is used to realize the prediction of weak periodic data. The embodiment of the present application determines the propensity score of the first sample data in the first data set, re-matches the first sample data in the first data set based on the propensity score, determines the sampling weight in the data sampling process based on the matching result, and performs data sampling based on the obtained sampling weight, thereby avoiding the problems of poor explanatory power of random sampling, the defects of prior knowledge in stratified sampling and the complexity of the algorithm caused by the increase of feature dimensions, and the defects of noise and outliers in cluster sampling, which helps to improve the various indicators and generalization performance of the trained model.

[0011] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0013] Figure 1 A flow chart of a data sampling method according to an exemplary embodiment of the present application is shown;

[0014] Figure 2 A trend chart showing average user profit in a data sampling method according to an exemplary embodiment of the present application is shown;

[0015] Figure 3 A schematic diagram of a process for determining a propensity score of each first sample data and second sample data in a data sampling method according to an exemplary embodiment of the present application is shown;

[0016] Figure 4 A schematic diagram showing how to match first sample data and second sample data using a radius matching method in a data sampling method according to an exemplary embodiment of the present application is shown;

[0017] Figure 5 A schematic block diagram of a data sampling device according to an exemplary embodiment of the present application is shown;

[0018] Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present application is shown. DETAILED DESCRIPTION

[0019] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0020] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0021] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0022] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0024] Data sampling refers to the process of extracting a portion of data from a large-scale data set, which has important applications in data mining, statistical analysis, market research, machine learning and other fields. For example, in the field of machine learning, data sampling can be used to obtain training data for model training. Specifically, taking the Internet finance industry as an example, data sampling can be used to obtain representative data from a large number of sample data sets, train the corresponding model, and implement corresponding data prediction and other functions through the trained model.

[0025] In the related art, the above data sampling process is mainly implemented through the following data sampling methods:

[0026] (1) Random sampling: This method randomly selects samples from the data set to represent the entire data set. Random sampling can cover the entire data set more evenly.

[0027] However, the above random sampling method has the following problems: it cannot fully reflect the distribution characteristics of the original data set, which may cause the trained model to perform poorly on a specific subset. For example, when there is a class imbalance problem in the data set, random sampling may further aggravate the imbalance. In addition, the results of random sampling often lack interpretability and cannot provide useful information about the distribution and characteristics of the data set, which may make it difficult to discover potential problems and patterns in the subsequent analysis and modeling process.

[0028] (2) Stratified sampling: This method divides the data set into multiple levels according to its characteristics and performs sampling at different levels. By dividing the population into several levels or subgroups and then randomly selecting samples from each level or subgroup in proportion, the characteristic attributes of the data can be better preserved and the characteristic distribution of the entire data set can be more accurately reflected.

[0029] The above stratified sampling method has the following problems: The stratified sampling method needs to divide the layers based on the prior knowledge of the characteristics of the data set. If the prior knowledge is inaccurate or insufficient, it may lead to unreasonable layer division, thus affecting the sampling effect. When the feature dimension of the data set is high or the number of layers is large, the implementation of stratified sampling may become complicated and difficult. In addition, when the difference between layers is small, the advantage of stratified sampling may not be obvious. In stratified sampling, each layer needs to take enough samples to maintain its representativeness. When the number of samples in some layers is small, more samples may need to be wasted to achieve sufficient sampling volume.

[0030] (3) Cluster sampling: Cluster the samples in the data set according to their similarity, and then sample them in different clusters. By dividing the samples in the data set into several clusters, and then randomly selecting one or more samples from each cluster as representative samples, the inherent structure of the data can be better preserved, and the correlation and similarity of the data can be better reflected.

[0031] The above cluster sampling methods have the following problems: Cluster sampling requires clustering operations on the entire data set, which may result in high computational costs. Especially on large data sets, the computational complexity and time complexity of clustering algorithms may be high. Cluster sampling is sensitive to noise and outliers. If there are many noises and outliers in the data set, the clustering results may be inaccurate, thus affecting the sampling effect.

[0032] In addition, the above data sampling method has poor characterization ability for weak periodicity. Among them, weak periodicity refers to a relatively gentle, small-growth cyclical fluctuation that occurs in a certain economic, financial or other dynamic system. For example, in the field of Internet finance, the overall user group has periodic characteristics in terms of risk, demand, profitability, etc. in the credit scenario. For example, user demand is often more vigorous at the end of the year and the beginning of the year, and is relatively sluggish in the middle of the year. In addition, there is some periodic information that belongs to weak periodicity, which is an insignificant periodicity and a characteristic that is difficult to show clearly. How to effectively extract the weak periodicity characteristics of data in the process of data sampling to improve the model effect is also a hot spot that data sampling needs to pay attention to. The above data sampling methods are all based on independent sampling of the training set samples of the model, that is, they do not consider the time relationship within the training set, nor do they consider the relationship with the test set, and they cannot reflect the weak periodicity of credit data.

[0033] In order to solve the problems existing in the above data sampling method and take into account the weak periodicity characteristics of the sampled data during the sampling process to improve the effect of model training, the embodiment of the present application provides a data sampling method, device, electronic device and computer-readable storage medium. The following describes the scheme of the present application with reference to the accompanying drawings for illustration.

[0034] The present application embodiment first provides a data sampling method. Figure 1 As shown, the data sampling method specifically includes the following steps:

[0035] S110: Acquire a sample data set, where the sample data set includes a first data set and a second data set, where the generation time of the second data set is later than the generation time of the first data set;

[0036] S120: Determine a propensity score for each first sample data in the first data set; wherein the propensity score is used to characterize the probability that the first sample data falls into the second data set;

[0037] S130: Matching the first sample data with the second sample data in the second data set based on the propensity score of the first sample data;

[0038] S140: Determine a first sampling weight of the first data set and a second sampling weight of each first sample data in the first data set according to the matching result;

[0039] S150: Sampling target sample data from the sample data set based on the first sampling weight and the second sampling weight, where the target sample data is used to train the target model.

[0040] In the data sampling method provided in the embodiment of the present application, by determining the propensity score of the first sample data in the first data set, re-matching the first sample data in the first data set based on the propensity score, determining the sampling weight in the data sampling process based on the matching result, and performing data sampling based on the obtained sampling weight, the method avoids the problems of poor interpretability of random sampling, the defects of prior knowledge in stratified sampling and the complexity of the algorithm as the feature dimension increases, as well as the defects of noise and outliers in cluster sampling, which helps to improve the various indicators and generalization performance of the trained model.

[0041] Next, in another embodiment, the above steps are described in more detail.

[0042] In step S110 , a sample data set is obtained, where the sample data set includes a first data set and a second data set, where the generation time of the second data set is later than the generation time of the first data set.

[0043] In an embodiment of the present application, the above-mentioned sample data set is a data set consisting of historical data of any business scenario. The data sampling method provided in the embodiment of the present application performs data sampling in the sample data set, and performs model training based on the sampled data, and the trained target model is used to implement the business functions required by the above-mentioned business scenario. Exemplarily, in the field of Internet finance, the above-mentioned sample data set may be a data set consisting of historical data of user credit data, and the above-mentioned target model may be used to predict user profitability. Then, the embodiment of the present application may perform data sampling on the data set consisting of historical data of the user credit data, and train the target model based on the sampled data, so as to predict user profitability through the trained target model.

[0044] In an embodiment of the present application, the sample data set may include a historical training data set and a historical test data set of the target model. The historical training data set is historical data used to train the target model, and the historical test data set is historical data used to test the model performance of the target model trained with the training data set.

[0045] In an embodiment of the present application, the above-mentioned first data set may be the above-mentioned training data set, and the above-mentioned second data set may be the above-mentioned test data set, and the generation time of the data in the test data set is later than the generation time of the data in the training data set. For example, taking the modeling and prediction of user profitability in the above-mentioned Internet finance credit scenario as an example, the above-mentioned first data set (training data set) may be the user credit data generated from January 2023 to July 2024, and the above-mentioned second data set may be the user credit data (test data set) generated in August 2024. In an embodiment of the present application, the target model may be obtained by sampling the data in the above-mentioned training data set and the test data set, and training based on the sampled data. The trained target model may be used to predict user profitability in September 2024.

[0046] In the embodiment of the present application, the acquisition of the sample data set can be implemented as follows: the first data set and the second data set are acquired in a time-sliced ​​manner. Specifically, the historical training data and the historical test data of the target model can be collected in a time-sliced ​​manner, for example, user credit data from January 2023 to July 2024 can be collected in units of months to obtain multiple first data sets, and user credit data from August 2024 can be collected to obtain the second data set.

[0047] In an embodiment of the present application, the above-mentioned user credit data may include user's credit analysis data, user loan data, user portrait and platform interaction and embedding data and other credit data.

[0048] In the embodiment of the present application, the credit analysis data includes the user's borrowing behavior and detailed borrowing records at financial institutions, which are used to characterize the user's credit risk and demand. For example, by performing operations such as feature screening and feature combination on the credit analysis data, the user's credit demand can be obtained.

[0049] In the embodiment of the present application, the above-mentioned user lending data includes the user's historical lending behavior on the Internet financial platform, such as the number of recent borrowings, recent credit limit utilization rate, recent credit limit, interest rate adjustment, historical credit granting times, etc., which are used to supplement the characterization of the user's short-term and cyclical credit needs.

[0050] In the embodiment of the present application, the above-mentioned user portrait includes basic information such as the user's education level, gender, age, and job.

[0051] In the embodiment of the present application, the above-mentioned platform interaction and buried data include the user's operations on the application software corresponding to the Internet financial institution, such as the number of clicks on certain buttons and the dwell time on certain pages, which are used to supplement the characterization of the user's current credit needs.

[0052] In the embodiments of the present application, Figure 2 This is a trend chart of the average user profit, such as Figure 2 As shown, the above user profitability has weak periodicity. By analyzing the historical data of user profitability, the embodiment of the present application can derive the average user profitability of the latest time slice, which is close to the average user profitability of the month-on-month time slice, and has the same certain characteristics as the average user profitability of the year-on-year time slice. Therefore, related technologies often need to adopt a longer time window modeling, or to enable the model to learn periodic information by refining the construction of periodic features. The embodiment of the present application can significantly improve the model effect in the modeling of actual business problems through an adaptive data set sampling method based on propensity score matching. Exemplarily, the above-mentioned adaptive data set sampling method based on propensity score matching can be implemented through the following steps S120 to S150.

[0053] In step S120, the propensity score of each first sample data in the first data set is determined; wherein the propensity score is used to characterize the probability that the first sample data falls into the second data set.

[0054] In the embodiment of the present application, the above-mentioned propensity score refers to the conditional probability that the research object is assigned to the treatment group under the condition of a given set of covariates. For example, taking the above-mentioned credit scenario in the field of Internet finance, modeling and predicting user profitability as an example, the target model corresponding to the test data set can be used as the treatment group, and the training data set of each single time slice can be used as the control group. The above-mentioned covariates are the credit analysis data, user loan data, user portraits, platform interaction and buried point data and other characteristic data in the above-mentioned user credit data, then the above-mentioned propensity score refers to the conditional probability that each sample data in the above-mentioned sample data set is assigned to the test set. For example, taking the training data set as the user credit data generated from January 2023 to July 2024, and the test data set as the user credit data generated in August 2024 as an example, the propensity score of a sample data in the training data set in January 2023 refers to the conditional probability that the sample data falls into the test data set in August 2024.

[0055] In the embodiment of the present application, the first sample data is the sample data in the first data set. Exemplarily, if the first data set is a training data set, the first sample data is the historical training data of the target model.

[0056] In the embodiment of the present application, illustratively, the above determination of the propensity score of each first sample data in the first data set can be implemented as follows: for each time slice of the first data set, the propensity score function corresponding to the first data set of the current time slice is obtained based on the first data set and the second data set of the current time slice training; the propensity score of each first sample data is determined based on the propensity score function. Specifically, the above second data set can be used as a processing group, and each first data set can be used as a control group, and the label of the processing group is set to 1, and the label of the control group is set to 0; the first data set and the second data set are used as training data to train the machine learning model to obtain a propensity score determination model, wherein the training of the machine learning model can adopt algorithms such as logistic regression, random forest, neural network, etc., and the embodiment of the present application does not make special restrictions on this; based on the propensity score determination model, the first sample data in the control group is re-scored to obtain the conditional probability of each first sample data being assigned to the processing group.

[0057] Furthermore, the embodiment of the present application can also score each second sample data in the second data set based on the propensity score function obtained by the above training to obtain the propensity score of each second sample data. Wherein, the above second sample data is the sample data in the second data set. Exemplarily, if the second data set is a test data set, the second sample data is the historical test data of the above target model.

[0058] Below, taking the above-mentioned modeling and prediction of user profitability in the credit scenario in the Internet finance field as an example, the process of determining the propensity score of each first sample data in the first data set and each second sample data in the second data set is described in detail:

[0059] Figure 3 Schematic diagram of the process of determining the propensity score of each first sample data and second sample data. Figure 3 As shown, the test set is represented by T, and the training data set of each time slice is represented by C. For example, the training data sets of time slices 202404, 202405, 202406, and 202407 can be represented by C1, C2, C3, and C4 respectively, and the test data set of 202408 can be represented by T. Then, the process of determining the propensity score of each first sample data in C1 can be implemented as follows: the original labels of T and C1 are added as one-dimensional features to the original feature set, and new labels are set for T and C1 respectively. Specifically, the new label of T is set to 1, and the new label of C1 is set to 0; T and C1 are input into the neural network and trained to obtain the propensity score model f1; the propensity score model f1 is used to score T and C1 to obtain the propensity score of the sample data in the T and C1 sample sets. The method for determining the propensity score of the sample data in the training data sets of other time slices such as C2, C3, and C4 is the same as that of C1, and will not be repeated here.

[0060] Among them, in the embodiment of the present application, the above-mentioned original feature set refers to the set composed of the features of each user data in the sample data set. Specifically, in the scenario of modeling and predicting user profitability, the above-mentioned user data is user credit data, and the features of the above-mentioned user data can be user credit analysis data, user loan data, user portrait and platform interaction and embedding data and other feature data. The above-mentioned original label is used to characterize the current user profitability corresponding to the user data.

[0061] It should be noted that the above scenario is only an exemplary description, and other methods for determining the propensity score of sample data, such as logistic regression, gradient boosting tree, neural network, etc., also fall within the protection scope of the embodiments of this application.

[0062] In step S130 , the first sample data and the second sample data in the second data set are matched based on the propensity score of the first sample data.

[0063] In the embodiment of the present application, after the propensity score of each first sample data is obtained, the first sample data can be matched with the second sample data based on the obtained propensity score of the first sample data.

[0064] In an embodiment of the present application, the process of matching the first sample data and the second sample data based on the propensity score of the first sample data can be implemented based on the radius matching method (Radius Matching, RM). The radius matching method improves the accuracy and stability of matching by limiting the difference range of the propensity score. Specifically, the radius matching method sets an absolute value (caliper value) of the propensity score difference as the matching range, and pairs the individuals in the treatment group and the control group whose propensity score differences are within the matching range. Among them, the above-mentioned caliper value needs to be set according to the specific circumstances of the actual scenario and the characteristics of the data.

[0065] In the embodiment of the present application, the radius matching method can ensure that the matched individuals have similarity in propensity scores by setting appropriate caliper values, thereby reducing the impact of confounding factors and improving the inter-group balance of the matching set. In addition, by limiting the difference range of propensity scores, the radius matching method can improve the accuracy and stability of matching, thereby improving the representativeness of the data obtained by data sampling, and further improving the model performance of the trained target model.

[0066] In the embodiment of the present application, the first sample data and the second sample data are matched based on the propensity score of the first sample data. Exemplarily, it can be implemented as follows: determining a matching radius; for each second sample data, determining a matching range corresponding to each second sample data based on the propensity score of the second sample data and the matching radius; matching the first sample data to the matching range of the corresponding second sample data based on the propensity score of each first sample data.

[0067] Below, taking the above-mentioned modeling and prediction of user profitability in the credit scenario in the field of Internet finance as an example, the process of matching the first sample data and the second sample data based on the propensity score of the first sample data is described in detail:

[0068] Figure 4 This is a schematic diagram of the result of matching the first sample data and the second sample data based on the propensity scores of the first sample data and the second sample data using the radius matching method, wherein the hollow circle is the second sample data in the second data set, and the solid circle is the first sample data in the first data set. Specifically, taking the above-mentioned training data set C1 and test data set T as an example, the above-mentioned radius matching method can be implemented as follows: set the caliper value, illustratively, the caliper value can be set to 0.05; traverse each second sample data in the T test data set, and match the samples in the C1 training data set according to the propensity score of the second sample data and the caliper value. For example, assuming that the propensity score of a second sample data in the T test data set is 0.65, it can be matched to the first sample data with a propensity score of (0.6, 0.7) in the C1 training data set.

[0069] It should be noted that the above scenario is only an exemplary description, and matching the first sample data and the second sample data by other matching methods also falls within the protection scope of the embodiments of the present application.

[0070] In step S140, a first sampling weight of the first data set and a second sampling weight of each first sample data in the first data set are determined according to the matching result.

[0071] In the embodiment of the present application, the first sampling weight is used to characterize the probability that the corresponding first data set is selected by the data sampling process.

[0072] In the embodiment of the present application, the second sampling weight is used to characterize the probability that the corresponding first sample data is selected by the data sampling process.

[0073] In the embodiment of the present application, the above-mentioned determination of the first sampling weight of the first data set based on the matching results can be implemented, exemplarily, as follows: for each first data set, determining the proportion of the first sample data in the first data set that is matched to the matching range of any second sample data; determining the first sampling weight of the first data set based on the proportion of the first sample data in the first data set that is matched to the matching range of any second sample data.

[0074] In the embodiment of the present application, the second sampling weight of each first sample data in the first data set is determined according to the matching results. Exemplarily, it can be implemented as follows: for each first sample data, the number of times the first sample data matches the matching range of the second sample data is determined; based on the number of times the first sample data matches the matching range of the second sample data, the second sampling weight of the first sample data is determined.

[0075] Below, taking the above-mentioned modeling and prediction of user profitability in the credit scenario in the Internet finance field as an example, the process of determining the first sampling weight of the first data set according to the matching results and the second sampling weight of each first sample data in the first data set is described in detail:

[0076] Specifically, the above-mentioned determination of the first sampling weight of the first data set according to the matching results, and the second sampling weight of each first sample data in the first data set can be implemented as follows: after traversing all the second sample data in the T test data set, the ratio r1 of the number of first sample data matched in the training data set C1 to the total number of first sample data in C1 is counted; the above operation is repeated for C2, C3, and C4 to obtain r2, r3, and r4, and r1, r2, r3, and r4 are normalized to obtain the first sampling weights w1, w2, w3, and w4 corresponding to the test data sets of their respective time slices; the ratio of the number of times each first sample data in C1 is matched by the radius to the number of second sample data in the T test data set is counted, and it is used as the second sampling weight of each first sample data.

[0077] In step S150, target sample data is sampled from the sample data set based on the first sampling weight and the second sampling weight. The target sample data is used to train a target model, and the target model is used to achieve prediction of weak periodic data.

[0078] In the embodiment of the present application, the above-mentioned target sample data is sample data sampled from the above-mentioned sample data set based on the data sampling method provided in the embodiment of the present application.

[0079] In the embodiment of the present application, the target model is a model trained based on the target sample data. Exemplarily, the target model can be the model for predicting user profitability.

[0080] In an embodiment of the present application, since the above-mentioned target sample data collected by the data sampling method provided in the embodiment of the present application takes into account the similarity and correlation between data sets, as well as the weak periodicity of the data, the above-mentioned target model can be used to realize the prediction of weak periodic data.

[0081] In an embodiment of the present application, the above-mentioned sampling of target sample data from the sample data set based on the first sampling weight and the second sampling weight can be implemented as follows: random weighted sampling is performed on each first data set based on the first sampling weight and the second sampling weight to obtain the target sample data.

[0082] The embodiment of the present application proposes a data sampling method based on propensity score matching for causal inference, which can realize an adaptive sampling scheme for weak periodicity data in the credit field. The adaptive sampling method avoids a series of drawbacks of existing data sampling methods, takes into account the similarity and correlation between data sets, is highly flexible, easy to implement, and can effectively improve various indicators of the model and alleviate the model attenuation phenomenon.

[0083] The above mainly introduces the solution provided by the embodiment of the present invention from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0084] The embodiment of the present invention can divide the data sampling device into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0085] The present application embodiment provides a data sampling device. Figure 5 As shown, the data sampling device 500 may include an acquisition module 510, a calculation module 520, a matching module 530 and a sampling module 540, wherein:

[0086] The acquisition module 510 may be used to acquire a sample data set, where the sample data set includes a first data set and a second data set, where the generation time of the second data set is later than the generation time of the first data set;

[0087] The operation module 520 may be used to determine the propensity score of each first sample data in the first data set; wherein the propensity score is used to characterize the probability that the first sample data falls into the second data set;

[0088] A matching module 530 may be used to match the first sample data with the second sample data in the second data set based on the propensity score of the first sample data;

[0089] The operation module 520 may also be used to determine a first sampling weight of the first data set and a second sampling weight of each first sample data in the first data set according to the matching result;

[0090] The sampling module 540 is used to sample target sample data from the sample data set based on the first sampling weight and the second sampling weight, and the target sample data is used to train the target model.

[0091] In an embodiment of the present application, the acquisition module is specifically used to: acquire a historical training data set and a historical test data set of the target model, and use the historical training data set as the first data set and the historical test data set as the second data set.

[0092] In an embodiment of the present application, the above-mentioned sample data set includes multiple first data sets of time slices, and the above-mentioned operation module is specifically used to: for each first data set of time slice, obtain the propensity score function corresponding to the first data set of the current time slice based on the first data set and the second data set of the current time slice for training; determine the propensity score of each first sample data and the second sample data based on the propensity score function.

[0093] In an embodiment of the present application, the above-mentioned matching module is specifically used to: determine the matching radius; for each second sample data, determine the matching range corresponding to each second sample data based on the propensity score and the matching radius of the second sample data; based on the propensity score of each first sample data, match the first sample data to the matching range of the corresponding second sample data.

[0094] In an embodiment of the present application, the above-mentioned matching module is specifically used to: determine, for each first data set, the proportion of first sample data in the first data set that is matched to the matching range of any second sample data; and determine the first sampling weight of the first data set based on the proportion of first sample data in the first data set that is matched to the matching range of any second sample data.

[0095] In an embodiment of the present application, the matching module is specifically used to: determine, for each first sample data, the number of times the first sample data matches the matching range of the second sample data; and determine the second sampling weight of the first sample data based on the number of times the first sample data matches the matching range of the second sample data.

[0096] In the embodiment of the present application, the sampling module is specifically used to: perform random weighted sampling on each first data set based on the first sampling weight and the second sampling weight to obtain target sample data.

[0097] In addition, the specific implementation details of the above data sampling device have been described in detail in the corresponding position of the data sampling method, so they will not be repeated here.

[0098] Figure 6 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. Figure 6 , which shows a structural schematic diagram of an electronic device 600 suitable for implementing the embodiments of the present application. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0099] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603 to implement the data sampling method of the embodiment described in the present application. In RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0100] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0101] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart, thereby implementing the data sampling method as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0102] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0103] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0104] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0105] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0106] Acquire a sample data set, where the sample data set includes a first data set and a second data set, where the generation time of the second data set is later than the generation time of the first data set;

[0107] Determine a propensity score for each first sample data in the first data set; wherein the propensity score is used to characterize the probability that the first sample data falls into the second data set;

[0108] Matching the first sample data with the second sample data in the second data set based on the propensity score of the first sample data;

[0109] Determine a first sampling weight of the first data set and a second sampling weight of each first sample data in the first data set according to the matching result;

[0110] Target sample data is sampled from the sample data set based on the first sampling weight and the second sampling weight, and the target sample data is used to train the target model.

[0111] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0112] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0113] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0114] The units involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0115] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0116] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0117] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0118] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or to be performed in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0119] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A data sampling method, characterized in that: include: Acquire a sample data set, where the sample data set includes a first data set and a second data set, where the generation time of the second data set is later than the generation time of the first data set; Determine a propensity score for each first sample data in the first data set; wherein the propensity score is used to characterize the probability that the first sample data falls into the second data set; Matching the first sample data with second sample data in the second data set based on the propensity score of the first sample data; Determine a first sampling weight of the first data set and a second sampling weight of each of the first sample data in the first data set according to the matching result; Target sample data is sampled from the sample data set based on the first sampling weight and the second sampling weight, and the target sample data is used to train a target model.

2. The data sampling method according to claim 1, characterized in that: The obtaining of the sample data set comprises: A historical training data set and a historical test data set of the target model are obtained, and the historical training data set is used as the first data set, and the historical test data set is used as the second data set.

3. The data sampling method according to claim 1, characterized in that: The sample data set includes the first data set of multiple time slices; The determining of the propensity score of each first sample data in the first data set includes: For each time slice, the first data set is trained based on the first data set and the second data set of the current time slice to obtain a propensity score function corresponding to the first data set of the current time slice; The propensity score of each of the first sample data is determined based on the propensity score function.

4. The data sampling method according to claim 1, characterized in that: The matching of the first sample data with the second sample data in the second data set based on the propensity score of the first sample data includes: Determine the matching radius; For each of the second sample data, determining a matching range corresponding to each of the second sample data based on the matching radius; Based on the propensity score of each of the first sample data, the first sample data is matched to the matching range of the corresponding second sample data.

5. The data sampling method according to claim 4, characterized in that: The determining a first sampling weight of the first data set according to the matching result includes: For each of the first data sets, determining a proportion of the first sample data in the first data set that is matched to the matching range of any of the second sample data; The first sampling weight of the first data set is determined based on the proportion of the first sample data matched to the matching range of any second sample data in the first data set.

6. The data sampling method according to claim 4, characterized in that: The determining, according to the matching result, a second sampling weight of each of the first sample data in the first data set includes: For each of the first sample data, determining the number of times the first sample data matches the matching range of the second sample data; The second sampling weight of the first sample data is determined based on the number of times the first sample data matches the matching range of the second sample data.

7. The data sampling method according to claim 1, characterized in that: The step of sampling target sample data from the sample data set based on the first sampling weight and the second sampling weight includes: Random weighted sampling is performed on each of the first data sets based on the first sampling weight and the second sampling weight to obtain the target sample data.

8. A data sampling device, characterized in that: include: An acquisition module, used to acquire a sample data set, wherein the sample data set includes a first data set and a second data set, and the generation time of the second data set is later than the generation time of the first data set; A calculation module, used to determine the propensity score of each first sample data in the first data set; wherein the propensity score is used to characterize the probability that the first sample data falls into the second data set; a matching module, configured to match the first sample data with second sample data in the second data set based on the propensity score of the first sample data; The operation module is further used to determine a first sampling weight of the first data set and a second sampling weight of each of the first sample data in the first data set according to the matching result; A sampling module is used to sample target sample data from the sample data set based on the first sampling weight and the second sampling weight, and the target sample data is used to train a target model.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images