A multi-objective balanced sampling method, system, device and storage medium

Abstract

Embodiments of the present application provide a multi-objective balanced sampling method, system, device and storage medium, relating to the field of data sampling, the method comprising: obtaining a full data set and a total sample size; dimension reduction to obtain a low-dimensional representation vector; constructing a multi-objective loss function based on the overall distribution of the full data set under the preset sampling target; clustering the samples based on the similarity between the low-dimensional representation vectors to obtain a plurality of clustering clusters, and determining the initial sampling quota according to the values calculated in the multi-objective loss function for each clustering cluster; selecting candidate samples based on the initial sampling quota and substituting them into the multi-objective loss function, and iteratively adjusting the sampling quota and the candidate samples based on the comparison of the target loss function value and the preset function threshold to determine the target sampling quota and the target samples and obtain a balanced sample set. Through the multi-objective loss function as feedback to adjust the samples, the distribution of the balanced sample set on the preset sampling target is close to the overall distribution, thereby improving the accuracy of reflecting the risks of various enterprises.

Description

Technical Field

[0001] This invention relates to the field of financial data sampling technology, and more specifically, to a multi-objective balanced sampling method, system, device, and storage medium. Background Technology

[0002] With the deepening of digital transformation in finance, banks and financial regulatory agencies have accumulated massive amounts of corporate data, including basic information, financial data, revenue and expenditure records, credit performance, and related-party transaction data. When building machine learning models for corporate credit assessment and risk warning, it is necessary to extract a representative sample set of enterprises from the full dataset of these massive amounts of data.

[0003] The common practice is to extract a small, representative sample of companies from the massive dataset of all enterprises, and use this sample to represent the entire dataset for model training. Currently, commonly used sampling methods include: simple random sampling, which involves completely random selection like drawing lots; traditional stratified sampling, which involves grouping companies according to a certain feature and then selecting them proportionally; and cluster sampling, which involves grouping similar companies into one category and then selecting a few categories.

[0004] However, the above methods rely entirely on luck with simple random sampling, failing to guarantee that the sampled companies will reflect the true picture across all dimensions, including industry, region, and scale. Traditional stratified sampling, when balancing more than 3-4 dimensions, results in an exponential increase in the number of stratified combinations. Three dimensions might generate hundreds of groups, while four dimensions could produce thousands or even tens of thousands. Many of these groups may not even contain any companies, leading to the dimensionality explosion problem. Cluster sampling groups similar companies into clusters, but only selects a few clusters to represent the whole, inevitably causing the groups represented by those unselected clusters to completely disappear from the sample. Resampling methods only focus on whether there are more good companies or fewer bad companies, regardless of whether other dimensions are balanced.

[0005] Therefore, existing sampling methods can generally only perform stratification on a single or a few dimensions. When faced with scenarios that require balancing the distribution of multiple dimensions at the same time, the distribution of the extracted samples in each dimension is difficult to be consistent with the distribution of each dimension in the full dataset, resulting in insufficient global representativeness of the sampling and thus difficulty in accurately reflecting the actual risk characteristics of various enterprises in the full dataset. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a multi-objective balanced sampling method, system, device and storage medium to solve the technical problem that the distribution of the extracted samples in multiple dimensions is difficult to be consistent with the distribution of the entire data, resulting in extremely low identification accuracy when identifying such niche high-risk enterprises.

[0007] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a multi-objective balanced sampling method, the method comprising: acquiring a full dataset and a total number of samples, wherein the full dataset includes multiple samples, each sample having feature data of multiple dimensions; performing dimensionality reduction processing on the full dataset to obtain low-dimensional representation vectors corresponding to each sample; constructing a multi-objective loss function based on the overall distribution of the full dataset under a preset sampling objective; the multi-objective loss function being used to measure the difference between the distribution of any sample set under the preset sampling objective and the distribution of the full dataset under the preset sampling objective; and clustering multiple samples in the full dataset based on the similarity between the low-dimensional representation vectors to obtain multiple clusters; Based on the distribution of samples in each cluster under the preset sampling target and the multi-objective loss function, an initial sampling quota for each cluster is determined; the sum of the initial sampling quotas for each cluster equals the total number of samples; candidate samples are selected from each cluster based on the initial sampling quota, and the candidate samples are substituted into the sampling sample set in the multi-objective loss function to obtain the corresponding target loss function value; the initial sampling quota and the candidate samples are adjusted based on the comparison between the target loss function value and the preset function threshold to determine the target sampling quota and target sample for each cluster; based on the target sampling quota and target sample for each cluster, samples are extracted from the full dataset to obtain a balanced sample set.

[0008] In this embodiment, after acquiring the full dataset, the total number of samples to be sampled is determined based on the full dataset. Then, the full dataset is dimensionality-reduced to obtain low-dimensional representation vectors for each sample, thus avoiding dimensionality explosion during computation. Next, a multi-objective loss function is constructed based on the overall distribution of the full dataset under a preset sampling objective, so as to measure the difference between the distributions of the sampled sample set and the full dataset under the preset sampling objective. Finally, the dimensionality-reduced low-dimensional representation vectors are clustered to obtain multiple clusters, so that samples from the clusters can be input into the multi-objective loss function to determine... Initial sampling quotas for each cluster are determined, ensuring that the sum of these quotas equals the total sample size. This facilitates the random selection of candidate samples from each cluster based on these initial quotas. These candidate samples are then substituted into a multi-objective loss function to obtain the corresponding target loss function value. By comparing the calculated target loss function value with a preset threshold, the target sampling quota and target sample for each cluster are determined. During this comparison, the randomly selected samples are adjusted to obtain a balanced sample set whose proportion under the preset sampling target is not significantly different from the proportion of the full dataset under the preset sampling target. By using the multi-objective loss function value as feedback, the final target sampling quotas and target samples for each cluster can simultaneously approximate the overall distribution of the full dataset under multiple preset sampling targets. This ensures that a balanced sample set with high global representativeness across all preset sampling targets is extracted from the full dataset, avoiding the technical shortcomings of insufficient global representativeness caused by stratified processing of only one or a few dimensions. This improves the accuracy of reflecting the actual risk characteristics of various enterprises in the full dataset.

[0009] In some embodiments, constructing a multi-objective loss function based on the overall distribution of the full dataset under a preset sampling target includes: using the distribution of any sample set under the preset sampling target as an independent variable, using the overall distribution of the full dataset under the preset sampling target as a reference benchmark, and using the preset weight coefficient corresponding to the preset sampling target as a weighting coefficient; and constructing the multi-objective loss function based on the divergence value between the independent variable and the reference benchmark, and the corresponding weighting coefficient.

[0010] This setting limits the distribution of the sample set, the overall distribution of the full dataset, and the preset weight coefficients to the independent variables, reference benchmarks, and weighting coefficients of the loss function, respectively, enabling the multi-objective loss function to simultaneously measure the distribution differences under multiple preset sampling objectives.

[0011] In some embodiments, the similarity is represented by Euclidean distance, and the step of clustering multiple samples in the full dataset based on the similarity between the low-dimensional representation vectors to obtain multiple clusters includes: clustering the low-dimensional representation vectors based on the Euclidean distance between the low-dimensional representation vectors of any two samples in the full dataset to obtain the multiple clusters; in the multiple clusters, the Euclidean distance between the low-dimensional representation vectors of any two samples within the same cluster is less than the Euclidean distance between the low-dimensional representation vectors of samples in different clusters.

[0012] This setup uses Euclidean distance as a measure of similarity between low-dimensional representation vectors and constrains the Euclidean distance between the low-dimensional representation vectors of any two samples within the same cluster to be less than the distance between samples from different clusters. This ensures the homogeneity of samples within clusters in terms of their original multidimensional features and the heterogeneity of samples between clusters, enabling each cluster to truly reflect the distribution of different feature groups in the full dataset.

[0013] In some embodiments, determining the initial sampling quota of each cluster based on the distribution of samples of each cluster under the preset sampling target and the multi-objective loss function includes: substituting the distribution of samples of each cluster under the preset sampling target into the sampling sample set in the multi-objective loss function to obtain the initial loss function value of each cluster; and determining the initial sampling quota of each cluster based on the ratio of the initial loss function value of each cluster to the sum of the initial loss function values ​​of all clusters.

[0014] This setup substitutes the distribution of each cluster under the preset sampling objective into the multi-objective loss function. The resulting initial loss function value quantifies the degree to which each cluster deviates from the overall distribution of the whole population. The initial sampling quota is the proportion of the initial loss function value of each cluster in the sum of all clusters. This allows clusters that deviate more from the population to receive higher quotas, thus enabling differentiated sampling attention to different clusters in the initial stage.

[0015] In some embodiments, substituting the distribution of samples from each cluster under the preset sampling target into the sampling sample set in the multi-objective loss function to obtain the initial loss function value for each cluster includes: calculating the divergence value corresponding to the preset sampling target based on the distribution of samples from each cluster under the preset sampling target and the overall distribution; multiplying the divergence value corresponding to the preset sampling target by the preset weight coefficient corresponding to the preset sampling target to obtain the difference value corresponding to the preset sampling target; and summing the difference values ​​corresponding to each preset sampling target to obtain the initial loss function value for the cluster.

[0016] This setup calculates the divergence value of clusters under a preset sampling target, and after calculating preset weighting coefficients, it finally summarizes them into a single value, so that the initial loss function value of each cluster can accurately reflect the comprehensive deviation between the sample distribution and the overall distribution.

[0017] In some embodiments, adjusting the initial sampling quota and the candidate samples based on a comparison between the target loss function value and a preset function threshold, and determining the target sampling quota and target sample for each cluster, includes: when the target loss function value is greater than the preset function threshold, adjusting the initial sampling quota of each cluster and updating the candidate samples; substituting the updated distribution of the candidate samples under the preset sampling target back into the sampling sample set in the multi-objective loss function until the calculated target loss function value is less than or equal to the preset function threshold; and determining the current sampling quota and candidate samples of each cluster as the target sampling quota and target sample, respectively.

[0018] With this setting, the sampling quotas and candidate samples of each cluster can gradually approach the sampling balance under each preset sampling target in multiple rounds of adjustment, effectively avoiding the difference problem caused by the mutual checks and balances between multiple sampling targets in a single allocation, so that the distribution of the final output balanced sample set on multiple preset sampling targets is close to the full dataset.

[0019] In some embodiments, adjusting the initial sampling quota of each cluster and updating the candidate samples includes: if, in any of the plurality of clusters, the number of samples belonging to a sample category under the preset sampling target in any cluster is less than a preset threshold relative to the total number of samples belonging to that sample category in the plurality of clusters, the sample category is determined as a scarce category of the current cluster; a preset number of clusters with the smallest Euclidean distance to the current cluster are designated as neighboring clusters, and samples belonging to the scarce category are extracted from the neighboring clusters to supplement the sampling quota of the current cluster; the supplemented current sampling quota is used as the adjusted initial sampling quota, and the current candidate samples are updated based on the adjusted initial sampling quota to obtain updated candidate samples.

[0020] This setup effectively ensures the full representativeness of niche categories that are easily overlooked during sample allocation by supplementing real samples from neighboring clusters into clusters with scarce categories. It also addresses the technical deficiency of existing proportional sampling methods that suffer from severe lack of samples from niche, high-risk groups.

[0021] Secondly, embodiments of the present invention provide a multi-objective balanced sampling system, the system comprising: The acquisition module is used to acquire the full dataset and the total number of samples. The full dataset includes multiple samples, and each sample has feature data in multiple dimensions. The dimensionality reduction module is used to perform dimensionality reduction processing on the full dataset to obtain the low-dimensional representation vector corresponding to each sample. A stratified sampling module is used to construct a multi-objective loss function based on the overall distribution of the full dataset under a preset sampling objective. The multi-objective loss function measures the difference between the distribution of any sample set under the preset sampling objective and the distribution of the full dataset under the preset sampling objective. Based on the similarity between the low-dimensional representation vectors, multiple samples in the full dataset are clustered to obtain multiple clusters. According to the distribution of samples in each cluster under the preset sampling objective and the multi-objective loss function, the initial sampling quota for each cluster is determined. The sum of the initial sampling quotas for each cluster equals the total number of samples. The processing module is configured to select candidate samples from each of the clusters based on the initial sampling quota, and substitute the candidate samples into the sampling sample set in the multi-objective loss function to obtain the corresponding target loss function value; adjust the initial sampling quota and the candidate samples according to the comparison between the target loss function value and the preset function threshold to determine the target sampling quota and target sample for each of the clusters; and extract samples from the full dataset based on the target sampling quota and target sample for each of the clusters to obtain a balanced sample set.

[0022] Thirdly, embodiments of the present invention provide an electronic device, including a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the multi-objective balanced sampling method as described in the first aspect.

[0023] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multi-objective balanced sampling method as described in the first aspect.

[0024] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating the steps of a multi-objective balanced sampling method provided in this embodiment of the invention; Figure 2 This is a flowchart of a multi-objective balanced sampling method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the functional modules of the multi-objective balanced sampling system provided in an embodiment of the present invention; Figure 4 A block diagram of an electronic device provided in an embodiment of the present invention.

[0027] Icons: 1000 - Multi-objective balanced sampling system; 1100 - Acquisition module; 1200 - Dimensionality reduction module; 1300 - Stratified sampling module; 1400 - Processing module; 2000 - Electronic equipment; 2100 - Processor; 2200 - Memory; 2300 - Bus; 2400 - Communication interface. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0029] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0030] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0031] As described in the background section, existing sampling methods can generally only perform stratification on a single or a few dimensions. When faced with scenarios that require balancing the distribution of multiple dimensions simultaneously, the distribution of the extracted samples in each dimension is difficult to be consistent with the distribution of each dimension in the full dataset, resulting in insufficient global representativeness of the sampling and thus difficulty in accurately reflecting the actual risk characteristics of various enterprises in the full dataset.

[0032] To address this, embodiments of the present invention provide a multi-objective balanced sampling method. By using the value of a multi-objective loss function as feedback, the final determined target sampling quotas and target samples for each cluster can simultaneously approximate the overall distribution of the full dataset under multiple preset sampling objectives. This allows for the extraction of a balanced sample set from the full dataset that possesses high global representativeness across all preset sampling objectives. This avoids the technical shortcomings of insufficient global representativeness caused by stratifying only a single or a few dimensions, thereby improving the accuracy of reflecting the actual risk characteristics of various enterprises in the full dataset. See also... Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating the steps of a multi-objective balanced sampling method provided in an embodiment of the present invention. Figure 2 This is a flowchart of a multi-objective balanced sampling method provided in an embodiment of the present invention. The multi-objective balanced sampling method includes steps S100 to S800: S100. Obtain the full dataset and total number of samples. The full dataset includes multiple samples, each with multiple dimensions of feature data.

[0033] In this embodiment, each sample in the full dataset corresponds to a single enterprise, and each enterprise sample has multi-dimensional feature data, including basic enterprise information, financial data, industry category, city / prefecture location, bank transaction history, and credit performance records. The total number of samples is determined to be 150,000 based on business needs, which is approximately 5% of the total number of samples in the full dataset.

[0034] In some embodiments, the full dataset is sourced from the complete enterprise data aggregated by a provincial enterprise revenue and expenditure credit reporting platform. This full dataset covers over three million operating enterprises across 29 commercial banks in the province, excluding those that have been deregistered or revoked. Before using the aforementioned full dataset, preprocessing is required. Preprocessing specifically includes outlier handling, missing value handling, feature encoding, and standardization.

[0035] For example, in terms of outlier handling, the 3σ principle is applied to numerical features, meaning that extreme outliers exceeding three standard deviations above or below the mean of all samples in the numerical feature field are removed. Taking registered capital and operating revenue as examples, the enterprise samples corresponding to extreme values ​​exceeding three standard deviations above the mean are removed, resulting in 12,847 outlier entities being removed, leaving 3,011,922 valid entities. Regarding missing value handling, missing fields of numerical features are filled with the median of enterprises of the same size and industry, while missing fields of categorical features are filled with the value that appears most frequently in the entire dataset, keeping the overall feature missing rate below 2%. For feature encoding, categorical features such as industry category, city, and enterprise type are first converted to integer numbers using label encoding, and then expanded into multi-dimensional vectors using one-hot encoding. For standardization, continuous features such as registered capital and operating revenue are uniformly standardized using Z-score processing, ensuring that each feature field follows a standard normal distribution with a mean of zero and a standard deviation of one, thereby eliminating the interference of different units on subsequent Euclidean distance calculations. After preprocessing, each sample in the full dataset has 136 feature dimensions, including 47 categorical features and 89 numerical features.

[0036] S200. Perform dimensionality reduction on the full dataset to obtain the low-dimensional representation vectors corresponding to each sample.

[0037] In this embodiment, an autoencoder is used as a nonlinear dimensionality reduction algorithm to map each sample from 136-dimensional multidimensional feature data to an 8-dimensional low-dimensional representation space, thereby obtaining the corresponding 8-dimensional low-dimensional representation vector for each sample.

[0038] For example, the autoencoder's network structure is a four-layer fully connected architecture, specifically including an input layer, a first hidden layer, a second hidden layer, and an output layer. The input layer has 136 neurons, corresponding to the 136-dimensional original features; the first hidden layer has 64 neurons, with ReLU activation; the second hidden layer is the representation layer, with 8 neurons, also with ReLU activation; and the output layer has 136 neurons, with Sigmoid activation. The autoencoder is trained by minimizing the reconstruction error between the input and output layers, forcing the 8 neurons in the representation layer to summarize the core information in the original 136-dimensional feature data with extremely low information loss.

[0039] Furthermore, regarding the training parameter configuration of the autoencoder, the optimizer was Adam, the learning rate was set to 0.001, the batch size was set to 2048, and the maximum number of training epochs was preset to one hundred. An early stopping strategy was adopted, meaning training was terminated early if the reconstruction error on the validation set did not decrease for five consecutive epochs to prevent overfitting. After training, the first half of the autoencoder network, from the input layer to the representation layer, was used as the encoder. The 136-dimensional feature data of each sample in the full dataset was input into this encoder, and the eight values ​​output by the representation layer became the 8-dimensional low-dimensional representation vector of that sample. After training, the mean squared error of reconstruction on the training set was 0.0127, and the mean squared error of reconstruction on the validation set was 0.01232. The information retention rate from the 8-dimensional representation vector to the 136-dimensional representation vector reached 94%, indicating that the 8-dimensional low-dimensional representation vector effectively preserved most of the discriminative information and nonlinear coupling relationships between features in the original high-dimensional feature data. In 8-dimensional space, the smaller the Euclidean distance between the low-dimensional representation vectors corresponding to any two samples, the higher the overall similarity between the two samples on the original 136-dimensional feature data.

[0040] In some other embodiments, dimensionality reduction can also be implemented using any nonlinear dimensionality reduction algorithm, either UMAP or t-SNE, and the dimension of the low-dimensional representation vector can range from three to ten dimensions.

[0041] S300. Based on the overall distribution of the full dataset under the preset sampling objective, construct a multi-objective loss function. The multi-objective loss function is used to measure the difference between the distribution of any sample set under the preset sampling objective and the distribution of the full dataset under the preset sampling objective.

[0042] In this embodiment, based on the business requirements of financial credit risk control modeling, the preset sampling objectives include at least two of the following: credit quality sample distribution, enterprise size distribution, industry category distribution, and municipal administrative division distribution. This embodiment uses four preset sampling objectives as an example, setting corresponding preset weight coefficients for each. The weight coefficient for the credit quality sample distribution is 0.4, while the weight coefficients for the enterprise size distribution, industry category distribution, and municipal administrative division distribution are all 0.2. The weight coefficients reflect the importance of each preset sampling objective in the multi-objective loss function; the larger the weight coefficient, the higher the degree to which the preset sampling objective is preferentially satisfied in the final balanced sample set. After determining the preset sampling objectives, the overall distribution corresponding to each preset sampling objective is statistically analyzed from the full dataset. That is, using the full dataset as the statistical scope, the proportion of samples in each category under each preset sampling objective to the total number of samples in the full dataset is calculated.

[0043] S400: Based on the similarity between low-dimensional representation vectors, multiple samples in the full dataset are clustered to obtain multiple clusters.

[0044] In this embodiment, the clustering process is based on the Euclidean distance of the low-dimensional representation vectors of each sample in 8-dimensional space. The cluster center positions are repeatedly adjusted and the cluster affiliation of each sample is updated, ensuring that the Euclidean distance between the low-dimensional representation vectors of any two samples within the same cluster is less than the Euclidean distance between the low-dimensional representation vectors of samples from different clusters. For example, after clustering, the more than three million enterprise samples in the full dataset are divided into two hundred clusters. Each cluster has a cluster center, which is the mean vector of the low-dimensional representation vectors of all samples within that cluster. Enterprise samples within a cluster are close to each other in 8-dimensional space, indicating that these samples have high similarity in the original 136-dimensional feature data, while samples from different clusters show significant differences in feature data.

[0045] In some other embodiments, the number of clusters can be selected from fifty to five hundred, depending on the total number of samples in the full dataset and the number of dimensions of the low-dimensional representation vectors. The clustering algorithm can be the k-medoids algorithm.

[0046] S500. Based on the distribution of samples in each cluster under the preset sampling objective and the multi-objective loss function, determine the initial sampling quota for each cluster; the sum of the initial sampling quotas for each cluster equals the total number of samples.

[0047] In this embodiment, after obtaining multiple clusters, the distribution of samples contained in each cluster under four preset sampling objectives is used as independent variables and substituted into the constructed multi-objective loss function to calculate an initial loss function value for each cluster. Then, the initial loss function value calculated for each cluster is used as the basis for the initial sampling quota, so as to determine the initial sampling quota of each cluster through the initial loss function value, so that the sum of the initial sampling quotas of each cluster is equal to the total number of samples.

[0048] For example, the initial loss function value of a single cluster is calculated as follows: First, based on the distribution of the cluster's samples under a preset sampling target and the overall distribution of the full dataset, the divergence value corresponding to each of the four preset sampling targets is calculated. The divergence value is used to characterize the degree to which the cluster's distribution deviates from the overall distribution under that target. Then, the divergence value corresponding to each preset sampling target is multiplied by the preset weight coefficient corresponding to that preset sampling target to obtain the difference value corresponding to each preset sampling target. Finally, the difference values ​​corresponding to the four preset sampling targets are summed to obtain the initial loss function value of this single cluster. The larger the initial loss function value, the more significant the difference between the distribution structure of the samples within the cluster and the overall distribution of the full dataset.

[0049] S600. Based on the initial sampling quota, candidate samples are selected from each cluster, and the candidate samples are substituted into the sampling sample set in the multi-objective loss function to obtain the corresponding objective loss function value.

[0050] In this embodiment, after allocating the initial sampling quotas for each cluster, a corresponding number of specific enterprise samples are randomly selected from each cluster according to their respective initial sampling quotas to form a candidate sample set. For example, the total number of candidate samples is equal to the total number of samples, 150,000. Subsequently, the distribution of the candidate samples under the four preset sampling objectives is substituted as an independent variable into the sampling sample set in the multi-objective loss function to calculate the target loss function value. The calculation process of the target loss function value is the same as the calculation process of the initial loss function value. First, the divergence value corresponding to each preset sampling objective is calculated based on the distribution of the candidate samples under each preset sampling objective and the overall distribution of the full dataset. Then, each divergence value is multiplied by the corresponding preset weight coefficient to obtain the difference value. Finally, the difference values ​​are added together to obtain the target loss function value.

[0051] It should be noted that in the initial stage, since the initial sampling quota is allocated based on the proportion of the initial loss function value and has not yet undergone iterative optimization, the target loss function value calculated for the first time is usually greater than the preset function threshold. For example, in this embodiment, the initial target loss function value is 0.0214, while the preset function threshold is 0.008. The difference between the target loss function value and the preset function threshold constitutes the optimization space for subsequent iterative adjustments.

[0052] S700. Based on the comparison between the target loss function value and the preset function threshold, the initial sampling quota and candidate samples are adjusted to determine the target sampling quota and target samples for each cluster.

[0053] In this embodiment, when the initial sampling quota and candidate samples are initially obtained, the initial target loss function value is usually greater than a preset function threshold, thus requiring an iterative adjustment process. The core of this iterative adjustment lies in using the multi-objective loss function value as a feedback signal to repeatedly adjust the sampling quota for each cluster and the specific enterprises selected as candidate samples until the target loss function value calculated after substitution is less than or equal to the preset function threshold. During the iterative adjustment process, the magnitude and direction of the adjustment are determined by the degree of deviation of the distribution of samples in each cluster relative to the overall distribution under each preset sampling objective.

[0054] For example, in each iteration, the divergence between the distribution of each cluster under each preset sampling objective and the overall population distribution is calculated. Clusters with greater deviation receive a larger adjustment in sampling quota in the next round. After the adjustment, the candidate sample list is updated, and the updated candidate samples are resubmitted into the multi-objective loss function to recalculate the target loss function value, which is then compared with the preset function threshold again.

[0055] Specifically, this embodiment uses a genetic algorithm as the implementation method for iterative adjustment. The population size is set to 100, the crossover probability is 0.8, the mutation probability is 0.05, the elite retention rate is 10%, and the maximum number of iterations is 500. In each iteration, each individual in the population represents a combination scheme of sampling quotas and candidate samples from two hundred clusters. The fitness is calculated by substituting the candidate sample corresponding to the individual into the multi-objective loss function. Individuals with better fitness have a higher probability of entering the next round. A new generation of the population is generated through selection, crossover, and mutation operations, and the target loss function value of the population decreases generation by generation. The initial average target loss function value of the population is 0.0214. After 327 iterations, the target loss function value of the best individual drops to 0.0071, which is lower than the preset function threshold of 0.008, satisfying the convergence condition, and the iteration stops.

[0056] In some other embodiments, iterative adjustments can also be implemented using simulated annealing or multi-objective particle swarm optimization.

[0057] S800: Based on the target sampling quota and target sample of each cluster, samples are extracted from the full dataset to obtain a balanced sample set.

[0058] In this embodiment, after the target loss function value satisfies the preset convergence condition, the sampling quota of each cluster at this time is determined as its respective target sampling quota, and each candidate sample at this time is determined as its respective target sample. According to the target sampling quota of each cluster and the specific list of target samples, the complete multidimensional feature data of 150,000 enterprises are extracted from the more than 3 million enterprises in the full dataset to obtain a balanced sample set. After the above-described full process of dimensionality reduction, clustering, sampling quota allocation, scarce category compensation and iterative adjustment, the difference between the distribution of this balanced sample set on the four preset sampling targets and the overall distribution of the full dataset has been iteratively optimized to a minimum value.

[0059] For example, the final output balanced sample set has a KL divergence of 0.0006 on the distribution of good and bad credit samples, corresponding to a similarity of 99.2%; a KL divergence of 0.0011 on the distribution of enterprise size, corresponding to a similarity of 98.7%; a KL divergence of 0.0020 on the distribution of major industry categories, corresponding to a similarity of 97.6%; and a KL divergence of 0.0017 on the distribution of prefecture-level administrative regions, corresponding to a similarity of 98.1%. The overall loss value is 0.0071, and the overall distribution similarity reaches 96.8%. When this balanced sample set is directly used to train the enterprise credit risk rating model, compared to the model trained using traditional two-dimensional stratified sampling, the model's AUC value increases from 0.782 to 0.817, and the KS value increases from 0.426 to 0.471.

[0060] In some embodiments, for step S300, one possible implementation of the present invention is that step S300 includes sub-steps S310~S320: S310. Take the distribution of any sample set under the preset sampling target as the independent variable, take the overall distribution of the full dataset under the preset sampling target as the reference benchmark, and take the preset weight coefficient corresponding to the preset sampling target as the weighting coefficient.

[0061] In this embodiment, the overall distribution of the entire dataset under the preset sampling objectives is first used as a constant standard reference. The distribution of any sample set under the four preset sampling objectives is used as the independent variable, and the value of the independent variable changes with different candidate sample sets. The preset weight coefficient corresponding to each preset sampling objective is used as the weighting coefficient. For example, the weight coefficient of the credit quality sample distribution mentioned in step S300 is 0.4, and the weight coefficients of the enterprise size distribution, industry category distribution, and prefecture-level administrative division distribution are all 0.2.

[0062] S320. Based on the divergence between the independent variable and the reference baseline, and the corresponding weighting coefficients, construct a multi-objective loss function.

[0063] In this embodiment, divergence is used as a measure of the difference between the independent variable and the reference benchmark. The divergence values ​​corresponding to each preset sampling target are weighted and combined with their corresponding weighting coefficients to construct a multi-objective loss function. The constructed multi-objective loss function is formally the sum of the divergence values ​​of each preset sampling target multiplied by their respective weighting coefficients. The divergence is measured using KL divergence, which is calculated by multiplying the probability distribution value of each category in the sample distribution by the logarithm of that probability distribution value relative to the corresponding category's probability distribution value in the overall distribution. The sum of all weighting coefficients is one. Preset sampling targets with larger weighting coefficients have greater influence in the multi-objective loss function, and their distribution differences contribute more significantly to the overall loss value.

[0064] For example, the multi-objective loss function is specifically as follows: ; in, Let be the weight coefficient of the i-th sampling target, and ; The KL divergence between the sample distribution and the population distribution under the i-th sampling objective is calculated using the following formula: ; in, The distribution of the sampled data under the preset sampling target. This represents the overall distribution of the full dataset under the preset sampling objective.

[0065] For example, taking the distribution of creditworthiness samples as an example, bad samples are defined as enterprises that have been overdue for 30 days or more in the past twelve months. Statistics show that there are 247,521 bad sample enterprises in the entire dataset, accounting for 8.22% of the total sample. Taking the distribution of enterprise size as an example, enterprises are divided into four categories according to the SME classification standards: micro, small, medium, and large. The proportions of these four categories in the entire dataset are 68.3%, 24.7%, 5.8%, and 1.2%, respectively.

[0066] For example, taking the distribution of major industry categories as an example, statistics are collected according to the twenty major industry categories of the economic industry classification. Taking the distribution of prefecture-level administrative divisions as an example, it covers fourteen prefecture-level cities in a certain province. The distribution of any sample set under four preset sampling objectives is used as the independent variable. The overall distribution of the above-mentioned full dataset under the four preset sampling objectives is used as the reference benchmark. The preset weight coefficients of each preset sampling objective are used as weighting coefficients. The divergence value is used to measure the difference between the independent variable and the reference benchmark. The divergence value corresponding to each preset sampling objective is weighted and combined with the corresponding weighting coefficient to construct a multi-objective loss function. The lower the value of the multi-objective loss function, the closer the distribution of the sample set is to the overall distribution of the full dataset.

[0067] In some embodiments, similarity is represented by Euclidean distance. For step S400, one possible implementation provided by the embodiments of the present invention includes sub-step S410: S410. Based on the Euclidean distance between the low-dimensional representation vectors of any two samples in the full dataset, cluster the low-dimensional representation vectors to obtain multiple clusters; among the multiple clusters, the Euclidean distance between the low-dimensional representation vectors of any two samples in the same cluster is less than the Euclidean distance between the low-dimensional representation vectors of samples in different clusters.

[0068] In this embodiment, the k-means++ algorithm is used to cluster the low-dimensional representation vectors. The number of clusters K is set to 200, and the maximum number of iterations is set to 300. During the clustering process, the algorithm uses the Euclidean distance between the low-dimensional representation vectors of each sample in 8-dimensional space as the criterion for determining sample affiliation. After multiple rounds of iteration, the cluster centers and the cluster membership relationships of each sample are updated, ultimately ensuring that the multiple clusters after clustering satisfy the following condition: the Euclidean distance between the low-dimensional representation vectors of any two samples belonging to the same cluster is less than the Euclidean distance between these two samples and any sample in any other cluster. In other words, intra-cluster distance is minimized, and inter-cluster distance is maximized.

[0069] In some embodiments, for step S500, one possible implementation of the present invention is that step S500 includes sub-steps S510~S520: S510. Substitute the distribution of samples of each cluster under the preset sampling objective into the sampling sample set in the multi-objective loss function to obtain the initial loss function value of each cluster.

[0070] In this embodiment, all samples contained in each cluster are treated as an independent small sample set. Using this small sample set as the statistical scope, its distribution under four preset sampling targets is statistically analyzed, forming the intra-cluster distribution of the cluster under each preset sampling target. Taking the distribution of credit quality samples as an example, the number of bad samples and good samples within the cluster is counted, and each is divided by the total number of samples in the cluster to obtain the intra-cluster distribution of the cluster in the credit quality dimension. Similarly, taking the enterprise size distribution as an example, the number of micro, small, medium, and large enterprises within the cluster is counted, and each is divided by the total number of samples in the cluster to obtain the intra-cluster distribution of the cluster in the enterprise size dimension. The industry category distribution and the distribution by prefecture-level city administrative division are handled similarly.

[0071] Furthermore, the intra-cluster distribution of the cluster under each preset sampling objective is substituted into the sampling sample set position in the multi-objective loss function as the independent variable. Using the overall distribution of the full dataset under each preset sampling objective as a fixed reference, the divergence value between the intra-cluster distribution and the overall distribution under each preset sampling objective is calculated one by one. Taking credit quality samples as an example, the intra-cluster distribution of the cluster in the credit quality dimension (i.e., the proportion of bad samples and good samples within the cluster) and the overall distribution of the full dataset under the credit quality sample distribution are substituted into the KL divergence formula. The difference in the logarithmic ratios of the bad sample category and the good sample category is calculated for each category and then summed to obtain the divergence value corresponding to the preset sampling objective of the credit quality sample distribution. The divergence values ​​corresponding to the three preset sampling objectives—enterprise size distribution, industry category distribution, and municipal administrative division distribution—are calculated in the same way. Then, the divergence values ​​corresponding to each of the four preset sampling objectives are multiplied by their respective preset weight coefficients to obtain their respective difference values. Finally, the difference values ​​are summed to obtain the initial loss function value of the cluster. The magnitude of the initial loss function value reflects the degree to which the cluster deviates from the overall distribution of the total population as an independent group across the four preset sampling target dimensions. The larger the value, the more significant the difference between the overall distribution characteristics of the cluster and the total population.

[0072] S520. Determine the initial sampling quota for each cluster based on the ratio of the initial loss function value of each cluster to the sum of the initial loss function values ​​of all clusters.

[0073] In this embodiment, the initial loss function values ​​of all clusters are summed to obtain a total. Then, for each cluster, the proportion of its initial loss function value to the total is calculated. The larger the proportion, the more urgent the need for representativeness correction in the entire population, and the more sampling quotas should be allocated to reduce its deviation. The initial sampling quota for each cluster is calculated by multiplying the proportion of each cluster by the total sample size of 150,000 and rounding down.

[0074] For example, taking one allocation result as an example, the cluster with the largest deviation from the overall distribution received the highest proportion of initial loss function values, obtaining the highest sampling quota of 1243; while the cluster with the smallest deviation received the lowest sampling quota of 87. The sum of the initial sampling quotas for all two hundred clusters equals 150,000. After the above quota allocation is completed, the weight of each cluster in the actual sampling has been initially matched with its representative necessity in the overall population, so as to give more sampling quotas to clusters with larger deviations from the overall distribution in the initial stage.

[0075] In some embodiments, for step S510, one possible implementation of the present invention is that step S510 includes sub-steps S511~S513: S511. Based on the distribution of samples from clusters under a preset sampling target and the overall distribution, calculate the divergence value corresponding to the preset sampling target.

[0076] S512. Multiply the divergence value corresponding to the preset sampling target by the preset weight coefficient corresponding to the preset sampling target to obtain the difference value corresponding to the preset sampling target.

[0077] S513. Add up the difference values ​​corresponding to each preset sampling target to obtain the initial loss function value of the cluster.

[0078] In S511~S513, taking a cluster as an example, the distribution of the cluster's samples under four preset sampling targets is used as the comparison object, and the overall distribution of the entire dataset under the four preset sampling targets is used as the comparison benchmark. The divergence value between the two is calculated one by one under each preset sampling target. The divergence value is calculated using the KL divergence formula, that is, based on the probability distribution value of each category in the overall distribution, the difference between the log ratio of the overall distribution and the cluster distribution in that category is calculated for each category and then accumulated. Each preset sampling target corresponds to one divergence value.

[0079] Furthermore, the divergence value corresponding to the preset sampling target is multiplied by the preset weight coefficient corresponding to the preset sampling target to obtain the difference value corresponding to the preset sampling target. The difference value reflects the degree of deviation and importance of the preset sampling target. Then, the difference values ​​corresponding to the four preset sampling targets are directly added together, and the single value obtained is the initial loss function value of the cluster. The initial loss function value is a dimensionless composite number, and its magnitude reflects the weighted composite degree of deviation of the cluster from the overall distribution in each preset sampling target dimension. Subsequently, the initial sampling quota of the cluster is determined based on the proportion of this value in all clusters.

[0080] In some embodiments, for step S700, one possible implementation of the present invention is that step S700 includes sub-step S710: S710. When the target loss function value is greater than the preset function threshold, adjust the initial sampling quota of each cluster and update the candidate samples. Substitute the distribution of the updated candidate samples under the preset sampling target back into the sampling sample set in the multi-objective loss function until the calculated target loss function value is less than or equal to the preset function threshold. Then, determine the current sampling quota and candidate samples of each cluster as the target sampling quota and target samples, respectively.

[0081] In this embodiment, the distribution of candidate samples under four preset sampling objectives is used as the independent variable and substituted into the multi-objective loss function for calculation. The resulting target loss function value is used as the fitness of the candidate sample; the lower the fitness, the better the candidate sample. After each round of adjustment, based on the newly generated sampling quota, the specific enterprises selected as candidate samples in each cluster are updated, and the updated distribution of candidate samples under the four preset sampling objectives is re-substituted into the multi-objective loss function to calculate a new target loss function value. As the number of iterations accumulates, the target loss function value of the optimal candidate sample decreases generation by generation.

[0082] For example, in the 327th iteration, the target loss function value drops to 0.0071, which is less than the preset function threshold of 0.008. The iteration stops, and the sampling quota and candidate sample list corresponding to this optimal target loss function value are determined as the target sampling quota and target samples, respectively. Throughout the iteration process, the sampling quotas of 127 clusters were adjusted, and 21,436 candidate samples were replaced, with the final total number of candidate samples remaining constant at 150,000.

[0083] In some embodiments, while updating the candidate samples each time according to the adjusted sampling quota, it is also necessary to check whether the number of samples belonging to each category under each preset sampling target in the current candidate samples meets the scarce category determination condition, and to perform a neighbor cluster supplementation operation on the scarce categories that meet the condition. Therefore, for step S710, one possible implementation of the present invention is that step S710 includes sub-steps S711~S713: S711. In multiple clusters, if the number of samples belonging to a sample category under a preset sampling target in any cluster is less than a preset threshold as a proportion of the total number of samples belonging to the sample category in multiple clusters, the sample category is determined as the scarce category of the current cluster.

[0084] In this embodiment, after updating the candidate samples in each iteration, two hundred clusters are traversed. For each cluster, the number of samples belonging to each sample category under each preset sampling target in the current candidate samples is checked one by one. If the proportion of the number of samples belonging to a certain sample category under a certain preset sampling target in a certain cluster is lower than the proportion of the total number of samples belonging to the same sample category in all two hundred clusters, it indicates that the cluster has a serious under-coverage of that sample category. This is likely to cause the number of samples of that sample category in the final balanced sample set to be seriously underestimated. Therefore, the sample category is determined as the scarce category corresponding to the cluster, so that the number of samples in the scarce category can be supplemented in subsequent steps.

[0085] Among them, the sample categories under the preset sampling objectives are, for example, good samples and bad samples in terms of credit quality; micro-enterprises, small enterprises, medium-sized enterprises, and large enterprises in terms of enterprise size; mining, manufacturing, and construction in terms of industry categories; and cities A, B, and C in terms of city regions.

[0086] S712. Select the cluster with the smallest Euclidean distance to the current cluster as the neighboring cluster, and extract samples belonging to the scarce category from the neighboring cluster to supplement the sampling quota of the current cluster.

[0087] In this embodiment, after determining the scarce category of the current cluster, to find supplementary sources from other clusters that are most similar to it in terms of features, the Euclidean distance between the cluster center of the current cluster and the cluster centers of each of the other clusters is calculated in the 8-dimensional low-dimensional representation space. The cluster center is the mean vector of the low-dimensional representation vectors of all samples within the cluster. The clusters are sorted according to their Euclidean distance from the current cluster in ascending order, and a preset number of clusters with the smallest distance, such as two clusters, are selected as the neighboring clusters of the current cluster. Since a smaller distance in the 8-dimensional low-dimensional representation space means a higher comprehensive similarity between the two clusters in terms of their original multidimensional features, extracting samples from these neighboring clusters to supplement the current cluster ensures that the supplemented samples are similar to the current cluster in terms of feature distribution. After determining the neighboring clusters, prioritize extracting real samples that belong to the scarce category from the neighboring clusters. If the number of samples belonging to the scarce category in the neighboring clusters is insufficient, then extract samples from the neighboring clusters that are closest to the characteristics of the scarce category to supplement them, until the proportion of the sample category in the candidate samples of the current cluster increases to no less than 0.5% of the preset threshold, and then stop supplementing the scarce category samples.

[0088] S713. Use the supplemented current sampling quota as the adjusted initial sampling quota, update the current candidate sample based on the adjusted initial sampling quota, and obtain the updated candidate sample.

[0089] In this embodiment, after extracting samples belonging to the scarce category from neighboring clusters and adding them to the sampling quota of the current cluster, the supplemented current sampling quota is used as the adjusted initial sampling quota. Since the supplemented samples are also real samples in the full dataset and have complete multidimensional feature data, the candidate sample list of the current cluster is updated based on the adjusted initial sampling quota. The updated candidate sample list includes the original candidate samples selected according to the quota and the samples supplemented from neighboring clusters. Based on this, the updated candidate samples are merged with the candidate samples of other clusters to form a complete candidate sample set. The multi-objective loss function is then re-applied to calculate the updated objective loss function value to determine whether the convergence condition is met.

[0090] It is important to note that this process is executed once after each round of candidate sample updates, thus ensuring that even as the quotas and candidate samples of each cluster are constantly adjusted throughout the entire iterative optimization process, the sample coverage of scarce categories is continuously monitored and replenished in a timely manner.

[0091] Based on the above method, embodiments of the present invention also provide a system corresponding to the above method, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of the functional modules of the multi-objective balanced sampling system provided in this embodiment of the invention. It should be noted that the basic principle and technical effects of the multi-objective balanced sampling system 1000 provided in this embodiment are the same as those in the above method embodiments. For the sake of brevity, parts not mentioned in this embodiment can be referred to the corresponding content in the method embodiments.

[0092] In this embodiment, the multi-objective balanced sampling system 1000 includes an acquisition module 1100, a dimensionality reduction module 1200, a stratified sampling module 1300, and a processing module 1400.

[0093] The acquisition module 1100 is used to acquire the full dataset and the total number of samples. The full dataset includes multiple samples, each with multi-dimensional feature data. It can be understood that the acquisition module 1100 is used to perform the above step S100.

[0094] Dimensionality reduction module 1200 is used to perform dimensionality reduction processing on the entire dataset to obtain the low-dimensional representation vector corresponding to each sample. It can be understood that dimensionality reduction module 1200 is used to perform the above step S200.

[0095] The stratified sampling module 1300 is used to construct a multi-objective loss function based on the overall distribution of the full dataset under a preset sampling objective. The multi-objective loss function measures the difference between the distribution of any sample set under the preset sampling objective and the distribution of the full dataset under the preset sampling objective. Based on the similarity between low-dimensional representation vectors, multiple samples in the full dataset are clustered to obtain multiple clusters. The initial sampling quota for each cluster is determined according to the distribution of samples in each cluster under the preset sampling objective and the multi-objective loss function. The sum of the initial sampling quotas for each cluster equals the total number of samples. It can be understood that the stratified sampling module 1300 is used to execute the above steps S300~S500.

[0096] Processing module 1400 is used to select candidate samples from each cluster based on an initial sampling quota, and substitute the candidate samples into the sampling sample set in the multi-objective loss function to obtain the corresponding target loss function value; adjust the initial sampling quota and candidate samples according to the comparison between the target loss function value and the preset function threshold to determine the target sampling quota and target sample for each cluster; and extract samples from the full dataset based on the target sampling quota and target sample for each cluster to obtain a balanced sample set. It can be understood that processing module 1400 is used to execute the above steps S600~S800.

[0097] In some embodiments, the stratified sampling module 1300 is used to take the distribution of any sample set under a preset sampling objective as the independent variable, the overall distribution of the entire dataset under the preset sampling objective as the reference benchmark, and the preset weight coefficients corresponding to the preset sampling objective as weighting coefficients; based on the divergence value between the independent variable and the reference benchmark, and the corresponding weighting coefficients, a multi-objective loss function is constructed. It can be understood that the stratification module is used to perform the above steps S310~S320.

[0098] In some embodiments, similarity is represented by Euclidean distance. The stratified sampling module 1300 is used to cluster the low-dimensional representation vectors based on the Euclidean distance between the low-dimensional representation vectors of any two samples in the full dataset, resulting in multiple clusters. Among the multiple clusters, the Euclidean distance between the low-dimensional representation vectors of any two samples within the same cluster is less than the Euclidean distance between the low-dimensional representation vectors of samples from different clusters. It can be understood that the stratified sampling module 1300 is used to perform the above step S410.

[0099] In some embodiments, the stratified sampling module 1300 is used to substitute the distribution of samples from each cluster under a preset sampling objective into the sampling sample set in the multi-objective loss function to obtain the initial loss function value of each cluster; and to determine the initial sampling quota of each cluster based on the ratio of the initial loss function value of each cluster to the sum of the initial loss function values ​​of all clusters. It can be understood that the stratified sampling module 1300 is used to perform the above steps S510~S520.

[0100] In some embodiments, the stratified sampling module 1300 is used to calculate the divergence value corresponding to the preset sampling target based on the distribution of samples of the cluster under the preset sampling target and the overall distribution; multiply the divergence value corresponding to the preset sampling target by the preset weight coefficient corresponding to the preset sampling target to obtain the difference value corresponding to the preset sampling target; and sum the difference values ​​corresponding to each preset sampling target to obtain the initial loss function value of the cluster. It can be understood that the stratified sampling module 1300 is used to perform the above steps S511~S513.

[0101] In some embodiments, the processing module 1400 is configured to adjust the initial sampling quota of each cluster and update the candidate samples when the target loss function value is greater than a preset function threshold. The updated candidate sample distribution under the preset sampling target is then re-substituted into the sampling sample set in the multi-objective loss function until the calculated target loss function value is less than or equal to the preset function threshold. The current sampling quota and candidate samples of each cluster are then determined as the target sampling quota and target samples, respectively. It can be understood that the processing module 1400 is configured to perform the above step S710.

[0102] In some embodiments, the processing module 1400 is configured to determine the sample category as a scarce category of the current cluster if, in any cluster, the proportion of samples belonging to a sample category under a preset sampling target is less than a preset threshold. The module then identifies the cluster with the smallest Euclidean distance to the current cluster as a neighboring cluster, extracts samples belonging to the scarce category from the neighboring clusters to supplement the sampling quota of the current cluster, uses the supplemented current sampling quota as an adjusted initial sampling quota, and updates the current candidate samples based on the adjusted initial sampling quota to obtain updated candidate samples. It is understood that the processing module 1400 is configured to perform the above steps S711~S713.

[0103] Based on the same inventive concept disclosed above, the present invention also provides a block diagram of an electronic device 2000 performing the above method. Please refer to... Figure 4 , Figure 4 This is a block diagram of an electronic device provided in an embodiment of the present invention. The electronic device 2000 includes a processor 2100, a memory 2200, a bus 2300, and a communication interface 2400. The processor 2100 and the memory 2200 are connected via the bus 2300, and the processor 2100 communicates with external devices via the communication interface 2400.

[0104] Processor 2100 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed through integrated logic circuits in the hardware of processor 2100 or through software instructions. The processor 2100 may be a general-purpose processor 2100, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0105] The memory 2200 is used to store computer programs. For example, the multi-objective balanced sampling system 1000 in this embodiment of the invention includes at least one software function module that can be stored in the memory 2200 in the form of software or firmware. After receiving the execution instruction, the processor 2100 executes the program to implement the multi-objective balanced sampling method in this embodiment of the invention.

[0106] The memory 2200 may include high-speed random access memory (RAM) or non-volatile memory. Optionally, the memory 2200 may be a storage device built into the processor 2100 or a storage device independent of the processor 2100.

[0107] Bus 2300 can be ISA bus 2300, PCI bus 2300 or EISA bus 2300, etc. Figure 4 It is indicated by only one double-headed arrow, but does not mean that there is only one bus 2300 or one type of bus 2300.

[0108] Electronic devices 2000 can be mobile phones, tablets, laptops, desktop computers, and other computer devices.

[0109] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program thereon, which, when executed by processor 2100, implements the multi-objective balanced sampling method described above. This computer-readable storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-objective balanced sampling method, characterized in that, The method includes: Obtain the full dataset and the total number of samples, wherein the full dataset includes multiple samples, and each sample has feature data in multiple dimensions; The full dataset is subjected to dimensionality reduction processing to obtain the low-dimensional representation vector corresponding to each sample. Based on the overall distribution of the full dataset under the preset sampling objective, a multi-objective loss function is constructed; the multi-objective loss function is used to measure the difference between the distribution of any sample set under the preset sampling objective and the distribution of the full dataset under the preset sampling objective; Based on the similarity between the low-dimensional representation vectors, multiple samples in the full dataset are clustered to obtain multiple clusters. Based on the distribution of samples in each cluster under the preset sampling objective and the multi-objective loss function, the initial sampling quota for each cluster is determined; the sum of the initial sampling quotas for each cluster is equal to the total number of samples. Based on the initial sampling quota, candidate samples are selected from each of the clusters, and the candidate samples are substituted into the sampling sample set in the multi-objective loss function to obtain the corresponding target loss function value. Based on the comparison between the target loss function value and the preset function threshold, the initial sampling quota and the candidate samples are adjusted to determine the target sampling quota and target samples for each cluster. Based on the target sampling quota and target sample of each cluster, samples are extracted from the full dataset to obtain a balanced sample set.

2. The method according to claim 1, characterized in that, The construction of a multi-objective loss function based on the overall distribution of the full dataset under a preset sampling objective includes: The distribution of any sample set under the preset sampling target is used as the independent variable, and the overall distribution of the full dataset under the preset sampling target is used as the reference benchmark. The preset weight coefficient corresponding to the preset sampling target is used as the weighting coefficient. The multi-objective loss function is constructed based on the divergence value between the independent variable and the reference benchmark, and the corresponding weighting coefficients.

3. The method according to claim 1, characterized in that, The similarity is represented by Euclidean distance. Based on the similarity between the low-dimensional representation vectors, multiple samples in the full dataset are clustered to obtain multiple clusters, including: Based on the Euclidean distance between the low-dimensional representation vectors of any two samples in the full dataset, the low-dimensional representation vectors are clustered to obtain the plurality of clusters; among the plurality of clusters, the Euclidean distance between the low-dimensional representation vectors of any two samples in the same cluster is less than the Euclidean distance between the low-dimensional representation vectors of samples in different clusters.

4. The method according to claim 1, characterized in that, The step of determining the initial sampling quota for each cluster based on the distribution of samples from each cluster under the preset sampling target and the multi-objective loss function includes: Substitute the distribution of samples from each cluster under the preset sampling objective into the sampling sample set in the multi-objective loss function to obtain the initial loss function value for each cluster. The initial sampling quota for each cluster is determined based on the ratio of the initial loss function value of each cluster to the sum of the initial loss function values ​​of all clusters.

5. The method according to claim 4, characterized in that, The step of substituting the distribution of samples from each cluster under the preset sampling objective into the sampling sample set in the multi-objective loss function to obtain the initial loss function value for each cluster includes: Based on the distribution of samples from the clusters under the preset sampling target and the overall distribution, calculate the divergence value corresponding to the preset sampling target; Multiply the divergence value corresponding to the preset sampling target by the preset weight coefficient corresponding to the preset sampling target to obtain the difference value corresponding to the preset sampling target. The initial loss function value of the cluster is obtained by summing the difference values ​​corresponding to each preset sampling target.

6. The method according to claim 1, characterized in that, The step of adjusting the initial sampling quota and the candidate samples based on the comparison between the target loss function value and the preset function threshold, and determining the target sampling quota and target samples for each cluster, includes: If the target loss function value is greater than the preset function threshold, the initial sampling quota of each cluster is adjusted and the candidate samples are updated. The distribution of the updated candidate samples under the preset sampling target is then substituted back into the sampling sample set in the multi-objective loss function until the calculated target loss function value is less than or equal to the preset function threshold. The current sampling quota and candidate samples of each cluster are then determined as the target sampling quota and target samples, respectively.

7. The method according to claim 6, characterized in that, The adjustment of the initial sampling quota for each of the clusters and the updating of the candidate samples include: If, in any of the plurality of clusters, the number of samples belonging to a sample category under the preset sampling target in any cluster is less than a preset threshold relative to the total number of samples belonging to that sample category in the plurality of clusters, then that sample category is determined as the scarce category of the current cluster. The clusters with the smallest Euclidean distance to the current cluster are designated as neighboring clusters, and samples belonging to the scarce category are extracted from the neighboring clusters to supplement the sampling quota of the current cluster. The supplemented current sampling quota is used as the adjusted initial sampling quota, and the current candidate samples are updated based on the adjusted initial sampling quota to obtain the updated candidate samples.

8. A multi-objective balanced sampling system, characterized in that, The system includes: The acquisition module is used to acquire the full dataset and the total number of samples. The full dataset includes multiple samples, and each sample has feature data in multiple dimensions. The dimensionality reduction module is used to perform dimensionality reduction processing on the full dataset to obtain the low-dimensional representation vector corresponding to each sample. A stratified sampling module is used to construct a multi-objective loss function based on the overall distribution of the full dataset under a preset sampling objective. The multi-objective loss function measures the difference between the distribution of any sample set under the preset sampling objective and the distribution of the full dataset under the preset sampling objective. Based on the similarity between the low-dimensional representation vectors, multiple samples in the full dataset are clustered to obtain multiple clusters. According to the distribution of samples in each cluster under the preset sampling objective and the multi-objective loss function, the initial sampling quota for each cluster is determined. The sum of the initial sampling quotas for each cluster equals the total number of samples. The processing module is configured to select candidate samples from each of the clusters based on the initial sampling quota, and substitute the candidate samples into the sampling sample set in the multi-objective loss function to obtain the corresponding target loss function value; adjust the initial sampling quota and the candidate samples according to the comparison between the target loss function value and the preset function threshold to determine the target sampling quota and target sample for each of the clusters; and extract samples from the full dataset based on the target sampling quota and target sample for each of the clusters to obtain a balanced sample set.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program that can be executed by the processor to implement the multi-objective balanced sampling method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-objective balanced sampling method as described in any one of claims 1-7.

Images