Diversified sampling algorithm with flexibly adjustable spatial range
By dividing data clusters in random sampling and filtering candidate data, the problem of insufficient performance of random sampling in terms of coverage and diversity is solved, and more efficient data sampling and model performance improvement is achieved.
Patent Information
- Application Number
- CN202411633552.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-14
- Publication Date
- 2025-04-15
AI Technical Summary
The random sampling method lacks coverage and diversity performance in data sampling, which affects the prediction accuracy and generalization ability of the model.
By obtaining each candidate data and dividing it into the data cluster based on the feature distance, the candidate data is filtered to match the feature distribution information in the data cluster, and finally hierarchical sampling is performed to improve the diversity of the sampling results.
It improves the coverage and diversity of sampling results, and enhances the prediction accuracy and generalization ability of the model.
Smart Images
Figure CN119833003A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a diversity sampling algorithm with flexibly adjustable spatial range. Background Art
[0002] The random sampling method can randomly extract samples from a large data set to achieve data sampling. It has been widely used in data processing and model training, data analysis and mining, etc.
[0003] However, due to the strong randomness of the random sampling method, it performs poorly in terms of coverage and diversity of the sampling results, which directly restricts the application effect of the sampling results in related fields. For example, when the sampling results are applied to model modeling, it will affect the model's prediction accuracy and generalization ability. Summary of the invention
[0004] The present invention provides a diversity sampling algorithm with flexibly adjustable spatial range, so as to solve the defects of random sampling in related technologies in terms of insufficient coverage and diversity.
[0005] The present invention provides a diversity sampling method with flexibly adjustable spatial range, comprising: Obtaining candidate data; Based on the characteristic distance between each candidate data and the center of each data cluster, the candidate data are divided into each data cluster, where each data cluster is obtained by performing characteristic clustering on each reference data; Based on the characteristic distribution information of the reference data in each data cluster, the candidate data in each data cluster is screened, and after the screening is completed, the candidate data in each data cluster is sampled.
[0006] According to a diversity sampling method with flexibly adjustable spatial range provided by the present invention, the acquisition of characteristic distribution information of reference data in the data cluster includes: Based on the characteristic distance between the center of the data cluster and each reference data in the data cluster, the maximum characteristic distance of the data cluster is determined, and the maximum characteristic distance of the data cluster is used as the characteristic distribution information of the reference data in the data cluster.
[0007] According to a diversity sampling method with flexible adjustable spatial range provided by the present invention, the candidate data in each data cluster is screened based on the characteristic distribution information of the reference data in each data cluster, including: Based on the maximum characteristic distance of each data cluster and the characteristic distance between the center of each data cluster and the candidate data in each data cluster, the candidate data in each data cluster are screened.
[0008] According to a diversity sampling method with flexibly adjustable spatial range provided by the present invention, the sampling of candidate data in each data cluster includes: For each of the data clusters, based on the characteristic distance between the candidate data in the data cluster and the center of the data cluster, the data cluster is layered to obtain each data layer of the data cluster; The candidate data in each data layer in the data cluster is sampled.
[0009] According to a diversity sampling method with flexibly adjustable spatial range provided by the present invention, the data cluster is layered based on the characteristic distance between the candidate data in the data cluster and the center of the data cluster to obtain each data layer of the data cluster, including: Acquire a plurality of preset numbers of layers, and divide the data cluster into the preset number of data layers based on the characteristic distance between the candidate data in the data cluster and the center of each data cluster; Based on the clustering effect of the candidate data of the preset number of data layers, determining a target number of layers from the plurality of preset numbers of layers; Based on the characteristic distance between the candidate data in the data cluster and the center of the data cluster, the data cluster is divided into the target number of data layers.
[0010] According to a diversity sampling method with flexibly adjustable spatial range provided by the present invention, the sampling of candidate data in each data layer in the data cluster includes: Randomly select one of the selected candidate data from the data layer as sampled data, and delete the sampled data from the data layer; Calculate the minimum value of the characteristic distance between each candidate data after screening in the data layer and all the sampled data, take the candidate data with the largest minimum value as the sampled data, delete the sampled data from the data layer, and return to calculate the minimum value of the characteristic distance between each remaining candidate data in the data layer and all the sampled data, until the selection is completed.
[0011] According to a diversity sampling method with flexibly adjustable spatial range provided by the present invention, the determination of each data cluster includes: Acquire a plurality of preset cluster numbers, perform feature clustering on each reference data, and obtain data clusters of the preset number of clusters; Based on the clustering effect of the preset number of clusters, determining a target number of clusters from the plurality of preset numbers of clusters; Perform feature clustering on the reference data to obtain the target cluster data clusters.
[0012] According to a diversity sampling method with flexibly adjustable spatial range provided by the present invention, the determination of the characteristic distance between each candidate data and the center of each data cluster includes: Based on the data features of each candidate data and the central data features of each data cluster, the characteristic distance between each candidate data and the center of each data cluster is determined. The central data features of each data cluster are determined based on the data features of the reference data in each data cluster. The data features of the candidate data and the data features of the reference data are in the same characteristic space.
[0013] The present invention also provides a diversity sampling device with a flexibly adjustable spatial range, comprising: An acquisition unit, used for acquiring each candidate data; A division unit, configured to divide each candidate data into each data cluster based on a characteristic distance between each candidate data and a center of each data cluster, wherein each data cluster is obtained by performing characteristic clustering on each reference data; The screening sampling unit is used to screen the candidate data in each data cluster based on the characteristic distribution information of the reference data in each data cluster, and after the screening is completed, sample the candidate data in each data cluster.
[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a diversity sampling method with flexibly adjustable spatial range as described in any one of the above.
[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a diversity sampling method with flexibly adjustable spatial range as described in any one of the above.
[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned diversity sampling methods with flexibly adjustable spatial range.
[0017] The diversity sampling algorithm with flexibly adjustable spatial range provided by the present invention divides each candidate data into each data cluster obtained by clustering the reference data, and screens the candidate data in each data cluster based on the characteristic distribution information of the reference data in each data cluster, and samples the candidate data in each data cluster after the screening is completed, thereby enabling data sampling to refer to the distribution and diversity of the data, thereby ensuring the coverage and diversity of the sampling results. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 It is a schematic flow chart of a diversity sampling method with flexibly adjustable spatial range provided by the present invention.
[0020] Figure 2 It is a distribution diagram of the data features provided by the present invention.
[0021] Figure 3 It is a structural schematic diagram of the data sampling device provided by the present invention.
[0022] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0024] Data sampling refers to selecting a sample set from the overall data (which can be a specific group, a set of data, a batch of products, etc.) to represent the entire population in order to infer and analyze the population. Its purpose is to reduce the amount of data, save time and cost, and improve the feasibility of data and analysis efficiency.
[0025] A common data sampling method is random sampling, which can randomly extract samples from the overall data. For example, in the field of data processing and model training, random sampling can achieve efficient sample extraction, thereby reducing the amount of calculation and accelerating model training. Another example is in the field of data analysis and mining, random sampling can assist in discovering the potential patterns of the overall data.
[0026] However, since the random sampling method has a strong randomness, some data in the overall data may not be fully sampled, and the sampling results may be concentrated on certain specific data points, resulting in poor coverage and diversity of the sampling results. The existence of these problems will directly limit the application effect of the sampling results.
[0027] For example, when modeling a yield prediction model in the field of chemistry, data sampling can be performed using simple random sampling. That is, samples can be drawn from a data set of organic chemical reactions (including information such as reactants, reaction conditions, and products) in a completely random manner, where each data point has an equal probability of being drawn. In this scenario, the sampling results may miss important information, that is, if there are small but very important organic chemical reaction patterns or special cases in the data set, simple random sampling may miss this information. For example, some rare reaction mechanisms or reactions that occur under specific extreme conditions may not be selected at all during the random sampling of small samples because they account for a small proportion of the data set. In addition, the representativeness of the sampling results is also very limited. For organic chemical reaction data sets with complex distributions (such as multimodal distributions), the samples obtained by simple random sampling may not represent the distribution characteristics of the entire data set well. For example, the yield data of organic reactions in the data set show a bimodal distribution, which may be due to the presence of two different reaction pathways. Random sampling may cause the reaction corresponding to one of the peaks to be insufficiently reflected in the sample.
[0028] Since the sampling results obtained by random sampling have insufficient coverage and diversity, the yield prediction model obtained based on this training has deficiencies in both prediction accuracy and generalization ability.
[0029] In view of the above problems, the present invention provides a diversity sampling method with flexibly adjustable spatial range. Figure 1 is a flow chart of a diversity sampling method with flexibly adjustable spatial range provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, obtaining each candidate data.
[0030] Here, candidate data refers to data to be sampled, and each candidate data may constitute the overall data to be sampled. The type of candidate data may be associated with the application scenario of data sampling. For example, when the application scenario of data sampling is modeling of a yield prediction model in the field of chemistry, each candidate data may be a product of a chemical reaction for which yield prediction is required; for another example, when the application scenario of data sampling is training of an image classification model, each candidate data may be a variety of images, which is not specifically limited in the embodiments of the present invention.
[0031] Step 120 , based on the feature distance between each candidate data and the center of each data cluster, the candidate data are divided into the data clusters, where each data cluster is obtained by performing feature clustering on each reference data.
[0032] Specifically, the reference data is data used as a reference for data sampling. The reference data and the candidate data are of the same type of data. For example, the reference data and the candidate data may both be products of a chemical reaction. For another example, the reference data and the candidate data may both be images. Each reference data may constitute a reference data set. The distribution and diversity of the reference data in the reference data set may serve as a reference for data sampling for each candidate data. It is understandable that the expectation for data sampling in the embodiments of the present invention is that the distribution and diversity of the sampled data may be as close as possible to the distribution and diversity of the reference data, thereby ensuring the rationality and reliability of data sampling.
[0033] The selection of reference data can also be associated with the application scenario of data sampling.
[0034] For example, when the application scenario of data sampling is the modeling of a yield prediction model in the chemical field, products in a database that can reflect the actual application of chemical reactions can be selected as reference data. For example, products in a patent database can be used as reference data, thereby ensuring that the results obtained from data sampling based on this reference can be equally close to the actual application of chemical reactions.
[0035] For example, when the application scenario of data sampling is the training of an image recognition model for cats, images in an image library that can reflect a certain breed of cats can be selected as reference data. For example, images of a tabby cat in various forms can be used as reference data, thereby ensuring that the results obtained from data sampling based on this reference are close to the actual situation.
[0036] After obtaining each reference data, feature clustering can be performed for each reference data, thereby obtaining multiple data clusters. Here, feature clustering can be performed for each reference data, specifically, feature extraction can be performed for each reference data, thereby obtaining data features of each reference data, and then clustering can be performed on each reference data based on the data features of each reference data. By clustering, reference data with similar data features are divided into the same data cluster, and reference data with different data features are divided into different data clusters. It can be understood that the multiple data clusters obtained by clustering can reflect the diversity of the reference data, that is, each data cluster can be understood as a category of the reference data.
[0037] After obtaining each candidate data through step 110, the characteristic distance between each candidate data and the center of each data cluster can be calculated. Here, for any candidate data, it is necessary to calculate the distance between the data feature of the candidate data and the data feature of the center of each data cluster, that is, the characteristic distance. It can be understood that, for any candidate data and any data cluster, the characteristic distance between the two is used to reflect the difference between the two at the data feature level. The larger the characteristic distance between the two, the greater the difference between the two at the data feature level, and the lower the probability that the candidate data belongs to the data cluster; the smaller the characteristic distance between the two, the smaller the difference between the two at the data feature level, and the higher the probability that the candidate data belongs to the data cluster.
[0038] Therefore, for each candidate data, after obtaining the characteristic distance between the candidate data and the center of each data cluster, the candidate data can be divided into data clusters. Specifically, the candidate data can be divided into the data cluster with the smallest characteristic distance; or the minimum characteristic distance value for the candidate data to be divided into the data cluster can be pre-set as the preset distance threshold, and the candidate data can be divided into the data cluster with a characteristic distance less than the preset distance threshold. The embodiment of the present invention does not make specific limitations on this.
[0039] Step 130 , based on the characteristic distribution information of the reference data in each data cluster, the candidate data in each data cluster is screened, and after the screening is completed, the candidate data in each data cluster is sampled.
[0040] Specifically, after completing the data cluster division for each candidate data in step 120 , further screening can be performed on the candidate data divided into each data cluster.
[0041] For any data cluster, the feature distribution information of the reference data in the data cluster, that is, the distribution of the data features of each reference data in the data cluster, can be expressed as the distribution of the distance between the data features of each reference data in the data cluster and the data features of the center of the data cluster. In the data cluster, the feature distribution information of the reference data can be used as a basis to determine whether the feature distribution information of each candidate data in the data cluster is similar to the feature distribution information of the reference data, thereby retaining the candidate data in the data cluster whose feature distribution information is similar to the reference data, and filtering out the candidate data in the data cluster whose feature distribution information is different from the reference data, thereby achieving the screening of the candidate data in the data cluster, and after this screening, the distribution of the candidate data in the data cluster can be close to the distribution of the reference data.
[0042] After the screening is completed, the candidate data in each data cluster can be sampled separately, thereby completing the data sampling of the candidate data. It can be understood that, due to the screening, the distribution of the candidate data in each data cluster can be close to the distribution of the reference data, and the sampling based on this can ensure the rationality of the distribution of the sampling results. In addition, since the candidate data are divided into each data cluster in advance, and the setting of each data cluster itself can reflect the diversity of the reference data, on this basis, sampling is performed separately for each data cluster, so that the data diversity of the sampling results can be consistent with the diversity of the reference data, thereby ensuring the diversity of the sampling results.
[0043] In the method provided in an embodiment of the present invention, each candidate data is divided into each data cluster obtained by clustering the reference data, and the candidate data in each data cluster is screened based on the characteristic distribution information of the reference data in each data cluster, and after the screening is completed, the candidate data in each data cluster is sampled, thereby enabling data sampling to refer to the distribution and diversity of the data, thereby ensuring the coverage and diversity of the sampling results.
[0044] Based on the above embodiment, in step 120, the determination of each data cluster includes: Acquire a plurality of preset cluster numbers, perform feature clustering on each reference data, and obtain data clusters of the preset number of clusters; Based on the clustering effect of the preset number of clusters, determining a target number of clusters from the plurality of preset numbers of clusters; Perform feature clustering on the reference data to obtain the target cluster data clusters.
[0045] Specifically, in the process of clustering each reference data to obtain data clusters, commonly used clustering algorithms may be K-means clustering, K-median clustering, EM (Expectation Maximization) clustering, etc. Taking K-means clustering as an example, feature clustering for each reference data can be expressed as follows: In the formula, is the data feature of the i-th reference data, is the central data feature of the jth data cluster, is the characteristic distance between the a-th reference data and the center of the j-th data cluster, Indicates the data cluster to which the i-th reference data belongs.
[0046] It should be noted that, before the clustering algorithm is performed, the number of data clusters obtained by clustering needs to be set in advance.
[0047] In order to ensure the rationality of the number of data clusters obtained by clustering, multiple candidate preset cluster numbers can be set in advance. Here, each preset cluster number is the number of data clusters that can be obtained by clustering each reference data. For example, 5 preset cluster numbers can be set, namely 8, 9, 10, 11, and 12.
[0048] For each preset number of clusters, each reference data can be clustered according to the preset number of clusters, thereby obtaining the preset number of data clusters. For example, when there are 5 preset numbers of clusters, namely 8, 9, 10, 11, and 12, each reference data can be clustered into 8 data clusters, 9 data clusters, 10 data clusters, 11 data clusters, and 12 data clusters, that is, each preset number of clusters can correspond to a clustering result.
[0049] After obtaining the clustering results corresponding to each preset number of clusters, the clustering effect of each clustering result can be evaluated. Among them, for any preset number of clusters, the clustering result corresponding to the preset number of clusters is the preset number of data clusters obtained by clustering the characteristics of each reference data. The clustering effect of the clustering result can be expressed as the clustering effect of each of the preset number of data clusters. For example, the SSE (Sum of Squared Errors) of the clustering result can be used as a representation of the clustering effect. Here, SSE refers to the sum of the squares of the characteristic distances from each reference data in the clustering result to the center of the data cluster to which it belongs. The smaller the value of SSE, the better the clustering effect.
[0050] After obtaining the clustering effect corresponding to each preset number of clusters, the preset number of clusters with the best clustering effect can be selected from all the preset numbers of clusters as the target number of clusters. For example, a curve graph that can reflect the SSE values of each preset number of clusters can be drawn, and then the point where the SSE decreases slowly, that is, the preset number of clusters corresponding to the elbow point, can be selected from the curve graph as the target number of clusters.
[0051] After determining the target number of clusters, each reference data may be clustered into data clusters of the target number of clusters, thereby achieving clustering of the reference data.
[0052] Based on any of the above embodiments, in step 120, determining the characteristic distance between each candidate data and the center of each data cluster includes: Based on the data features of each candidate data and the central data features of each data cluster, the characteristic distance between each candidate data and the center of each data cluster is determined. The central data features of each data cluster are determined based on the data features of the reference data in each data cluster. The data features of the candidate data and the data features of the reference data are in the same characteristic space.
[0053] Specifically, for any candidate data, the data feature of the candidate data is the feature obtained by extracting the feature of the candidate data, and the type of the data feature is determined based on the type of the candidate data itself.
[0054] For example, when the application scenario of data sampling is the modeling of a yield prediction model in the field of chemistry, for the case where the candidate data is a product, the simplified molecular input line entry system (SMILES) string of the product can be converted into a molecular fingerprint, and the molecular fingerprint can be used as the data feature of the product, or the feature obtained by further feature extraction of the molecular fingerprint can be used as the data feature of the product, or the feature after dimensionality reduction of the molecular fingerprint can be used as the data feature of the product, and the embodiment of the present invention does not specifically limit this. Here, the molecular fingerprint refers to a series of numbers or strings converted from the structure and chemical properties of a molecule, such as a Morgan fingerprint, an extended connectivity fingerprint, etc.
[0055] Similarly, for any reference data, the data features of the reference data are the features obtained by extracting features from the reference data, and the type of the data features is based on the type of the reference data itself. Considering that the reference data and the candidate data are of the same type, the method of extracting features from the reference data is consistent with the method of extracting features from the candidate data. Therefore, the data features of the reference data and the data features of the candidate data belong to the same type of features, that is, the data features of the reference data and the data features of the candidate data can be mapped to the same feature space to facilitate feature comparison. For example, the data features of the reference data and the data features of the candidate data are both molecular fingerprints, or are both features after dimensionality reduction of the molecular fingerprints.
[0056] After the data feature extraction for each reference data is completed, the data features of each reference data can be clustered to obtain each data cluster. It can be understood that for any data cluster, the data cluster includes at least one data feature of the reference data, and the center of the data cluster can also be understood as the centroid of the data cluster, specifically indicating the average position of the data in the data cluster. The central data feature is the data feature at the center, which can be based on the data features of all the reference data in the data cluster, for example, it can be the mean of the data features of all the reference data in the data cluster.
[0057] Therefore, for any candidate data and any data cluster, the characteristic distance between the candidate data and the center of the data cluster can be understood as the distance between the data feature of the candidate data and the central data feature of the data cluster. The distance between the two can be calculated by Euclidean distance.
[0058] Accordingly, in step 120, the dividing the candidate data into the data clusters based on the characteristic distance between the candidate data and the center of each data cluster may specifically include: For any candidate data, the characteristic distances between the candidate data and the centers of all data clusters are calculated, and the center of the data cluster with the closest characteristic distance is determined based on the above characteristic distances, and then the candidate data is divided into the data cluster to achieve the nearest allocation of the candidate data, which can be specifically expressed as the following formula: In the formula, is the data feature of the i-th candidate data, is the central data feature of the jth data cluster, is the characteristic distance between the i-th candidate data and the center of the j-th data cluster, represents the data cluster to which the i-th candidate data belongs, Specifically, The data cluster with the smallest value.
[0059] Based on any of the above embodiments, in step 130, acquiring the characteristic distribution information of the reference data in the data cluster includes: Based on the characteristic distance between the center of the data cluster and each reference data in the data cluster, the maximum characteristic distance of the data cluster is determined, and the maximum characteristic distance of the data cluster is used as the characteristic distribution information of the reference data in the data cluster.
[0060] Specifically, for any data cluster, the characteristic distribution information of the reference data in the data cluster can be expressed as the maximum value of the characteristic distance between each reference data in the data cluster and the center of the data cluster, that is, the maximum value of the distance between the data feature of each reference data in the data cluster and the central data feature of the data cluster, which can be specifically expressed as the following formula: in, is the data feature of the i-th reference data, is the central data feature of the jth data cluster, Indicates that the i-th reference data belongs to the j-th data cluster , is the characteristic distance between the i-th reference data and the center of the j-th data cluster. represents the maximum characteristic distance of the jth data cluster, Specifically The maximum value of .
[0061] It can be understood that by using the maximum feature distance as the feature distribution information of the reference data in the data cluster, the distance between the data in the data cluster and the center of the data cluster can be used as a standard to measure whether the data conforms to the feature distribution information of the data cluster, thereby realizing the screening of candidate data in the data cluster.
[0062] Based on any of the above embodiments, in step 130, screening the candidate data in each data cluster based on the characteristic distribution information of the reference data in each data cluster includes: Based on the maximum characteristic distance of each data cluster and the characteristic distance between the center of each data cluster and the candidate data in each data cluster, the candidate data in each data cluster are screened.
[0063] Specifically, when screening the candidate data in each data cluster, for any data cluster, the maximum characteristic distance of the data cluster can be determined, that is, the maximum value of the characteristic distance between the center of the data cluster and each reference data in the data cluster. In addition, the characteristic distance between each candidate data in the data cluster and the center of the data cluster can also be determined.
[0064] After respectively obtaining the maximum feature distance of the data cluster and the feature distance between each candidate data in the data cluster and the center of the data cluster, the feature distance between each candidate data and the center of the data cluster can be compared with the maximum feature distance of the data cluster, thereby retaining the candidate data whose feature distance is less than or equal to the maximum feature distance and screening out the candidate data whose feature distance is greater than the maximum feature distance.
[0065] It can be understood that in this process, the maximum feature distance is the threshold for candidate data screening. For candidate data whose feature distance is less than or equal to the maximum feature distance, it can be understood that the feature distribution of the candidate data in the data cluster is consistent with the feature distribution of the reference data in the data cluster, so the candidate data is retained; for candidate data whose feature distance is greater than the maximum feature distance, it can be understood that the feature distribution of the candidate data in the data cluster exceeds the range of the feature distribution of the reference data in the data cluster, so the candidate data can be screened out.
[0066] The specific method of screening candidate data can be expressed as follows: in, is the characteristic distance between the i-th candidate data and the center of the j-th data cluster, represents the maximum characteristic distance of the jth data cluster, if , the i-th candidate data can be removed from the j-th data cluster.
[0067] It can be understood that the candidate data obtained by screening can meet the following conditions: In an embodiment of the present invention, the candidate data in each data cluster is screened in combination with the maximum characteristic distance of each data cluster and the characteristic distance between the center of each data cluster and the candidate data in each data cluster, so that the characteristic distribution of the candidate data in each data cluster after screening can be consistent with the characteristic distribution of the reference data in the data cluster, thereby ensuring the coverage of subsequent sampling.
[0068] Based on any of the above embodiments, in step 130, sampling the candidate data in each data cluster includes: For each of the data clusters, based on the characteristic distance between the candidate data in the data cluster and the center of the data cluster, the data cluster is layered to obtain each data layer of the data cluster; The candidate data in each data layer in the data cluster is sampled.
[0069] Specifically, after the candidate data in each data cluster is screened, the screened candidate data in each data cluster can be sampled. It can be understood that the sampling here is performed for each data cluster separately. For ease of explanation, the implementation of sampling candidate data is described below by taking any data cluster as an example.
[0070] For any data cluster, in order to sample candidate data within the data cluster, the data cluster can be stratified first, thereby dividing the data cluster into multiple data layers, and then sampling is performed on each data layer respectively, that is, stratified sampling is performed on the data cluster.
[0071] Here, the data cluster can be layered according to the feature distance between the candidate data in the data cluster and the center of the data cluster. That is, different feature distance ranges can be divided according to the feature distance between each candidate data in the data cluster and the center, so that the candidate data whose feature distance falls within the same feature distance range is taken as a layer in the data cluster, thereby obtaining candidate data belonging to different feature distance ranges, that is, obtaining different data layers in the data cluster.
[0072] After the data clusters are stratified, the candidate data in each data layer in the data cluster can be sampled. For example, a certain number of candidate data can be randomly selected from each data layer as the sampling result. This achieves stratified sampling for the data cluster, thereby further ensuring that the sampling results can be consistent with the distribution of the reference data, thereby ensuring the coverage and diversity of the sampling results.
[0073] Based on any of the above embodiments, in step 130, the data cluster is layered based on the characteristic distance between the candidate data in the data cluster and the center of the data cluster to obtain each data layer of the data cluster, including: Acquire a plurality of preset numbers of layers, and divide the data cluster into the preset number of data layers based on the characteristic distance between the candidate data in the data cluster and the center of each data cluster; Based on the clustering effect of the candidate data of the preset number of data layers, determining a target number of layers from the plurality of preset numbers of layers; Based on the characteristic distance between the candidate data in the data cluster and the center of the data cluster, the data cluster is divided into the target number of data layers.
[0074] Specifically, in order to ensure the rationality of data cluster stratification, multiple candidate preset numbers of layers can be set in advance. Here, each preset number of layers is the number of data layers obtainable by stratifying the data cluster. For example, three preset numbers of layers can be set, namely 3 layers, 4 layers, and 5 layers.
[0075] For each preset number of layers, the data cluster can be divided into a preset number of data layers based on the characteristic distance between the candidate data in the data cluster and the center. For example, when there are three preset numbers of layers, namely, 3 layers, 4 layers, and 5 layers, 3 data layers can be obtained by dividing the data cluster into 3 layers, 4 data layers can be obtained by dividing the data cluster into 4 layers, and 5 data layers can be obtained by dividing the data cluster into 5 layers, that is, each preset number of layers can correspond to a stratification result.
[0076] After obtaining the stratification results corresponding to each preset number of layers, the stratification effect of each stratification result can be evaluated. Among them, for any preset number of layers, the stratification result corresponding to the preset number of layers, that is, the preset number of data layers obtained by stratifying the data clusters, the stratification effect of the stratification result can be expressed as the clustering effect of the candidate data in the preset number of data layers.
[0077] It can be understood that for any data layer, the data layer may contain multiple candidate data, and the similarity of the data features of the multiple candidate data in the data layer, that is, can reflect the clustering effect of the candidate data in the data layer, that is, the closeness and separation of the candidate data in the data layer when clustered. For example, the Silhouette coefficient of the data layer can be calculated based on the data features of each candidate data in the data layer, and the calculated Silhouette coefficient is used as a representation of the clustering effect of the candidate data in the data layer. Here, the Silhouette coefficient is also called the contour coefficient, which is an indicator for evaluating the clustering effect. The Silhouette coefficient can measure the closeness between each candidate data in the data layer and the data layer, and the separation between each candidate data in the data layer and the data layer. The value of the Silhouette coefficient is between -1 and 1. The closer the Silhouette coefficient is to 1, the better the clustering effect.
[0078] Moreover, for any preset number of layers, after obtaining the clustering effect of candidate data in the preset number of data layers, the clustering effect of candidate data in each data layer can be combined to represent the effect of applying the preset number of layers for stratification. For example, the mean value of the Silhouette coefficients of the preset number of data layers can be used as the effect of applying the preset number of layers for stratification.
[0079] Thus, the target number of layers can be determined from multiple preset numbers of layers based on the clustering effect of the candidate data of the preset number of layers of data, that is, the effect of applying the preset number of layers for stratification can be determined based on the clustering effect of the candidate data of the preset number of layers of data, and then the preset number of layers with the best stratification effect can be selected from multiple preset numbers of layers as the target number of layers. It can be understood that the target number of layers thus determined is the preset number of layers with the best effect when stratifying data clusters among multiple preset numbers of layers, and the target numbers of layers corresponding to different data clusters can be the same or different.
[0080] After determining the target number of layers, the data cluster can be divided into the target number of data layers, thereby achieving stratification of the data cluster, thereby providing conditions for achieving stratified sampling of the data cluster.
[0081] Based on any of the above embodiments, in step 130, sampling candidate data in each data layer in the data cluster includes: Randomly select one of the selected candidate data from the data layer as sampled data, and delete the sampled data from the data layer; Calculate the minimum value of the characteristic distance between each candidate data after screening in the data layer and all the sampled data, take the candidate data with the largest minimum value as the sampled data, delete the sampled data from the data layer, and return to calculate the minimum value of the characteristic distance between each remaining candidate data in the data layer and all the sampled data, until the selection is completed.
[0082] Specifically, for any data layer in any data cluster, in the process of sampling the candidate data in the data layer, the candidate data obtained by sampling can be recorded as sampled data.
[0083] In addition, for two data, the feature distance between the two data is the distance between the data features of the two data, which can be specifically the Euclidean distance. It can be understood that for two data, the closer the feature distance is, the higher the similarity of the data features of the two data is, and the farther the feature distance is, the lower the similarity of the data features of the two data is.
[0084] In order to select representative candidate data as sampling data from a large number of candidate data in the data layer, so as to ensure uniform and comprehensive distribution of the sampling data, in an embodiment of the present invention, the process of selecting sampling data from the candidate data in the data layer can be understood as starting from a random candidate data and selecting candidate data one by one as sampling data, and in this process, it is necessary to ensure that the minimum value of the feature distance between the sampling data selected previously and the candidate data to be selected later as sampling data is maximized, thereby ensuring that the data features of the finally selected sampling data are as dispersed as possible in the feature space.
[0085] It can be understood that the minimum value of the feature distance between the previously selected sampling data and the candidate data to be selected as the sampling data reflects the similarity between the sampling data that is most similar to the candidate data to be selected and the candidate data. Maximizing the minimum value of the feature distance between the previously selected sampling data and the candidate data to be selected as the sampling data means that it is necessary to select the candidate data with the smallest similarity with all the already selected sampling data from all the selectable candidate data in the data layer as the new sampling data, that is, the sampling data selected each time is the candidate data with the greatest difference from all the already selected sampling data among all the selectable candidate data.
[0086] In a specific implementation, when sampling data is selected for the first time, a candidate data can be randomly selected from all optional candidate data in the data layer as the first sampling data, and after the first sampling data is selected, the sampling data is removed from all optional candidate data.
[0087] Subsequently, when selecting the next sampling data, the characteristic distances between all optional candidate data and the already selected sampling data can be calculated, and when there are multiple sampling data, for each optional candidate data, the minimum value of the characteristic distances between the candidate data and each sampling data is counted, that is, the characteristic distance between the candidate data and the sampling data that is most similar to itself.
[0088] Next, from all the optional candidate data, the candidate data with the largest minimum feature distance, that is, the candidate data with the largest feature distance to the sample data with the highest similarity to itself, is selected as the new sample data. And after the new sample data is selected, the sample data is removed from all the optional candidate data.
[0089] Then, you can return to the next sampling data selection process. The above steps can be repeated until the selection is completed.
[0090] Here, the condition for ending the selection may be that the number of selected sampling data reaches a preset quantity threshold, or that the distribution of data features of all selected sampling data meets the uniform and comprehensive requirements for generating reaction samples. The embodiment of the present invention does not specifically limit this.
[0091] In the method provided by the embodiment of the present invention, the uniformity and comprehensiveness of the distribution of the finally selected sampling data are ensured by maximizing the minimum feature distance between the previously selected sampling data and the candidate data to be selected as the sampling data.
[0092] Based on any of the above embodiments, an embodiment of the present invention provides a diversity sampling method with flexibly adjustable spatial range. The diversity sampling method with flexibly adjustable spatial range here is specifically applied to the modeling scenario of the yield prediction model in the chemical field to realize the product sampling of chemical reactions that require yield prediction.
[0093] The method specifically comprises the following steps: First, candidate data and reference data are obtained separately.
[0094] Here, the candidate data refers to various products to be sampled, and the reference data refers to various products summarized from the patent database. The reference data here can specifically come from the USPTO reaction data set in the patent database.
[0095] On this basis, data features can be extracted for the candidate data and the reference data respectively, and through the data features of the two, the candidate data and the reference data can be mapped to the same feature space.
[0096] Furthermore, considering that both the candidate data and the reference data are products of chemical reactions, the SMILES string of the product can be converted into a molecular fingerprint, and then the dimensionality reduction is performed on the molecular fingerprint to obtain data features. The dimensionality reduction here can be achieved through principal component analysis (PCA), or can also be achieved through linear discriminant analysis (LDA), t-distributed stochastic neighbor embedding (t-SNE) and other algorithms, which are not specifically limited in the embodiments of the present invention.
[0097] For example, Figure 2 It is a distribution diagram of the data features provided by the present invention. Figure 2 As shown, the molecular fingerprints of the candidate data and the reference data can be reduced in dimension through principal component analysis, thereby projecting the data features onto a two-dimensional plane for display, with principal component 1 of the data features obtained by dimensionality reduction as the horizontal coordinate and principal component 2 as the vertical coordinate.
[0098] Figure 2 Each point in the figure is a product of a chemical reaction. The blue points represent products obtained as reference data from a patent database, the red triangles represent products obtained from a high-throughput experiment (HTE), and the yellow squares represent products obtained by random sampling of the above products. Figure 2 It can be seen that the yellow blocks obtained based on random sampling perform poorly in terms of coverage and diversity.
[0099] After the data feature extraction is completed, each reference data can be feature clustered based on the data features of each reference data, thereby dividing each reference data into n data clusters. Here, the number n can be the target cluster number determined based on the above embodiment. For example, for a product of the order of 50k, the target cluster number n can be 10.
[0100] After obtaining n data clusters, the distance between the data features of each candidate data and the central data features of the n data clusters can be calculated, and the candidate data can be assigned to the data cluster with the closest distance. Thus, each data cluster contains not only the product as reference data, but also the product assigned nearby as candidate data.
[0101] After completing the data cluster division for the candidate data, for each data cluster, the distance between the center of the data cluster and each reference data in the data cluster can be calculated, and the maximum value of the distance is used as the maximum characteristic distance of the data cluster. On this basis, it can be determined whether the distance between each candidate data in the data cluster and the center exceeds the above maximum characteristic distance. If it exceeds the maximum characteristic distance, the candidate data needs to be screened out. If it does not exceed the maximum characteristic distance, the candidate data can be retained, thereby achieving the screening of candidate data.
[0102] After the screening is completed, stratified sampling can be performed for each data cluster.
[0103] In the process of stratified sampling, each data cluster can first be stratified according to the distance between the candidate data and the center of the data cluster, thereby obtaining multiple data layers of each data cluster. Subsequently, for each data layer, the candidate data in the data layer can be sampled according to the principle of maximizing the minimum value of the characteristic distance between the previously selected sampled data and the candidate data to be selected as the sampled data later, thereby obtaining sampled data. It can be understood that what is obtained here is the sampled data of all data layers in all data clusters, that is, the product used for modeling the yield prediction model.
[0104] The data sampling device provided by the present invention is described below. The diversity sampling device with flexibly adjustable spatial range described below and the diversity sampling method with flexibly adjustable spatial range described above can correspond to each other.
[0105] Figure 3 Schematic diagram of the structure of the data sampling device provided by the present invention. Figure 3 As shown, the device comprises: An acquisition unit 310 is used to acquire each candidate data; A division unit 320, configured to divide each candidate data into each data cluster based on a characteristic distance between each candidate data and a center of each data cluster, wherein each data cluster is obtained by performing characteristic clustering on each reference data; The screening sampling unit 330 is used to screen the candidate data in each data cluster based on the characteristic distribution information of the reference data in each data cluster, and sample the candidate data in each data cluster after the screening is completed.
[0106] In the device provided by an embodiment of the present invention, each candidate data is divided into each data cluster obtained by clustering the reference data, and the candidate data in each data cluster is screened based on the characteristic distribution information of the reference data in each data cluster, and after the screening is completed, the candidate data in each data cluster is sampled, thereby enabling data sampling to refer to the distribution and diversity of the data, thereby ensuring the coverage and diversity of the sampling results.
[0107] Based on any of the above embodiments, the screening sampling unit is further used for: Based on the characteristic distance between the center of the data cluster and each reference data in the data cluster, the maximum characteristic distance of the data cluster is determined, and the maximum characteristic distance of the data cluster is used as the characteristic distribution information of the reference data in the data cluster.
[0108] Based on any of the above embodiments, the screening sampling unit is specifically used for: Based on the maximum characteristic distance of each data cluster and the characteristic distance between the center of each data cluster and the candidate data in each data cluster, the candidate data in each data cluster are screened.
[0109] Based on any of the above embodiments, the screening sampling unit is specifically used for: For each of the data clusters, based on the characteristic distance between the candidate data in the data cluster and the center of the data cluster, the data cluster is layered to obtain each data layer of the data cluster; The candidate data in each data layer in the data cluster is sampled.
[0110] Based on any of the above embodiments, the screening sampling unit is specifically used for: Acquire a plurality of preset numbers of layers, and divide the data cluster into the preset number of data layers based on the characteristic distance between the candidate data in the data cluster and the center of each data cluster; Based on the clustering effect of the candidate data of the preset number of data layers, determining a target number of layers from the plurality of preset numbers of layers; Based on the characteristic distance between the candidate data in the data cluster and the center of the data cluster, the data cluster is divided into the target number of data layers.
[0111] Based on any of the above embodiments, the screening sampling unit is specifically used for: Randomly select one of the selected candidate data from the data layer as sampled data, and delete the sampled data from the data layer; Calculate the minimum value of the characteristic distance between each candidate data after screening in the data layer and all the sampled data, take the candidate data with the largest minimum value as the sampled data, delete the sampled data from the data layer, and return to calculate the minimum value of the characteristic distance between each remaining candidate data in the data layer and all the sampled data, until the selection is completed.
[0112] Based on any of the above embodiments, the dividing unit is further used for: Acquire a plurality of preset cluster numbers, perform feature clustering on each reference data, and obtain data clusters of the preset number of clusters; Based on the clustering effect of the preset number of clusters, determining a target number of clusters from the plurality of preset numbers of clusters; Perform feature clustering on the reference data to obtain the target cluster data clusters.
[0113] Based on any of the above embodiments, the dividing unit is further used for: Based on the data features of each candidate data and the central data features of each data cluster, the characteristic distance between each candidate data and the center of each data cluster is determined. The central data features of each data cluster are determined based on the data features of the reference data in each data cluster. The data features of the candidate data and the data features of the reference data are in the same characteristic space.
[0114] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a diversity sampling method with a flexibly adjustable spatial range, the method comprising: Obtaining candidate data; Based on the characteristic distance between each candidate data and the center of each data cluster, the candidate data are divided into each data cluster, where each data cluster is obtained by performing characteristic clustering on each reference data; Based on the characteristic distribution information of the reference data in each data cluster, the candidate data in each data cluster is screened, and after the screening is completed, the candidate data in each data cluster is sampled.
[0115] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0116] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the diversity sampling method with flexible adjustable spatial range provided by the above methods, the method comprising: Obtaining candidate data; Based on the characteristic distance between each candidate data and the center of each data cluster, the candidate data are divided into each data cluster, where each data cluster is obtained by performing characteristic clustering on each reference data; Based on the characteristic distribution information of the reference data in each data cluster, the candidate data in each data cluster is screened, and after the screening is completed, the candidate data in each data cluster is sampled.
[0117] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the diversity sampling method with flexibly adjustable spatial range provided by the above methods, the method comprising: Obtaining candidate data; Based on the characteristic distance between each candidate data and the center of each data cluster, the candidate data are divided into each data cluster, where each data cluster is obtained by performing characteristic clustering on each reference data; Based on the characteristic distribution information of the reference data in each data cluster, the candidate data in each data cluster is screened, and after the screening is completed, the candidate data in each data cluster is sampled.
[0118] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0119] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.
[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A diversity sampling method with flexible spatial range, characterized in that: include: Obtaining candidate data; Based on the characteristic distance between each candidate data and the center of each data cluster, the candidate data are divided into each data cluster, where each data cluster is obtained by performing characteristic clustering on each reference data; Based on the characteristic distribution information of the reference data in each data cluster, the candidate data in each data cluster is screened, and after the screening is completed, the candidate data in each data cluster is sampled.
2. The diversity sampling method with flexible adjustable spatial range according to claim 1, characterized in that: The acquisition of characteristic distribution information of the reference data in the data cluster includes: Based on the characteristic distance between the center of the data cluster and each reference data in the data cluster, the maximum characteristic distance of the data cluster is determined, and the maximum characteristic distance of the data cluster is used as the characteristic distribution information of the reference data in the data cluster.
3. The diversity sampling method with flexible adjustable spatial range according to claim 2, characterized in that: The screening of candidate data in each data cluster based on the characteristic distribution information of the reference data in each data cluster includes: Based on the maximum characteristic distance of each data cluster and the characteristic distance between the center of each data cluster and the candidate data in each data cluster, the candidate data in each data cluster are screened.
4. The diversity sampling method with flexible adjustable spatial range according to claim 1, characterized in that: The sampling of candidate data in each data cluster includes: For each of the data clusters, based on the characteristic distance between the candidate data in the data cluster and the center of the data cluster, the data cluster is layered to obtain each data layer of the data cluster; The candidate data in each data layer in the data cluster is sampled.
5. The diversity sampling method with flexible adjustable spatial range according to claim 4, characterized in that: The step of stratifying the data cluster based on the characteristic distance between the candidate data in the data cluster and the center of the data cluster to obtain each data layer of the data cluster includes: Acquire a plurality of preset numbers of layers, and divide the data cluster into the preset number of data layers based on the characteristic distance between the candidate data in the data cluster and the center of each data cluster; Based on the clustering effect of the candidate data of the preset number of data layers, determining a target number of layers from the plurality of preset numbers of layers; Based on the characteristic distance between the candidate data in the data cluster and the center of the data cluster, the data cluster is divided into the target number of data layers.
6. The diversity sampling method with flexible adjustable spatial range according to claim 4, characterized in that: The sampling of candidate data in each data layer in the data cluster includes: Randomly select one of the selected candidate data from the data layer as the sampled data, and delete the sampled data from the data layer; Calculate the minimum value of the characteristic distance between each candidate data after screening in the data layer and all the sampled data, take the candidate data with the largest minimum value as the sampled data, delete the sampled data from the data layer, and return to calculate the minimum value of the characteristic distance between each remaining candidate data in the data layer and all the sampled data, until the selection is completed.
7. The diversity sampling method with flexibly adjustable spatial range according to any one of claims 1 to 6, characterized in that: The determination of each data cluster includes: Acquire a plurality of preset cluster numbers, perform feature clustering on each reference data, and obtain data clusters of the preset number of clusters; Based on the clustering effect of the preset number of clusters, determining a target number of clusters from the plurality of preset numbers of clusters; Perform feature clustering on the reference data to obtain the target cluster data clusters.
8. The diversity sampling method with flexibly adjustable spatial range according to any one of claims 1 to 6, characterized in that: The determination of the characteristic distance between each candidate data and the center of each data cluster includes: Based on the data features of each candidate data and the central data features of each data cluster, the characteristic distance between each candidate data and the center of each data cluster is determined. The central data features of each data cluster are determined based on the data features of the reference data in each data cluster. The data features of the candidate data and the data features of the reference data are in the same characteristic space.
9. A diversity sampling device with a flexibly adjustable spatial range, characterized in that: include: An acquisition unit, used for acquiring each candidate data; A division unit, configured to divide each candidate data into each data cluster based on a characteristic distance between each candidate data and a center of each data cluster, wherein each data cluster is obtained by performing characteristic clustering on each reference data; The screening sampling unit is used to screen the candidate data in each data cluster based on the characteristic distribution information of the reference data in each data cluster, and after the screening is completed, sample the candidate data in each data cluster.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the diversity sampling method with flexibly adjustable spatial range as described in any one of claims 1 to 8 is implemented.
11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the diversity sampling method with flexibly adjustable spatial range as claimed in any one of claims 1 to 8 is implemented.
12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the diversity sampling method with flexibly adjustable spatial range as claimed in any one of claims 1 to 8 is implemented.