Sample data generation method and apparatus, electronic device, and storage medium
By generating and filtering recombinant sample sets using clustering models, the problem of poor sample quality in existing training sample augmentation methods is solved, thereby improving sample diversity and the accuracy of deep learning models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 中移信息技术有限公司
- Filing Date
- 2023-07-31
- Publication Date
- 2026-06-05
AI Technical Summary
Existing training sample augmentation methods generate poor-quality samples, resulting in low accuracy of deep learning models.
The preset first clustering model is used to cluster the preset merged element set to generate a recombined sample set. Then, the clustering core of the second clustering model is used to screen out the target recombined sample set that is highly related to the original sample, so as to ensure the quality of the recombined sample.
This improved the quality and diversity of generated samples, enhancing the training effect of deep learning models.
Smart Images
Figure CN116992287B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device and storage medium for generating sample data. Background Technology
[0002] The accuracy of deep learning models depends on the amount and diversity of training samples. A large number of diverse training samples generally result in a high-accuracy deep learning model; conversely, a small number of diverse training samples lead to low accuracy. Therefore, before training a deep learning model, it is generally necessary to augment the training samples to improve its performance. However, current methods for augmenting training samples typically only combine features to obtain composite samples to increase the number of samples. The resulting sample set may contain some samples that differ significantly from the actual sample set, resulting in poor sample quality. Summary of the Invention
[0003] The main purpose of this application is to provide a sample data generation method, apparatus, electronic device and storage medium, which aims to solve the technical problem of poor sample quality obtained by current training sample augmentation methods.
[0004] To achieve the above objectives, this application provides a sample data generation method, the sample data generation method comprising:
[0005] The first clustering result is obtained by clustering the elements in the preset merged element set using a preset first clustering model, wherein the preset merged element set is determined by the features of the original samples in the original sample set.
[0006] A recombined sample set is generated for each group based on the first clustering result, wherein the recombined sample set includes recombined samples, and the constituent elements of the recombined samples come from the same group in the first clustering result.
[0007] The recombined sample set is screened based on the clustering core of the second clustering model. Recombined samples in the recombined sample set whose correlation with the clustering core is less than a preset correlation threshold are removed to obtain the target recombined sample set. The clustering core is determined based on the original samples in the original sample set.
[0008] Optionally, the step of filtering the recombinant sample set based on the clustering core of the second clustering model includes:
[0009] Clustering is performed on the original samples in the original sample set to obtain the clustering result of the original sample set;
[0010] The target cluster core is determined based on the clustering results of the original sample set, and the target cluster core is used as the cluster core of the second clustering model.
[0011] The recombinant samples in the recombinant sample set are clustered based on the second clustering model to obtain a second clustering result, which includes each group obtained based on the target clustering core cluster.
[0012] The weakly associated recombinant samples in the recombinant sample set are removed to obtain the target recombinant sample set, wherein the weakly associated recombinant samples are those whose distance from the target cluster core of their respective group is greater than a preset distance threshold.
[0013] Optionally, before the step of clustering the elements in the preset merged element set using a preset first clustering model to obtain the first clustering result, the method further includes:
[0014] The original samples in the original sample set are decomposed into a basic element set by feature decomposition.
[0015] Mutate the basic elements in the basic element set to generate a mutated element set;
[0016] The set of basic elements and the set of mutated elements are combined to obtain the preset merged element set.
[0017] Optionally, the step of clustering the elements in the preset merged element set using a preset first clustering model to obtain the first clustering result includes:
[0018] The elements in the preset merged element set are vectorized to obtain an element vector set, wherein the element vector set is composed of element vectors, and the element vector of any element is generated by the element and other elements in the pre-constructed knowledge graph that have a relationship with the element.
[0019] The first clustering result is obtained by clustering the element vector set using the first clustering model.
[0020] Optionally, the step of generating a recombinant sample set based on each group according to the first clustering result includes:
[0021] Elements are selected from the set of elements belonging to the same group in the first clustering result to generate recombined samples, wherein the number distribution of each class of samples in the recombined sample set is consistent with the number distribution of each class of samples in the original sample set;
[0022] The recombinant samples generated by the combination are screened to remove recombinant samples with homologous elements, and the recombinant sample set is generated based on the screened recombinant samples.
[0023] Optionally, when the original sample is a sample of a data type, the basic element includes a feature field and the feature value of the feature field; when the original sample is a text type sample, the basic element is text segmentation; the mutation element set is composed of mutation elements; and the step of generating the mutation element set by mutating the basic elements in the basic element set includes:
[0024] Modify the feature values of the feature fields in the basic element to generate a variant element;
[0025] Alternatively, the word order features of the text segmentation, the part-of-speech features of the text segmentation, or the text segmentation itself can be changed in the basic element to generate a variant element.
[0026] Optionally, after the step of filtering the recombinant sample set based on the clustering core of the second clustering model to remove recombinant samples in the recombinant sample set that are weakly associated with the clustering core to obtain the target recombinant sample set, the method further includes:
[0027] The recombined samples in the target recombined sample set are added to the original sample set as original samples to obtain a new original sample set;
[0028] Based on the new original sample set, the step of decomposing the original samples in the original sample set into a basic element set is repeated until the number of original samples in the original sample set reaches a preset threshold.
[0029] Furthermore, to achieve the above objectives, this application also provides a sample data generation apparatus, the sample data generation apparatus comprising:
[0030] The first clustering module is used to cluster the elements in the preset merged element set using a preset first clustering model to obtain the first clustering result, wherein the preset merged element set is determined by the features of the original samples in the original sample set.
[0031] A recombination module is used to generate a recombination sample set based on each group of the first clustering result, wherein the recombination sample set includes recombination samples, and the constituent elements of the recombination samples come from the same group of the first clustering result.
[0032] The second clustering module is used to filter the recombinant sample set based on the clustering core of the second clustering model, and remove recombinant samples in the recombinant sample set whose correlation with the clustering core is less than a preset correlation threshold, to obtain the target recombinant sample set. The clustering core is determined based on the original samples in the original sample set.
[0033] In addition, to achieve the above objectives, this application also provides an electronic device, which includes: a memory, a processor, and a sample data generation program stored in the memory and executable on the processor. When the sample data generation program is executed by the processor, it implements the steps of the sample data generation method described above.
[0034] In addition, to achieve the above objectives, this application also provides a storage medium storing a sample data generation program, which, when executed by a processor, implements the steps of the sample data generation method described above.
[0035] This application proposes a sample data generation method, apparatus, electronic device, and storage medium. In this embodiment, elements in a preset merged element set are clustered using a preset first clustering model to obtain a first clustering result, wherein the preset merged element set is determined by the characteristics of the original samples in the original sample set; a recombined sample set is generated based on the groups of the first clustering result, wherein the recombined sample set includes recombined samples, and the constituent elements of the recombined samples come from the same group in the first clustering result; the recombined sample set is screened based on the clustering core of a second clustering model, removing recombined samples whose correlation with the clustering core is less than a preset correlation threshold, to obtain a target recombined sample set, wherein the clustering core is determined based on the original samples in the original sample set. In this example, the recombined samples are generated from the clustering results of the clustering model. The constituent elements of the recombined samples all come from the same cluster in the clustering results, thus exhibiting a high degree of correlation between the constituent elements. This ensures the quality of the recombined samples in the recombined sample set. Furthermore, after obtaining the recombined sample set, the target recombined sample set is obtained by filtering the recombined sample set using the clustering core of the second clustering model. Since the clustering core of the second clustering model is determined from the original sample set, the recombined samples in the target recombined sample set have a strong correlation with the original samples. This means the recombined samples are more closely aligned with the actually generated samples. Therefore, the quality of the recombined samples generated in this embodiment is more guaranteed, ensuring the value of the recombined samples for model training. Attached Figure Description
[0036] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of this application;
[0037] Figure 2 This is a flowchart illustrating the first embodiment of the sample data generation method of this application;
[0038] Figure 3 This is a flowchart illustrating the second embodiment of the sample data generation method in this application;
[0039] Figure 4 This is a schematic diagram of the knowledge graph structure in the sample data generation method of this application;
[0040] Figure 5 This is a schematic diagram illustrating the adjustment of vector elements in the sample data generation method of this application;
[0041] Figure 6 This is a schematic diagram illustrating the screening process based on the second clustering model in the sample data generation method of this application;
[0042] Figure 7 This is a schematic diagram of the sample data generation device in the sample data generation method of this application.
[0043] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0044] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.
[0045] like Figure 1 As shown, Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of this application.
[0046] The electronic device in this application embodiment can be a server, or it can be an electronic terminal device such as a smartphone, PC, tablet computer, or portable computer.
[0047] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0048] Optionally, the electronic device may also include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, and so on. The terminal may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, thermometer, and infrared sensor, which will not be elaborated upon here. Those skilled in the art will understand that... Figure 1 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0049] Those skilled in the art will understand that Figure 1 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0050] In addition, such as Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a sample data generation program.
[0051] exist Figure 1 In the illustrated electronic device, network interface 1004 is mainly used to connect to the backend server and communicate data with it; user interface 1003 is mainly used to connect to the client (user terminal) and communicate data with it; while processor 1001 can be used to call the sample data generation program stored in memory 1005 and perform the following operations:
[0052] The first clustering result is obtained by clustering the elements in the preset merged element set using a preset first clustering model, wherein the preset merged element set is determined by the features of the original samples in the original sample set.
[0053] A recombined sample set is generated for each group based on the first clustering result, wherein the recombined sample set includes recombined samples, and the constituent elements of the recombined samples come from the same group in the first clustering result.
[0054] The recombined sample set is screened based on the clustering core of the second clustering model. Recombined samples in the recombined sample set whose correlation with the clustering core is less than a preset correlation threshold are removed to obtain the target recombined sample set. The clustering core is determined based on the original samples in the original sample set.
[0055] In one feasible implementation, the processor 1001 may invoke the sample data generation program stored in the memory 1005 and further perform the following operations:
[0056] The step of filtering the recombinant sample set using the clustering core based on the second clustering model includes:
[0057] Clustering is performed on the original samples in the original sample set to obtain the clustering result of the original sample set;
[0058] The target cluster core is determined based on the clustering results of the original sample set, and the target cluster core is used as the cluster core of the second clustering model.
[0059] The recombinant samples in the recombinant sample set are clustered based on the second clustering model to obtain a second clustering result, which includes each group obtained based on the target clustering core cluster.
[0060] The weakly associated recombinant samples in the recombinant sample set are removed to obtain the target recombinant sample set, wherein the weakly associated recombinant samples are those whose distance from the target cluster core of their respective group is greater than a preset distance threshold.
[0061] In one feasible implementation, the processor 1001 may invoke the sample data generation program stored in the memory 1005 and further perform the following operations:
[0062] Before the step of clustering the elements in the preset merged element set using a preset first clustering model to obtain the first clustering result, the method further includes:
[0063] The original samples in the original sample set are decomposed into a basic element set by feature decomposition.
[0064] Mutate the basic elements in the basic element set to generate a mutated element set;
[0065] The set of basic elements and the set of mutated elements are combined to obtain the preset merged element set.
[0066] In one feasible implementation, the processor 1001 may invoke the sample data generation program stored in the memory 1005 and further perform the following operations:
[0067] The step of clustering the elements in the preset merged element set using a preset first clustering model to obtain the first clustering result includes:
[0068] The elements in the preset merged element set are vectorized to obtain an element vector set, wherein the element vector set is composed of element vectors, and the element vector of any element is generated by the element and other elements in the pre-constructed knowledge graph that have a relationship with the element.
[0069] The first clustering result is obtained by clustering the element vector set using the first clustering model.
[0070] In one feasible implementation, the processor 1001 may invoke the sample data generation program stored in the memory 1005 and further perform the following operations:
[0071] The step of generating a recombinant sample set for each group based on the first clustering result includes:
[0072] Elements are selected from the set of elements belonging to the same group in the first clustering result to generate recombined samples, wherein the number distribution of each class of samples in the recombined sample set is consistent with the number distribution of each class of samples in the original sample set;
[0073] The recombinant samples generated by the combination are screened to remove recombinant samples with homologous elements, and the recombinant sample set is generated based on the screened recombinant samples.
[0074] In one feasible implementation, the processor 1001 may invoke the sample data generation program stored in the memory 1005 and further perform the following operations:
[0075] When the original sample is a sample of a data type, the basic element includes a feature field and the feature value of the feature field. When the original sample is a text type sample, the basic element is text segmentation. The set of mutated elements consists of mutated elements. The step of generating the set of mutated elements by mutating the basic elements in the set of basic elements includes:
[0076] Modify the feature values of the feature fields in the basic element to generate a variant element;
[0077] Alternatively, the word order features of the text segmentation, the part-of-speech features of the text segmentation, or the text segmentation itself can be changed in the basic element to generate a variant element.
[0078] In one feasible implementation, the processor 1001 may invoke the sample data generation program stored in the memory 1005 and further perform the following operations:
[0079] After the step of filtering the recombinant sample set using the clustering core based on the second clustering model to remove recombinant samples weakly associated with the clustering core to obtain the target recombinant sample set, the method further includes:
[0080] The recombined samples in the target recombined sample set are added to the original sample set as original samples to obtain a new original sample set;
[0081] Based on the new original sample set, the step of decomposing the original samples in the original sample set into a basic element set is repeated until the number of original samples in the original sample set reaches a preset threshold.
[0082] To clearly illustrate the advantages of this approach, a brief explanation of traditional sample expansion methods is provided below.
[0083] For example, in traditional sample expansion schemes, the sample expansion generation process includes: first, extracting the features of the samples; each feature is composed of several other features, and these features have certain dependencies; then, multivariate fitting is used to construct the relationships between these features, and the fitting results are used to expand the feature set of new samples. According to this sample expansion process, the expansion method only fits and recombines the features of existing samples to obtain new samples, without mutating the features. Therefore, the expanded samples obtained are limited by the number of features in the existing samples. Consequently, the richness of the expanded samples obtained by this scheme is insufficient. Furthermore, this method does not include a screening mechanism for the expanded samples, resulting in a lack of guarantee for sample quality.
[0084] To address the aforementioned issues, this application proposes a sample data generation method that can enrich the types of generated samples while ensuring the quality of the generated samples, thereby guaranteeing the performance of the model.
[0085] Reference Figure 2 The first embodiment of the sample data generation method of this application includes:
[0086] Step S10: Cluster the elements in the preset merged element set using a preset first clustering model to obtain the first clustering result, wherein the preset merged element set is determined by the features of the original samples in the original sample set.
[0087] It should be noted that the types of original samples mentioned above vary depending on the application scenario. An original sample typically contains different features (i.e., more than one feature). In this embodiment, the recombined sample is obtained by recombining the different features, i.e., elements from a preset merged element set. Typically, each element in the preset merged element set is obtained by disassembling the original samples in the original sample set.
[0088] For example, the first clustering model mentioned above can be a k-means clustering model. The elements in the preset merged element set obtained by merging the basic element set and the variant element set are vectorized and input into the first clustering model. The first clustering model performs clustering of the elements to obtain the first clustering result. The first clustering result includes different groups, and each group includes different similar, related or related elements.
[0089] Step S20: Generate a recombinant sample set based on each group of the first clustering result, wherein the recombinant sample set includes recombinant samples, and the constituent elements of the recombinant samples come from the same group of the first clustering result.
[0090] For example, the first clustering result consists of multiple groups, with elements within the same group having a higher correlation. In the newly formed recombinant sample, each element comes from a single group. This ensures the correlation between elements in each recombinant sample, thereby guaranteeing the quality of the recombinant sample.
[0091] Step S30: Based on the clustering core of the second clustering model, the recombinant sample set is screened, and recombinant samples in the recombinant sample set whose correlation with the clustering core is less than a preset correlation threshold are removed, to obtain the target recombinant sample set. The clustering core is determined based on the original samples in the original sample set.
[0092] For example, after generating the recombinant sample set, the samples in the recombinant sample set can be screened again. For instance, the recombinant samples in the recombinant sample set can be clustered again using a second clustering model. However, it should be noted that the second clustering model usually only performs one clustering operation, that is, it performs one clustering operation based on the clustering cores determined in the original sample set (which can be any original sample). It can be understood that the clustering model usually performs clustering based on the clustering cores, that is, it judges the correlation between each sample and the clustering cores (the correlation can be measured by the distance between the sample and the clustering cores; the closer the distance, the higher the correlation). For example, recombinant samples with high correlation with the clustering cores can be used as target recombinant samples in the recombinant sample set, or recombinant samples with low correlation with the clustering cores (correlation with the clustering cores is also called low correlation) can be deleted, thus obtaining the target recombinant sample set. Since the clustering cores of the second clustering model are determined by the original sample set, the target recombinant samples in the screened recombinant sample set have a high correlation with the original samples, thus ensuring the quality of the recombinant samples again.
[0093] In this embodiment, a first clustering result is obtained by clustering elements in a preset merged element set using a preset first clustering model, wherein the preset merged element set is determined by the features of the original samples in the original sample set; a recombined sample set is generated based on each group of the first clustering result, wherein the recombined sample set includes recombined samples, and the constituent elements of the recombined samples come from the same group of the first clustering result; the recombined sample set is screened based on the clustering core of a second clustering model, and recombined samples in the recombined sample set whose correlation with the clustering core is less than a preset correlation threshold are removed, thereby obtaining a target recombined sample set, wherein the clustering core is determined based on the original samples in the original sample set. In this example, the recombined samples are generated from the clustering results of the clustering model. The constituent elements of the recombined samples all come from the same cluster in the clustering results, thus exhibiting a high degree of correlation between the constituent elements. This ensures the quality of the recombined samples in the recombined sample set. Furthermore, after obtaining the recombined sample set, the target recombined sample set is obtained by filtering the recombined sample set using the clustering core of the second clustering model. Since the clustering core of the second clustering model is determined from the original sample set, the recombined samples in the target recombined sample set have a strong correlation with the original samples. This means the recombined samples are more closely aligned with the actually generated samples. Therefore, the quality of the recombined samples generated in this embodiment is more guaranteed, ensuring the value of the recombined samples for model training.
[0094] In one feasible implementation, before the step of clustering the elements in the preset merged element set using a preset first clustering model to obtain a first clustering result, the method includes:
[0095] Step S01: Decompose the original samples in the original sample set into a basic element set by feature decomposition.
[0096] Step S02: Mutate the basic elements in the basic element set to generate a mutated element set;
[0097] Step S03: Combine the basic element set and the mutated element set to obtain the preset merged element set.
[0098] For example, the features of each original sample are decomposed by a preset feature decomposition model. The decomposition model can be a feature extraction model, and a feature obtained by decomposing the original sample is a basic element. The basic elements can be composed of the original samples in the original sample set.
[0099] If the original sample is a data type sample, decomposing the original sample into a basic element can be composed of a feature field and the corresponding feature value. For example, the original sample 1 is (A:a, B:b, C:c), where A, B, and C are the three feature fields corresponding to the original sample 1, and a, b, and c are the specific feature values (or feature values of the elements) corresponding to the feature fields. In the sample decomposition process, the original sample 1 can be decomposed into three basic elements: A:a, B:b, and C:c.
[0100] In addition, to enrich the variety of elements, the basic elements can be mutated to obtain variant elements.
[0101] In one feasible implementation, when the original sample is a sample of a data type, the basic element includes a feature field and the feature value of the feature field; when the original sample is a text type sample, the basic element is text segmentation; the mutated element set is composed of mutated elements; and the step of generating the mutated element set by mutating the basic elements in the basic element set includes:
[0102] Step S04: Change the feature value of the feature field in the basic element to generate a variant element;
[0103] Step S05, or, change the word order features of the text segmentation in the basic element, the part-of-speech features of the text segmentation, or the text segmentation itself, to generate a variant element.
[0104] For example, when the original sample is a sample of a data type, the basic elements obtained from the decomposition will include feature fields and the feature values of those feature fields. For instance, based on the above example, a basic element is B:b. Changing the feature value b of feature field B generates a variant element B:b1. The feature values of the feature fields in the variant element can be obtained through random changes, and the feature values of the feature fields in the variant element are within the preset value range of that feature field.
[0105] For example, when the original sample is a text sample, the decomposition model for decomposing the original sample can be a word segmentation model. The word segmentation model segments the text of the original sample into words, with the basic element being the text word. Correspondingly, mutation can be performed by changing the word order features, part-of-speech features, or the text word itself. The word order feature can be the positional feature of the word in the text, and the part-of-speech feature can be the part of speech of the word, such as noun or verb. Specifically, the modification logic can be selectively set according to the business scenario, for example, replacing the original text word with synonyms or homophones. For image samples, feature decomposition can be performed by dividing the image into regions. The set of pixels in a region of the image can be used as the feature (i.e., a basic element) of that region. In addition to the methods described above, generating variant elements can also be achieved by exchanging the basic elements of two samples. The resulting basic elements become the variant elements. For example, during the collection of cellular data samples, the basic elements obtained from successfully connecting data samples from areas with good network connectivity can be combined with the basic elements obtained from failing to connect data samples from areas with poor network connectivity. This effectively adds a sample corresponding to a terminal that alternates between areas with good and poor network connectivity. The generated variant elements then form a variant element set. Furthermore, the specific mutation methods can be set by technical personnel according to requirements, which will not be elaborated here.
[0106] It is understood that in this embodiment, the features in the samples will be mutated to generate new features. Compared with traditional expansion schemes, this embodiment can greatly enrich the types of generated samples and ensure the quality of samples used to expand the number of samples.
[0107] In one feasible implementation, the step of clustering the elements in the preset merged element set using a preset first clustering model to obtain the first clustering result includes:
[0108] Step S11: Vectorize each element in the preset merged element set to obtain an element vector set. The element vector set is used for clustering in the first clustering model. The element vector set is composed of element vectors. The element vector of any element is composed of the element and other elements in the pre-constructed knowledge graph that have a relationship with the element.
[0109] Step S12: Cluster the element vector set using the first clustering model to obtain the first clustering result.
[0110] For example, in this embodiment, each element in a preset merged element set, consisting of a basic element set and a mutated element set, is vectorized to obtain an element vector set used by the first clustering model to classify the elements in the preset merged element set. The vectorization process may include the construction of a knowledge graph. Nodes in the knowledge graph can be elements in the preset merged element set or feature fields of elements. For example, for samples of data types, if a node is a feature field of an element, the corresponding node feature value is the feature value corresponding to the feature field (or possibly the feature value of the element). See reference... Figure 4 This is a schematic diagram of the knowledge graph structure in this embodiment. Let an original sample be (A:a1, B:b1, C:c1), and a mutated sample (A:a2, B:b2, C:c2) be obtained based on this original sample. Another original sample is (A`:a`, D:d2, F:f, E:e2), and a mutated element E:e2 is obtained based on this sample. Edge features represent the relationships between feature fields. These relationships can be determined based on the specific fields corresponding to the sample and the business scenario in which the sample was generated, thus establishing the relationships between the feature fields. For example, refer to... Figure 4 The edge features representing sample relationships can be AB, BC, AC, etc.; the edge features representing mutation relationships can be AA, BB, CC, etc.; and the edge features representing preset business relationships are AA` and BE, etc. The element vector of any element consists of the element's feature values and the feature values of other elements in the pre-built knowledge graph that are related to that element. These relationships can include sample relationships (representing the positional relationship between two elements in the same sample or within the same sample), mutation relationships (one element is derived from the mutation of another element), and preset business relationships (set by business personnel according to specific business needs). For example, refer to... Figure 4 For element A: a1, its element vector can be (a1, b1, c1, a2, a`). Furthermore, for text-type samples, nodes in the knowledge graph are derived from basic elements, such as word segmentation, while edges represent text within the same text or adjacent text. Similarly, for graph-type samples, nodes in the knowledge graph can also be basic elements, while edges represent adjacent basic elements, etc. Currently, knowledge graph technology is relatively mature; the nodes of a knowledge graph are determined based on the decomposed elements, and the edges between nodes represent relationships between elements.
[0111] Furthermore, it should be noted that the quantized element vector can be adjusted using the TransE model. For example, a function d(h+r,t) can be defined to measure the distance between h+r and t, as shown in [reference needed]. Figure 5As shown, let h be the vector value of element A:a1 in the vector space, t be the vector value of element B:b1, and r be the edge relationship between nodes A and B. By adjusting the parameters and training the model, we can make h+r≈t, which means that elements A:a1 and B:b1 have a factual relationship.
[0112] Furthermore, the aforementioned first clustering model can be a k-means clustering model. Elements in a pre-defined merged element set, obtained by merging the basic element set and the mutated element set, are vectorized and input into the first clustering model, which then performs the clustering. The k-value of the first clustering model (i.e., the number of cluster cores) can be determined based on the sum of squared errors (SSE), calculated using the following formula:
[0113]
[0114] In the formula, Ci is the i-th cluster of the first clustering model, p is the sample point in Ci, mi is the centroid of Ci (the mean of the samples in Ci), and SSE is the clustering error of all samples, representing the quality of the clustering. The specific process of determining the value of k includes determining the corresponding SSE value as k changes, to determine how the SSE value changes with k. After determining the change in SSE, the k value corresponding to the inflection point where SSE tends to stabilize is taken as the k value of the first clustering model.
[0115] Understandably, the clustering granularity of the first clustering model can be controlled by adjusting the k-value of the first clustering model. This, in turn, adjusts the correlation between elements in each cluster corresponding to the first clustering result.
[0116] Once the value of k in the first clustering model is determined, the first clustering model will select k clustering cores (which can be randomly selected initially) during clustering. Based on the k clustering cores, k clusters will be generated (the samples in a cluster are closest to the clustering core of their own cluster compared to other clustering cores). Based on the generated k clusters, the clustering core of each cluster will be re-determined (such as selecting the centroid of the cluster, which is the mean of the samples in the cluster). This process will be repeated until the clustering cores no longer change, at which point the clustering will be completed, resulting in k clusters.
[0117] In one feasible implementation, the step of generating a recombinant sample set based on each group according to the first clustering result includes:
[0118] Step S21: Select elements from the set of elements belonging to the same group in the first clustering result to generate a recombined sample, wherein the number distribution of each class of samples in the recombined sample set is consistent with the number distribution of each class of samples in the original sample set.
[0119] For example, for any recombinant sample, its constituent elements (including basic elements and variant elements) all come from the set of relevant elements in the same cluster. That is, elements are selected from the set of relevant elements in the same cluster from the first clustering result to form the recombinant sample. The selection method can be random selection, and the recombinant samples thus form a recombinant sample set. It should be noted that, in this embodiment, elements in the same cluster have a strong correlation. Therefore, selecting elements from the same cluster to form the recombinant sample can improve the correlation between elements in the recombinant sample, thereby ensuring the quality of the recombinant sample. For example, refer to... Figure 4 Suppose there exist recombinant sample 1 (A:a1, B:b1, C:c2) and recombinant sample 2 (A:a1, B:b1, F:f), and according to... Figure 4 From the perspective of the relationships between elements, the relationships between elements in recombined sample 1 (A:a1, B:b1, C:c2) are closer than those between elements in recombined sample 2 (A:a1, B:b1, F:f). Therefore, element set 1 (A:a1, B:b1, C:c2) is more likely to be assigned to the same group than element set 2 (A:a1, B:b1, F:f). This also means that the elements in recombined sample 1 (A:a1, B:b1, F:f) are more likely to be assigned to the same group. The probability of generating recombinant sample 1 (A:a1, B:b1, F:f) is much greater than that of generating recombinant sample 2 (A:a1, B:b1, F:f). This shows that recombinant sample 1 (A:a1, B:b1, C:c2) is more consistent with the structure of the original sample (A:a1, B:b1, C:f) than recombinant sample 2 (A:a1, B:b1, F:f), and the recombinant sample is more consistent with the sample generated in actual application. Therefore, the recombinant sample generation method of this application can guarantee the quality of the sample.
[0120] The recombined samples then form a recombined sample set. This recombined sample set is used to expand the sample data, increasing the number of samples used for model training and ensuring the training effect of the model.
[0121] Furthermore, it should be noted that the distribution of the number of samples of each class in the preferred recombined sample set should be consistent with the distribution of the number of samples of each class in the original sample set. For example, if there are N original samples P, the number of samples corresponding to each element quantity in the N original samples can be counted. If the possible sample types (or sample label types) in the N samples are n1, n2, n3…ni (i∈N+), and the original sample numbers corresponding to sample types n1, n2, n3…ni are m1, m2, m3…mi (i∈N+), respectively. Where:
[0122]
[0123] It is understandable that keeping the quantity distribution of each class of samples in the reconstructed sample set consistent with that in the original sample set ensures the correlation between the reconstructed sample set and the original sample set, thereby guaranteeing the training effect.
[0124] Step S22: Filter the recombinant samples generated by combination, remove recombinant samples with homologous elements, and generate the recombinant sample set based on the recombinant samples that have passed the filtering.
[0125] It should be noted that in this embodiment, after generating the recombinant samples, the initial recombinants can be screened, that is, recombinant samples containing homologous elements are deleted. Homologous elements are elements that have a mutation relationship or variant elements generated based on the same basic element. For example, assuming that variant elements A:a1 and A:a2 are obtained by mutating a basic element A:a, the basic element A:a and variant element A:a1 have a mutation relationship, as do the basic element A:a and variant element A:a2. Since variant elements A:a1 and A:a2 are obtained by mutating the same basic element A:a, A:a and A:a1, A:a and A:a2, and A:a1 and A:a2 are all homologous elements. It can be understood that the expressive power of homologous elements in a sample is similar, so the quality of samples containing homologous elements is relatively low, and therefore they can be screened out. The recombinant samples after screening constitute the recombinant sample set.
[0126] In one feasible implementation, the step of using the clustering core based on the second clustering model to screen the recombinant samples in the recombinant sample set to obtain the recombinant sample set includes:
[0127] Step S310: Cluster the original samples in the original sample set to obtain the clustering result of the original sample set;
[0128] Step S320: Determine the target clustering core based on the clustering results of the original sample set, and use the target clustering core as the clustering core of the second clustering model;
[0129] Step S330: Based on the second clustering model, the recombinant samples in the recombinant sample set are clustered to obtain a second clustering result. The second clustering result includes each group obtained based on the target clustering core cluster.
[0130] Step S340: Remove weakly associated recombinant samples from the recombinant sample set to obtain the target recombinant sample set, wherein the weakly associated recombinant samples are recombinant samples whose distance from the target cluster core of their respective group is greater than a preset distance threshold.
[0131] For example, clustering can be achieved by grouping the original samples in the original sample set. This grouping process can involve vectorizing each original sample and using it as input to a k-means clustering model, which then performs the clustering. The specific clustering process can be referenced above or existing solutions, and will not be elaborated further here. The initial clustering core for clustering the original sample set is determined based on the average distance between samples in the original sample set. For instance, any original sample in the original sample set can be used as a basic original sample. The distances between the basic original sample and other original samples in the original sample set are determined to obtain a basic distance set. The distances in the basic distance set are the distances between the basic original sample and each other original sample. If the proportion of the number of distances in the basic distance set exceeding the average distance to the total number of distances in the basic distance set exceeds a preset proportion (e.g., 0.6), then the basic original sample is used as the initial clustering core. It is understandable that determining the initial clustering core is actually determining the k value of the k-means clustering model when clustering the original sample set based on the k-means clustering model.
[0132] The clustering results of the original sample set will include multiple clusters and the cluster cores of each cluster (i.e., the cluster cores of the original sample set's clustering results). The cluster cores of the original sample set can be directly used as the target cluster cores, or the sample mean of each cluster in the original sample set's clustering results can be used as the target cluster cores. These target cluster cores are also the cluster cores of the second clustering model (or the initial cluster cores of the second clustering model).
[0133] After determining the clustering cores of the second clustering model, the recombined sample set is then clustered based on the second clustering model. It should be noted that the second clustering model can also be a k-means clustering model; the difference is that the clustering cores remain unchanged. That is, the second clustering model performs clustering once based on the clustering cores (the initial clustering cores, which are also the target clustering cores) to obtain the second clustering result. The second clustering result includes different groups generated based on the target clustering cores. Then, samples weakly correlated with the target clustering cores are removed from the recombined sample set to obtain the recombined sample set. It can be understood that weakly correlated samples refer to recombined samples whose distance from the clustering core of their respective group in the second clustering result is greater than a preset distance threshold. For example, refer to... Figure 6The diagram illustrates the screening process based on the second clustering model. Black dots represent cluster cores (target cluster cores), white dots represent recombinant samples, and r is a preset distance threshold. If the distance between a recombinant sample and the cluster core of its group exceeds the preset distance threshold (understandably, if a recombinant sample is classified in this group, it means that the recombinant sample is closest to the cluster core of its group relative to other cluster cores; therefore, screening only requires comparing the distance between the recombinant sample and the cluster core of its group), then the recombinant sample can be removed from the recombinant sample set to obtain the target recombinant sample set.
[0134] Reference Figure 3 Based on the first embodiment of this application, a second embodiment of this application is proposed. The parts identical to those described above can be referred to the above content and will not be repeated here. After the step of generating a recombinant sample set based on each group of the first clustering result, the method includes:
[0135] Step A10: Add the recombined samples from the target recombined sample set as original samples to the original sample set to obtain a new original sample set;
[0136] Step A20: Based on the new original sample set, return to the step of decomposing the original samples in the original sample set into a basic element set, until the number of original samples in the original sample set reaches a preset threshold.
[0137] For example, in this embodiment, after obtaining the recombined sample set, samples in the recombined sample set (i.e., target recombined samples) can be added to the original sample set as original samples (or a portion of samples can be selected from the recombined sample set as original samples) to obtain a new original sample set. Based on the new original sample set, the step of decomposing the original samples in the original sample set into features to obtain a basic element set and subsequent steps are performed until the number of original samples in the original sample set reaches a preset threshold, i.e., sufficient sample data is generated to expand the sample quantity.
[0138] For example, if a portion of the recombined samples are selected as original samples and added to the original sample set, the number of recombined samples is updated to Xw, where W is the total number of recombined samples obtained. Then, a% of the recombined samples are randomly selected and added to the original samples. This process is repeated until X recombined samples are output, or until X+b original samples are output, where b is the number of original samples in the initial original sample set. Where a = [(Xw) / W]*100.
[0139] Please see Figure 7 Furthermore, this application embodiment also provides a sample data generation device 100, the sample data generation device 100 comprising:
[0140] The first clustering module 10 is used to cluster the elements in the preset merged element set using a preset first clustering model to obtain the first clustering result, wherein the preset merged element set is determined by the features of the original samples in the original sample set.
[0141] The recombination module 20 is used to generate a recombination sample set based on each group of the first clustering result, wherein the recombination sample set includes recombination samples, and the constituent elements of the recombination samples come from the same group of the first clustering result.
[0142] The second clustering module 30 is used to filter the recombinant sample set based on the clustering core of the second clustering model, and remove recombinant samples in the recombinant sample set whose correlation with the clustering core is less than a preset correlation threshold, to obtain the target recombinant sample set, wherein the clustering core is determined based on the original samples in the original sample set.
[0143] Optionally, the second clustering module 30 is also used for:
[0144] Clustering is performed on the original samples in the original sample set to obtain the clustering result of the original sample set;
[0145] The target cluster core is determined based on the clustering results of the original sample set, and the target cluster core is used as the cluster core of the second clustering model.
[0146] The recombinant samples in the recombinant sample set are clustered based on the second clustering model to obtain a second clustering result, which includes each group obtained based on the target clustering core cluster.
[0147] The weakly associated recombinant samples in the recombinant sample set are removed to obtain the target recombinant sample set, wherein the weakly associated recombinant samples are those whose distance from the target cluster core of their respective group is greater than a preset distance threshold.
[0148] Optionally, the sample data generation device 100 further includes a mutation disassembly module 40, which is used to further:
[0149] The original samples in the original sample set are decomposed into a basic element set by feature decomposition.
[0150] Mutate the basic elements in the basic element set to generate a mutated element set;
[0151] The set of basic elements and the set of mutated elements are combined to obtain the preset merged element set.
[0152] Optionally, the first clustering module 10 is further configured to:
[0153] The elements in the preset merged element set are vectorized to obtain an element vector set. The element vector set is composed of element vectors. The element vector of any element is generated by the element and other elements in the pre-constructed knowledge graph that have a relationship with the element.
[0154] The first clustering result is obtained by clustering the element vector set using the first clustering model.
[0155] Optionally, the combination module 20 is further configured to:
[0156] Elements are selected from the set of elements belonging to the same group in the first clustering result to generate recombined samples, wherein the number distribution of each class of samples in the recombined sample set is consistent with the number distribution of each class of samples in the original sample set;
[0157] The recombinant samples generated by the combination are screened to remove recombinant samples with homologous elements, and the recombinant sample set is generated based on the screened recombinant samples.
[0158] Optionally, when the original sample is a sample of a data type, the basic element includes a feature field and the feature value of the feature field; when the original sample is a text type sample, the basic element is text segmentation; the set of mutated elements consists of mutated elements; and the decomposition and mutation module 40 is further used for:
[0159] Modify the feature values of the feature fields in the basic element to generate a variant element;
[0160] Alternatively, the word order features of the text segmentation, the part-of-speech features of the text segmentation, or the text segmentation itself can be changed in the basic element to generate a variant element.
[0161] Optionally, the sample data generation device 100 further includes a loop module 50, which is used for:
[0162] The recombined samples in the target recombined sample set are added to the original sample set as original samples to obtain a new original sample set;
[0163] Based on the new original sample set, the step of decomposing the original samples in the original sample set into a basic element set is repeated until the number of original samples in the original sample set reaches a preset threshold.
[0164] The sample data generation apparatus provided in this application employs the sample data generation method described in the above embodiments, aiming to solve the technical problem of poor sample quality obtained by current training sample augmentation methods. Compared with the prior art, the beneficial effects of the sample data generation apparatus provided in this application are the same as those of the sample data generation method provided in the above embodiments, and other technical features in this sample data generation apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.
[0165] In addition, to achieve the above objectives, this application also provides an electronic device, which includes: a memory, a processor, and a sample data generation program stored in the memory and executable on the processor. When the sample data generation program is executed by the processor, it implements the steps of the sample data generation method as described above.
[0166] The specific implementation of the electronic device in this application is basically the same as the various embodiments of the above-described sample data generation method, and will not be repeated here.
[0167] In addition, to achieve the above objectives, this application also provides a storage medium storing a sample data generation program, which, when executed by a processor, implements the steps of the sample data generation method described above.
[0168] The specific implementation of the storage medium in this application is basically the same as the various embodiments of the above-described sample data generation method, and will not be repeated here.
[0169] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0170] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0171] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0172] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A method for generating sample data, characterized in that, The sample data generation method includes: The first clustering result is obtained by clustering the elements in the preset merged element set using a preset first clustering model, wherein the preset merged element set is determined by the features of the original samples in the original sample set. A recombined sample set is generated for each group based on the first clustering result, wherein the recombined sample set includes recombined samples, and the constituent elements of the recombined samples come from the same group in the first clustering result. The original samples in the original sample set are clustered based on the initial clustering core to obtain the clustering result of the original sample set. The initial clustering core is determined based on the average distance between the samples in the original sample set. The target cluster core is determined based on the clustering results of the original sample set, and the target cluster core is used as the cluster core of the second clustering model. The recombinant samples in the recombinant sample set are clustered based on the second clustering model to obtain a second clustering result, which includes each group obtained based on the target clustering core cluster. Weakly associated recombinant samples are removed from the recombinant sample set to obtain the target recombinant sample set, wherein the weakly associated recombinant samples are those whose distance from the target cluster core of their respective group is greater than a preset distance threshold.
2. The sample data generation method as described in claim 1, characterized in that, Before the step of clustering the elements in the preset merged element set using a preset first clustering model to obtain the first clustering result, the method further includes: The original samples in the original sample set are decomposed into a basic element set by feature decomposition. Mutate the basic elements in the basic element set to generate a mutated element set; The set of basic elements and the set of mutated elements are combined to obtain the preset merged element set.
3. The sample data generation method as described in claim 2, characterized in that, The step of clustering the elements in the preset merged element set using a preset first clustering model to obtain the first clustering result includes: The elements in the preset merged element set are vectorized to obtain an element vector set, wherein the element vector set is composed of element vectors, and the element vector of any element is generated by the element and other elements in the pre-constructed knowledge graph that have a relationship with the element. The first clustering result is obtained by clustering the element vector set using the first clustering model.
4. The sample data generation method according to any one of claims 1 to 3, characterized in that, The step of generating a recombinant sample set for each group based on the first clustering result includes: Elements are selected from the set of elements belonging to the same group in the first clustering result to generate recombined samples, wherein the number distribution of each class of samples in the recombined sample set is consistent with the number distribution of each class of samples in the original sample set; The recombinant samples generated by the combination are screened to remove recombinant samples with homologous elements, and the recombinant sample set is generated based on the screened recombinant samples.
5. The sample data generation method as described in claim 2, characterized in that, When the original sample is a sample of a data type, the basic element includes a feature field and the feature value of the feature field. When the original sample is a text type sample, the basic element is text segmentation. The set of mutated elements consists of mutated elements. The step of generating the set of mutated elements by mutating the basic elements in the set of basic elements includes: Modify the feature values of the feature fields in the basic element to generate a variant element; Alternatively, the word order features of the text segmentation, the part-of-speech features of the text segmentation, or the text segmentation itself can be changed in the basic element to generate a variant element.
6. The sample data generation method as described in claim 2, characterized in that, After the step of removing weakly associated recombination samples from the recombination sample set to obtain the target recombination sample set, the method further includes: The recombined samples in the target recombined sample set are added to the original sample set as original samples to obtain a new original sample set; Based on the new original sample set, the step of decomposing the original samples in the original sample set into a basic element set is repeated until the number of original samples in the original sample set reaches a preset threshold.
7. A sample data generation device, characterized in that, The sample data generation device includes: The first clustering module is used to cluster the elements in the preset merged element set using a preset first clustering model to obtain the first clustering result, wherein the preset merged element set is determined by the features of the original samples in the original sample set. A recombination module is used to generate a recombination sample set based on each group of the first clustering result, wherein the recombination sample set includes recombination samples, and the constituent elements of the recombination samples come from the same group of the first clustering result. The second clustering module is used to cluster the original samples in the original sample set based on the initial clustering core to obtain the clustering result of the original sample set, wherein the initial clustering core is determined based on the average distance between the samples in the original sample set; to determine the target clustering core based on the clustering result of the original sample set, and to use the target clustering core as the clustering core of the second clustering model; to cluster the recombined samples in the recombined sample set based on the second clustering model to obtain the second clustering result, wherein the second clustering result includes each group obtained by clustering based on the target clustering core; and to remove weakly associated recombined samples in the recombined sample set to obtain the target recombined sample set, wherein the weakly associated recombined samples are those whose distance from the target clustering core of their respective group is greater than a preset distance threshold.
8. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a sample data generation program stored in the memory and executable on the processor, wherein: when the sample data generation program is executed by the processor, it implements the steps of the sample data generation method as described in any one of claims 1 to 6.
9. A storage medium, characterized in that, The storage medium stores a sample data generation program, which, when executed by a processor, implements the steps of the sample data generation method as described in any one of claims 1 to 6.