Data cutting method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data cutting method and device, electronic equipment and a storage medium, and the method comprises the steps: selecting a target sample from a plurality of samples of a sample data set, and obtaining a target label of the target sample; determining a plurality of neighborhood samples of the target sample in the sample data set, and obtaining an adjacent feature matrix according to feature codes of the plurality of neighborhood samples; performing feature propagation processing on the adjacent feature matrix based on the feature codes to obtain an updated adjacent feature matrix, and calculating an adjacent predicted value of each neighborhood sample based on the updated adjacent feature matrix; based on the adjacency prediction value and the label probability value of the target label, calculating to obtain a similarity estimation value between the plurality of neighborhood samples and the target sample; and carrying out data cutting on the sample data set based on the similarity estimation value to obtain a core data set, and carrying out label re-labeling on the target sample of the core data set based on the similarity estimation value so as to improve the accuracy of carrying out data cutting in the sample data set to retain important sample data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data clipping method, device, electronic device and storage medium. Background Art

[0002] The explosive growth of data volume is one of the key factors that have enabled deep neural networks to succeed in various applications. However, model training on large-scale data is not only time-consuming, but also requires huge energy costs. In fact, a considerable portion of these training data is redundant, which means that redundant data can be removed through data pruning without affecting model performance to retain core data with high information content.

[0003] In the related art, during the data clipping process, each sample data is usually labeled with a predetermined gradient value to reflect the importance of the sample data, so that the sample data with high importance is selected for data clipping to improve the efficiency of model training. However, since the gradient values ​​of some noise samples themselves are relatively large, there is a situation where the noise samples are labeled as important sample data and saved, resulting in low accuracy in data clipping to retain important sample data. Summary of the invention

[0004] The embodiments of the present application provide a data clipping method, device, electronic device and storage medium, which can improve the accuracy of data clipping in a sample data set to retain important sample data.

[0005] To achieve the above objective, a first aspect of an embodiment of the present application provides a data clipping method, the method comprising: Selecting a target sample from multiple samples in a sample data set, and obtaining a target label for the target sample; Determine a plurality of neighboring samples of the target sample in the sample data set, and obtain an adjacency feature matrix according to feature codes of the plurality of neighboring samples; Performing feature propagation processing on the adjacency feature matrix based on the feature coding to obtain an updated adjacency feature matrix, and calculating an adjacency prediction value of each of the neighborhood samples based on the updated adjacency feature matrix; Based on the neighboring prediction value and the label probability value of the target label, a similarity estimation value between the plurality of neighborhood samples and the target sample is calculated; The sample data set is pruned based on the similarity estimation value to obtain a core data set, and at least one target sample in the core data set is relabeled based on the similarity estimation value.

[0006] In some embodiments, determining a plurality of neighboring samples of the target sample in the sample data set includes: Obtaining a feature code for each of the samples; Calculate the cosine distance between the feature code of the target sample and the feature codes of other samples; Based on the order of the cosine distance values, a preset number of neighboring samples are selected from the multiple samples.

[0007] In some embodiments, the performing feature propagation processing on the adjacency feature matrix based on the feature coding to obtain an updated adjacency feature matrix includes: Generate a transition matrix based on the cosine distance value between every two feature codes; Obtaining a diagonal matrix based on the cumulative sum of the row vector elements in the transition matrix; The adjacency feature matrix is ​​updated based on the diagonal matrix and the transition matrix to obtain the updated adjacency feature matrix.

[0008] In some embodiments, the updating the adjacency feature matrix based on the diagonal matrix and the transition matrix to obtain the updated adjacency feature matrix includes: Multiplying the diagonal matrix, the transition matrix, and the diagonal matrix to obtain an adjacency matrix; Based on the sum of the unit matrix and the adjacency matrix, multiplying it by the adjacency feature matrix, the updated adjacency feature matrix is ​​obtained.

[0009] In some embodiments, the calculating the adjacency prediction value of each of the neighborhood samples based on the updated adjacency feature matrix includes: Get the fully connected layer parameters and probability activation function; Obtaining prediction parameters based on the product of the updated adjacency feature matrix and the fully connected layer parameters; Substituting the prediction parameters into the probability activation function, the adjacent prediction value of each of the neighborhood samples is obtained.

[0010] In some embodiments, the calculating, based on the neighboring prediction value and the label probability value of the target label, to obtain the estimated similarity value between the plurality of neighborhood samples and the target sample comprises: Selecting a target adjacent prediction value from multiple adjacent prediction values ​​one by one; Calculating a divergence value between the target adjacent prediction value and the label probability value; All the divergence values ​​are averaged to obtain the similarity estimation value.

[0011] In some embodiments, there are multiple target samples, and relabeling at least one sample in the sample data set based on the similarity estimation value includes: Based on the numerical ranking of the similarity estimation values, performing stratified sampling processing on the similarity estimation values ​​of the plurality of target samples to obtain a core data set including at least one target sample; The sample label of the target sample in the core data set is obtained, and when the similarity estimation value indicates that the sample label corresponding to the target sample is incorrectly labeled, the sample label of the target sample is relabeled.

[0012] In some embodiments, performing stratified sampling processing according to the similarity estimation values ​​of the plurality of target samples to obtain cropped data containing at least one target sample includes: dividing the plurality of similarity estimation values ​​into a plurality of estimation value clusters, each of the estimation value clusters including a plurality of the similarity estimation values; Based on the numerical ranking of the plurality of similarity estimation values, selecting similarity estimation values ​​of a preset number of clusters from each of the estimation value clusters one by one as target similarity estimation values, wherein the target similarity estimation value is less than the preset similarity value; The target samples corresponding to all the target similarity estimation values ​​are taken as the core data set.

[0013] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a data clipping device, the device comprising: A target sample selection module is used to select a target sample from multiple samples in a sample data set and obtain a target label of the target sample; A neighborhood sample selection module, used to determine a plurality of neighborhood samples of the target sample in the sample data set, and obtain an adjacency feature matrix according to feature codes of the plurality of neighborhood samples; A prediction value calculation module, used for performing feature propagation processing on the adjacency feature matrix based on the feature coding to obtain an updated adjacency feature matrix, and calculating the adjacency prediction value of each of the neighborhood samples based on the updated adjacency feature matrix; A similarity calculation module, configured to calculate a similarity estimation value between the plurality of neighborhood samples and the target sample based on the neighboring prediction value and the label probability value of the target label; A data clipping module is used to perform data clipping on the sample data set based on the similarity estimation value to obtain a core data set, and to relabel at least one target sample in the core data set based on the similarity estimation value.

[0014] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the data clipping method described in the first aspect when executing the computer program.

[0015] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, it implements the data clipping method described in the first aspect above.

[0016] The data clipping method, device, electronic device and storage medium proposed in the embodiments of the present application include: first, selecting a target sample from multiple samples in a sample data set and obtaining a target label of the target sample; then, determining multiple neighborhood samples of the target sample in the sample data set, and obtaining an adjacency feature matrix based on the feature codes of the multiple neighborhood samples; then, performing feature propagation processing on the adjacency feature matrix based on the feature codes to obtain an updated adjacency feature matrix, and calculating the adjacency prediction value of each neighborhood sample based on the updated adjacency feature matrix; next, calculating the similarity estimation value between the multiple neighborhood samples and the target sample based on the adjacency prediction value and the label probability value of the target label; finally, performing data clipping on the sample data set based on the similarity estimation value to obtain a core data set, and relabeling the label of at least one target sample in the core data set based on the similarity estimation value. The embodiment of the present application selects samples from the sample data set one by one as target samples, and further targets multiple neighboring samples closest to the target samples, performs feature propagation processing using feature encoding between neighboring samples, so as to effectively mine data information between neighboring samples, and then accurately calculates the neighboring prediction value of whether each neighboring sample is a noise sample based on the updated neighboring feature matrix after the feature propagation processing to correct the annotation label of the neighboring sample, and then accurately corrects the data annotation of the target sample in combination with the similarity between the neighboring sample and the target sample, thereby improving the accuracy of the target sample label annotation, and then uses the accurate similarity estimation value of the target sample to select a target sample with a small similarity estimation value, that is, a target sample with high importance, from multiple target samples in multiple sample data sets as a core data set, so as to improve the accuracy of data clipping in the sample data set to retain important sample data, and uses the similarity estimation value to re-label the data of erroneous target samples in the core data set, thereby improving the reliability and accuracy of the selected core data set.

[0017] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of a data clipping method provided in one embodiment of the present application.

[0019] Figure 2 yes Figure 1 Flow chart of step 102 in FIG.

[0020] Figure 3 yes Figure 1 Flow chart of step 103 in FIG.

[0021] Figure 4 yes Figure 3 Flow chart of step 303 in FIG.

[0022] Figure 5 yes Figure 1 Another flow chart of step 102 in FIG.

[0023] Figure 6 yes Figure 1 Flow chart of step 103 in FIG.

[0024] Figure 7 yes Figure 1 Flow chart of step 104 in FIG.

[0025] Figure 8 yes Figure 7 Flowchart of step 701 in FIG.

[0026] Fig. 9 This is a flow chart of a data clipping method provided in another embodiment of the present application.

[0027] Fig.10 It is a schematic diagram of the process framework of a data clipping method provided in yet another embodiment of the present application.

[0028] Fig.11 This is a schematic diagram of the process structure of another data clipping method provided by yet another embodiment of the present application.

[0029] Fig.12 It is a structural schematic diagram of a data clipping device provided in one embodiment of the present application.

[0030] Fig.13 It is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0034] The explosive growth of data volume is one of the key factors that have enabled deep neural networks to succeed in various applications. However, model training on large-scale data is not only time-consuming, but also requires huge energy costs. In fact, a considerable portion of these training data is redundant, which means that redundant data can be removed through data pruning without affecting model performance to retain core data with high information content.

[0035] There has been a lot of research on data pruning, which uses a variety of carefully designed metrics to identify important samples, including distribution distance, uncertainty, loss, and gradient. Although these methods are effective in their respective scenarios, they are limited in scope and usually rely on the prior assumption that the data is perfectly labeled. For example, some methods assume that samples with large loss values ​​are difficult samples that are critical to improving model performance, while samples with small loss values ​​are easy samples that can be removed.

[0036] However, when the data does not meet the assumption of being perfectly labeled (that is, the data set contains mislabeled noise samples), samples with larger gradient values ​​may be mislabeled samples. From the perspective of robustness, samples with smaller losses will be more conducive to improving the robustness of the model. In addition, in real-world scenarios, data collection usually involves complex processes, such as crowdsourcing labeling, semi-automatic labeling, etc. These labeling methods usually have mislabeling phenomena, and basically do not meet the prior assumption that the data is perfectly labeled, which makes it possible to label noise samples as important sample data for preservation, resulting in low accuracy in data clipping to retain important sample data.

[0037] In order to improve the accuracy of data clipping in a sample data set to retain important sample data, an embodiment of the present application selects samples from the sample data set one by one as target samples, and further targets multiple neighborhood samples closest to the target sample, uses feature encoding between neighborhood samples to perform feature propagation processing to effectively mine data information between neighborhood samples, and then accurately calculates the adjacency prediction value of whether each neighborhood sample is a noise sample based on the updated adjacency feature matrix after feature propagation processing to correct the labeling label of the neighborhood sample, and then accurately corrects the data labeling of the target sample in combination with the similarity between the neighborhood sample and the target sample, thereby improving the accuracy of the target sample label annotation, and then uses the accurate similarity estimation value of the target sample to select a target sample with a small similarity estimation value, that is, a target sample with high importance, from multiple target samples in multiple sample data sets as a core data set, so as to improve the accuracy of data clipping in the sample data set to retain important sample data, and uses the similarity estimation value to re-label the data of the erroneous target samples in the core data set, thereby improving the reliability and accuracy of the selected core data set.

[0038] The data clipping method, device, electronic device and storage medium provided in the embodiments of the present application will be further described below. The data clipping method provided in the embodiments of the present application can be applied to any server or intelligent processing terminal, etc.

[0039] The data clipping method in the embodiment of the present application will be described in detail below. Figure 1 , is an optional flow chart of the data clipping method provided in the embodiment of the present application, Figure 1 The method may include but is not limited to steps 101 to 105. It can also be understood that this embodiment Figure 1 The order of step 101 to step 105 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0040] Step 101: Select a target sample from multiple samples in a sample data set, and obtain a target label of the target sample.

[0041] The following is a detailed description of step 101.

[0042] In some embodiments, in response to data annotation for a sample data set containing multiple samples to facilitate subsequent data tailoring, Multiple samples of Select samples one by one as target samples , and obtain the target sample Target label .

[0043] Among them, for classification tasks, the selection of data sets is a hot topic in the field of machine learning. This embodiment is aimed at a sample data set and a core dataset ,in represents the input sample, Representation sample Labels (including noisy labels), and are the number of samples in the two data sets respectively. Definition is the feature extractor, for Classifier of the class.

[0044] The purpose of the data clipping method provided in the embodiment of the present application is to extract the sample data set from the Select the most representative core data set , so that the core data set can be used The model obtained by training Using the entire sample dataset The model obtained by training It has closer generalization performance, thereby improving training efficiency while ensuring training accuracy.

[0045] Step 102: Determine a plurality of neighboring samples of the target sample in the sample data set, and obtain an adjacency feature matrix according to feature codes of the plurality of neighboring samples.

[0046] The following is a detailed description of step 102.

[0047] In some embodiments, when determining a target sample Finally, in order to accurately label the target sample, it is necessary to use the approximation between the target sample’s neighborhood samples and the target sample for calculation. Therefore, it is necessary to first determine the multiple neighborhood samples closest to the target sample from the multiple samples in the sample data set. , and based on the multiple field samples The feature encoding of , as described below.

[0048] Reference Figure 2 , determining multiple neighborhood samples of a target sample in a sample data set, including the following steps 201 to 203.

[0049] Step 201: Obtain the feature code of each sample.

[0050] Step 202: Calculate the cosine distance between the feature code of the target sample and the feature codes of other samples.

[0051] Step 203: Based on the order of the cosine distance values, select a preset number of neighboring samples from multiple samples.

[0052] Steps 201 to 203 are described in detail below.

[0053] In some embodiments, when obtaining a sample data set And determine the target sample After that, we also get the sample data set Each sample (including , etc.) in the feature space (including Corresponding ,and Corresponding etc).

[0054] Then, the K nearest neighbor algorithm is used to calculate the cosine distance value between the feature code of the target sample and the feature code of each other sample, which is used to characterize the distance between the features, as shown in the following formula (1).

[0055]

[0056] in, represents the transposed symbol. Then, based on the target sample The samples are sorted by the cosine distance values ​​between each other, and a preset number of neighboring samples are selected from multiple samples as shown in the following formula (2).

[0057]

[0058] in Defined as target sample Multiple neighborhood samples in the feature space, It is expressed as K nearest neighbor algorithm (i.e. formula (1)), Represents a set of samples in the feature space.

[0059] Through the above steps 201 to 203, the cosine distance value between the encoding of the features of the target sample and other samples in the sample data set is used to accurately determine the multiple neighborhood samples closest to the target sample, so as to facilitate the subsequent calculation of the labeling result of the target sample using the similarity between the neighborhood samples and the target sample, thereby improving the accuracy of data labeling.

[0060] After KNN is used to calculate multiple neighborhood samples of the target Then, the target sample and multiple neighborhood samples are Defined to a graph Then based on multiple neighborhood samples The feature encoding of ,in is the number of neighborhood samples, The dimension to encode the features.

[0061] Step 103: Perform feature propagation processing on the adjacency feature matrix based on feature coding to obtain an updated adjacency feature matrix, and calculate the adjacency prediction value of each neighborhood sample based on the updated adjacency feature matrix.

[0062] Step 103 is described in detail below.

[0063] In some embodiments, when determining a target sample Multiple neighborhood samples of And the adjacency feature matrix Finally, in order to further mine the data information between neighborhood samples, the adjacency feature matrix is ​​coded based on the feature Perform feature propagation processing to obtain the updated adjacency feature matrix , as described below.

[0064] Reference Figure 3 , based on feature coding, feature propagation processing is performed on the adjacent feature matrix to obtain an updated adjacent feature matrix, including the following steps 301 to 303.

[0065] Step 301: Generate a transition matrix based on the cosine distance value between every two feature codes.

[0066] Step 302: Obtain a diagonal matrix based on the cumulative sum of row vector elements in the transition matrix.

[0067] Step 303: Update the adjacency feature matrix based on the diagonal matrix and the transition matrix to obtain an updated adjacency feature matrix.

[0068] Steps 301 to 303 are described in detail below.

[0069] In some embodiments, when determining the adjacency feature matrix After that, we first generate a transition matrix based on the cosine distance value of each two feature codes. As shown in the following formula (3).

[0070]

[0071] in, express No. The first The feature encoding of the neighborhood samples.

[0072] Next, based on the transition matrix The cumulative sum of the row vector elements in the diagonal matrix As shown in the following formula (4).

[0073]

[0074] Among them, the diagonal matrix It is also represented as a degree matrix.

[0075] Then further based on the diagonal matrix and the transition matrix Adjacency feature matrix Update to get the updated adjacency feature matrix , as described below.

[0076] Reference Figure 4 , updating the adjacency feature matrix based on the diagonal matrix and the transition matrix to obtain an updated adjacency feature matrix, including the following steps 401 to 402.

[0077] Step 401: Multiply the diagonal matrix, the transition matrix and the diagonal matrix to obtain an adjacency matrix.

[0078] Step 402: Based on the sum of the identity matrix and the adjacency matrix, multiply it by the adjacency feature matrix to obtain an updated adjacency feature matrix.

[0079] Steps 401 to 402 are described in detail below.

[0080] In some embodiments, based on the adjacency feature matrix Get the diagonal matrix and the transition matrix Then, multiply the diagonal matrix by the negative power of , transition matrix and the negative half power of the diagonal matrix , to obtain the adjacency matrix As shown in the following formula (5).

[0081]

[0082] Then based on the adjacency matrix Adjacency feature matrix Perform feature propagation, that is, based on the unit matrix and the adjacency matrix The sum of , then multiplied by the adjacency feature matrix , get the updated adjacency feature matrix As shown in the following formula (6).

[0083]

[0084] Through the above steps 301 to 303 and steps 401 to 402, a transition matrix is ​​generated using an adjacency feature matrix composed of feature codes of multiple adjacent samples, and a diagonal matrix and an adjacency matrix are further generated in sequence using the transition matrix, so that the adjacency matrix can be used to fully mine the data information between multiple adjacent samples, so as to facilitate the subsequent calculation of the prediction value of whether each adjacent sample is noise by updating the adjacency feature matrix, thereby improving the accuracy of data labeling.

[0085] After getting the updated adjacency feature matrix After that, we will further use the updated adjacency feature matrix Calculate the adjacent prediction value of whether each neighborhood sample is a noise sample , as described below.

[0086] Reference Figure 5 , calculating the adjacency prediction value of each neighborhood sample based on the updated adjacency feature matrix, including the following steps 501 to 503.

[0087] Step 501: Obtain fully connected layer parameters and probability activation function.

[0088] Step 502: Obtain prediction parameters based on the product of the updated adjacency feature matrix and the fully connected layer parameters.

[0089] Step 503: Substitute the prediction parameters into the probability activation function to obtain the neighbor prediction value of each neighborhood sample.

[0090] Steps 501 to 503 are described in detail below.

[0091] In some embodiments, when performing feature propagation to obtain an updated adjacency feature matrix In addition, the fully connected layer parameters are obtained And the probability activation function .

[0092] Among them, the parameters of the fully connected layer include the weight matrix and the bias vector. The size of the weight matrix is ​​the number of input neurons multiplied by the number of output neurons, and each element represents the connection weight between the input and output neurons; the size of the bias vector is the same as the number of output neurons, and each element represents the bias value of the output neuron. When calculating, the input vector is first multiplied by the weight matrix, then the bias vector is added, and finally a nonlinear transformation is performed through the activation function to produce the output. The number of parameters of the fully connected layer is usually large, especially between the input layer and the output layer, which makes the fully connected layer prone to overfitting when processing large-scale data, so regularization techniques are often required to control the complexity of the model.

[0093] Probabilistic Activation Function It is an activation function commonly used in multi-classification problems. It converts a real-valued vector into a probability distribution so that the value of each element is between 0 and 1, and the sum of all elements is 1. Specifically, for each element in the input vector, the Softmax function calculates its exponential value and divides it by the sum of the exponential values ​​of all elements of the entire vector to obtain a probability value. This process ensures that the output can be interpreted as a probability distribution, so that the output of the model can be directly used for probability interpretation and classification decisions. In deep learning, the Softmax function is often used in the output layer of a neural network, especially when dealing with multi-classification tasks. It can convert the original output (logits) of the model into the probability of each category for classification prediction.

[0094] Next, using the results of feature propagation Perform label propagation, that is, based on updating the adjacency feature matrix and the fully connected layer parameters The product of gets the prediction parameter , and further the prediction parameter Substitute the probability activation function To obtain each neighborhood sample The adjacent prediction value of is shown in the following formula (7).

[0095]

[0096] Represents the network's response to neighborhood samples Whether the data is a predicted value for noisy data.

[0097] Through the above steps 501 to 503, the updated adjacency feature matrix obtained after feature propagation is used And the fully connected layer parameters of the combined model , and further use the probability activation function , label propagation of neighborhood samples is performed to accurately correct the label information (i.e., neighbor prediction value) of neighborhood samples by using the data information between domain samples, so as to facilitate the subsequent calculation of the similarity between the corrected neighbor prediction value of the domain sample and the target sample, so as to accurately determine the target sample annotation label.

[0098] Step 104: Based on the neighboring prediction value and the label probability value of the target label, the similarity estimation value between the multiple neighborhood samples and the target sample is calculated.

[0099] Step 104 is described in detail below.

[0100] In some embodiments, after obtaining multiple neighborhood samples The neighbor prediction value of After that, we will further use the neighbor prediction value and target sample exist Target tag under Type The label probability value , calculate multiple neighborhood samples and target sample The estimated similarity between , to determine the distribution difference between the target sample and the neighborhood samples, as described below.

[0101] Reference Figure 6 , based on the neighboring prediction value and the label probability value of the target label, the similarity estimation value between multiple neighborhood samples and the target sample is calculated, including the following steps 601 to 603.

[0102] Step 601: Select target adjacent prediction values ​​from multiple adjacent prediction values ​​one by one.

[0103] Step 602: Calculate the divergence value between the target neighbor prediction value and the label probability value.

[0104] Step 603: average all the divergence values ​​to obtain a similarity estimation value.

[0105] Steps 601 to 603 are described in detail below.

[0106] In some embodiments, after obtaining multiple neighborhood samples The neighbor prediction value of Then, predict the values ​​from multiple neighbors one by one Select the target neighbor prediction value , and then use the JS divergence formula to calculate the target adjacent prediction value and target sample The label probability value The divergence value between , and then further average all the divergence values ​​to obtain the target sample The estimated similarity between multiple neighborhood samples is shown in the following formula (8).

[0107]

[0108] Generally speaking, the smaller the similarity estimate value is, the more likely the target sample is to be correctly labeled. Conversely, the larger the similarity estimate value is, the more likely the target sample is to be incorrectly labeled.

[0109] It can be understood that Jensen-Shannon divergence (JS divergence) is an indicator to measure the similarity of two probability distributions. It is a symmetric version of Kullback-Leibler divergence (KL divergence) and its calculation formula is ,in is the average distribution of P and Q, and They are the KL divergence from P to M and from Q to M. The value range of JS divergence is between 0 and log(2). When the two distributions are exactly the same, JS divergence is 0; when the two distributions are completely different, that is, there is no overlap, the value of JS divergence tends to log(2).

[0110] Through the above steps 601 to 603, the average value of the JS divergence between the adjacent prediction values ​​of multiple adjacent samples obtained by feature propagation and label propagation and the label probability value of the target sample is used to accurately obtain the similarity estimation value that characterizes the distribution difference between the target sample and the adjacent samples, so as to use the similarity estimation value to accurately obtain the label annotation result of the target sample to distinguish the noise sample.

[0111] After obtaining the target sample The similarity estimate of After that, we will further extract the sample data set Select the next target sample To calculate the target sample , so as to facilitate the subsequent use of similarity estimates of multiple target samples to analyze the sample data set Perform data trimming to obtain a suitable core data set , and the data set (including the sample data set and core datasets ) to improve the accuracy of the label annotation of the dataset.

[0112] Step 105: based on the similarity estimation value, the sample data set is trimmed to obtain a core data set, and based on the similarity estimation value, at least one target sample in the core data set is relabeled.

[0113] Step 105 is described in detail below.

[0114] In order to further improve the accuracy of selecting the core data set from the sample data set, after obtaining the similarity estimation values ​​of multiple target samples, these similarity estimation values ​​will be used to perform data trimming on multiple samples in the sample data set to obtain the core data set. Then, in order to further improve the labeling accuracy of the target samples in the core data set, after selecting the core data set, the target sample data with abnormal labeling in the core data set will be relabeled according to the similarity estimation value, thereby improving the training efficiency of the subsequent model in the model training process based on the core data set, and also improving the reliability and accuracy. This will be further described below.

[0115] Reference Figure 7 , based on the similarity estimation value, the sample data set is trimmed to obtain a core data set, and based on the similarity estimation value, at least one target sample in the core data set is relabeled, including the following steps 701 to 702.

[0116] Step 701: Based on the numerical sorting of the similarity estimation values, stratified sampling is performed on the similarity estimation values ​​of multiple target samples to obtain a core data set containing at least one target sample.

[0117] Step 701 is described in detail below.

[0118] In some embodiments, after obtaining similarity estimates of multiple target samples, in order to ensure that samples in different intervals are taken into account when selecting data to improve the coverage of the trimmed data, a density-based sampling method is used, based on the numerical sorting of the similarity estimates, and a stratified sampling method is used to sample these similarity estimates to obtain a core data set containing multiple target samples, as described below.

[0119] Reference Figure 8 , based on the numerical sorting of the similarity estimation values, stratified sampling processing is performed on the similarity estimation values ​​of multiple target samples to obtain a core data set containing at least one target sample, including the following steps 801 to 803.

[0120] Step 801: Divide a plurality of similarity estimation values ​​into a plurality of estimation value clusters.

[0121] Step 802: Based on the numerical ranking of the plurality of similarity estimation values, similarity estimation values ​​of a preset number of clusters are selected one by one from each estimation value cluster as target similarity estimation values.

[0122] Step 803: Take the target samples corresponding to all target similarity estimation values ​​as the core data set.

[0123] Steps 801 to 803 are described in detail below.

[0124] In some embodiments, after obtaining similarity estimation values ​​of multiple target samples in the sample data set, these similarity estimation values ​​are divided into multiple estimation value clusters, each estimation value cluster includes multiple similarity estimation values. In this step, cluster division can be performed based on certain features, such as a threshold value of the similarity estimation value, the position order of the target samples, etc.

[0125] Then, based on the numerical sorting of multiple similarity estimation values, a preset number of clusters are selected from each estimation value cluster one by one, and similarity estimation values ​​whose similarity estimation values ​​are less than the preset similarity value are selected as target similarity estimation values, and target samples corresponding to all target similarity estimation values ​​are taken as core data sets, so as to ensure that the core data sets obtained after stratified sampling contain simple samples and difficult samples, and ensure that the similarity estimation values ​​of the selected target samples are low, that is, they correspond to more reliable target samples, so as to improve the accuracy of retaining important sample data in the core data sets obtained after trimming the data, and improve the coverage of sample data in the core data sets.

[0126] Step 702: Obtain the sample label of the target sample in the core data set. When the similarity estimation value indicates that the sample label corresponding to the target sample is incorrectly labeled, relabel the sample label of the target sample.

[0127] Step 702 is described in detail below.

[0128] In some embodiments, after data trimming is performed on the sample data set to obtain a core data set, the original sample labels of the target samples in the core data set are confirmed one by one. Then, when there is a similarity estimate value of a target sample label in the core data set (i.e., the similarity estimate value is too high), indicating that the corresponding sample label is incorrectly labeled, the target sample label is relabeled to update the sample label corresponding to the target sample label, thereby improving the accuracy of the target sample label labeling in the core data set, and further improving the training efficiency and accuracy of subsequent model training using the core data set.

[0129] Reference Fig. 9 and Fig.10 , is a schematic diagram of the process structure and process architecture of a data clipping method provided in an embodiment of the present application. Fig. 9 and Fig.10As shown in , after obtaining the sample data set of complete data, the sample data set is input into the encoder to obtain the target sample (i.e., the given sample) and multiple neighborhood samples of the target sample, and then the feature encoding between the neighborhood samples is used for feature propagation to fully explore the data information between the neighborhood samples; after feature propagation, the domain samples are label propagated (i.e., the domain prediction value of the neighborhood samples is calculated) to obtain the predicted label of the neighborhood samples, and then the distribution difference measure (i.e., the similarity estimate, also known as the difference score) between the target sample and the neighborhood sample is calculated based on the predicted label of the neighborhood sample. Then, after obtaining the original similarity estimates of multiple targets, the coverage is trimmed based on the similarity estimates and the stratified sampling method to obtain a subset (i.e., the core data set), and the similarity estimates are used to re-label the incorrect sample data in the core data set, so as to trim the data to obtain a core data set that only includes sample data that is highly important and reliable.

[0130] Reference Fig.11 , is a schematic diagram of the flow structure of another data clipping method provided in an embodiment of the present application. Fig.11 As shown in , after obtaining accurate sample label annotations and further obtaining a suitable subset (i.e., core data set), the core data set is input into the encoder to perform model training tasks for downstream tasks (including target classification, target detection, target segmentation, etc.) to reduce the overfitting phenomenon caused by the presence of noise samples in the selected data during the model training process, so as to improve the accuracy and training efficiency of the model training.

[0131] The data clipping method, device, electronic device and storage medium proposed in the embodiments of the present application include: first, selecting a target sample from multiple samples in a sample data set and obtaining a target label of the target sample; then, obtaining a feature code of each sample, calculating a cosine distance value between the feature code of the target sample and the feature code of other samples, sorting based on the size of the cosine distance value, selecting a preset number of neighboring samples from multiple samples, and obtaining an adjacency feature matrix according to the feature codes of multiple neighboring samples; then, generating a transition matrix based on the cosine distance value of every two feature codes, obtaining a diagonal matrix based on the cumulative sum of row vector elements in the transition matrix, accumulating and multiplying the diagonal matrix, the transition matrix and the diagonal matrix to obtain an adjacency matrix, multiplying the adjacency feature matrix by the sum of the unit matrix and the adjacency matrix to obtain an updated adjacency feature matrix, and obtaining fully connected layer parameters and a probability activation function, and performing an activation function based on the updated adjacency feature matrix and the fully connected layer. The product of the layer parameters is used to obtain the prediction parameters, and the prediction parameters are substituted into the probability activation function to obtain the neighbor prediction value of each neighborhood sample; next, the target neighbor prediction value is selected one by one from the multiple neighbor prediction values, the divergence value of the target neighbor prediction value and the label probability value is calculated, and all the divergence values ​​are averaged to obtain the similarity estimation value; finally, the multiple similarity estimation values ​​are divided into multiple estimation value clusters, each estimation value cluster includes multiple similarity estimation values, based on the numerical sorting of the multiple similarity estimation values, the similarity estimation values ​​of a preset number of clusters are selected from each estimation value cluster one by one as the target similarity estimation value, the target similarity estimation value is less than the preset similarity value, the target samples corresponding to all the target similarity estimation values ​​are taken as the core data set, and the sample labels of the target samples in the core data set are obtained. When the similarity estimation value indicates that the sample label corresponding to the target sample is incorrectly labeled, the sample label of the target sample is relabeled.

[0132] The embodiment of the present application targets each target sample one by one, and further targets multiple neighboring samples closest to the target sample, and uses the cosine distance value between the feature codes of the target sample and other samples in the sample data set to accurately determine the multiple neighboring samples closest to the target sample, so as to facilitate the subsequent calculation of the labeling result of the target sample using the similarity between the neighboring sample and the target sample, thereby improving the accuracy of data labeling; and, using the adjacency feature matrix composed of the feature codes of multiple adjacent samples to generate a transition matrix, and using the transition matrix to further generate a diagonal matrix and an adjacency matrix in sequence, so as to facilitate the use of the adjacency matrix to fully mine the data information between multiple adjacent samples, so as to facilitate the subsequent use of the updated adjacency feature matrix to calculate the predicted value of whether each adjacent sample is noise, thereby improving the accuracy of data labeling; and, using the updated adjacency feature matrix obtained after feature propagation And the fully connected layer parameters of the combined model , and further use the probability activation function , perform label propagation of neighborhood samples, and use the data information between domain samples to accurately correct the label information of neighborhood samples (i.e., neighbor prediction value), so as to facilitate the subsequent use of the corrected neighbor prediction value of the domain sample to calculate the similarity with the target sample, so as to accurately determine the target sample label; and use the average value of the JS divergence between the neighbor prediction value of multiple neighbor samples obtained by feature propagation and label propagation and the label probability value of the target sample to accurately obtain the similarity estimation value that characterizes the distribution difference between the target sample and the neighboring samples, so as to use the similarity estimation value to accurately obtain the label annotation result of the target sample to distinguish the noise sample; finally, based on the similarity estimation value, and using the stratified sampling method to maintain It is to ensure that the trimmed data obtained after stratified sampling contains simple samples and difficult samples, thereby ensuring that the core data set obtained after stratified sampling contains simple samples and difficult samples, and ensuring that the similarity estimate of the selected target samples is low, that is, corresponding to more reliable target samples, so as to improve the accuracy of retaining important sample data in the core data set obtained after trimming the data, and improve the coverage of sample data in the core data set, and update the label annotation of erroneous sample data in the core data set, so as to perform data trimming to obtain a core data set that only includes important and reliable sample data, so as to reduce the overfitting phenomenon caused by the presence of noise samples in the selected data during the model training process, so as to improve the accuracy and training efficiency of model training.

[0133] The present application also provides a data clipping device, which can implement the above data clipping method. Fig.12 , the device 1200 comprises: A target sample selection module 1210 is used to select a target sample from multiple samples in a sample data set and obtain a target label of the target sample; A neighborhood sample selection module 1220 is used to determine multiple neighborhood samples of a target sample in a sample data set, and obtain an adjacency feature matrix according to feature codes of the multiple neighborhood samples; A prediction value calculation module 1230 is used to perform feature propagation processing on the adjacency feature matrix based on feature coding to obtain an updated adjacency feature matrix, and calculate the adjacency prediction value of each neighborhood sample based on the updated adjacency feature matrix; A similarity calculation module 1240 is used to calculate similarity estimates between multiple neighborhood samples and a target sample based on the neighboring prediction value and the label probability value of the target label; The data clipping module 1250 is configured to perform data clipping on the sample data set based on the similarity estimation value to obtain a core data set, and to relabel at least one target sample in the core data set based on the similarity estimation value.

[0134] In some embodiments, the neighborhood sample selection module 1220 is further used to: Get the feature code of each sample; Calculate the cosine distance between the feature code of the target sample and the feature code of other samples; Based on the order of cosine distance values, neighborhood samples with a preset number of neighbors are selected from multiple samples.

[0135] In some embodiments, the prediction value calculation module 1230 is further used to: Generate a transition matrix based on the cosine distance value between every two feature codes; Based on the cumulative sum of the row vector elements in the transition matrix, a diagonal matrix is ​​obtained; The adjacency feature matrix is ​​updated based on the diagonal matrix and the transition matrix to obtain an updated adjacency feature matrix.

[0136] In some embodiments, the prediction value calculation module 1230 is further used to: Multiply the diagonal matrix, the transition matrix, and the diagonal matrix to obtain the adjacency matrix; Based on the sum of the identity matrix and the adjacency matrix, multiplied by the adjacency feature matrix, the updated adjacency feature matrix is ​​obtained.

[0137] In some embodiments, the prediction value calculation module 1230 is further used to: Get the fully connected layer parameters and probability activation function; The prediction parameters are obtained based on the product of the updated adjacency feature matrix and the fully connected layer parameters; Substitute the prediction parameters into the probability activation function to obtain the neighbor prediction value of each neighborhood sample.

[0138] In some embodiments, the similarity calculation module 1240 is further used to: Selecting a target adjacent prediction value from multiple adjacent prediction values ​​one by one; Calculate the divergence between the target neighbor prediction value and the label probability value; All divergence values ​​are averaged to obtain a similarity estimate.

[0139] In some embodiments, the data clipping module 1250 is further configured to: Based on the numerical ranking of the similarity estimation values, performing stratified sampling processing on the similarity estimation values ​​of the plurality of target samples to obtain a core data set including at least one target sample; The sample label of the target sample in the core data set is obtained, and when the similarity estimation value indicates that the sample label corresponding to the target sample is incorrectly labeled, the sample label of the target sample is relabeled.

[0140] In some embodiments, the data clipping module 1250 is further configured to: dividing the plurality of similarity estimation values ​​into a plurality of estimation value clusters, each of the estimation value clusters including a plurality of the similarity estimation values; Based on the numerical ranking of the plurality of similarity estimation values, selecting similarity estimation values ​​of a preset number of clusters from each of the estimation value clusters one by one as target similarity estimation values; The target samples corresponding to all the target similarity estimation values ​​are taken as the core data set, and the similarity estimation values ​​of all the target samples in the core data set are less than a preset similarity value.

[0141] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, the specific implementation of the data clipping device is basically the same as the specific implementation of the above data clipping method, and will not be repeated here.

[0142] In the embodiment of the present application, the data clipping device targets each target sample one by one, and further targets multiple neighboring samples closest to the target sample, and uses the cosine distance value between the feature codes of the target sample and other samples in the sample data set to accurately determine the multiple neighboring samples closest to the target sample, so as to facilitate the subsequent calculation of the labeling result of the target sample using the similarity between the neighboring sample and the target sample, thereby improving the accuracy of data labeling; and, using the adjacency feature matrix composed of the feature codes of multiple adjacent samples to generate a transition matrix, and using the transition matrix to further generate a diagonal matrix and an adjacency matrix in sequence, so as to facilitate the use of the adjacency matrix to fully mine the data information between multiple adjacent samples, so as to facilitate the subsequent use of the updated adjacency feature matrix to calculate the predicted value of whether each adjacent sample is noise, thereby improving the accuracy of data labeling; and, using the updated adjacency feature matrix obtained after feature propagation And the fully connected layer parameters of the combined model , and further use the probability activation function , perform label propagation of neighborhood samples, and use the data information between domain samples to accurately correct the label information of neighborhood samples (i.e., neighbor prediction value), so as to facilitate the subsequent use of the corrected neighbor prediction value of the domain sample to calculate the similarity with the target sample, so as to accurately determine the target sample label; and use the average value of the JS divergence between the neighbor prediction value of multiple neighbor samples obtained by feature propagation and label propagation and the label probability value of the target sample to accurately obtain the similarity estimation value that characterizes the distribution difference between the target sample and the neighboring samples, so as to use the similarity estimation value to accurately obtain the label annotation result of the target sample to distinguish the noise sample; finally, based on the similarity estimation value, and using the stratified sampling method to maintain It is to ensure that the trimmed data obtained after stratified sampling contains simple samples and difficult samples, thereby ensuring that the core data set obtained after stratified sampling contains simple samples and difficult samples, and ensuring that the similarity estimate of the selected target samples is low, that is, corresponding to more reliable target samples, so as to improve the accuracy of retaining important sample data in the core data set obtained after trimming the data, and improve the coverage of sample data in the core data set, and update the label annotation of erroneous sample data in the core data set, so as to perform data trimming to obtain a core data set that only includes important and reliable sample data, so as to reduce the overfitting phenomenon caused by the presence of noise samples in the selected data during the model training process, so as to improve the accuracy and training efficiency of model training.

[0143] The present application also provides an electronic device, including: at least one memory; at least one processor; at least one program; The program is stored in the memory, and the processor executes the at least one program to implement the data clipping method implemented in the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0144] See also Fig.13 , Fig.13 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes: The processor 1301 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 1302 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device or RAM (Random Access Memory). The memory 1302 can store operating systems and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1302, and the processor 1301 calls and executes the data clipping method of the embodiments of this application; Input / output interface 1303, used to implement information input and output; The communication interface 1304 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.); A bus 1305 that transmits information between various components of the device (e.g., the processor 1301, the memory 1302, the input / output interface 1303, and the communication interface 1304); The processor 1301 , the memory 1302 , the input / output interface 1303 and the communication interface 1304 are connected to each other in communication within the device via a bus 1305 .

[0145] An embodiment of the present application further provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the above-mentioned data clipping method is implemented.

[0146] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0148] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0149] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0151] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0152] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0153] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0154] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0157] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A data clipping method, characterized in that: The method comprises: Selecting a target sample from multiple samples in a sample data set, and obtaining a target label for the target sample; Determine a plurality of neighboring samples of the target sample in the sample data set, and obtain an adjacency feature matrix according to feature codes of the plurality of neighboring samples; Performing feature propagation processing on the adjacency feature matrix based on the feature coding to obtain an updated adjacency feature matrix, and calculating an adjacency prediction value of each of the neighborhood samples based on the updated adjacency feature matrix; Based on the neighboring prediction value and the label probability value of the target label, a similarity estimation value between the plurality of neighborhood samples and the target sample is calculated; The sample data set is pruned based on the similarity estimation value to obtain a core data set, and at least one target sample in the core data set is relabeled based on the similarity estimation value.

2. The data clipping method according to claim 1, characterized in that: The step of determining a plurality of neighboring samples of the target sample in the sample data set comprises: Obtaining a feature code for each of the samples; Calculate the cosine distance between the feature code of the target sample and the feature codes of other samples; Based on the order of the cosine distance values, a preset number of neighboring samples are selected from the multiple samples.

3. The data clipping method according to claim 1, characterized in that: The step of performing feature propagation processing on the adjacency feature matrix based on the feature coding to obtain an updated adjacency feature matrix includes: Generate a transition matrix based on the cosine distance value between every two feature codes; Obtaining a diagonal matrix based on the cumulative sum of the row vector elements in the transition matrix; The adjacency feature matrix is ​​updated based on the diagonal matrix and the transition matrix to obtain the updated adjacency feature matrix.

4. The data clipping method according to claim 3, characterized in that: The updating of the adjacency feature matrix based on the diagonal matrix and the transition matrix to obtain the updated adjacency feature matrix includes: Multiplying the diagonal matrix, the transition matrix, and the diagonal matrix to obtain an adjacency matrix; Based on the sum of the unit matrix and the adjacency matrix, multiplying it by the adjacency feature matrix, the updated adjacency feature matrix is ​​obtained.

5. The data clipping method according to claim 1, characterized in that: The step of calculating the adjacency prediction value of each of the neighborhood samples based on the updated adjacency feature matrix includes: Get the fully connected layer parameters and probability activation function; Obtaining prediction parameters based on the product of the updated adjacency feature matrix and the fully connected layer parameters; Substituting the prediction parameters into the probability activation function, the adjacent prediction value of each of the neighborhood samples is obtained.

6. The data clipping method according to claim 1, characterized in that: The calculating, based on the neighboring prediction value and the label probability value of the target label, a similarity estimation value between the plurality of neighborhood samples and the target sample comprises: Selecting a target adjacent prediction value from multiple adjacent prediction values ​​one by one; Calculating a divergence value between the target adjacent prediction value and the label probability value; All the divergence values ​​are averaged to obtain the similarity estimation value.

7. The data clipping method according to claim 1, characterized in that: There are multiple target samples, and the data clipping of the sample data set based on the similarity estimation value to obtain a core data set, and label re-labeling of at least one target sample in the core data set based on the similarity estimation value include: Based on the numerical ranking of the similarity estimation values, performing stratified sampling processing on the similarity estimation values ​​of the plurality of target samples to obtain a core data set including at least one target sample; The sample label of the target sample in the core data set is obtained, and when the similarity estimation value indicates that the sample label corresponding to the target sample is incorrectly labeled, the sample label of the target sample is relabeled.

8. The data clipping method according to claim 7, characterized in that: The step of performing stratified sampling on the similarity estimation values ​​of the plurality of target samples based on the numerical sorting of the similarity estimation values ​​to obtain a core data set including at least one target sample comprises: dividing the plurality of similarity estimation values ​​into a plurality of estimation value clusters, each of the estimation value clusters including a plurality of the similarity estimation values; Based on the numerical ranking of the plurality of similarity estimation values, selecting similarity estimation values ​​of a preset number of clusters from each of the estimation value clusters one by one as target similarity estimation values, wherein the target similarity estimation value is less than the preset similarity value; The target samples corresponding to all the target similarity estimation values ​​are taken as the core data set.

9. A data cutting device, characterized in that: The device comprises: A target sample selection module is used to select a target sample from multiple samples in a sample data set and obtain a target label of the target sample; A neighborhood sample selection module, used to determine a plurality of neighborhood samples of the target sample in the sample data set, and obtain an adjacency feature matrix according to feature codes of the plurality of neighborhood samples; A prediction value calculation module, used for performing feature propagation processing on the adjacency feature matrix based on the feature coding to obtain an updated adjacency feature matrix, and calculating the adjacency prediction value of each of the neighborhood samples based on the updated adjacency feature matrix; A similarity calculation module, configured to calculate a similarity estimation value between the plurality of neighborhood samples and the target sample based on the neighboring prediction value and the label probability value of the target label; A data clipping module is used to perform data clipping on the sample data set based on the similarity estimation value to obtain a core data set, and to relabel at least one target sample in the core data set based on the similarity estimation value.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the data clipping method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data clipping method according to any one of claims 1 to 8 is implemented.

Images