Data feature enhancement method and device for corpus data and computer equipment

Abstract

The invention discloses a data feature enhancement method and device for corpus data, computer equipment and a storage medium, and relates to an artificial intelligence technology. After a full corpusdata set is obtained, data grouping is carried out to obtain multiple groups of corpus data subsets; A to-be-trained user intention recognition model is trained after deleting each group of corpus data subsets in sequence to obtain a plurality of user intention recognition models, and each piece of data in the full corpus data set is taken as training sample data and test sample data; a model average correct rate difference value, a sample recall rate difference value and a prediction correct rate difference value are calculated correspondingly to obtain a sample contribution degree triple corresponding to each corpus data; and if three difference values in the sample contribution degree triad corresponding to the corpus data are negative values, target corpus data is acquired to form a to-be-deleted corpus data set so as to delete the target corpus data from the full corpus data set. Automatic cleaning of the negative contribution corpus data is achieved, human intervention is not needed in the cleaning process, and the obtaining efficiency of a high-quality training set is improved.

Description

technical field [0001] The present invention relates to the technical field of artificial intelligence model trusteeship, in particular to a data feature enhancement method, device, computer equipment and storage medium for corpus data. Background technique [0002] Traditional conversational robots use corpus data to train deep learning models to complete tasks such as user intent recognition. The quality of the training corpus is the key to the effect of the model. The quality of the corpus is generally measured by two aspects: "quality" and "quantity". "Quality" is to ensure the correctness of the corpus and the boundaries between different intentions are clear, and "quantity" is to ensure that the model can fully learn the distribution of data features. , the two complement each other and are indispensable. [0003] When sorting out the training data, the R&D personnel found that when expanding the "quantity" of the training set, adding a sample to the training set does...

AI Technical Summary

Problems solved by technology

[0005] Embodiments of the present invention provide a data feature enhancement method, device, computer equipment, and storage medium for corpus data, aiming at solving the problem of manually expanding training corpus in the prior art, requiring high labor costs, and expanding the expected data process. The data cleaning process in is also done manually, which leads to the problem of low efficiency in obtaining high-quality training sets

Images