Corpus processing method, related device and equipment

Abstract

The embodiment of the invention discloses a corpus processing method, a related device and equipment, which are used for sufficiently expanding corpora to be expanded so as to meet the requirement of model training for the number of the corpora. The method of the embodiment of the invention comprises the steps that: corpora to be expanded are obtained, K candidate corpora are obtained according to the corpora to be expanded, and the K candidate corpora and the corpora to be expanded are input into a semantic recognition model so as to obtain K semantic recognition results, wherein each semantic recognition result is a similarity score or a similarity classification, the similarity score represents the semantic similarity degree between the candidate corpus and the corpus to be expanded, and the similarity classification represents the semantic category between the candidate corpus and the corpus to be expanded; and if at least one semantic recognition result in the K semantic recognition results meets the corpus extraction condition, the candidate corpus corresponding to the at least one semantic recognition result is determined as the target corpus to obtain at least one target corpus belonging to the corpus to be expanded.

Description

technical field [0001] The embodiments of the present application relate to the technical field of artificial intelligence, and in particular to a method for processing corpus, related devices and equipment. Background technique [0002] With the popularization of artificial intelligence, more and more artificial intelligence technologies can bring convenience to people's life. For example, the user inputs some request sentences through the smart assistant, and the smart assistant analyzes and processes these request sentences and transmits the processing results. Give corresponding feedback to subsequent services, so as to complete an interaction process with user voice. [0003] When the current semantic analysis platform processes the request sentence, it usually uses the intent classification model to analyze the sentence. The intent classification model needs to be trained through a certain amount of training corpus, but because the existing training corpus only A smal...

AI Technical Summary

Problems solved by technology

[0003] When the current semantic analysis platform processes the request sentence, it usually uses the intent classification model to analyze the sentence. The intent classification model needs to be trained through a certain amount of training corpus, but because the existing training corpus only A small amount of seed corpus can be provided, which is not enough to support the normal training of the model, or the trained model is too fit to the training data, resulting in poor model generalization

Images