Sentence vector model-based sentence vector generation method and apparatus, and computer device

Abstract

The embodiment of the invention belongs to the field of artificial intelligence, and relates to a sentence vector model-based sentence vector generation method and device, computer equipment and a storage medium, and the method comprises the steps of obtaining TF-IDF information of each initial text in an initial text set to determine a target adjustment word, and adjusting the initial text based on the target adjustment word to generate a similar text; inputting the initial text into an initial sentence vector model to obtain an initial sentence vector, and inputting the similar text into the initial sentence vector model to obtain a similar sentence vector; taking the similar sentence vector as a positive sample of the current initial sentence vector, taking other initial sentence vectors and the similar sentence vector as a negative sample of the current initial sentence vector, and performing comparative learning on the initial sentence vector model to obtain a sentence vector model; and inputting the to-be-processed text into the sentence vector model to obtain a sentence vector of the to-be-processed text. In addition, the invention also relates to a block chain technology, and the initial text set can be stored in a block chain. According to the invention, the accurate and available sentence vectors can be efficiently generated.

Description

technical field [0001] The present application relates to the technical field of artificial intelligence, in particular to a sentence vector generation method, device, computer equipment and storage medium based on a sentence vector model. Background technique [0002] Sentence embedding is one of the hot research fields in natural language processing in recent years. Sentence vectors can be obtained by mapping the words, tokens, and semantic information between words in a sentence to a quantifiable space. The generated sentence vectors are usually provided to downstream tasks for further processing, such as similarity calculation, classification and clustering based on sentence vectors. [0003] Some of the existing sentence vector generation technologies simply separate the words in the sentence, convert the words into word vectors, and then calculate the average value to obtain the sentence vector. However, this will lose the semantic information in the sentence and affe...

AI Technical Summary

Problems solved by technology

[0003] Some of the existing sentence vector generation technologies simply separate the words in the sentence, convert the words into word vectors, and then calculate the average value to obtain the sentence vector. However, this will lose the semantic information in the sentence and affect the accuracy of the sentence vector. characteristics; some generate sentence vectors through supervised learning, but in reality it is difficult to obtain a large amount of labeled text corpus for supervised learning

Images