A short text search similar long text method without labeled data

Abstract

The present application belongs to the technical field of natural language search processing, and particularly relates to a short text search similar long text method without labeled data. That is, according to a batch of original long texts without labeling, sentence division, keyword extraction, long-short sentence mapping relationship establishment, and long-short sentence relationship pair as input source are performed; long text encoding and short text encoding are respectively performed on the long-short sentence relationship pair, and text feature representation is respectively obtained; model learning is performed, and a CLIP means is used to make the cosine similarity of a batch of training data existing long-short sentence mapping relationship maximum, otherwise as small as possible, and the cosine similarity of the data not existing long-short sentence mapping relationship minimum; after saving the model each time, the data is shuffled, and the probability of occurrence of negative samples is increased. The main innovation points of the present application are as follows: the keyword extraction and the CLIP architecture-based contrast learning technology are combined, a search mode based on deep learning semantic representation without labeling of user data is realized, and gMLP is used as a text encoder, so that the gMLP can be effectively used for knowledge retrieval.

Description

Technical Field

[0001] This invention belongs to the field of natural language search processing technology, specifically relating to a method for searching similar long texts from short texts without requiring labeled data. Background Technology

[0002] Text search refers to the process by which an application receives text input from a user and returns search results. This generally includes text-to-text searches, image-to-text searches, and recommendations. Tasks like short-text search for long-text are significant in application scenarios such as recommendation search, chatbots, and knowledge-based question answering. While deep learning-based short-text search for long-text has gradually become mainstream, data annotation consumes substantial human resources, and the tedious annotation work reduces user satisfaction. Furthermore, current search solutions based on word segmentation and full-text retrieval fail to provide satisfactory feedback to users. These are the pain points and challenges of supervised deep learning-based short-text search for long-text. This invention proposes a short-text search for long-text without requiring labeled data, effectively solving this problem. Summary of the Invention

[0003] This invention provides a technical solution for searching similar long texts from short texts without requiring labeled data, overcoming, to some extent, the technical problem of difficulty in labeling data in the initial stages of supervised methods in related technologies. Other features and advantages of this invention will become clear from the following detailed description, or in part will be illustrated by specific implementation.

[0004] This invention proposes a short-text search technique for long texts that does not require labeled data, specifically including the following steps:

[0005] Step 1, Text Data Processing: The data is processed accordingly to ensure that it is not affected by other additional features. Specifically, there are various kinds of noisy text that do not conform to standard text, such as HTML tags, blank text, repeated words, inconsistent capitalization. Common noise is processed, including but not limited to filtering HTML tags, unifying capitalization, unifying text encoding format, removing blank characters, and deleting duplicate characters, to process the noisy text into relatively clean text data.

[0006] Step 2, Keyword Extraction: Based on a custom algorithm or third-party tools, perform keyword extraction on each text in the data to obtain a keyword list for each text. For example, given a series of texts (multiple texts), use the jieba word segmentation tool to segment each text into words, count the word frequency of each word, calculate the inverse document frequency (IDF) of each word, and store the results. When new text appears, segment it again, count the word frequency, and then calculate the TF-IDF value with the pre-calculated IDF values. The higher the score, the more keywords are considered for that text. By taking multiple keywords, a keyword list can be obtained.

[0007] Step 3, Sentence Segmentation and Short Sentence Filtering: Each data entry is segmented according to custom punctuation marks to obtain short sentences. Short sentences containing keywords are then filtered out as the input short text set. In this case, long text refers to each data entry without sentence segmentation, i.e., the original text data. Short text refers to each sentence of the original text data segmented according to punctuation marks. In particular, when there are no punctuation marks in the text data that can be used for sentence segmentation, both short text and long text refer to that data entry.

[0008] Step 4, Generate relation pairs: Establish a one-to-one mapping relationship between the set of short sentences of each text in Step 3 and the corresponding long text, and use them as the input source for the model;

[0009] Step 5, Text Encoding Representation: Input the long and short texts from the input source in Step 4 into the long text encoder and the short text encoder, respectively. Specifically, the long text encoder and the short text encoder adopt the same gMLP encoder structure according to the convention, that is, they adopt 6 identical blocks. Each block includes a multi-head self-attention layer, a normalized residual connection layer, a feedforward neural network layer, and a normalized residual connection layer. Finally, the long and short texts are obtained through such long and short text encoders, respectively.

[0010] Step 6, train the model: input the long and short text encoding representations from Step 5 into the model, and train the model according to the pre-defined network structure and loss function optimization, etc.

[0011] Step 7, Text Prediction: The short text input by the user is semantically represented by a short text encoder, and the model searches for the most similar long text to obtain the final output result.

[0012] The technical solution of this invention has the following advantages: it can train a deep learning short text search long text model without the need for labeled data, which solves the problems of low efficiency caused by the scarcity of data in the early stage of the project and the need for a large number of labelers.

[0013] Short text search for long text is widely used in knowledge retrieval. For example, given a batch of articles related to artificial intelligence, when a user inputs a short passage of knowledge in the field of artificial intelligence, the system can recommend relevant technical texts in that field, allowing users to quickly grasp the knowledge they need. This invention proposes a technical solution for short text search of similar long text without the need for labeled data. This involves using a batch of unlabeled original long text for sentence segmentation, keyword extraction, establishing long-short sentence mapping relationships, and using long-short sentence relationship pairs as input sources. The long-short sentence relationship pairs are then encoded using both long and short text encoding to obtain text feature representations. Model learning utilizes the CLIP (Contrastive Learning Platform) technique to maximize the cosine similarity of long-short sentence mapping relationships in a batch of training data, minimize it otherwise, and minimize the cosine similarity of no long-short sentence mapping relationships. After each model saving, the data is shuffled to increase the probability of negative samples. The main innovation of this invention lies in combining keyword extraction with CLIP-based contrastive learning technology to achieve a search mode based on deep learning semantic representation that requires no labeling of user data. Furthermore, the use of gMLP as a text encoder makes it effective for knowledge retrieval. Attached Figure Description

[0014] Figure 1 This is a diagram of the overall network architecture of the present invention;

[0015] Figure 2 The usage process for the gMLP text encoder. Detailed Implementation Plan

[0016] The following will be combined with the appendix Figure 1 and 2 The technical solution of the present invention will be further described below.

[0017] This invention proposes a short-text search technique for long texts that does not require labeled data, specifically including the following steps:

[0018] Step 1, Text Data Processing: The data is processed accordingly to ensure that it is not affected by other additional features. Specifically, there are various kinds of noisy text that do not conform to standard text, such as HTML tags, blank text, repeated words, inconsistent capitalization. Common noise is processed, including but not limited to filtering HTML tags, unifying capitalization, unifying text encoding format, removing blank characters, and deleting duplicate characters, to process the noisy text into relatively clean text data.

[0019] Step 2, Keyword Extraction: Based on a custom algorithm or third-party tools, perform keyword extraction on each text in the data to obtain a keyword list for each text. For example, given a series of texts (multiple texts), use the jieba word segmentation tool to segment each text into words, count the word frequency of each word, calculate the inverse document frequency (IDF) of each word, and store the results. When new text appears, segment it again, count the word frequency, and then calculate the TF-IDF value with the pre-calculated IDF values. The higher the score, the more keywords are considered for that text. By taking multiple keywords, a keyword list can be obtained.

[0020] Step 3, Sentence Segmentation and Short Sentence Filtering: Each data entry is segmented according to custom punctuation marks to obtain short sentences. Short sentences containing keywords are then filtered out as the input short text set. In this case, long text refers to each data entry without sentence segmentation, i.e., the original text data. Short text refers to each sentence of the original text data segmented according to punctuation marks. In particular, when there are no punctuation marks in the text data that can be used for sentence segmentation, both short text and long text refer to that data entry.

[0021] Step 4, Generate relation pairs: Establish a one-to-one mapping relationship between the set of short sentences of each text in Step 3 and the corresponding long text, and use them as the input source for the model;

[0022] Step 5, Text Encoding Representation: Input the long and short texts from the input source in Step 4 into the long text encoder and the short text encoder, respectively. Specifically, the long text encoder and the short text encoder adopt the same gMLP encoder structure according to the convention, that is, they adopt 6 identical blocks. Each block includes a multi-head self-attention layer, a normalized residual connection layer, a feedforward neural network layer, and a normalized residual connection layer. Finally, the long and short texts are obtained through such long and short text encoders, respectively.

[0023] Step 6, train the model: input the long and short text encoding representations from Step 5 into the model, and train the model according to the pre-defined network structure and loss function optimization, etc.

[0024] Step 7, Text Prediction: The short text input by the user is semantically represented by a short text encoder, and the model searches for the most similar long text to obtain the final output result.

[0025] In step 1, the specific method of the corresponding operation is to perform noise reduction processing on the data text by unifying the case, unifying the text encoding format, and removing whitespace characters, so as to eliminate the influence of these features on the subsequent results.

[0026] In step 2, the specific method for keyword extraction is as follows: the tf-idf algorithm is used to calculate the importance of words in each text (tf is the term frequency, idf is the inverse document frequency of all documents, and tf-idf is the product of the two), and the words are sorted according to their importance. A certain number of keywords are selected as the candidate keyword set for the text.

[0027] In step 3, the specific operation of sentence segmentation is as follows: predefined delimiters, i.e., relevant punctuation marks, generally periods, exclamation marks, semicolons, etc., are used as delimiters to segment the text into sentences. In particular, when there are no punctuation marks as delimiters in the text, the original text is used as the result of sentence segmentation. The specific operation of filtering short sentences is as follows: using the keyword list in step 2 as a benchmark, it is determined whether there are relevant keywords in the short sentences. If there are, the short sentence is retained; otherwise, the irrelevant short sentence is removed, thereby obtaining the final set of short sentences.

[0028] In step 4, the specific operation of generating relation pairs is as follows: Let each text be x, and the list of short sentences after x is divided into clauses be [x1, x2, x3, ..., xn]. Then the relation pairs generated for this text are (x1, x), (x2, x), (x3, x), etc., which are short text-long text relation pairs. Each text is subjected to relevant operations, and finally a series of short text-long text relation pairs are obtained as the text input source before encoding.

[0029] In step 5, the specific operation of the text encoding representation is as follows: The long and short texts in the relation pair undergo initial text embedding, i.e., text vectorization, and are input into two text encoders, a long text encoder and a short text encoder, respectively. Both text encoders use a gMLP network structure for encoding representation, with data dimensions of [batch_size, max_len_long, d_model] and [batch_size, max_len_short, d_model], respectively. The data is then processed through multiple layers of gMLP feature extractors to obtain the output representation, with data dimensions of [batch_size, max_len_long, d_model] and [batch_size, max_len_short, d_model], respectively. The vector of the first token in the output representation is then used as the semantic representation of the entire text encoder (the first token is chosen because after passing through the gMLP encoder, the vector representation of the first token in the output vector can usually be used as the semantic representation of the entire text; also, eliminating one dimension of representation is more convenient for subsequent tensor calculations). At this point, the dimension of the output vector becomes [batch_size, The second dimension, d_model, is then divided by its L2 norm (used for vector normalization) to obtain the semantic representation of the text for both long and short texts. Parameter descriptions: batch_size is the number of texts in the batch; max_len_long is the maximum text sequence length set for the input long text encoder model; max_len_short is the maximum text sequence length set for the input short text encoder model; d_model is the dimension of the text feature representation.

[0030] In step 6, the specific operation method is as follows: The semantic representation vectors of the short and long texts obtained in step 5 both have dimensions of [batch_size, d_model]. The semantic representation dimension of the short text remains unchanged, while the semantic representation dimension of the long text is transposed, resulting in a semantic representation dimension of [d_model, batch_size]. Then, the semantic representation vectors of the short and long texts are multiplied by a tensor to obtain a tensor with dimensions of [batch_size, batch_size]. The diagonal elements of this tensor represent the cosine similarity between the short text and its corresponding long text. The model optimization objective is to maximize the value of each diagonal element and minimize the values ​​at other positions. (The diagonal elements represent the similarity of positive samples, and the off-diagonal elements represent the similarity of negative samples; samples with a given relationship are considered positive samples.) The model's loss function uses a stacked cross-entropy loss function. Through this objective and loss function, the model is trained using data to ultimately obtain the model parameters.

[0031] In step 7, according to the method of claim 1, the specific operation method in step S6 is as follows: the dimensions of the semantic representation vectors of the short and long texts obtained in step S5 are both [batch_size, d_model]. The semantic representation tensor of the short text is of size [d_model, batch_size]. The most similar semantic vector with the highest similarity is searched using the Faiss search tool, and the corresponding long text corresponding to the index of the vector is extracted, thus completing the task of searching for long texts from short texts.

Claims

1. A method for searching similar long texts from short texts without requiring labeled data, characterized in that: Includes the following steps: S1. Text data preprocessing: The text data contains noisy text that does not conform to standard text, such as HTML tags, blank text, repeated words, inconsistent capitalization, etc. The noise is processed by filtering HTML tags, ensuring consistent capitalization and text encoding format, removing blank characters, and deleting duplicate characters, so as to process the noisy text into relatively clean text data. S2. Keyword extraction: Using third-party tools such as jieba, keywords are extracted from each text in the data to obtain a keyword list for each text. For a series of texts (multiple texts), the jieba word segmentation tool is used to segment each text into words, and the word frequency of each word is counted. The inverse document frequency (IDF) of each word in the text is calculated and stored. When a new text appears, it is segmented, word frequency is counted, and then the TFIDF value is calculated with the pre-counted IDF values. The higher the score, the more keywords are selected for that text. Multiple keywords are selected to obtain the keyword list. S3. Sentence segmentation and sentence filtering: Each data entry is segmented according to custom punctuation marks to obtain short sentences. Short sentences containing keywords are then filtered out as the input short text set. Here, long text refers to each data entry without sentence segmentation, i.e., the original text data. Short text refers to each sentence segmented from the original text data according to punctuation marks. When there are no punctuation marks in the text data that can be used for sentence segmentation, both short text and long text refer to that data entry. In the model, the maximum length of short text input to the model is defined as 25, and the maximum length of long text input to the model is defined as 200. S4. Generate long and short text mapping pairs. Establish a one-to-one mapping relationship between the set of short sentences of each text in step S3 and the corresponding long text, and use them as the input source of the model. S5. Text Encoding Representation: The long and short texts from the input source in step S4 are input into the long text encoder and the short text encoder, respectively. Specifically, the long text encoder and the short text encoder adopt the same gMLP encoder structure, using 6 identical blocks. Each block includes a multi-head self-attention layer, a normalized residual connection layer, a feedforward neural network layer, and a normalized residual connection layer. The long and short texts are respectively encoded by the long and short text encoders to obtain the text semantic encoding representation. gMLP has a spatial projection gating unit, which has less computation than the multi-head attention mechanism in Transformer. Through layer-by-layer encoding, the token in each text can sense the information of other tokens through this gating unit. S6. Train the model. Input the long and short text encoding representations from step S5 into the model. Train the model according to the pre-set network structure and loss function optimization, so that the cosine similarity between the semantic representation vector of the short text and the corresponding long text is as close to 1 as possible, while the cosine similarity between the semantic representation vector of the short text and the corresponding long text is as close to 0 as possible. Finally, save the model. S7. Text search: All long texts input by the user are semantically represented by a long text encoder and stored offline; the short text input by the user is semantically represented by a short text encoder, and the model searches for the most similar long text to obtain the final output result.

2. The method for searching similar long texts from short texts without requiring labeled data, as described in claim 1, is characterized in that: The specific method for keyword extraction in step S2 is as follows: the tf-idf algorithm is used to calculate the importance of words in each text, where tf is the term frequency, idf is the inverse document frequency of all documents, and tf-idf is the product of the two terms. Multiple keywords are selected as the candidate keyword set for the text.

3. The method for searching similar long texts from short texts without requiring labeled data, as described in claim 1, is characterized in that: In step S3, the specific operation of sentence segmentation is as follows: predefined delimiters, i.e., relevant punctuation marks, generally periods, exclamation marks, and semicolons, are used as delimiters to segment the text into sentences. When there are no punctuation marks as delimiters in the text, the original text is used as the result after sentence segmentation. The specific operation of filtering short sentences is as follows: using the keyword list in step S2 as a benchmark, it is determined whether there are relevant keywords in the short sentences. If there are, the short sentence is retained; otherwise, the irrelevant short sentence is removed, thereby obtaining the final set of short sentences.

4. The method for searching similar long texts from short texts without requiring labeled data, as described in claim 1, is characterized in that: In step S4, the specific operation of generating relation pairs is as follows: Let each text be x, and the list of short sentences after x is divided into clauses be [x1, x2, x3, ..., xn]. Then the relation pairs generated for this text are (x1, x), (x2, x), (x3, x), etc., which are short text-long text relation pairs. Each text performs its own short sentence and its own mapping operation, and finally obtains a series of short text-long text mapping relation pairs, which are used as the text input source before encoding.

5. The method for searching similar long texts from short texts without requiring labeled data, as described in claim 1, is characterized in that: In step S5, the specific text encoding operation is as follows: The long and short texts in the relation pair undergo initial text embedding, i.e., text vectorization, and are input into two text encoders: a long text encoder and a short text encoder. Both encoders use a gMLP network structure for encoding. The tensor dimensions of the long and short texts input to the encoders are [batch_size, max_len_long] and [batch_size, max_len_short], respectively. The long and short text data are then processed through multiple layers of gMLP feature extractors to obtain the output representations. At this point, the semantic representation tensor dimensions are [batch_size, max_len_long, d_model] and [batch_size, max_len_short, d_model], respectively. Finally, the vector of the first token in the output representation is used as the semantic representation of the entire text encoder. The dimension of the output tensor then becomes [batch_size, max_len_long, d_model]. The second dimension is then divided by the L2 norm of that dimension. The L2 norm is used for vector normalization to obtain the final semantic representation of the text for both long and short texts. Parameter descriptions: batch_size is the number of texts in the batch; max_len_long is the maximum text sequence length set for the input long text encoder model; max_len_short is the maximum text sequence length set for the input short text encoder model; d_model is the dimension of the text feature representation.

6. The method for searching similar long texts from short texts without requiring labeled data, as described in claim 1, is characterized in that: The specific operation method in step S6 is as follows: The semantic representation vectors of the long and short texts obtained in step S5 both have dimensions of [batch_size, d_model]. The semantic representation tensor of the short text remains unchanged, while the semantic representation tensor of the long text is transposed, so the semantic representation of the long text becomes [d_model, batch_size]. At this time, the semantic representations of the long and short texts are multiplied by tensor to obtain a tensor with dimensions of [batch_size, batch_size]. The diagonal elements of this tensor are the cosine similarity between the short text and the corresponding long text. The model optimization objective is to maximize the value of the diagonal elements and minimize the values ​​of other positions. The diagonal elements are the similarity of positive samples, and the off-diagonal elements are the so-called similarity of negative samples. That is, samples with a relationship pair are positive samples. The loss function of the model is the stacked cross-entropy loss function. With such an objective and loss function, the model is trained with data to finally obtain the model parameters.

7. The method for searching similar long texts from short texts without requiring labeled data, as described in claim 1, is characterized in that: The specific operation method in step S6 is as follows: the semantic representation vectors of the short and long texts obtained in step S5 are both [batch_size, d_model]. The semantic representation tensor of the short text is [d_model, batch_size]. The most similar semantic vector with the highest similarity is searched using the Faiss search tool, and the corresponding long text corresponding to the index of the vector is extracted, thus completing the task of searching for long texts from short texts.

Images