A method for fuzzy retrieval of string-encoded data

Abstract

The application relates to the technical field of text retrieval, and particularly discloses a string coding fuzzy retrieval method. The method comprises the following steps: collecting and obtaining document string coding, obtaining corresponding sentences and words, and establishing an inverted index based on word distribution; a fusion matching network is constructed, and the matching network is trained; the sentences and words obtained by the document string coding are taken as inputs of the fusion matching network, matched with known key coding, and a matching result is obtained; the matching result is used to determine position information of matching words by using the inverted index, and the matching result and the confidence degree are used to jointly create a coding result set, and matching content in an original document is labeled. The method has better matching accuracy and robustness, avoids complicated multiple processing procedures in other retrieval methods, and reduces retrieval time in an end-to-end manner.

Description

Technical Field

[0001] This invention belongs to the field of text retrieval technology, specifically relating to a string-encoded fuzzy retrieval method. Background Technology

[0002] Text retrieval involves organizing text from a collection in a specific way, then using keywords or sentences provided by the user, and employing retrieval algorithms to organize the retrieved documents according to the user's desired format and return them to the user. Text retrieval is one of the most important functions of information retrieval, and the most crucial aspect of text retrieval is the text matching model architecture. The quality of the model architecture directly affects the effectiveness of text retrieval. Therefore, building an accurate text matching model and implementing fuzzy search based on string encoding greatly helps improve the efficiency and practicality of information retrieval.

[0003] Existing deep learning-based text matching methods mostly rely on a single network, resulting in significant fluctuations in robustness and accuracy. They are easily affected by the length and complexity of the characters being searched, making it difficult to meet the retrieval requirements for strings encoding a mixture of numbers, English letters, and Chinese characters. Furthermore, the matching accuracy is insufficient for such searches. Moreover, existing fuzzy string retrieval methods often require multiple cumbersome processing steps that are independent and cannot be optimized, further reducing retrieval accuracy and significantly increasing retrieval time. Therefore, there is still considerable room for improvement in the accuracy, robustness, and time efficiency of text retrieval. Summary of the Invention

[0004] The purpose of this invention is to provide a string-encoded fuzzy retrieval method that solves the technical problems of low accuracy and poor robustness caused by the use of a single network in existing text matching networks, as well as the problem of long retrieval time caused by the use of multiple independent processing steps in existing text retrieval methods.

[0005] The technical solution of the present invention is as follows: a string-encoded fuzzy retrieval method, the method comprising:

[0006] The document string encoding is collected and preprocessed to obtain the corresponding sentence and word segments, and an inverted index based on the word distribution is established.

[0007] A fusion matching network consisting of a convolutional latent semantic model and a matching pyramid model is constructed, and the matching network is trained using the acquired data;

[0008] The sentence segments and word segments obtained by encoding the document string are used as input to the trained fusion matching network, matched with known key encodings, and the matching results are obtained.

[0009] The matching results are used to determine the location information containing the matching terms using the inverted index. Together with the matching results and confidence scores, an encoded result set is created, and the matching content in the original document is annotated.

[0010] The fusion matching network includes an input layer, a representation layer, and a matching layer. The input layer and representation layer of the convolutional latent semantic model and the input layer and representation layer of the matching pyramid model are constructed in the fusion matching network, respectively. The matching layer is used to concatenate the feature vectors obtained by the convolutional latent semantic model and the matching pyramid model into a one-dimensional feature vector. Two fully connected layers are used to classify the feature vector.

[0011] The specific steps for constructing the input layer and representation layer of the convolutional latent semantic model in the fusion matching network include:

[0012] The document strings are segmented and combined to form the vector space of the input layer of the convolutional latent semantic model;

[0013] The input layer of the convolutional latent semantic model is constructed, and two sentences from different documents are output as two vector spaces through the input layer;

[0014] A representation layer is constructed using convolutional layers, global max pooling layers, and fully connected layers with a finite number of neurons, which are used to extract one-dimensional sentence feature vectors from two different texts.

[0015] The specific steps for constructing the input layer and representation layer of the matching pyramid model are as follows:

[0016] Construct the input layer of the matching pyramid model, use the pre-trained word vector model to obtain the input word vectors, and obtain the similarity matrix by calculating the cosine distance between word vectors of different texts;

[0017] The representation layer is constructed by two convolutional layers, two pooling layers, and a fully connected network. The features extracted by the representation layer are used as one-dimensional similarity feature vectors for two different texts.

[0018] The specific method for segmenting and combining document strings to form the vector space of the input layer of the convolutional latent semantic model is as follows:

[0019] For English data in the document, sentences are segmented using a sliding window with a window length of 3, and the segments containing three words are further segmented into words. The vector space of each word after segmentation is 3-dimensional. The vectors of the three words after segmentation are combined to form an input layer with a vector space of 9 dimensions.

[0020] For Chinese data in the document, a 1.5-dimensional vector space is obtained simply by segmenting the characters;

[0021] The method of constructing a representation layer using convolutional layers, global max-pooling layers, and fully connected layers with a finite number of neurons, specifically for extracting one-dimensional sentence feature vectors from two different texts, is as follows:

[0022] A representation layer is constructed using convolutional layers, global max-pooling layers, and a fully connected layer with 128 neurons. These layers are used to extract one-dimensional sentence feature vectors from two different texts. The activation function in the fully connected layer is...

[0023] The process of using a pre-trained word vector model to obtain input word vectors and calculating the similarity matrix by measuring the cosine distance between word vectors of different texts is as follows:

[0024] The word vector α is obtained using the trained word vector model Word2Vec. i ,β j Then, the similarity matrix M is obtained by calculating the cosine distance between word vectors of different texts. ij (i*j dimensional matrix), the formula is:

[0025]

[0026] The representation layer, consisting of two convolutional layers, two pooling layers, and a fully connected network, is used to construct a one-dimensional similarity feature vector for two different texts.

[0027] A representation layer for the matching pyramid model in a fusion matching network is constructed, consisting of two convolutional layers, two pooling layers, and a fully connected layer. The first convolutional layer has a 3x3 kernel, a stride of 1, and two layers. Subsequent pooling layers employ a dynamic pooling strategy to obtain a fixed-length feature map. The pooling layer size is... The second convolutional layer has a 4x4 kernel, a stride of 1, and a layer number of 1. Subsequent pooling layers are max-pooling layers. The fully connected network has 128 neurons, and its activation function is... The features extracted from the representation layer are used as one-dimensional similarity feature vectors for two different texts.

[0028] The specific steps of using a matching layer to concatenate the feature vectors obtained from the convolutional latent semantic model and the matching pyramid model into a one-dimensional feature vector are as follows:

[0029] A matching layer of a fusion matching network is constructed. This layer concatenates the two one-dimensional sentence feature vectors and one one-dimensional similarity feature vector obtained from the input and representation layers of the convolutional latent semantic model and the matching pyramid model, respectively, into a single one-dimensional feature vector. Two fully connected layers are then used to classify this feature vector, with the activation function being... The probability Pk of outputting each label class is calculated using softmax, with the following formula:

[0030]

[0031] Among them, S k This is the output of the matching layer, where k is the predicted category and D is the number of categories.

[0032] The constructed fusion matching network also requires a training step:

[0033] The pre-constructed fusion matching network is trained using pre-labeled sample data with varying matching levels, where the loss function is:

[0034]

[0035] The trained neural network is directly used as the matching model for subsequent steps.

[0036] The specific steps for obtaining sentence segmentation and word segmentation through document string encoding are as follows:

[0037] Collect the string encoding of the document and perform sentence segmentation and word segmentation for English characters and / or Chinese characters respectively;

[0038] Use existing tools to read strings from documents such as Word or PDF documents. For example, you can use the open-source toolkit Apache POI to read strings from documents. For English characters in the document, use the natural language processing toolkit NLTK to segment sentences and words; for Chinese characters in the document, use the Chinese word segmentation toolkit jieba to segment sentences and words.

[0039] The steps of the inverted index based on term distribution are as follows:

[0040] An inverted full index is constructed using Hadoop MapReduce, where the first item of the index is a word, and the second item is the document index and location containing that word.

[0041] The significant advantages of this invention are as follows: The string-encoded fuzzy retrieval method described herein employs a fusion matching network, which is composed of a convolutional latent semantic model (CLSM) and a matching pyramid model (MatchPyramid). This network combines the advantages of both networks, utilizing both individual information from the two texts and the information between them, thus minimizing the disadvantages of using only one network. Consequently, it exhibits better matching accuracy and robustness. Furthermore, the string-encoded fuzzy retrieval method described herein avoids the cumbersome multiple processing steps found in other retrieval methods, reducing retrieval time through an end-to-end approach. Moreover, the end-to-end network, compared to multiple independent processing steps, can be further optimized to improve retrieval accuracy. Attached Figure Description

[0042] Figure 1 This is a flowchart of a string encoding fuzzy retrieval method according to the present invention. Detailed Implementation

[0043] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] like Figure 1 As shown, a string encoding fuzzy retrieval method is described, which specifically includes:

[0045] S1. Collect and obtain the document string encoding, and after preprocessing the string encoding, obtain the corresponding sentence segmentation and word segmentation, and establish an inverted index based on the word distribution;

[0046] S1.1 Collect the string encoding of the document and perform sentence segmentation and word segmentation for English characters and / or Chinese characters respectively;

[0047] Utilize existing tools to read strings from documents such as Word or PDF documents. For example, the open-source toolkit Apache POI can be used to read strings from documents. For English characters in the document, the natural language processing toolkit NLTK can be used for sentence and word segmentation; for Chinese characters in the document, the Chinese word segmentation toolkit jieba can be used for sentence and word segmentation.

[0048] S1.2 Utilize the obtained sentence and word segments to construct a reverse full index and realize an inverted index based on term distribution;

[0049] A reverse full index can be built using Hadoop MapReduce, wherein the first item of the index is a word, and the second item is the document index and location containing that word;

[0050] S2. Construct a fusion matching network and train the matching network using the acquired data;

[0051] A fusion matching network consisting of a convolutional latent semantic model (CLSM) and a matching pyramid model (MatchPyramid) is constructed, which includes an input layer, a representation layer, and a matching layer, and is trained using the acquired data.

[0052] S2.1 Construct the input layer and representation layer of the convolutional latent semantic model in the fusion matching network;

[0053] S2.1.1. Segment and combine document strings to form the vector space of the input layer of the convolutional latent semantic model;

[0054] For English data in the document, sentences are segmented using a sliding window with a window length of 3, and the segments containing three words are further segmented into words. The vector space of each word after segmentation is 3-dimensional. The vectors of the three words after segmentation are combined to form an input layer with a vector space of 9 dimensions.

[0055] For Chinese data in the document, a 1.5-dimensional vector space is obtained simply by segmenting the characters;

[0056] The input layer of the convolutional latent semantic model is constructed, and two sentences from different documents are output as two vector spaces through the input layer;

[0057] S2.1.2 Utilize convolutional layers, global max pooling layers, and fully connected layers with a finite number of neurons to construct representation layers, which are used to extract one-dimensional sentence feature vectors from two different texts;

[0058] A representation layer is constructed using convolutional layers, global max-pooling layers, and a fully connected layer with 128 neurons. These layers are used to extract one-dimensional sentence feature vectors from two different texts. The activation function in the fully connected layer is... For example, the above convolutional layer is a convolutional layer with a kernel size of 3*90000, a stride of 1, and a number of layers of 300;

[0059] S2.2 Construct the input layer and representation layer of the matching pyramid model in the fusion matching network;

[0060] S2.2..1 Construct the input layer of the matching pyramid model, use the pre-trained word vector model to obtain the input word vectors, and obtain the similarity matrix by calculating the cosine distance between different text word vectors;

[0061] The trained word vector model Word2Vec can be used to obtain the input word vector α. i ,β j Then, the similarity matrix M is obtained by calculating the cosine distance between word vectors of different texts. ij (i*j dimensional matrix), the formula is:

[0062]

[0063] S2.2..2 A representation layer is constructed by two convolutional layers, two pooling layers, and a fully connected network. The features extracted by the representation layer are used as one-dimensional similarity feature vectors for two different texts.

[0064] A representation layer for the matching pyramid model in a fusion matching network is constructed, consisting of two convolutional layers, two pooling layers, and a fully connected layer. The first convolutional layer has a 3x3 kernel, a stride of 1, and two layers. Subsequent pooling layers employ a dynamic pooling strategy to obtain a fixed-length feature map (size: N). i′j′ The pooling layer size is The second convolutional layer has a 4x4 kernel, a stride of 1, and a layer number of 1. Subsequent pooling layers are max-pooling layers. The fully connected network has 128 neurons, and its activation function is... The features extracted from the representation layer are used as one-dimensional similarity feature vectors for two different texts;

[0065] S2.3 Construct the matching layer of the fusion matching network, and concatenate the two one-dimensional sentence feature vectors and one one-dimensional similarity feature vector obtained in the above steps into a one-dimensional feature vector. Use two fully connected layers to classify the feature vector.

[0066] A matching layer of a fusion matching network is constructed. The two one-dimensional sentence feature vectors and one one-dimensional similarity feature vector obtained in steps 2.1.2 and 2.2.2 are concatenated into a one-dimensional feature vector. Two fully connected layers are then used to classify this feature vector, with the activation function being... And use softmax to calculate the probability P of the output being each class label. k The formula is:

[0067]

[0068] Among them, S k This is the output of the matching layer, where k is the predicted category and D is the number of categories.

[0069] S2.4 Training the constructed fusion matching network;

[0070] The pre-constructed fusion matching network is trained using pre-labeled sample data with varying matching levels, where the loss function is:

[0071]

[0072] The trained neural network is directly used as the matching model for subsequent steps;

[0073] S3. Use the document string encoding obtained in step S1 as input to the trained fusion matching network and match it with the known key encoding.

[0074] The sentence and word segments obtained from the document string encoding in step S1 are sequentially input into the fusion matching network that has been trained in step S2, and matched with the known key encodings. The corresponding matching degree results and confidence scores are output to achieve the matching of documents with known keywords.

[0075] S4. Generate a set of encoded results and annotate the encodings in the original document;

[0076] The matching results obtained in step S3 are used to determine the location information containing the matching terms using the inverted index constructed in step S1. Together with the matching degree results and confidence obtained in step S3, an encoded result set is created, and the matching content in the original document is labeled. The encoded result set includes the location and confidence of the matched terms.

Claims

1. A string-encoded fuzzy retrieval method, characterized in that, The method includes: The document string encoding is collected and preprocessed to obtain the corresponding sentence and word segments, and an inverted index based on the word distribution is established. A fusion matching network consisting of a convolutional latent semantic model and a matching pyramid model is constructed, and the matching network is trained using the acquired data; The fusion matching network includes an input layer, a representation layer, and a matching layer. The input layer and representation layer of the convolutional latent semantic model and the input layer and representation layer of the matching pyramid model are constructed in the fusion matching network, respectively. The matching layer is used to concatenate the feature vectors obtained by the convolutional latent semantic model and the matching pyramid model into a one-dimensional feature vector. Two fully connected layers are used to classify the feature vector. The specific steps for constructing the input layer and representation layer of the convolutional latent semantic model in the fusion matching network include: The document strings are segmented and combined to form the vector space of the input layer of the convolutional latent semantic model; The input layer of the convolutional latent semantic model is constructed, and two sentences from different documents are output as two vector spaces through the input layer; A representation layer is constructed using convolutional layers, global max pooling layers, and fully connected layers with a finite number of neurons, which are used to extract one-dimensional sentence feature vectors from two different texts. The sentence segments and word segments obtained by encoding the document string are used as input to the trained fusion matching network, matched with known key encodings, and the matching results are obtained. The matching results are used to determine the location information containing the matching terms using the inverted index. Together with the matching results and confidence scores, an encoded result set is created, and the matching content in the original document is annotated.

2. The string encoding fuzzy retrieval method according to claim 1, characterized in that, The specific steps for constructing the input and representation layers of the matching pyramid model are as follows: Construct the input layer of the matching pyramid model, use the pre-trained word vector model to obtain the input word vectors, and obtain the similarity matrix by calculating the cosine distance between word vectors of different texts; The representation layer is constructed by two convolutional layers, two pooling layers, and a fully connected network. The features extracted by the representation layer are used as one-dimensional similarity feature vectors for two different texts.

3. The string encoding fuzzy retrieval method according to claim 1, characterized in that, The specific method for segmenting and combining document strings to form the vector space of the input layer of the convolutional latent semantic model is as follows: For English data in the document, sentences are segmented using a sliding window with a window length of 3, and the segments containing three words are further segmented into words. The vector space of each word after segmentation is 3-dimensional. The vectors of the three words after segmentation are combined to form an input layer with a vector space of 9 dimensions. For Chinese data in the document, a vector space with a dimension of 1.5 is obtained simply by segmenting the characters; The method of constructing a representation layer using convolutional layers, global max-pooling layers, and fully connected layers with a finite number of neurons, specifically for extracting one-dimensional sentence feature vectors from two different texts, is as follows: A representation layer is constructed using convolutional layers, global max-pooling layers, and a fully connected layer with 128 neurons. These layers are used to extract one-dimensional sentence feature vectors from two different texts. The activation function in the fully connected layer is... .

4. The string encoding fuzzy retrieval method according to claim 2, characterized in that, The process of using a pre-trained word vector model to obtain input word vectors and calculating the similarity matrix by measuring the cosine distance between word vectors of different texts is as follows: Obtain the input word vectors using the trained word vector model Word2Vec. Then, the similarity matrix is ​​obtained by calculating the cosine distance between word vectors of different texts. ( i*j (dimensional matrix), the formula is: ； The representation layer, consisting of two convolutional layers, two pooling layers, and a fully connected network, is used to construct a one-dimensional similarity feature vector for two different texts. A representation layer for the matching pyramid model in a fusion matching network is constructed, consisting of two convolutional layers, two pooling layers, and a fully connected layer. The first convolutional layer has a 3x3 kernel, a stride of 1, and two layers. Subsequent pooling layers employ a dynamic pooling strategy to obtain a fixed-length feature map. The pooling layer size is... The second convolutional layer has a 4x4 kernel, a stride of 1, and a layer number of 1. Subsequent pooling layers are max-pooling layers. The fully connected network has 128 neurons, and its activation function is... The features extracted from the representation layer are used as one-dimensional similarity feature vectors for two different texts.

5. The string encoding fuzzy retrieval method according to claim 1, characterized in that, The specific steps of using a matching layer to concatenate the feature vectors obtained from the convolutional latent semantic model and the matching pyramid model into a one-dimensional feature vector are as follows: A matching layer of a fusion matching network is constructed. This layer concatenates the two one-dimensional sentence feature vectors and one one-dimensional similarity feature vector obtained from the input and representation layers of the convolutional latent semantic model and the matching pyramid model, respectively, into a single one-dimensional feature vector. Two fully connected layers are then used to classify this feature vector, with the activation function being... And use softmax to calculate the probability that the output is a label of each class. The formula is: ， in, For the output of the matching layer, k For the predicted category, D This represents the number of categories.

6. The string encoding fuzzy retrieval method according to claim 1, characterized in that, The constructed fusion matching network still requires a training step: The pre-constructed fusion matching network is trained using pre-labeled sample data with varying matching levels, where the loss function is: ， The trained neural network is directly used as the matching model for subsequent steps.

7. The string encoding fuzzy retrieval method according to claim 1, characterized in that, The specific steps for obtaining sentence segmentation and word segmentation through document string encoding are as follows: Collect the string encoding of the document and perform sentence segmentation and word segmentation for English characters and / or Chinese characters respectively; We use existing tools to read strings from documents such as Word or PDF documents, and the open-source toolkit Apache POI to read strings from documents. For English characters in the documents, we use the natural language processing toolkit NLTK to perform sentence and word segmentation; for Chinese characters in the documents, we use the Chinese word segmentation toolkit jieba to perform sentence and word segmentation.

8. The string encoding fuzzy retrieval method according to claim 1, characterized in that, The steps of the inverted index based on term distribution are as follows: An inverted full index is constructed using Hadoop MapReduce, where the first item of the index is a word, and the second item is the document index and location containing that word.

Images