Approximate nearest neighbor text semantic retrieval method, device and equipment and storage medium

Abstract

The application relates to a kind of approximate nearest neighbor text semantic retrieval method, device, equipment and storage medium, the method comprises: obtaining text data to be retrieved, the text data to be retrieved is preprocessed using bag-of-words strategy to generate text vector;The weight of the text vector in the text library is calculated;The text vector is subjected to product quantization coding processing, and product quantization coding data is generated;Based on the weight and the product quantization coding data, the text vector is processed using a multi-table product quantization algorithm, and an approximate nearest neighbor text semantic retrieval result set is generated.The application is based on the nearest neighbor retrieval problem of semantic text, and a fast approximate nearest neighbor retrieval method is designed, which can greatly improve the speed of semantic-based text approximate nearest neighbor retrieval in a large-scale text database while ensuring the accuracy of the nearest neighbor retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text data processing, and particularly relates to a nearest neighbor text semantic retrieval method and device, equipment and a storage medium. BACKGROUND

[0002] Approximate nearest neighbor search (ANNS) on large-scale datasets is increasingly becoming a key problem in database-related applications, and encoding high-dimensional data and performing approximate nearest neighbor search based on the encoding has become an effective method to solve related problems.

[0003] The product quantization (PQ) method provides an effective way for high-dimensional data encoding, which can quickly and accurately realize the quantization encoding of high-dimensional data. Using a hash table structure, existing methods greatly improve the processing speed of approximate nearest neighbor search based on PQ encoding, and realize the approximate nearest neighbor search operation of a billion-scale dataset within hundreds of milliseconds. In the existing retrieval strategy design based on product quantization encoding, product quantization encoding is usually regarded as the terminal point of retrieval acceleration, and the short code value of product quantization encoding is considered to be more suitable for accelerating retrieval than the original data. However, in fact, although the dimension space of the quantization encoding is usually much lower than that of the original data, the retrieval time overhead of using traversal method for approximate nearest neighbor search in the encoding space is also huge, and the related technology cannot be directly applied to actual production scenarios (such as fingerprint matching and face recognition). This puts forward requirements for the approximate nearest neighbor search strategy design based on encoding. The IVFADC algorithm (inverted file with asymmetric distance calculation) proposed in the product quantization method uses coarse clustering and asymmetric distance calculation strategy to retrieve data. The inverted file with asymmetric distance calculation mechanism solves the approximate nearest neighbor search problem based on quantization encoding to a certain extent, but due to the failure to effectively utilize the local sensitivity characteristics of product quantization encoding, and the involvement of a certain degree of flooding operation in the retrieval process, the retrieval efficiency still has room for improvement.

[0004] In addition, approximate nearest neighbor search of long text data at the semantic level is an important issue in the field of information processing. After preprocessing of Chinese long text by word segmentation and stop word removal, the text is vectorized by bag-of-words model, TF-IDF (a commonly used weighting technique for information retrieval and data mining) and other algorithms, and the semantic similarity of the text is calculated by calculating the Euclidean distance, Jaccard distance and Hamming distance of the vectors. However, in the actual vector distance calculation process, the traversal calculation based on vector distance in the entire database is extremely time-consuming and cannot meet the demand of production environment for text similarity calculation and approximate nearest neighbor search. In addition, due to the difference between text vectorization algorithm and image label vectorization task, the weights of different identification bits in the text vector are often different. Some frequently appearing words cannot contribute enough difference in distinguishing the similarity of the text (for example, the word "happy" has lower distinguishing ability than the word "ecstatic"). This text vector distance calculation mode considering the weight further increases the overall time consumption of the approximate nearest neighbor search of the text vector.

[0005] The multi-table product quantization algorithm is an optimization of the existing product quantization coding approximate nearest neighbor search algorithm PQTable (an algorithm for quickly searching product quantization codes through hash table). The multi-table product quantization algorithm mainly optimizes the speed of the vector product quantization hash search algorithm through multi-table query. In the case of the same search accuracy, the search speed can be improved by 1-2 orders of magnitude, which can efficiently solve the approximate nearest neighbor search problem of data vectors. The multi-table product quantization algorithm has good scalability and parameter adjustment properties. However, since it is an algorithm applied to the approximate nearest neighbor search problem of image data, the multi-table product quantization algorithm does not have the ability to adjust the search results according to the weight information of the vector identification bits. Therefore, the direct application in text search has relatively poor effect and obvious optimization space. SUMMARY

[0006] Therefore, it is necessary to provide an approximate nearest neighbor text semantic search method, device, computer equipment and storage medium capable of greatly improving the speed of semantic-based text approximate nearest neighbor search in a large-scale text database while ensuring accuracy.

[0007] In one aspect, an approximate nearest neighbor text semantic search method is provided, which comprises:

[0008] Step A: obtaining text data to be searched, and preprocessing the text data to be searched by bag-of-words strategy to generate a text vector;

[0009] Step B: calculating the weight of the text vector in the text library;

[0010] Step C: product quantization encoding processing is performed on the text vector to generate product quantization encoding data;

[0011] Step D: based on the weight and the product quantization encoding data, the text vector is processed by using a multi-table product quantization algorithm to generate an approximate nearest neighbor text semantic retrieval result set.

[0012] In one of the embodiments, further comprising: before the text data is preprocessed by using the bag-of-words strategy to generate the text vector, the method further comprises: extracting text data from a preset text library; performing word segmentation processing on the text data, counting all appearing words, and removing stop words, and selecting quantitative words as mark bits based on a preset rule.

[0013] In one of the embodiments, further comprising: the obtaining of the to-be-retrieved text data and the preprocessing of the to-be-retrieved text data by using the bag-of-words strategy to generate the text vector comprises: obtaining the to-be-retrieved text data; performing word segmentation processing on the to-be-retrieved text data, counting all appearing words, and removing stop words; verifying whether the to-be-retrieved text data related words after removing the stop words are contained in the mark bits; encoding the words contained in the mark bits to generate corresponding text vectors.

[0014] In one of the embodiments, further comprising: the calculation of the weight of the text vector in the text library comprises: counting the frequency of occurrence of different words in the text library; based on the frequency of occurrence of different words, the weight of the words is calculated, and the calculation formula of the weight is:

[0015] w=e ^(-p)

[0016] wherein p represents the word frequency, and w represents the weight;

[0017] The sum of the weights of all mark bit words is defined as the weight of the paragraph i composed of the mark bits, denoted as W i .

[0018] In one of the embodiments, further comprising: the product quantization encoding processing of the text vector to generate product quantization encoding data comprises: performing segmentation processing on the text vector; clustering based on the data in each segment and selecting k clustering cores; selecting the vector segment most adjacent to the clustering core, and performing quantization encoding on the vector segment by using the corresponding clustering core number to generate product quantization encoding data.

[0019] In one of the embodiments, the method further comprises: the processing of the text vector by using the multi-table product quantization algorithm based on the weight and the product quantization encoded data to generate an approximate nearest neighbor text semantic retrieval result set comprises: the product quantization encoded data is segmented and stored in each hash bucket by using n hash buckets, and the product quantization encoded data in the hash coding table is retrieved and sorted by using the hash table lookup mechanism and the asymmetric distance coding algorithm to generate the approximate nearest neighbor text semantic retrieval result set; the data set meeting the retrieval return quantity requirement is filtered out from the approximate nearest neighbor text semantic retrieval result set by using the or operation, which is the approximate nearest neighbor result.

[0020] In one of the embodiments, the method further comprises: the asymmetric distance calculation method comprises: comparing the distance between the text vector and the k cluster cores, and selecting the shortest distance as D i ; the asymmetric distance is obtained based on the weight W i and the distance D i , and the calculation formula is:

[0021] R = D i * W i

[0022] Wherein, R represents the asymmetric distance.

[0023] On the other hand, an approximate nearest neighbor text semantic retrieval device is provided, and the device comprises:

[0024] A preprocessing module is configured to extract text data from a text library, and generate a text vector by preprocessing the text data by using a bag-of-words strategy.

[0025] A weight calculation module is configured to calculate the weight of the text vector in the text library.

[0026] A product quantization encoding module is configured to perform product quantization encoding processing on the related data of the text vector to generate product quantization encoded data.

[0027] An approximate nearest neighbor retrieval module is configured to process the text vector by using a multi-table product quantization algorithm based on the weight and the product quantization encoded data, so as to realize fast approximate nearest neighbor retrieval for the text vector.

[0028] In another aspect, a computer device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0029] Step A: obtaining text data to be retrieved, and preprocessing the text data to be retrieved by using a bag-of-words strategy to generate a text vector.

[0030] Step B: calculating the weight of the text vector in the text library;

[0031] Step C: performing product quantization coding processing on the text vector to generate product quantization coding data;

[0032] Step D: based on the weight and the product quantization coding data, processing the text vector using a multi-table product quantization algorithm to generate an approximate nearest neighbor text semantic retrieval result set.

[0033] In another aspect, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the following steps:

[0034] Step A: obtaining text data to be retrieved, and preprocessing the text data to be retrieved using a bag-of-words strategy to generate a text vector;

[0035] Step B: calculating the weight of the text vector in the text library;

[0036] Step C: performing product quantization coding processing on the text vector to generate product quantization coding data;

[0037] Step D: based on the weight and the product quantization coding data, processing the text vector using a multi-table product quantization algorithm to generate an approximate nearest neighbor text semantic retrieval result set.

[0038] The above approximate nearest neighbor text semantic retrieval method, device, equipment and storage medium, the method comprises: obtaining text data to be retrieved, and preprocessing the text data to be retrieved using a bag-of-words strategy to generate a text vector; calculating the weight of the text vector in the text library; performing product quantization coding processing on the text vector to generate product quantization coding data; based on the weight and the product quantization coding data, processing the text vector using a multi-table product quantization algorithm to generate an approximate nearest neighbor text semantic retrieval result set. Based on the semantic text nearest neighbor retrieval problem of the present application, a fast approximate nearest neighbor retrieval method is designed, which can greatly improve the speed of semantic-based text approximate nearest neighbor retrieval in a large-scale text database while ensuring the accuracy of nearest neighbor retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 It is an application environment diagram of the approximate nearest neighbor text semantic retrieval method in one embodiment;

[0040] Figure 2 It is a flowchart of the approximate nearest neighbor text semantic retrieval method in one embodiment;

[0041] Figure 3A structural block diagram of an approximate nearest neighbor text semantic retrieval device in one embodiment;

[0042] Figure 4 An internal structural diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0043] For the purpose, technical solutions and advantages of the present application to be more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0044] The approximate nearest neighbor text semantic retrieval method provided by the present application can be applied in an application environment as shown in Figure 1 . Wherein, the terminal 102 communicates with the data processing platform set on the server 104 through the network, wherein the terminal 102 can be but is not limited to various personal computers, notebook computers, smart phones and tablet computers, and the server 104 can be realized by an independent server or a server cluster composed of multiple servers.

[0045] Embodiment 1

[0046] In one embodiment, as shown in Figure 2 , an approximate nearest neighbor text semantic retrieval method is provided, which is applied to the terminal in Figure 1 for example, including the following steps:

[0047] S1: obtaining text data to be retrieved, and pre-processing the text data to be retrieved by using a bag-of-words strategy to generate a text vector.

[0048] It should be noted that before the text data is pre-processed by using the bag-of-words strategy to generate the text vector, the method further includes:

[0049] extracting text data from a preset text library, wherein the text library is all related data stored in the server;

[0050] performing word segmentation processing on the text data, counting all appearing words, and removing stop words (removing words such as "of", "and" and the like which have poor effect on distinguishing word meaning), and selecting quantitative words as mark bits based on a preset rule, wherein the preset rule can be the frequency of word appearance or other preset rules.

[0051] Further, the obtaining text data to be retrieved and pre-processing the text data to be retrieved by using the bag-of-words strategy to generate a text vector includes:

[0052] obtaining text data to be retrieved;

[0053] The text data to be searched is processed by word segmentation, all appearing words are counted, and stop words are removed. This step is the same as above;

[0054] It is verified whether the words of the text data to be searched after removing stop words are contained in the mark bit;

[0055] The words contained in the mark bit are encoded to generate a corresponding text vector;

[0056] For example, there are 10k words in the word library, and 1024 common words are taken. The encoding is 1024 bits. If a sentence contains some rarely used words, only the information in the 1024 common words is encoded. Stop words include: words with too high frequency, such as “de, is, not” (too common to affect similarity); words with too low frequency, such as “jiqu yudi, liyi” (too sparse to calculate similarity). For example, if the text contains eight mark bits “spring, summer, autumn, winter, clothing, food, housing, and travel”, the text encoding of text 1 “spring summer clothing spring food housing” is “21001110”, and the text encoding of text 2 “spring autumn housing travel travel” is “10100012”. The text encoding is the text vector.

[0057] S2: Calculate the weight of the text vector in the text library.

[0058] It should be noted that for the text retrieval task, there are two problems: one is that the words used to construct the text vector should appear more frequently in different texts, so words with relatively high frequency should be selected; two is that words with too high frequency (such as “de” and “or”) have relatively poor contribution to distinguishing text semantics, so the calculation of text semantic similarity should be based on words with lower frequency to a certain extent. This step calculates the frequency of different words in the text library to identify the importance of different words, specifically:

[0059] The frequency of different words in the text library is counted;

[0060] Based on the frequency of different words, the weight of the word is calculated, and the calculation formula of the weight is:

[0061] w = e ^(-p)

[0062] Where p represents the frequency, and w represents the weight;

[0063] The sum of the weights of all mark bit words is defined as the weight of the paragraph i composed of the mark bits, denoted as W i .

[0064] S3: Perform product quantization coding processing on the text vector to generate product quantization coding data.

[0065] Specifically, the text vector is segmented;

[0066] Based on the data in each segment, clustering is performed and k cluster cores are selected;

[0067] Select the vector segment closest to the cluster core, and use the corresponding cluster core number to quantize and encode the vector segment to generate product quantization coding data.

[0068] For example, in this embodiment, a 32-bit segmentation scheme is used for the characteristics of long text vector encoding length and discrete encoding, and a k-means clustering algorithm is used to form 256 cluster cores. Specifically, for a 1024-bit text code used in the system, it is divided into 32 groups according to 32-bit segmentation, and the text vector 1 is closest to the cluster core in 8 groups, which is recorded as its 8-bit product quantization coding data C (the encoding value of each bit ranges from 0 to 255).

[0069] S4: Based on the weight and the product quantization coding data, use a multi-table product quantization algorithm to process the text vector to generate an approximate nearest neighbor text semantic retrieval result set.

[0070] Specifically, the product quantization coding data is stored in n hash buckets, for example, in this embodiment, the product quantization coding is stored in 4 hash buckets, and each hash bucket contains 8 corresponding positions of the data with 0-255 encoding values.

[0071] In each hash bucket, a Keygen calculation module is used to return a series of data with the smallest difference between the corresponding encoding bits in the bucket and the data being searched. The Keygen module uses a hash table lookup mechanism and an asymmetric distance encoding algorithm to perform asymmetric distance retrieval and sorting on the product quantization coding data in the hash coding table to generate an approximate nearest neighbor text semantic retrieval result set. The asymmetric distance calculation method includes:

[0072] Compare the distance between the text vector and the k cluster cores, and select the shortest distance as D i ;

[0073] Since the text vocabulary labels corresponding to different cluster centers have different weights, during the calculation of this distance, the asymmetric distance is obtained based on the weight W i and the distance D i The calculation formula is:

[0074] R = D i x Wi

[0075] Wherein, R represents the asymmetric distance, that is, the distance between the weighted text encoding and the cluster center, and the asymmetric distance retrieval and sorting in the hash table are performed according to the distance;

[0076] All the retrieval results are combined, and the combined results are sorted according to the distance.

[0077] The data set meeting the retrieval return quantity requirement is filtered from the approximate nearest neighbor text semantic retrieval result set through an AND operation, that is, the approximate nearest neighbor result.

[0078] Wherein, the AND operation: first, the data set contained in the feedback results of different hash buckets is selected, and then the data set unique to each is selected according to the quantity requirement. The appearance of a data in the feedback results of a hash bucket means that the encoding of the data on the bit corresponding to the hash bucket is close to the data to be retrieved. If the feedback data in multiple buckets all contain a certain data, the relevance of the data to the data to be retrieved is relatively greater. The data set meeting the requirement is selected, that is, the approximate nearest neighbor result.

[0079] In the above approximate nearest neighbor text semantic retrieval method, the method comprises: obtaining a text data to be retrieved, pre-processing the text data to be retrieved to generate a text vector by using a bag-of-words strategy; calculating the weight of the text vector in the text library; performing product quantization encoding processing on the text vector to generate product quantization encoding data; and processing the text vector by using a multi-table product quantization algorithm based on the weight and the product quantization encoding data to generate an approximate nearest neighbor text semantic retrieval result set. Based on the semantic text nearest neighbor retrieval problem, a fast approximate nearest neighbor retrieval method is designed, which can greatly improve the speed of semantic text approximate nearest neighbor retrieval in a large-scale text database while ensuring the accuracy of the nearest neighbor retrieval.

[0080] It should be understood that, although Figure 2 The steps in the flowchart of FIG. 1 are displayed in sequence according to the arrows, but these steps are not necessarily executed in sequence according to the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, Figure 2 At least part of the steps in the flowchart of FIG. 1 can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times. The execution sequence of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with other steps or sub-steps or stages of other steps.

[0081] Embodiment 2

[0082] In one embodiment, as shown in Figure 3 An approximate nearest neighbor text semantic retrieval device is provided, comprising a preprocessing module, a weight calculation module, a product quantization encoding module and an approximate nearest neighbor search module, wherein:

[0083] The preprocessing module is configured to extract text data from a text library, and preprocess the text data to generate a text vector by using a bag-of-words strategy.

[0084] The weight calculation module is configured to calculate the weight of the text vector in the text library.

[0085] The product quantization encoding module is configured to perform product quantization encoding processing on the relevant data of the text vector to generate product quantization encoding data.

[0086] The approximate nearest neighbor search module is configured to process the text vector based on the weight and the product quantization encoding data by using a multi-table product quantization algorithm, so as to realize fast approximate nearest neighbor search for the text vector.

[0087] The device further comprises a marker bit generation module configured to, before preprocessing the text data to generate a text vector by using a bag-of-words strategy, extract text data from a preset text library, perform word segmentation processing on the text data, count all appearing words, remove stop words, and select a certain number of words as marker bits based on a preset rule.

[0088] As a more preferred embodiment, in the embodiment of the present application, the preprocessing module is specifically configured to:

[0089] Obtain text data to be searched;

[0090] Perform word segmentation processing on the text data to be searched, count all appearing words, and remove stop words;

[0091] Verify whether the text data to be searched after removing stop words is included in the marker bits;

[0092] Encode the words included in the marker bits to generate a corresponding text vector.

[0093] As a more preferred embodiment, in the embodiment of the present application, the weight calculation module is specifically configured to:

[0094] Count the frequency of occurrence of different words in the text library;

[0095] Calculate the weight of the words based on the frequency of occurrence of different words, and the calculation formula of the weight is:

[0096] w=e ^(-p)

[0097] wherein, p represents the word frequency, and w represents the weight;

[0098] define the sum of the weights of all the marked bit vocabularies as the weight of the paragraph i composed of the marked bits, denoted as W i .

[0099] As a more preferred embodiment, the product quantization encoding module is specifically used for:

[0100] segmenting the text vector;

[0101] performing clustering based on the data in each segment and selecting k cluster cores;

[0102] selecting the vector segment closest to the cluster core, using the corresponding cluster core number to quantize and encode the vector segment, and generating product quantization encoding data.

[0103] As a more preferred embodiment, the approximate nearest neighbor search module is specifically used for:

[0104] segmenting the product quantization encoding data using n hash buckets for storage;

[0105] in each hash bucket, using a hash table lookup mechanism and an asymmetric distance encoding algorithm to perform asymmetric distance search and sorting on the product quantization encoding data in the hash encoding table, and generating an approximate nearest neighbor text semantic search result set;

[0106] filtering out a data set meeting the search return quantity requirement from the approximate nearest neighbor text semantic search result set through an AND operation, which is the approximate nearest neighbor result.

[0107] wherein, the asymmetric distance calculation method comprises:

[0108] comparing the distance between the text vector and the k cluster cores, and selecting the shortest distance as D i ;

[0109] based on the weight W i and the distance D i , the asymmetric distance is obtained, and the calculation formula is:

[0110] R=D i ×W i

[0111] wherein, R represents the asymmetric distance.

[0112] The specific definition of the approximate nearest neighbor text semantic retrieval device can refer to the definition of the approximate nearest neighbor text semantic retrieval method in the foregoing, and will not be described here. Each module in the approximate nearest neighbor text semantic retrieval device described above can be realized by software, hardware, and a combination thereof, in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each of the above modules.

[0113] Embodiment 3

[0114] In an embodiment, a computer device, which can be a terminal, has an internal structure diagram as shown in Figure 4 The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement an approximate nearest neighbor text semantic retrieval method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad provided on the shell of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0115] Those skilled in the art can understand that Figure 4 The structure shown in the foregoing is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the diagram, or combine certain components, or have a different arrangement of components.

[0116] In an embodiment, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor implements the following steps when executing the computer program:

[0117] S1: obtaining text data to be retrieved, and pre-processing the text data to be retrieved by using a bag-of-words strategy to generate a text vector;

[0118] S2: calculating the weight of the text vector in the text library;

[0119] S3: performing a product quantization encoding process on the text vector to generate product quantization encoding data;

[0120] S4: based on the weight and the product quantization encoding data, processing the text vector by using a multi-table product quantization algorithm to generate an approximate nearest neighbor text semantic retrieval result set.

[0121] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0122] Before the text data is preprocessed by using the bag-of-words strategy to generate the text vector, text data is extracted from a preset text library, the text data is processed by using word segmentation, all appearing words are counted, and stop words are removed, and quantitative words are selected as mark bits based on a preset rule.

[0123] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0124] Obtaining text data to be searched;

[0125] Processing the text data to be searched by using word segmentation, counting all appearing words, and removing stop words;

[0126] Verifying whether the mark bits contain words related to the text data to be searched after the stop words are removed;

[0127] Encoding the words contained in the mark bits to generate corresponding text vectors.

[0128] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0129] Counting the frequency of occurrence of different words in the text library;

[0130] Based on the frequency of occurrence of different words, the weight of the word is calculated, and the calculation formula of the weight is:

[0131] w=e ^(-p)

[0132] Wherein, p represents the word frequency, and w represents the weight;

[0133] The sum of the weights of all mark bit words is defined as the weight of the paragraph i composed of the mark bits, denoted as W i .

[0134] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0135] Segmenting the text vector;

[0136] Clustering and selecting k cluster cores based on data in each segment;

[0137] Selecting the vector segment closest to the cluster core, quantizing the vector segment using the corresponding cluster core number to generate product quantization coding data.

[0138] In one embodiment, the processor also implements the following steps when executing the computer program:

[0139] Segmenting and storing the product quantization coding data using n hash buckets;

[0140] In each hash bucket, using a hash table lookup mechanism and an asymmetric distance coding algorithm to perform asymmetric distance retrieval and sorting on the product quantization coding data in the hash coding table, to generate an approximate nearest neighbor text semantic retrieval result set;

[0141] Filtering out a data set that meets the retrieval return quantity requirement from the approximate nearest neighbor text semantic retrieval result set through an AND operation, which is the approximate nearest neighbor result.

[0142] The asymmetric distance calculation method includes:

[0143] Comparing the distance between the text vector and the k cluster cores, and selecting the shortest distance as D i ;

[0144] Based on the weight W i and the distance D i , the asymmetric distance is obtained, and the calculation formula is:

[0145] R = D i × W i

[0146] Where R represents the asymmetric distance.

[0147] Embodiment 4

[0148] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0149] S1: Obtain the text data to be retrieved, and pre-process the text data to be retrieved using the bag-of-words strategy to generate a text vector;

[0150] S2: Calculate the weight of the text vector in the text library;

[0151] S3: Perform product quantization coding processing on the text vector to generate product quantization coding data;

[0152] S4: quantizing and encoding data based on the weight and the product, processing the text vector by using a multi-table product quantization algorithm to generate an approximate nearest neighbor text semantic retrieval result set.

[0153] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0154] Before the text data is preprocessed by using the bag-of-words strategy to generate the text vector, the text data is extracted from a preset text library, word segmentation processing is performed on the text data, all appearing words are counted, and stop words are removed, and quantitative words are selected as mark bits based on a preset rule.

[0155] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0156] Obtaining text data to be searched;

[0157] Performing word segmentation processing on the text data to be searched, counting all appearing words, and removing stop words;

[0158] Verifying whether the text data to be searched after the stop words are removed is related to the words in the mark bits;

[0159] Encoding the words contained in the mark bits to generate corresponding text vectors.

[0160] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0161] Counting the frequency of occurrence of different words in the text library;

[0162] Based on the frequency of occurrence of different words, the weight of the words is calculated, and the calculation formula of the weight is:

[0163] w=e ^(-p)

[0164] Wherein, p represents the word frequency, and w represents the weight;

[0165] Defining the sum of the weights of all mark bit words as the weight of a paragraph i composed of the mark bits, denoted as W i .

[0166] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0167] Segmenting the text vector;

[0168] Clustering based on the data in each segment and selecting k cluster cores;

[0169] Selecting a vector segment closest to the cluster core, using the corresponding cluster core number to quantitatively encode the vector segment, and generating product quantization encoding data.

[0170] In one embodiment, the computer program, when executed by the processor, also implements the following steps:

[0171] Segmenting and storing the product quantization encoding data using n hash buckets;

[0172] Within each hash bucket, using a hash table lookup mechanism and an asymmetric distance encoding algorithm to perform asymmetric distance retrieval and ordering on the product quantization encoding data in the hash coding table, and generating an approximate nearest neighbor text semantic retrieval result set;

[0173] Filtering a data set that meets the retrieval return quantity requirement from the approximate nearest neighbor text semantic retrieval result set through an AND operation, which is the approximate nearest neighbor result.

[0174] The asymmetric distance calculation method includes:

[0175] Comparing the distance between the text vector and the k cluster cores, and selecting the shortest distance as D i ;

[0176] Based on the weight W i and the distance D i , the asymmetric distance is obtained, and the calculation formula is:

[0177] R = D i × W i

[0178] Where R represents the asymmetric distance.

[0179] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0180] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0181] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for approximate nearest neighbor text semantic retrieval, the method comprising: The method comprises: acquiring text data to be searched, and preprocessing the text data to be searched using a bag-of-words strategy to generate a text vector; calculating the weight of the text vector in a text library; performing product quantization coding processing on the text vector to generate product quantization coding data; processing the text vector based on the weight and the product quantization coding data using a multi-table product quantization algorithm to generate a set of approximate nearest neighbor text semantic search results, comprising: segmenting and storing the product quantization coding data using n hash buckets; in each hash bucket, performing asymmetric distance retrieval and sorting on the product quantization coding data in the hash coding table using a hash table lookup mechanism and an asymmetric distance coding algorithm to generate a set of approximate nearest neighbor text semantic search results, wherein the asymmetric distance is determined by the product of the shortest distance between the text vector and k cluster cores and the weight corresponding to the text vocabulary label bit, and the k cluster cores are obtained by clustering based on the segmented processing results of the text vector; filtering a data set that meets the search return quantity requirement from the set of approximate nearest neighbor text semantic search results through an or operation, which is the set of approximate nearest neighbor text semantic search results.

2. The approximate nearest neighbor text semantic search method according to claim 1, wherein, Before preprocessing the text data using the bag-of-words strategy to generate a text vector, the method further comprises: extracting text data from a preset text library; performing word segmentation processing on the text data, counting all appearing words, removing stop words, and selecting a quantitative word as a label bit based on a preset rule.

3. The approximate nearest neighbor text semantic search method according to claim 1 or 2, characterized in that, The method of acquiring text data to be searched and preprocessing the text data to be searched using a bag-of-words strategy to generate a text vector comprises: acquiring text data to be searched; performing word segmentation processing on the text data to be searched, counting all appearing words, and removing stop words; verifying whether the text data to be searched after removing stop words is included in the label bit; encoding the words included in the label bit to generate a corresponding text vector.

4. The method of claim 1, wherein the step of calculating the similarity between the query and the text data is performed by using a cosine similarity function. The method of calculating the weight of the text vector in the text library comprises: counting the frequency of different words appearing in the text library; based on the frequency of different words, calculating the weight of the words, and the calculation formula of the weight is: ； where p represents the word frequency and w represents the weight. The sum of the weights of all the token position vocabularies is defined as the weight of the paragraph composed of the token positions, denoted as . .

5. The method of claim 1, wherein the approximate nearest neighbor text semantic search method is characterized by, The method of performing product quantization coding processing on the text vector to generate product quantization coding data comprises: performing segmented processing on the text vector; based on the data in each segment, clustering and selecting k cluster cores; selecting a vector segment that is nearest to the cluster core, quantization coding the vector segment using the corresponding cluster core number to generate product quantization coding data.

6. The approximate nearest neighbor text semantic search method of claim 1, wherein, The asymmetric distance calculation method comprises: comparing the distance of the text vector with the k cluster cores, and selecting the shortest distance as ; based on the weight and the distance the asymmetric distance is obtained, and the calculation formula is: ； wherein denotes the asymmetric distance.

7. An apparatus for implementing the method of claim 1, wherein the method of approximate nearest neighbor text semantic retrieval is implemented by the apparatus. The device comprises: a preprocessing module configured to extract text data from a text library, and preprocess the text data using a bag-of-words strategy to generate a text vector; a weight calculation module configured to calculate the weight of the text vector in the text library; a product quantization coding module configured to perform product quantization coding processing on the relevant data of the text vector to generate product quantization coding data; An approximate nearest neighbor search module is configured to quantize the text vector using a multi-table product quantization algorithm based on the weights and the product quantized encoding data, to achieve fast approximate nearest neighbor search for the text vector.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

Images