Data retrieval method and apparatus, computer device, and storage medium

Abstract

The application relates to a data retrieval method and device, computer equipment and a storage medium. The application relates to the technical field of big data. The method comprises the following steps: acquiring text information input by a user; dividing the text information into multiple sentences by using a sentence division model, and performing coding conversion processing on each sentence to obtain a high-dimensional sentence feature vector corresponding to each sentence; generating a hash code corresponding to the high-dimensional sentence feature vector by using a hash function, querying a database for a hash code identical to the hash code of the high-dimensional sentence feature vector, and taking candidate data corresponding to the identical hash code as target retrieval data corresponding to the text information. The method can improve the accuracy of retrieval.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data retrieval method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the development of big data, the increasing complexity and diversity of data has made refined data processing very difficult. Therefore, an efficient data retrieval method is a key area of ​​research at this stage.

[0003] Traditional data retrieval methods use hash functions to transform high-dimensional feature vectors into low-dimensional binary hash codes. During retrieval, the data to be retrieved is simply converted into a hash code using a trained hash function, and then this hash code is compared with the hash codes of all data in the database. The retrieval result is returned based on the calculated similarity. However, the process of converting the data into a binary hash code reduces the dimensionality of the text information, resulting in a hash code containing less textual information and thus lower accuracy in data retrieval. Summary of the Invention

[0004] Therefore, it is necessary to provide a data retrieval method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0005] Firstly, this application provides a data retrieval method. The method includes:

[0006] Obtain text information input by the user;

[0007] The text information is divided into multiple sentences using a sentence segmentation model, and each sentence is encoded and converted to obtain a high-dimensional sentence feature vector corresponding to each sentence.

[0008] A hash code corresponding to the high-dimensional statement feature vector is generated using a hash function. The database is then searched for hash codes that are the same as those of the high-dimensional statement feature vector. The candidate data corresponding to the same hash code is then used as the target retrieval data corresponding to the text information.

[0009] Optionally, the sentence segmentation model includes a word embedding layer and a segmentation layer. The text information is divided into multiple sentences using the sentence segmentation model, and each sentence is encoded and converted to obtain a high-dimensional sentence feature vector corresponding to each sentence, including:

[0010] For each word segment in the text information, multiple word embedding vectors are determined through the word embedding layer, and the positional encoding of each word segment in the text information is performed to obtain the positional encoding vector of the word segment.

[0011] Each statement in the text information is divided into multiple short statements through the segmentation layer, and each short statement is transformed to obtain multiple short statement vectors corresponding to the statement.

[0012] Based on the word embedding vectors of the words contained in the sentences in the text information, the position encoding vectors of the words contained in the sentences, and the short sentence vectors corresponding to the sentences, the high-dimensional sentence feature vectors of each sentence in the text information are determined.

[0013] Optionally, determining the high-dimensional sentence feature vector of each sentence in the text information based on the word embedding vector of the words contained in the sentences, the position encoding vector of the words contained in the sentences, and the short sentence vector corresponding to the sentences includes:

[0014] For each statement, the word embedding vectors of the words contained in the statement, the position encoding vectors of the words contained in the statement, and the short statement vectors corresponding to the statement are superimposed to obtain the high-dimensional statement feature vector of the statement.

[0015] Optionally, the step of generating a hash code corresponding to the high-dimensional statement feature vector using a hash function, querying the database for candidate data corresponding to the hash code that is the same as the hash code of the high-dimensional statement feature vector, and using the candidate data with the same hash code as the target retrieval data corresponding to the text information includes:

[0016] For each high-dimensional statement feature vector, a hash code corresponding to the high-dimensional statement feature vector is generated using a hash function;

[0017] Select any candidate data in the database and determine whether the hash bit in the hash code of the candidate data is consistent with the hash bit in the hash code corresponding to the high-dimensional statement feature vector;

[0018] If the hash bit in the hash code of the candidate data does not match the hash bit in the hash code corresponding to the high-dimensional statement feature vector, return to the step of selecting any candidate data in the database until the hash bit in the hash code of the selected candidate data matches the hash bit in the hash code corresponding to the high-dimensional statement feature vector. Then, the candidate data with the matching hash bit is used as the target retrieval data corresponding to the text information.

[0019] Optionally, the method further includes:

[0020] Obtain sample text information and the hash codes of the feature vectors of each high-dimensional sample sentence in the sample text information;

[0021] The sample text information is divided into multiple sentences using an initial sentence segmentation model, and each sentence is encoded and converted to obtain a high-dimensional sentence feature vector for each sentence.

[0022] A hash code is generated for each of the high-dimensional sentence feature vectors using a hash function. The initial sentence segmentation model is then trained based on the hash codes of the high-dimensional sample sentence feature vectors and the hash codes of the high-dimensional sentence feature vectors to obtain the sentence segmentation model.

[0023] Optionally, training the initial sentence segmentation model based on the hash codes of the feature vectors of each of the high-dimensional sample sentences and the hash codes of the feature vectors of each of the high-dimensional sentences to obtain the sentence segmentation model includes:

[0024] Determine whether there exists a hash code for a high-dimensional statement feature vector that is different from the hash code of the high-dimensional sample statement feature vector;

[0025] If there is a hash code for a high-dimensional sentence feature vector that is different from the hash code of the high-dimensional sample sentence feature vector, the partitioning parameters of the initial sentence partitioning model are adjusted according to the difference between the hash code of the high-dimensional sample sentence feature vector and the hash code of the high-dimensional sentence feature vector. Then, the process of dividing the sample text information into multiple sentences through the initial sentence partitioning model and encoding and converting each sentence to obtain the high-dimensional sentence feature vector of each sentence is repeated until there is no hash code for a high-dimensional sentence feature vector that is different from the hash code of the high-dimensional sample sentence feature vector.

[0026] The initial statement partitioning model is used as the statement partitioning model if there is no hash code corresponding to the hash code of the high-dimensional statement feature vector that is different from the hash code of the high-dimensional sample statement feature vector.

[0027] Optionally, the method further includes:

[0028] Obtain the high-dimensional sample sentence feature vector and the sample data corresponding to the high-dimensional sample sentence feature vector;

[0029] Using an initial hash function, calculate the hash code of the feature vector of each high-dimensional sample statement and the hash code of each sample data, and calculate the similarity between the hash code of the feature vector of each high-dimensional sample statement and the hash code of each sample data.

[0030] If there are different hash codes of high-dimensional sample sentence feature vectors and the similarity between them and the hash codes of sample data is greater than the similarity threshold, adjust the hash code generation parameters of the initial hash function, and return to the step of calculating the hash code of each high-dimensional sample sentence feature vector and the hash code of each sample data through the initial hash function, until only the hash code of the high-dimensional sample sentence feature vector and the hash code of the sample data corresponding to the high-dimensional sample sentence feature vector have a similarity greater than the similarity threshold.

[0031] The initial hash function is defined as the hash function that has a similarity greater than a similarity threshold between the hash code of the feature vector of the high-dimensional sample statement and the hash code of the sample data corresponding to the feature vector of the high-dimensional sample statement.

[0032] Secondly, this application also provides a data retrieval device. The device includes:

[0033] The acquisition module is used to acquire text information input by the user; the text information is the text data that the user needs to retrieve.

[0034] The segmentation module is used to divide the text information into multiple sentences using a sentence segmentation model, and to encode and convert each sentence to obtain a high-dimensional sentence feature vector corresponding to each sentence.

[0035] The filtering module is used to generate a hash code corresponding to the high-dimensional statement feature vector using a hash function, query the database for candidate data corresponding to the hash code that is the same as the hash code of the high-dimensional statement feature vector, and use the candidate data with the same hash code as the target data.

[0036] Optionally, the partitioning module is specifically used for:

[0037] For each word segment in the text information, multiple word embedding vectors are determined through the word embedding layer, and the positional encoding of each word segment in the text information is performed to obtain the positional encoding vector of the word segment.

[0038] Each statement in the text information is divided into multiple short statements through the segmentation layer, and each short statement is transformed to obtain multiple short statement vectors corresponding to the statement.

[0039] Based on the word embedding vectors of the words contained in the sentences in the text information, the position encoding vectors of the words contained in the sentences, and the short sentence vectors corresponding to the sentences, the high-dimensional sentence feature vectors of each sentence in the text information are determined.

[0040] Optionally, the partitioning module is specifically used for:

[0041] For each statement, the word embedding vectors of the words contained in the statement, the position encoding vectors of the words contained in the statement, and the short statement vectors corresponding to the statement are superimposed to obtain the high-dimensional statement feature vector of the statement.

[0042] Optionally, the filtering module is specifically used for:

[0043] For each high-dimensional statement feature vector, a hash code corresponding to the high-dimensional statement feature vector is generated using a hash function;

[0044] Select any candidate data in the database and determine whether the hash bit in the hash code of the candidate data is consistent with the hash bit in the hash code corresponding to the high-dimensional statement feature vector;

[0045] If the hash bit in the hash code of the candidate data does not match the hash bit in the hash code corresponding to the high-dimensional statement feature vector, return to the step of selecting any candidate data in the database until the hash bit in the hash code of the selected candidate data matches the hash bit in the hash code corresponding to the high-dimensional statement feature vector. Then, the candidate data with the matching hash bit is used as the target retrieval data corresponding to the text information.

[0046] Optionally, the device further includes:

[0047] The sample acquisition module is used to acquire sample text information and the hash codes of the feature vectors of each high-dimensional sample sentence in the sample text information;

[0048] The encoding module is used to divide the sample text information into multiple sentences using an initial sentence segmentation model, and to encode and transform each sentence to obtain a high-dimensional sentence feature vector for each sentence.

[0049] The training module is used to generate hash codes corresponding to the feature vectors of each high-dimensional sentence using a hash function, and to train the initial sentence segmentation model based on the hash codes of the feature vectors of each high-dimensional sample sentence and the hash codes of the feature vectors of each high-dimensional sentence to obtain the sentence segmentation model.

[0050] Optionally, the training module is specifically used for:

[0051] Determine whether there exists a hash code for a high-dimensional statement feature vector that is different from the hash code of the high-dimensional sample statement feature vector;

[0052] If there is a hash code for a high-dimensional sentence feature vector that is different from the hash code of the high-dimensional sample sentence feature vector, the partitioning parameters of the initial sentence partitioning model are adjusted according to the difference between the hash code of the high-dimensional sample sentence feature vector and the hash code of the high-dimensional sentence feature vector. Then, the process of dividing the sample text information into multiple sentences through the initial sentence partitioning model and encoding and converting each sentence to obtain the high-dimensional sentence feature vector of each sentence is repeated until there is no hash code for a high-dimensional sentence feature vector that is different from the hash code of the high-dimensional sample sentence feature vector.

[0053] The initial statement partitioning model is used as the statement partitioning model if there is no hash code corresponding to the hash code of the high-dimensional statement feature vector that is different from the hash code of the high-dimensional sample statement feature vector.

[0054] Optionally, the device further includes:

[0055] The vector acquisition module is used to acquire the feature vector of the high-dimensional sample statement and the sample data corresponding to the feature vector of the high-dimensional sample statement.

[0056] The similarity calculation module is used to calculate the hash code of the feature vector of each high-dimensional sample sentence and the hash code of each sample data through an initial hash function, and to calculate the similarity between the hash code of the feature vector of each high-dimensional sample sentence and the hash code of each sample data.

[0057] The judgment module is used to adjust the hash code generation parameters of the initial hash function when there are different hash codes of high-dimensional sample sentence feature vectors and the similarity between them and the hash codes of sample data is greater than a similarity threshold. Then, it returns to the step of calculating the hash code of each high-dimensional sample sentence feature vector and the hash code of each sample data through the initial hash function until only the hash code of the high-dimensional sample sentence feature vector and the hash code of the sample data corresponding to the high-dimensional sample sentence feature vector have a similarity greater than a similarity threshold.

[0058] The determination module is used to select an initial hash function whose similarity to the hash code of the high-dimensional sample statement feature vector and the hash code of the sample data corresponding to the high-dimensional sample statement feature vector is greater than a similarity threshold, as the hash function.

[0059] Thirdly, this application provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in any one of the first aspects.

[0060] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0061] Fifthly, this application provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0062] The aforementioned data retrieval method, apparatus, computer equipment, and storage medium acquire text information input by the user; divide the text information into multiple sentences using a sentence segmentation model, and encode and transform each sentence to obtain a high-dimensional sentence feature vector corresponding to each sentence; generate a hash code corresponding to the high-dimensional sentence feature vector using a hash function; query a database for a hash code that matches the hash code of the high-dimensional sentence feature vector; and use the candidate data corresponding to the same hash code as the target retrieval data corresponding to the text information. By dividing the text information into multiple sentences and transforming each sentence into a high-dimensional sentence feature vector, the dimensionality of the text information is increased, and the dimensionality of the feature information of each sentence is strengthened. This allows the hash function to generate hash codes for multiple high-dimensional sentence feature vectors corresponding to the text information based on the high-dimensional sentence feature vector, thereby avoiding a reduction in the dimensionality of the text information during the hash function transformation, and thus improving the accuracy of the retrieval. Attached Figure Description

[0063] Figure 1 This is a flowchart illustrating a data retrieval method in one embodiment;

[0064] Figure 2 This is a flowchart illustrating the steps for determining the feature vector of a high-dimensional statement in one embodiment.

[0065] Figure 3 This is a flowchart illustrating the training steps of a hash function in one embodiment;

[0066] Figure 4 This is a flowchart illustrating a data retrieval example in one embodiment;

[0067] Figure 5 This is a structural block diagram of a data retrieval device in one embodiment;

[0068] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0070] The data retrieval method provided in this application can also be applied to a server, or to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc. The terminal increases the dimensionality of the text information by dividing it into multiple sentences and converting each sentence into a high-dimensional sentence feature vector. This strengthens the dimensionality of the feature information of each sentence, allowing the hash function to generate hash codes for multiple high-dimensional sentence feature vectors corresponding to the text information based on these high-dimensional sentence feature vectors. This avoids reducing the dimensionality of the text information during the hash function's transformation, thereby improving the accuracy of the retrieval.

[0071] In one embodiment, such as Figure 1 As shown, a data retrieval method is provided. Taking the application of this method to a terminal as an example, the method includes the following steps:

[0072] Step S101: Obtain the text information input by the user.

[0073] In this embodiment, the terminal obtains the text information input by the user by receiving the text data entered by the user at the display port.

[0074] Step S102: The text information is divided into multiple sentences using a sentence segmentation model, and each sentence is encoded and converted to obtain a high-dimensional sentence feature vector corresponding to each sentence.

[0075] In this embodiment, the terminal divides the acquired text information into multiple sentences using a sentence segmentation model. These sentences are long sentences. The terminal performs encoding and transformation processing on each long sentence to obtain a high-dimensional sentence feature vector corresponding to each sentence. The sentence segmentation model is the BERT model (Bidirectional Encoder Representation from Transformers), and the specific segmentation process will be explained in detail later.

[0076] Step S103: Generate a hash code corresponding to the high-dimensional statement feature vector using a hash function, query the database for the hash code that is the same as the hash code of the high-dimensional statement feature vector, and use the candidate data corresponding to the same hash code as the target retrieval data corresponding to the text information.

[0077] In this embodiment, the terminal uses a hash function to convert the high-dimensional statement feature vector corresponding to each statement into a hash code, and then queries the terminal's database for the hash code that matches the hash code of each high-dimensional statement feature vector. The terminal selects the candidate data with the same hash code in the database as the target retrieval data corresponding to the text information.

[0078] Based on the above scheme, by dividing the text information into multiple sentences and converting each sentence into a high-dimensional sentence feature vector, the dimensionality of the text information is increased and the dimensionality of the feature information of each sentence is strengthened. This allows the hash function to generate hash codes of multiple high-dimensional sentence feature vectors corresponding to the text information based on the high-dimensional sentence feature vectors, thereby avoiding the reduction of the dimensionality of the text information when the hash function transforms the text information, thus improving the accuracy of retrieval.

[0079] Optional, such as Figure 2 As shown, the sentence segmentation model includes a word embedding layer and a segmentation layer. Through this model, text information is divided into multiple sentences, and each sentence is encoded and transformed to obtain a high-dimensional sentence feature vector, including:

[0080] Step S201: For each word segment in the text information, multiple word embedding vectors are determined through the word embedding layer, and the positional encoding of the order of each word segment in the text information is performed to obtain the positional encoding vector of the word segment.

[0081] In this embodiment, the terminal inserts the start and end points of each word segment in the text information into the word embedding layer, and then encodes the words in the two word embedding layers to obtain multiple word embedding vectors. The terminal then performs position encoding according to the position of each word in the text information, following a sequential order, to obtain the position encoding vector for each word.

[0082] Step S202: Divide each statement in the text information into multiple short statements by dividing the text into layers, and transform each short statement to obtain multiple short statement vectors corresponding to the statement.

[0083] In this embodiment, the terminal divides each sentence in the text information into multiple short sentences through a segmentation layer to increase the dimensionality of the sentence, and performs encoding and conversion processing on each short sentence to obtain multiple short sentence vectors corresponding to each sentence.

[0084] Step S203: Determine the high-dimensional sentence feature vector of each sentence in the text information based on the word embedding vector of the words contained in the sentence, the position encoding vector of the words contained in the sentence, and the short sentence vector corresponding to the sentence.

[0085] In this embodiment, the terminal integrates and superimposes the word embedding vectors of each word segment contained in the text information, the position encoding vectors of the word segment contained in the text information, and the short sentence vectors corresponding to each sentence to obtain the high-dimensional sentence feature vector of each sentence in the text information.

[0086] Based on the above scheme, by segmenting each sentence into words and short sentences, and encoding each word and each short sentence into a vector, a high-dimensional sentence feature vector is obtained for each sentence, which improves the dimensionality of each sentence and thus improves the accuracy of retrieval.

[0087] Optionally, based on the word embedding vectors of the segments contained in the sentences in the text information, the position encoding vectors of the segments contained in the sentences, and the short sentence vectors corresponding to the sentences, the high-dimensional sentence feature vectors of each sentence in the text information are determined, including:

[0088] For each statement, the word embedding vectors of the words contained in the statement, the position encoding vectors of the words contained in the statement, and the short statement vectors corresponding to the statement are superimposed to obtain the high-dimensional statement feature vector.

[0089] In this embodiment, for each statement, the terminal performs vector concatenation processing on the word embedding vectors of the words contained in the statement, the position encoding vectors of the words contained in the statement, and the short statement vectors corresponding to the statement, to obtain the high-dimensional statement feature vector of the statement. Similarly, through the above steps, the high-dimensional statement feature vector of each statement is obtained.

[0090] Based on the above scheme, a high-dimensional sentence feature vector is obtained by superimposing the word segmentation vectors of each sentence and the vectors of short sentences, which improves the convenience of dimensionality enhancement for each sentence and avoids overly complicated operations.

[0091] Optionally, a hash code corresponding to a high-dimensional statement feature vector is generated using a hash function. The database is then queried for candidate data corresponding to hash codes that match the hash codes of the high-dimensional statement feature vectors. These candidate data with the same hash codes are then used as the target retrieval data for the text information. This process includes: generating a hash code corresponding to each high-dimensional statement feature vector using a hash function; selecting any candidate data from the database and determining whether the hash bit in the candidate data's hash code matches the hash bit in the hash code corresponding to the high-dimensional statement feature vector; if the hash bit in the candidate data's hash code does not match the hash bit in the hash code corresponding to the high-dimensional statement feature vector, returning to the step of selecting any candidate data from the database, until the hash bit in the selected candidate data matches the hash bit in the hash code corresponding to the high-dimensional statement feature vector, then using the candidate data with the matching hash bits as the target retrieval data for the text information.

[0092] In this embodiment, when the terminal obtains the high-dimensional statement feature vector corresponding to each statement, it generates a hash code corresponding to each high-dimensional statement feature vector using a hash function. The terminal randomly selects candidate data from its database and determines whether the hash code of the candidate data matches all the hash bits in the hash code of the high-dimensional statement feature vector. If the hash bits in the hash code of the candidate data match the hash bits in the hash code corresponding to the high-dimensional statement feature vector, the terminal uses the candidate data as the target retrieval data corresponding to the text information. If the hash bits in the hash code of the candidate data match the hash bits in the hash code corresponding to the high-dimensional statement feature vector, the terminal returns to the step of randomly selecting candidate data from its database until the hash bits in the hash code of the selected candidate data match the hash bits in the hash code corresponding to the high-dimensional statement feature vector. Finally, the terminal uses the candidate data with matching hash bits as the target retrieval data corresponding to the text information.

[0093] Based on the above scheme, by filtering the target retrieval data corresponding to the text information through hash codes, it is only necessary to compare whether the hash bits of the two statements are consistent. This is equivalent to obtaining the similarity between the two statements by performing an XOR operation, thus reducing the response time during retrieval.

[0094] Optionally, the method further includes: obtaining sample text information and hash codes of the feature vectors of each high-dimensional sample sentence in the sample text information; dividing the sample text information into multiple sentences using an initial sentence segmentation model, and encoding and converting each sentence to obtain a high-dimensional sentence feature vector for each sentence; generating hash codes corresponding to each high-dimensional sentence feature vector using a hash function, and training the initial sentence segmentation model based on the hash codes of each high-dimensional sample sentence feature vector and the hash codes of each high-dimensional sentence feature vector to obtain a sentence segmentation model.

[0095] In this embodiment, the terminal retrieves previously input text information from the historical search records as sample text information. It then queries the hash codes of the high-dimensional sample sentence feature vectors within this sample text information from the hash codes generated by the previously generated hash function. The terminal uses an initial sentence segmentation model to divide the sample text information into multiple sentences and performs encoding and transformation processing on each sentence to obtain high-dimensional sentence feature vectors for each sentence. The terminal uses a hash function to generate a hash code corresponding to the feature vector of each high-dimensional sentence. Based on the hash codes of the feature vectors of each high-dimensional sample sentence and the hash codes of the feature vectors of each high-dimensional sentence, the terminal trains the initial sentence segmentation model to obtain the sentence segmentation model.

[0096] Based on the above scheme, the accuracy of the sentence segmentation model was improved by training the sentence segmentation model.

[0097] Optionally, based on the hash codes of the feature vectors of each high-dimensional sample sentence and the hash codes of the feature vectors of each high-dimensional sentence, the initial sentence segmentation model is trained to obtain a sentence segmentation model, including: determining whether there is a hash code of a high-dimensional sentence feature vector that is different from the hash code of the feature vector of the high-dimensional sample sentence; if there is a hash code of a high-dimensional sentence feature vector that is different from the hash code of the feature vector of the high-dimensional sample sentence, adjusting the segmentation parameters of the initial sentence segmentation model according to the difference value between the hash codes of the feature vectors of the high-dimensional sample sentence and the hash codes of the feature vectors of the high-dimensional sentence, and returning to execute the steps of dividing the sample text information into multiple sentences through the initial sentence segmentation model, and encoding and converting each sentence to obtain the high-dimensional sentence feature vector of each sentence, until there is no hash code of a high-dimensional sentence feature vector that is different from the hash code of the feature vector of the high-dimensional sample sentence; the initial sentence segmentation model corresponding to the hash code of the high-dimensional sentence feature vector that is different from the hash code of the feature vector of the high-dimensional sample sentence is used as the sentence segmentation model.

[0098] In this embodiment, when the terminal trains the initial sentence segmentation model, it determines whether there exists a hash code for a high-dimensional sentence feature vector that is different from the hash code of the high-dimensional sample sentence feature vector. If there is a hash code for a high-dimensional sentence feature vector that is different from the hash code of the high-dimensional sample sentence feature vector, the terminal adjusts the segmentation parameters of the initial sentence segmentation model based on the difference between the hash codes of the high-dimensional sample sentence feature vector and the hash codes of the high-dimensional sentence feature vector. It then returns to the previous step of dividing the sample text information into multiple sentences using the initial sentence segmentation model and encoding and converting each sentence to obtain the high-dimensional sentence feature vector of each sentence, until no hash code for a high-dimensional sentence feature vector that is different from the hash code of the high-dimensional sample sentence feature vector exists. The terminal uses the initial sentence segmentation model corresponding to the hash code of the high-dimensional sentence feature vector that is not different from the hash code of the high-dimensional sample sentence feature vector as the sentence segmentation model.

[0099] Based on the above scheme, the partitioning parameters of the initial sentence partitioning model are adjusted by using the difference between the hash code of the feature vector of the high-dimensional sample sentence and the hash code of the feature vector of the high-dimensional sentence, thereby improving the accuracy of the initial sentence partitioning model adjustment.

[0100] Optional, such as Figure 3 As shown, the method also includes:

[0101] Step S301: Obtain the high-dimensional sample sentence feature vector and the sample data corresponding to the high-dimensional sample sentence feature vector.

[0102] In this embodiment, the terminal obtains the high-dimensional sample sentence feature vectors divided by the sentence segmentation model from the historical search records, as well as the sample data corresponding to each high-dimensional sample sentence feature vector.

[0103] Step S302: Using the initial hash function, calculate the hash code of each high-dimensional sample sentence feature vector and the hash code of each sample data, and calculate the similarity between the hash code of each high-dimensional sample sentence feature vector and the hash code of each sample data.

[0104] In this embodiment, the terminal calculates the hash code of each high-dimensional sample statement feature vector and the hash code of each sample data through an initial hash function, and calculates the similarity between the hash code of each high-dimensional sample statement feature vector and the hash code of each sample data.

[0105] Specifically, when training the hash function, the terminal uses labeled high-dimensional sample sentence feature vectors to learn the hash code. For example, if high-dimensional sample sentence feature vector a is "car", high-dimensional sample sentence feature vector b is "truck", and high-dimensional sample sentence feature vector c is "road", then the terminal determines that the similarity between high-dimensional sample sentence feature vector a and high-dimensional sample sentence feature vector b is higher than the similarity between high-dimensional sample sentence feature vector a and high-dimensional sample sentence feature vector c. That is, the terminal determines that high-dimensional sample sentence feature vector a and high-dimensional sample sentence feature vector b belong to the same category (car), and labels Label_ab = 1; high-dimensional sample sentence feature vector a and high-dimensional sample sentence feature vector c belong to different categories, and labels Label_ac = 0. Therefore, when training the hash function, the terminal aims to make the hash code of each high-dimensional sample sentence feature vector infinitely close to the pre-labeled Label information. For example, in an ideal state, the hash code of the high-dimensional sample sentence feature vector a is h_a = [1,1,0], the hash code of the high-dimensional sample sentence feature vector b is h_b = [1,1,0], and the hash code of the high-dimensional sample sentence feature vector c is h_c = [0,0,1]. Then, the similarity between h_a and h_b can be obtained by using the algorithm similarity_ab = (h_a_bit1(xor)h_b_bit1 + h_a_bit2(xor)h_b_bit2 + h_a_bit3(xor)h_b_bit3) / h_length, which gives similarity_ab = 1. Similarly, similarity_ac = 0 can be calculated. Similarity_ab is equal to Label_ab = 1, and similarity_ac is equal to Label_ac = 0. That is, to make the hash code similarity_xy of any two high-dimensional sample sentence feature vector data x and y infinitely close to Label_xy, the hash function is trained through the above steps to improve the features of the hash code that stores the semantic similarity information of the data, so that the trained hash function can convert the high-dimensional feature vector of any high-dimensional sample sentence feature vector data into a hash code.

[0106] Step S303: If there are different hash codes of high-dimensional sample sentence feature vectors and the similarity between them and the hash codes of sample data is greater than the similarity threshold, adjust the hash code generation parameters of the initial hash function, and return to the step of calculating the hash code of each high-dimensional sample sentence feature vector and the hash code of each sample data through the initial hash function, until only the hash code of the high-dimensional sample sentence feature vector and the similarity between them and the hash code of the sample data corresponding to the high-dimensional sample sentence feature vector are greater than the similarity threshold.

[0107] In this embodiment, the terminal determines whether there exist different high-dimensional sample statement feature vectors whose hash codes are greater than a similarity threshold with the hash codes of sample data. If there are different high-dimensional sample statement feature vectors whose hash codes are greater than the similarity threshold with the hash codes of sample data, the terminal adjusts the hash code generation parameters of the initial hash function and returns to step S302. The iteration operation stops when only the hash codes of high-dimensional sample statement feature vectors with the same hash code are greater than the similarity threshold with the hash codes of the sample data corresponding to the high-dimensional sample statement feature vectors, or the hash codes of high-dimensional sample statement feature vectors and the sample data corresponding to the high-dimensional sample statement feature vectors are greater than the similarity threshold.

[0108] Step S304: Use the hash function that has a similarity greater than a similarity threshold between the hash code of the feature vector of the high-dimensional sample statement and the hash code of the sample data corresponding to the feature vector of the high-dimensional sample statement.

[0109] In this embodiment, the terminal uses an initial hash function that has a similarity greater than a similarity threshold between the hash code of the high-dimensional sample statement feature vector and the hash code of the sample data corresponding to the high-dimensional sample statement feature vector.

[0110] Based on the above scheme, by training the hash function, we can ensure that the hash function can accurately generate hash codes, thereby improving the accuracy of retrieval.

[0111] This application also provides a data retrieval example, as shown in Figure 4, the specific processing steps of which include:

[0112] Step S401: Obtain the text information input by the user.

[0113] Step S402: For each word segment in the text information, multiple word embedding vectors are determined through the word embedding layer, and the positional encoding of the order of each word segment in the text information is performed to obtain the positional encoding vector of the word segment.

[0114] Step S403: Divide each statement in the text information into multiple short statements by dividing the text into layers, and transform each short statement to obtain multiple short statement vectors corresponding to the statement.

[0115] Step S404: Determine the high-dimensional sentence feature vector of each sentence in the text information based on the word embedding vector of the words contained in the sentence, the position encoding vector of the words contained in the sentence, and the short sentence vector corresponding to the sentence.

[0116] Step S405: For each high-dimensional statement feature vector, generate the hash code corresponding to the high-dimensional statement feature vector using a hash function.

[0117] Step S406: Select any candidate data from the database and determine whether the hash bit in the hash code of the candidate data is consistent with the hash bit in the hash code corresponding to the high-dimensional statement feature vector.

[0118] In step S407, if the hash bit in the hash code of the candidate data does not match the hash bit in the hash code corresponding to the high-dimensional statement feature vector, return to the step of selecting any candidate data in the database until the hash bit in the hash code of the selected candidate data matches the hash bit in the hash code corresponding to the high-dimensional statement feature vector. Then, use the candidate data with the matching hash bits as the target retrieval data corresponding to the text information.

[0119] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0120] Based on the same inventive concept, this application also provides a data retrieval device for implementing the data retrieval method described above. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data retrieval device embodiments provided below can be found in the limitations of the data retrieval method described above, and will not be repeated here.

[0121] In one embodiment, such as Figure 5 As shown, a data retrieval device is provided, including: an acquisition module 510, a segmentation module 520, and a filtering module 530, wherein:

[0122] The acquisition module 510 is used to acquire text information input by the user; the text information is the text data that the user needs to retrieve.

[0123] The segmentation module 520 is used to divide the text information into multiple sentences using a sentence segmentation model, and to encode and convert each sentence to obtain a high-dimensional sentence feature vector corresponding to each sentence.

[0124] The filtering module 530 is used to generate a hash code corresponding to the high-dimensional statement feature vector through a hash function, query the database for candidate data corresponding to the hash code that is the same as the hash code of the high-dimensional statement feature vector, and use the candidate data with the same hash code as the target data.

[0125] Optionally, the partitioning module 520 is specifically used for:

[0126] For each word segment in the text information, multiple word embedding vectors are determined through the word embedding layer, and the positional encoding of each word segment in the text information is performed to obtain the positional encoding vector of the word segment.

[0127] Each statement in the text information is divided into multiple short statements through the segmentation layer, and each short statement is transformed to obtain multiple short statement vectors corresponding to the statement.

[0128] Based on the word embedding vectors of the words contained in the sentences in the text information, the position encoding vectors of the words contained in the sentences, and the short sentence vectors corresponding to the sentences, the high-dimensional sentence feature vectors of each sentence in the text information are determined.

[0129] Optionally, the partitioning module 520 is specifically used for:

[0130] For each statement, the word embedding vectors of the words contained in the statement, the position encoding vectors of the words contained in the statement, and the short statement vectors corresponding to the statement are superimposed to obtain the high-dimensional statement feature vector of the statement.

[0131] Optionally, the filtering module 530 is specifically used for:

[0132] For each high-dimensional statement feature vector, a hash code corresponding to the high-dimensional statement feature vector is generated using a hash function;

[0133] Select any candidate data in the database and determine whether the hash bit in the hash code of the candidate data is consistent with the hash bit in the hash code corresponding to the high-dimensional statement feature vector;

[0134] If the hash bit in the hash code of the candidate data does not match the hash bit in the hash code corresponding to the high-dimensional statement feature vector, return to the step of selecting any candidate data in the database until the hash bit in the hash code of the selected candidate data matches the hash bit in the hash code corresponding to the high-dimensional statement feature vector. Then, the candidate data with the matching hash bit is used as the target retrieval data corresponding to the text information.

[0135] Optionally, the device further includes:

[0136] The sample acquisition module is used to acquire sample text information and the hash codes of the feature vectors of each high-dimensional sample sentence in the sample text information;

[0137] The encoding module is used to divide the sample text information into multiple sentences using an initial sentence segmentation model, and to encode and transform each sentence to obtain a high-dimensional sentence feature vector for each sentence.

[0138] The training module is used to generate hash codes corresponding to the feature vectors of each high-dimensional sentence using a hash function, and to train the initial sentence segmentation model based on the hash codes of the feature vectors of each high-dimensional sample sentence and the hash codes of the feature vectors of each high-dimensional sentence to obtain the sentence segmentation model.

[0139] Optionally, the training module is specifically used for:

[0140] Determine whether there exists a hash code for a high-dimensional statement feature vector that is different from the hash code of the high-dimensional sample statement feature vector;

[0141] If there is a hash code for a high-dimensional sentence feature vector that is different from the hash code of the high-dimensional sample sentence feature vector, the partitioning parameters of the initial sentence partitioning model are adjusted according to the difference between the hash code of the high-dimensional sample sentence feature vector and the hash code of the high-dimensional sentence feature vector. Then, the process of dividing the sample text information into multiple sentences through the initial sentence partitioning model and encoding and converting each sentence to obtain the high-dimensional sentence feature vector of each sentence is repeated until there is no hash code for a high-dimensional sentence feature vector that is different from the hash code of the high-dimensional sample sentence feature vector.

[0142] The initial statement partitioning model is used as the statement partitioning model if there is no hash code corresponding to the hash code of the high-dimensional statement feature vector that is different from the hash code of the high-dimensional sample statement feature vector.

[0143] Optionally, the device further includes:

[0144] The vector acquisition module is used to acquire the feature vector of the high-dimensional sample statement and the sample data corresponding to the feature vector of the high-dimensional sample statement.

[0145] The similarity calculation module is used to calculate the hash code of the feature vector of each high-dimensional sample sentence and the hash code of each sample data through an initial hash function, and to calculate the similarity between the hash code of the feature vector of each high-dimensional sample sentence and the hash code of each sample data.

[0146] The judgment module is used to adjust the hash code generation parameters of the initial hash function when there are different hash codes of high-dimensional sample sentence feature vectors and the similarity between them and the hash codes of sample data is greater than a similarity threshold. Then, it returns to the step of calculating the hash code of each high-dimensional sample sentence feature vector and the hash code of each sample data through the initial hash function until only the hash code of the high-dimensional sample sentence feature vector and the hash code of the sample data corresponding to the high-dimensional sample sentence feature vector have a similarity greater than a similarity threshold.

[0147] The determination module is used to select an initial hash function whose similarity to the hash code of the high-dimensional sample statement feature vector and the hash code of the sample data corresponding to the high-dimensional sample statement feature vector is greater than a similarity threshold, as the hash function.

[0148] Each module in the aforementioned data retrieval device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0149] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a data retrieval method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0150] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0151] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any one of the first aspects.

[0152] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0153] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0155] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0156] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data retrieval method, characterized in that, The method includes: Obtain text information input by the user; The text information is divided into multiple sentences using a sentence segmentation model, and each sentence is encoded and converted to obtain a high-dimensional sentence feature vector corresponding to each sentence. A hash code corresponding to the high-dimensional statement feature vector is generated using a hash function. The same hash code as the hash code of the high-dimensional statement feature vector is queried in the database, and the candidate data corresponding to the same hash code is used as the target retrieval data corresponding to the text information. The sentence segmentation model includes a word embedding layer and a segmentation layer. The text information is divided into multiple sentences using the sentence segmentation model, and each sentence is encoded and converted to obtain a high-dimensional sentence feature vector corresponding to each sentence, including: For each word segment in the text information, multiple word embedding vectors are determined through the word embedding layer, and the positional encoding of the order of each word segment in the text information is performed to obtain the positional encoding vector of the word segment; for each sentence in the text information, multiple short sentences are divided through the segmentation layer, and each short sentence is transformed to obtain multiple short sentence vectors corresponding to the sentence. For each statement, the word embedding vectors of the words contained in the statement, the position encoding vectors of the words contained in the statement, and the short statement vectors corresponding to the statement are superimposed to obtain the high-dimensional statement feature vector of the statement.

2. The method according to claim 1, characterized in that, The process of generating a hash code corresponding to the high-dimensional statement feature vector using a hash function, querying the database for candidate data corresponding to the hash code that is the same as the hash code of the high-dimensional statement feature vector, and using the candidate data with the same hash code as the target retrieval data corresponding to the text information includes: For each high-dimensional statement feature vector, a hash code corresponding to the high-dimensional statement feature vector is generated using a hash function; Select any candidate data in the database and determine whether the hash bit in the hash code of the candidate data is consistent with the hash bit in the hash code corresponding to the high-dimensional statement feature vector; If the hash bit in the hash code of the candidate data does not match the hash bit in the hash code corresponding to the high-dimensional statement feature vector, return to the step of selecting any candidate data in the database until the hash bit in the hash code of the selected candidate data matches the hash bit in the hash code corresponding to the high-dimensional statement feature vector. Then, the candidate data with the matching hash bit is used as the target retrieval data corresponding to the text information.

3. The method according to claim 1, characterized in that, The method further includes: Obtain sample text information and the hash codes of the feature vectors of each high-dimensional sample sentence in the sample text information; The sample text information is divided into multiple sentences using an initial sentence segmentation model, and each sentence is encoded and converted to obtain a high-dimensional sentence feature vector for each sentence. A hash code is generated for each of the high-dimensional sentence feature vectors using a hash function. The initial sentence segmentation model is then trained based on the hash codes of the high-dimensional sample sentence feature vectors and the hash codes of the high-dimensional sentence feature vectors to obtain the sentence segmentation model.

4. The method according to claim 3, characterized in that, The step of training the initial sentence segmentation model based on the hash codes of the feature vectors of each high-dimensional sample sentence and the hash codes of the feature vectors of each high-dimensional sentence to obtain the sentence segmentation model includes: Determine whether there exists a hash code for a high-dimensional statement feature vector that is different from the hash code of the high-dimensional sample statement feature vector; If there is a hash code for a high-dimensional sentence feature vector that is different from the hash code of the high-dimensional sample sentence feature vector, the partitioning parameters of the initial sentence partitioning model are adjusted according to the difference between the hash code of the high-dimensional sample sentence feature vector and the hash code of the high-dimensional sentence feature vector. Then, the process of dividing the sample text information into multiple sentences through the initial sentence partitioning model and encoding and converting each sentence to obtain the high-dimensional sentence feature vector of each sentence is repeated until there is no hash code for a high-dimensional sentence feature vector that is different from the hash code of the high-dimensional sample sentence feature vector. The initial statement partitioning model is used as the statement partitioning model if there is no hash code corresponding to the hash code of the high-dimensional statement feature vector that is different from the hash code of the high-dimensional sample statement feature vector.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the high-dimensional sample sentence feature vector and the sample data corresponding to the high-dimensional sample sentence feature vector; Using an initial hash function, calculate the hash code of the feature vector of each high-dimensional sample statement and the hash code of each sample data, and calculate the similarity between the hash code of the feature vector of each high-dimensional sample statement and the hash code of each sample data. If there are different hash codes of high-dimensional sample sentence feature vectors and the similarity between them and the hash codes of sample data is greater than the similarity threshold, adjust the hash code generation parameters of the initial hash function, and return to the step of calculating the hash code of each high-dimensional sample sentence feature vector and the hash code of each sample data through the initial hash function, until only the hash code of the high-dimensional sample sentence feature vector and the hash code of the sample data corresponding to the high-dimensional sample sentence feature vector have a similarity greater than the similarity threshold. The initial hash function is defined as the hash function that has a similarity greater than a similarity threshold between the hash code of the feature vector of the high-dimensional sample statement and the hash code of the sample data corresponding to the feature vector of the high-dimensional sample statement.

6. A data retrieval device, characterized in that, The device includes: The acquisition module is used to acquire text information input by the user; the text information is the text data that the user needs to retrieve. The segmentation module is used to divide the text information into multiple sentences using a sentence segmentation model, and to encode and convert each sentence to obtain a high-dimensional sentence feature vector corresponding to each sentence. The filtering module is used to generate a hash code corresponding to the high-dimensional statement feature vector through a hash function, query the database for candidate data corresponding to the hash code that is the same as the hash code of the high-dimensional statement feature vector, and use the candidate data with the same hash code as the target data. The sentence segmentation model includes a word embedding layer and a segmentation layer. The segmentation module is specifically used for: For each word segment in the text information, multiple word embedding vectors are determined through the word embedding layer, and the positional encoding of the order of each word segment in the text information is performed to obtain the positional encoding vector of the word segment; for each sentence in the text information, multiple short sentences are divided through the segmentation layer, and each short sentence is transformed to obtain multiple short sentence vectors corresponding to the sentence. For each statement, the word embedding vectors of the words contained in the statement, the position encoding vectors of the words contained in the statement, and the short statement vectors corresponding to the statement are superimposed to obtain the high-dimensional statement feature vector of the statement.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images