Search method, device, equipment, storage medium and computer program

Abstract

This application provides at least one retrieval method, apparatus, device, storage medium, and computer program. The method includes: obtaining information to be queried; determining at least one target document identifier corresponding to the information to be queried from a set of document identifiers based on semantic analysis of the information to be queried; wherein each document identifier in the set of document identifiers has a semantic mapping relationship with its corresponding document; and determining at least one target statement matching the information to be queried from the document corresponding to the target document identifier based on the mapping relationship.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of computer technology, and in particular to a retrieval method, apparatus, device, storage medium, and computer program. Background Technology

[0002] To locate a specific document and / or sentences within a document (e.g., documents stored on a File Transfer Protocol (FTP) server on the Internet), users need to retrieve this document data using appropriate queries or keywords. However, with the dramatic increase in the amount of document data on the Internet, retrieving the desired information from this massive amount of data has become a pressing problem. Summary of the Invention

[0003] In view of the above, embodiments of this application provide at least one retrieval method, apparatus, device, storage medium, and computer program.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] On the one hand, embodiments of this application provide a retrieval method, the method comprising:

[0006] Obtain the information to be queried;

[0007] Based on semantic analysis of the information to be queried, at least one target document identifier corresponding to the information to be queried is determined from the document identifier set; wherein, each document identifier in the document identifier set has a semantic mapping relationship with the corresponding document;

[0008] Based on the mapping relationship, at least one target statement that matches the query information is determined from the document corresponding to the target document identifier.

[0009] In some embodiments, determining at least one target document identifier corresponding to the query information from a set of document identifiers based on semantic analysis of the query information includes:

[0010] Based on the semantic analysis of the information to be queried, the first semantic information is obtained;

[0011] Identify at least one target document identifier in the document identifier set that matches the first semantic information.

[0012] In some embodiments, the method further includes:

[0013] Each document in the document set is clustered to obtain at least one cluster; wherein the second semantic information of each document in each cluster satisfies a specific condition;

[0014] According to the numbering rules, determine the number of each cluster.

[0015] For each cluster, a first document identifier is determined for each document in the cluster based on the cluster number;

[0016] The set of first document identifiers for each document in the document set is taken as the document identifier set.

[0017] In some embodiments, determining the first document identifier for each document in the cluster based on the cluster number includes at least one of the following:

[0018] If the number of documents in the cluster is not greater than the number threshold, for each document in the cluster, determine the document number, and determine the first document identifier of the document based on the document number and the cluster number;

[0019] If the number of documents in the cluster exceeds the threshold, each document in the cluster is clustered to obtain at least one sub-cluster. Based on the numbering rules and the number of the cluster, the number of each sub-cluster is determined. For each sub-cluster, based on the number of the sub-cluster, the first document identifier of each document in the sub-cluster is determined.

[0020] In some embodiments, the method further includes:

[0021] The neural network is pre-trained using a document set and the document identifier tag corresponding to each document in the document set to obtain a pre-trained retrieval model; the document identifier tag corresponding to each document represents the document identifier corresponding to the document in the document identifier set.

[0022] The retrieval model is fine-tuned using a query information sample set and the document identifier tag corresponding to each query information sample in the query information sample set, to obtain the fine-tuned retrieval model; the document identifier tag corresponding to the query information sample represents the document identifier corresponding to the query information sample in the document identifier set.

[0023] The step of determining at least one target document identifier corresponding to the query information from the document identifier set based on semantic analysis of the query information includes:

[0024] Using the finely tuned retrieval model, based on semantic analysis of the information to be queried, at least one target document identifier corresponding to the information to be queried is determined from the document identifier set.

[0025] In some embodiments, the step of pre-training the neural network using a document set and document identifier tags corresponding to each document in the document set to obtain a pre-trained retrieval model includes:

[0026] Using a neural network, based on semantic analysis of each document in the document set, at least one predicted document identifier is determined for each document; wherein the at least one predicted document identifier belongs to the document identifier set.

[0027] A first loss is determined based on the document identifier tag of each document and at least one corresponding predicted document identifier;

[0028] Based on the first loss, the network parameters of the neural network are updated to obtain the pre-trained retrieval model.

[0029] In some embodiments, the step of fine-tuning the pre-trained retrieval model using a query information sample set and document identifier tags corresponding to each query information sample in the query information sample set to obtain the fine-tuned retrieval model includes:

[0030] Using the pre-trained retrieval model, based on semantic analysis of each query information sample in the query information sample set, at least one predicted document identifier is determined for each query information sample; wherein, the at least one predicted document identifier belongs to the document identifier set;

[0031] The second loss is determined based on the document identifier label corresponding to each of the query information samples and at least one corresponding predicted document identifier;

[0032] Based on the second loss, the model parameters of the pre-trained retrieval model are updated to obtain the fine-tuned retrieval model.

[0033] In some embodiments, determining at least one target document identifier corresponding to the query information from a set of document identifiers based on semantic analysis of the query information includes: performing semantic analysis on the query information to obtain first semantic information; determining multiple document identifiers with the highest matching probability to the first semantic information from the set of document identifiers according to a preset search method, and determining the multiple document identifiers as the target document identifier; determining at least one target statement matching the query information from the documents corresponding to the target document identifier based on the mapping relationship includes: determining the target document corresponding to the target document identifier from a set of documents based on the mapping relationship; determining the importance of each candidate word in each statement in the target document and the importance of each query word in the query information; determining the similarity between each statement and the query information based on the importance of each candidate word in each statement and the importance of each query word; and determining at least one statement with the highest similarity to the query information as the target statement.

[0034] On the other hand, embodiments of this application provide a retrieval device, the device comprising:

[0035] The retrieval module is used to retrieve the information to be queried.

[0036] The first determining module is used to determine at least one target document identifier corresponding to the query information from a set of document identifiers based on semantic analysis of the query information; wherein each document identifier in the set of document identifiers has a semantic mapping relationship with the corresponding document;

[0037] The second determining module is used to determine, based on the mapping relationship, at least one target statement that matches the query information from the document corresponding to the target document identifier.

[0038] In another aspect, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0039] In another aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0040] In another aspect, embodiments of this application provide a computer program including computer-readable code, wherein when the computer-readable code is run in a computer device, a processor in the computer device performs some or all of the steps for implementing the above-described method.

[0041] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0043] Figure 1 A schematic diagram illustrating the implementation process of a retrieval method provided in an embodiment of this application;

[0044] Figure 2 This is a schematic diagram illustrating the implementation process of clustering a document set in a retrieval method provided in an embodiment of this application;

[0045] Figure 3 This is a schematic diagram illustrating the implementation process of training a retrieval model in a retrieval method provided in an embodiment of this application.

[0046] Figure 4 This is a schematic diagram illustrating the implementation process of an application embodiment of a retrieval method using the Transformer model as the retrieval model, provided in this application.

[0047] Figure 5 A schematic diagram illustrating the implementation process of the retrieval model training process in a retrieval method provided in this application embodiment;

[0048] Figure 6 This is a schematic diagram illustrating the implementation process of a document retrieval application using a retrieval model in an embodiment of this application.

[0049] Figure 7 This is a schematic diagram of the composition of a retrieval device provided in an embodiment of this application;

[0050] Figure 8 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0053] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0055] In related technologies, to retrieve matching documents from the massive amount of documents on the Internet, and then retrieve similar sentences from the matching documents, thereby achieving sentence-level retrieval and more accurately locating relevant information within the documents, the following two main approaches are used:

[0056] Option 1: Perform full-text search by building an inverted index. That is, first determine the words contained in the query statement, and then quickly retrieve a list of documents containing the words in the query statement from the document library.

[0057] Option 2: A text matching method based on a representation model, which first converts two text segments into semantic vectors, and then calculates the similarity between the two semantic vectors. This option places greater emphasis on constructing the text semantic vector representation layer.

[0058] The first approach described above searches on a word-by-word basis, meaning the search space is at the character level, thus offering high search efficiency. However, because the first approach is a literal matching-based document retrieval method, it does not utilize the contextual and semantic information of the document. Consequently, it fails to consider the semantic information of the document, and the search results may differ significantly from the semantics of the query statement.

[0059] The second approach described above determines the similarity between two statements by calculating the semantic vectors of sentences in the document and the semantic vector of the query statement. Therefore, it can express semantic information at the sentence level, but it does not utilize the semantic information of the entire document. Furthermore, the second approach requires storing the semantic representations of all sentences in the document, resulting in a large search space and a high computational cost during the search process.

[0060] Based on this, embodiments of this application provide a retrieval method. This method is based on semantic analysis of the information to be queried. It determines at least one target document identifier corresponding to the information to be queried from a set of document identifiers, and determines the target sentence matching the information to be queried based on the document corresponding to the target document identifier. Each document identifier and its corresponding document have a semantic mapping relationship. Thus, in the retrieval process, document-level semantic retrieval is performed first, and then sentence-level retrieval is performed, so that the retrieved documents and sentences have semantic relevance to the query statement, thereby improving the retrieval accuracy.

[0061] The retrieval method provided in this application embodiment can be executed by the processor of a computer device. The computer device refers to a device with data processing capabilities, such as a server, laptop, tablet, desktop computer, smart TV, set-top box, or mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device).

[0062] Figure 1 This is a schematic diagram illustrating the implementation flow of a retrieval method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps S101 to S103:

[0063] Step S101: Obtain the information to be queried.

[0064] Here, the information to be queried can be a statement to be queried, which has its corresponding semantic content.

[0065] In practical applications, the information to be queried can be obtained through user input or from other electronic devices. No specific method of obtaining the information is specified here.

[0066] Step S102: Based on the semantic analysis of the information to be queried, determine at least one target document identifier corresponding to the information to be queried from the document identifier set; wherein, each document identifier in the document identifier set has a semantic mapping relationship with the corresponding document.

[0067] Here, semantic analysis refers to the in-depth analysis of natural language sentences and their contexts to build multi-level models encompassing vocabulary, grammar, and semantics, aiming to achieve automatic understanding and intelligent processing of text. It is a text analysis method that combines natural language processing and machine learning techniques.

[0068] In some embodiments, semantic analysis can be performed using word vector models, topic models, or rule-based analysis methods. Word vector models represent each word as a vector and represent each vector in a low-dimensional space, thereby enabling the understanding and processing of textual information. Topic models divide the text into multiple topics through topic analysis. Rule-based methods refer to analysis and processing using pre-defined grammatical and semantic rules.

[0069] A document is data composed of multiple sentences, and each document has its corresponding document-level semantic information.

[0070] The document identifier has a semantic mapping relationship with the corresponding document. Therefore, the document identifier in this application embodiment is not a random identifier composed of discrete numbers or characters, but an identifier with semantic content based on the semantic encoding of the corresponding document.

[0071] In practical applications, clustering algorithms can be used to determine the semantic mapping relationship between document identifiers and their corresponding documents. For example, firstly, each document in the document set is vectorized using a word vector model; then, hierarchical clustering is performed on the vectorized documents; finally, each document is encoded according to the hierarchical relationship of the clustered documents, thus obtaining the document identifier corresponding to each document. It is evident that since the encoding of document identifiers is based on the vector representation of the documents, there is a semantic mapping relationship between document identifiers and their corresponding documents.

[0072] Since each document identifier in the document identifier set has semantic content, by performing semantic analysis on the information to be queried, at least one target document identifier corresponding to the information to be queried can be determined from the document identifier set. In some embodiments, the semantic information contained in at least one target document identifier satisfies a preset similarity requirement with the semantic information corresponding to the query statement.

[0073] Step S103: Based on the mapping relationship, determine at least one target statement that matches the query information from the document corresponding to the target document identifier.

[0074] Here, after determining at least one target document identifier corresponding to the query statement, based on the semantic mapping relationship between the document identifier and the corresponding document, at least one target document corresponding to the query statement can be determined.

[0075] After identifying at least one target document, a sentence-level retrieval is performed on each target document using the information to be queried; that is, at least one target statement that matches the information to be queried is identified. The target statement is a statement that meets a preset similarity requirement with the information to be queried.

[0076] In practical applications, sentence-level retrieval for each target document can be achieved through methods such as inverted indexes and text matching based on representation models.

[0077] In the retrieval method provided in this application embodiment, based on semantic analysis of the information to be queried, at least one target document identifier corresponding to the information to be queried is determined from the document identifier set, and each document identifier has a semantic mapping relationship with the corresponding document, thereby realizing document-level semantic retrieval of the information to be queried; then, using the semantic mapping relationship between the document identifier and the corresponding document, at least one target statement matching the information to be queried is determined from the document corresponding to the target document identifier. In this way, statement retrieval is performed on the basis of document-level semantic retrieval, avoiding the separation of semantic relationship between the information to be queried and the document containing the statement that occurs when the document is directly broken down into statement vectors and the statement vectors are directly used for semantic matching. Therefore, the retrieval method provided in this application embodiment can effectively ensure that the information to be queried and the document still have semantic relevance when performing statement retrieval, making the retrieval results more accurate.

[0078] In some embodiments, step S102 can be implemented by the following steps S1021 to S1022:

[0079] Step S1021: Based on the semantic analysis of the information to be queried, first semantic information is obtained.

[0080] Here, the first semantic information can be a set of vectors obtained by analyzing the query information using a neural network model.

[0081] In practical applications, any suitable neural network, such as the Transformer model, is used to perform semantic analysis on the query information to obtain the first semantic information. Specifically: First, each word in the query information is represented as a corresponding representation vector, and a word representation vector matrix is ​​formed by these representation vectors (each row of this matrix corresponds to the representation vector of one word). Then, the word representation vector matrix is ​​fed into the encoder of the Transformer model to obtain the encoded information matrix of all words in the query information. This encoded information matrix of all words in the query information is the first semantic information corresponding to the query information.

[0082] Step S1022: Determine at least one target document identifier in the document identifier set that matches the first semantic information.

[0083] Here, since each document identifier in the document identifier set has a semantic mapping relationship with the corresponding document, each document identifier has semantic content. Therefore, at least one target document identifier can be determined from the document identifier set based on the first semantic information of the information to be queried.

[0084] In practical applications, neural networks, such as the Transformer model, can be used to determine at least one target document identifier that matches the first semantic information of the query information. Specifically, firstly, the first semantic information (e.g., the encoded information matrix of all words in the query information) is input into the decoder of the Transformer model; then, the decoder of the Transformer model generates each identifier bit of the query information identifier corresponding to the query information in sequence according to the encoded information matrix of all words. When generating the query information identifier, each identifier bit is determined from the possible value range of the corresponding bit in the aforementioned set of document identifiers, thus ensuring that the generated query information identifier belongs to the set of document identifiers.

[0085] In some embodiments, when using the decoder of the Transformer model to generate each identifier bit in the identifier of the query information corresponding to the query information, only the prediction result with the highest probability or similarity can be used as the output of the corresponding identifier bit. That is, each identifier bit is determined by using a greedy search algorithm, thereby generating the query information identifier with the highest probability or similarity for the query information.

[0086] In some embodiments, when generating each identifier bit in the query information identifier corresponding to the query information using the decoder of the Transformer model, the top n prediction results can be sorted by probability or similarity as the output of the corresponding identifier bit (where n is a positive integer greater than 1). That is, each identifier bit is determined using the Beam Search algorithm, thereby generating at least one query information identifier for the query information. In some embodiments, at least one query information identifier is sorted according to probability or similarity.

[0087] In some embodiments, the retrieval method provided in this application further includes determining the document identifier set, which can be implemented through the following steps S104 to S107:

[0088] Step S104: Cluster each document in the document set to obtain at least one cluster; wherein, the second semantic information of each document in each cluster satisfies a specific condition.

[0089] Here, before clustering each document in the document set, a corresponding semantic vector is generated for each document, and this semantic vector is used as the second semantic information for each document. In practical applications, models such as the One-Hot Model, Bag of Words Model, Term Frequency-Inverse Document Frequency (TF-IDF), N-gram, Word2vec, or Document2vec can be used to generate the corresponding semantic vector for each document.

[0090] Clustering each document in a document set refers to clustering documents based on the similarity between the semantic vectors corresponding to each document, in order to obtain at least one cluster. In practical applications, the k-means clustering algorithm or other commonly used clustering algorithms in this field can be used to cluster the semantic vectors corresponding to each document.

[0091] The specific condition refers to the similarity between the semantic vectors of the documents in each cluster reaching a specified similarity threshold.

[0092] Step S105: Determine the number of each cluster according to the numbering rules.

[0093] Here, the numbering rule refers to the pre-defined rules for numbering clusters.

[0094] In some embodiments, clusters can be numbered using Arabic numerals, for example, by numbering clusters in the order of Arabic numerals 0 to N (where N is a positive integer greater than 0).

[0095] In some embodiments, clusters can be numbered using specified words or letters, for example, clusters can be numbered in the order of the English letters A to Z.

[0096] In practical applications, the numbering rules can be set by the operators according to the actual application scenario, and there are no restrictions here.

[0097] Step S106: For each cluster, determine the first document identifier for each document in the cluster based on the cluster number.

[0098] Here, determining the first document identifier of each document in a cluster based on the cluster number can be done by numbering each document in the cluster according to the numbering rules, and determining the first document identifier of the corresponding document based on the cluster number and the number of each document. For example, the combination of the cluster number and the document number corresponding to each document can be used to determine the first document identifier of the corresponding document.

[0099] In some embodiments, the numbering rule for each document in a cluster may be the same as the numbering rule for each individual cluster, for example, numbering each cluster and each document in each cluster according to the Arabic numerals 0 to N. In other embodiments, the numbering rule for each document in a cluster may be the same as the numbering rule for each individual cluster, for example, numbering the clusters according to the Arabic numerals 0 to N, and numbering each document in the cluster according to the English letters A to Z.

[0100] In some embodiments, documents can be numbered based on their name, generation time, similarity to the centroid of a cluster, etc.

[0101] In some embodiments, determining the first document identifier of each document in the cluster based on the cluster number in step S106 can be achieved by at least one of the following steps S1061 and S1062:

[0102] Step S1061: If the number of documents in the cluster is not greater than the number threshold, for each document in the cluster, determine the document number, and determine the first document identifier of the document based on the document number and the cluster number.

[0103] Here, if the number of documents in a cluster is c, and c is not greater than the number threshold, the documents in the cluster can be numbered according to a preset document numbering rule, for example, the documents in the cluster can be numbered in the order from 0 to c-1.

[0104] In practical applications, determining the first document identifier of a document based on its document number and cluster number can be achieved by combining the document number and cluster number to obtain the first document identifier.

[0105] Step S1062: If the number of document data in the cluster is greater than the number threshold, each document in the cluster is clustered to obtain at least one cluster sub-cluster. Based on the numbering rule and the number of the cluster, the number of each cluster sub-cluster is determined. For each cluster sub-cluster, based on the number of the cluster sub-cluster, the first document identifier of each document in the cluster sub-cluster is determined.

[0106] Here, if the number of documents in a cluster is c, and c is greater than the number threshold, the documents in the cluster are clustered to obtain at least one sub-cluster.

[0107] In some embodiments, when numbering the cluster subclusters, the value of at least one first number bit of each cluster subcluster can be determined using a preset number. Then, the value of at least one second number bit of the corresponding cluster subcluster can be determined based on the number of the cluster to which the cluster subcluster belongs. Finally, the at least one first number bit and the at least one second number bit are combined to obtain the number of the corresponding cluster subcluster.

[0108] In some embodiments, each document in a cluster can be numbered based on a preset numbering rule. Then, a first document identifier for each document can be determined based on the cluster number to which it belongs and the corresponding document number. For example, the first document identifier can be obtained by combining the cluster number to which each document belongs and the corresponding document number.

[0109] Step S107: The set of first document identifiers for each document in the document set is taken as the document identifier set.

[0110] Here, the first document identifier of each document is obtained by clustering the documents in the document set. The clustering of documents in the document set is based on the semantic vector of each document. Therefore, there is a semantic mapping relationship between the first document identifier of each document and the document.

[0111] Below, in conjunction with Figure 2 This application provides a detailed description of an application embodiment of a retrieval method that performs clustering processing on a document set.

[0112] like Figure 2 As shown, document set 201 includes N documents, where N is an integer greater than 1.

[0113] First, perform first-level clustering 202 on N documents to obtain 10 first-level clusters, and number the 10 clusters in the order of Arabic numerals from 0 to 9;

[0114] Then, after the first-level clustering process 202, each resulting cluster contains x+1 documents. Since x+1 is greater than the preset threshold of 10, a second-level clustering process 203 is performed on the documents in each cluster. In the second-level clustering process 203, the documents in each cluster are clustered to obtain two secondary clusters. Simultaneously, the secondary clusters are numbered based on the number of the primary cluster to which each secondary cluster belongs, such as... Figure 2As shown, in the second-level cluster with the first-level cluster number 0, the second-level cluster numbers are "00" and "01" respectively;

[0115] Secondly, after the second-level clustering process 203, each secondary cluster contains y+1 documents. Since y+1 is no greater than the quantity threshold of 10, each document in each secondary cluster is numbered. For example... Figure 2 As shown, each document in each secondary cluster is numbered in the order of a00 to a0y, a10 to a1y, b00 to b0y, b10 to b1y, c00 to c0y, c10 to c1y, d00 to d0y, and d10 to d1y.

[0116] Finally, based on the cluster number of the secondary cluster and the document number, the first document identifier corresponding to each document is determined. For example, the first document identifier of a document with a secondary cluster number of 00 and a document number of a00 can be represented as "00a00".

[0117] As described above, the retrieval method provided in this application embodiment can be implemented using a neural network model, such as the Transformer model. Therefore, the following will be combined with... Figure 3 The training process of the retrieval model in the retrieval method provided in the embodiments of this application will be described in detail.

[0118] like Figure 3 As shown, the training process of the retrieval model in this embodiment can be carried out through the following steps S301 to S302:

[0119] Step S301: Using the document set and the document identifier tag corresponding to each document in the document set, the neural network is pre-trained to obtain the pre-trained retrieval model; the document identifier tag corresponding to the document represents the document identifier corresponding to the document in the document identifier set.

[0120] Here, the document set corresponds to the document set used to determine the document identifier set in the previous text, and the document identifier label corresponds to the first document identifier in the previous text.

[0121] Pre-training is to enable the retrieval model to recognize the mapping relationship between documents and document identifier tags.

[0122] In practical applications, the documents input to the neural network can be preprocessed, namely: first, stop words in the document are removed, mainly including English characters, numbers, numeric characters, punctuation marks, and frequently used single Chinese characters; then, the title and the first M words of the document with stop words removed are used as input for the document representation corresponding to the document; finally, the document is pre-trained based on the document representation corresponding to the document.

[0123] In some embodiments, the process of pre-training the neural network to obtain the pre-trained retrieval model, i.e., step S301, can be implemented by the following steps S3011 to S3013:

[0124] Step S3011: Using a neural network, based on semantic analysis of each document in the document set, determine at least one predicted document identifier for each document; wherein the at least one predicted document identifier belongs to the document identifier set.

[0125] Here, the semantic analysis method for each document in the document set can utilize the semantic analysis method used in the step of determining the first document identifier described above, which will not be elaborated here.

[0126] Predicted document identifiers are document identifiers generated using neural networks based on semantic analysis of each document. Since the neural network determines each identifier bit in the predicted document identifier based on the value range of each identifier bit in the document identifier set, at least one predicted document identifier for each document belongs to the document identifier set.

[0127] When determining at least one predicted document identifier for each document, the neural network uses the semantic analysis results of each document to generate the predicted document identifier for the corresponding document bit by bit.

[0128] In practical applications, either a greedy algorithm or a beam search algorithm can be used to determine the prediction result for each identifier bit in the predicted document identifier. When using a greedy algorithm to determine the prediction result for each identifier bit, the neural network only uses the prediction result with the highest probability or similarity as the final prediction result for the corresponding identifier bit, thus generating a corresponding predicted document identifier for each document. When using a beam search algorithm to determine the prediction result for each identifier bit, the neural network sorts the prediction results according to probability or similarity and uses a specified number of prediction results as the final prediction result for the corresponding identifier bit, thus generating at least one predicted document identifier for each document.

[0129] Step S3012: Determine the first loss based on the document identifier tag of each document and at least one corresponding predicted document identifier.

[0130] Here, the first loss is used to measure the error between the document identifier label of each document and at least one predicted document identifier. In practical applications, a loss function is pre-set, and the first loss between the document identifier label of each document and at least one predicted document identifier is calculated using the loss function.

[0131] In some embodiments, a loss function can be established using squared loss and mean squared error loss.

[0132] Step S3013: Based on the first loss, update the network parameters of the neural network to obtain the pre-trained retrieval model.

[0133] Here, the parameters of the neural network are updated using the first loss to ensure that the updated neural network outputs at least one predicted document identifier for each document that is closer to the document's document identifier label. In other words, the model parameters of the pre-trained retrieval model accurately represent the mapping relationship between documents and document identifiers. Simultaneously, since the document identifiers are obtained through clustering to derive the semantic encoding of the document, the model actually learns the mapping relationship between the document and its semantic encoding.

[0134] Step S302: Using the query information sample set and the document identifier tag corresponding to each query information sample in the query information sample set, the pre-trained retrieval model is fine-tuned to obtain the fine-tuned retrieval model; the document identifier tag corresponding to the query information sample represents the document identifier corresponding to the query information sample in the document identifier set.

[0135] Here, the query information sample may include at least one of the following: query statement sample, query word sample.

[0136] The document identifier tag corresponding to each query information sample can be the document identifier of at least one similar document pre-determined for each query information sample from the document identifier set. That is, the document identifier tag corresponding to the query information sample represents the document identifier corresponding to the query information sample in the document identifier set. In practical applications, at least one document identifier tag can be determined for each query information sample through manual annotation.

[0137] Fine-tuning the pre-trained retrieval model aims to make it more suitable for document retrieval targeting specific types of queries. For example, when the query sample is a query statement, fine-tuning can make the retrieval model more suitable for document retrieval targeting statement-type queries, thereby improving the accuracy of document retrieval.

[0138] In some embodiments, step S302 can be implemented by the following steps S3021 to S3023:

[0139] Step S3021: Using the pre-trained retrieval model, based on semantic analysis of each query information sample in the query information sample set, determine at least one predicted document identifier that matches each query information sample; wherein, the at least one predicted document identifier belongs to the document identifier set.

[0140] Here, when the greedy algorithm is used to determine the final prediction result for each identifier bit in the pre-trained retrieval model, the number of predicted document identifiers for the query information sample is one; when the Beam Search algorithm is used to determine the final prediction result for each identifier bit in the pre-trained retrieval model, the number of predicted document identifiers for the query information sample is at least one. In practical applications, the number of predicted document identifiers determined for each query information sample can be determined according to the specific use case.

[0141] Step S3022: Determine the second loss based on the document identifier label corresponding to each of the query information samples and at least one corresponding predicted document identifier.

[0142] Here, a loss function is pre-set, and the loss function is used to calculate the second loss between the document identifier label corresponding to each query information sample and at least one predicted document identifier.

[0143] In practical applications, the loss function used to calculate the first loss and the loss function used to calculate the second loss may or may not be the same.

[0144] Step S3023: Based on the second loss, update the model parameters of the pre-trained retrieval model to obtain the fine-tuned retrieval model.

[0145] Here, based on the second loss, the model parameters of the pre-trained retrieval model are updated so that the updated retrieval model outputs at least one predicted document identifier for each query information sample that is closer to the corresponding document identifier label.

[0146] In this embodiment of the application, by fine-tuning the pre-trained retrieval model, the fine-tuned retrieval model can be made more suitable for document retrieval for specific types of query information, thereby improving the accuracy of document retrieval.

[0147] In some implementations, step S102 can be implemented as follows: using the fine-tuned retrieval model, based on semantic analysis of the information to be queried, to determine at least one target document identifier corresponding to the information to be queried from the document identifier set.

[0148] Here, because the pre-trained retrieval model learns the mapping relationship between documents and document identifiers / document semantic codes during the pre-training stage, the retrieval model can directly generate at least one target document identifier corresponding to the query information in a generative manner during the usage stage. This at least one target document identifier also contains the semantic association between the query information and the target document and belongs to the document identifier set.

[0149] In the above embodiments of this application, firstly, the neural network is pre-trained using a document set and document identifier tags corresponding to each document in the document set, so that the pre-trained retrieval model can accurately represent the mapping relationship between documents and document identifiers; then, the pre-trained retrieval model is fine-tuned using a query information sample set and document identifier tags corresponding to each query information sample in the query information sample set, so that the fine-tuned retrieval model is more adapted to document retrieval for specific query information; finally, in the model usage stage, the fine-tuned retrieval model is used to determine at least one target document identifier corresponding to the query information.

[0150] Below, refer to Figure 4 Taking the Transformer model as an example, this paper provides a detailed description of the application of the retrieval method provided in this application.

[0151] like Figure 4 As shown, in the pre-training process, firstly, N documents 401 and the document identifier label corresponding to each document 401 are input into the Transformer model 402; then, the encoder 4021 in the Transformer model 402 performs semantic analysis on each document to obtain the corresponding vector representation; next, the decoder 4022 of the Transformer model 402 generates at least one predicted document identifier 403 for each document based on the vector representation corresponding to each document; finally, the parameters of the Transformer model 402 are updated based on the predicted document identifier and document identifier label corresponding to each document to obtain the pre-trained Transformer model 402.

[0152] In the fine-tuning training phase, firstly, the query information sample set and the document identifier label 401 corresponding to each query information sample in the query information sample set are input into the Transformer model 402; then, the encoder 4021 in the Transformer model 402 performs semantic analysis on each query information sample to obtain the corresponding vector representation; next, the decoder 4022 of the Transformer model 402 determines at least one predicted document identifier 403 for each query information sample from the document identifier set based on the vector representation corresponding to each query information sample; finally, based on the predicted document identifier and document identifier label corresponding to each query information sample, the parameters of the Transformer model 402 are updated to obtain the fine-tuned Transformer model 402.

[0153] In the stage of document retrieval based on the query statement using the fine-tuned Transformer model 402, firstly, the query information 401 is input into the Transformer model 402. The encoder 4021 in the Transformer model 402 performs semantic analysis on the query information 401 to obtain the semantic information corresponding to the query information 401, that is, the vector representation of the query information 401. Then, the vector representation of the query information 401 is input into the decoder 4022 of the Transformer model 402. The Beam Search algorithm is added to the decoder 4022 so that the decoder 4022 determines multiple target document identifiers 403 corresponding to the query information 401 from the document identifier set corresponding to multiple documents 402 based on the vector representation of the query information 401.

[0154] In some embodiments, step S102 can also be implemented by the following steps S1023 to S1024:

[0155] Step S1023: Perform semantic analysis on the information to be queried to obtain the first semantic information.

[0156] Here, the method for performing semantic analysis on the query information to obtain the first semantic information can be referred to the description of step S1021 above, and will not be repeated here.

[0157] Step S1024: According to a preset search method, determine the multiple document identifiers with the highest probability of matching the first semantic information from the document identifier set, and determine the multiple document identifiers as the target document identifiers.

[0158] Here, the preset search method refers to how to determine the value corresponding to each identifier bit from multiple candidate values ​​when determining multiple document identifiers corresponding to the information to be queried. For example, when determining the value of one identifier bit in the document identifier corresponding to the information to be queried, there are multiple candidate values, and each candidate value has a corresponding similarity to the first semantic information of the information to be queried. In this case, the preset search method can be used to determine the value corresponding to the one identifier bit from multiple candidate values.

[0159] In some embodiments, the preset search method may be to search using the Beam Search algorithm. Based on the Beam Search algorithm, when determining each identifier corresponding to the information to be queried, k candidate values ​​with the highest similarity are retained. Then, when determining the next identifier, the k candidate values ​​to be retained for the current identifier are determined based on the k candidate values ​​retained for the previous identifier.

[0160] Here, when determining each identifier bit in the identifier of the information to be queried using a preset search method, the search space is based on all document identifiers in the document identifier set. Therefore, the multiple document identifiers determined by the information to be queried are multiple document identifiers in the document identifier set.

[0161] In some embodiments, step S103 can be implemented by the following steps S1031 to S1034:

[0162] Step S1031: Based on the mapping relationship, determine the target document corresponding to the target document identifier from the document set.

[0163] Here, each document identifier in the document identifier set has a semantic mapping relationship with the corresponding document. Therefore, based on the target document identifier, the corresponding target document can be determined from the document set.

[0164] The number of target documents can be at least one.

[0165] Step S1032: Determine the importance of each candidate word in each sentence of the target document and the importance of each query word in the query information.

[0166] Here, the candidate words in each sentence are obtained by segmenting each sentence in the target document; each query word in the query information is obtained by segmenting the query information.

[0167] In practical applications, dictionary-based segmentation, understanding-based segmentation, or statistical segmentation can be used to segment sentences and query information in the target document; no specific method is specified here.

[0168] The importance of candidate terms and query terms can be determined by the weights of each candidate term and each query term.

[0169] In practical applications, weighted algorithms can be used to determine the importance of each candidate word and each query word. For example, the Term Frequency-Inverse Document Frequency (TF-IDF) weighted algorithm can be used to determine the weight of each candidate word and each query word. That is, the TF-IDF value of each word in each target document and the TF-IDF value of each query word in the information to be searched can be determined.

[0170] In practical applications, a corresponding information list can be created for each target document. This information list includes at least one candidate word in the target document and the attribute information of each candidate word. In some embodiments, the attribute information of each candidate word may include the identifier of the sentence to which the candidate word belongs, the number of times the candidate word appears in the sentence, and the position of the candidate word in the sentence.

[0171] Table 1 below is a schematic diagram of the information list established for a target document by the retrieval method based on the embodiments of this application:

[0172] Word ID word Document frequency Inverted index (sentence ID, TF, POS) 1 Word 1 3 (1,1,<1>),(3,2,<1>) 2 Word 2 6 (6,1,<1>),(9,2,<1>),(11,1,<1>),(13,2,<1>) ...... ...... ...... ...... T word t 9 (5,4,<1>),(23,2,<1>),(71,1,<1>),(32,2,<1>)

[0173] As shown in Table 1, for each word in the target document's information list, the document frequency (i.e., the number of times the word appears in the document) and the inverted list are recorded. The inverted list records the sentence identifier (Identity document, ID) of the sentence to which the word belongs, the number of times it appears in the sentence (i.e., TF value), and the absolute position of the word from the beginning of the sentence (i.e., POS value). Taking word 1 as an example, its word ID is 1, and its frequency in the current target document is 3. Based on the inverted list, we know that word 1 appears once in sentence 1, with an absolute distance of 1 from the beginning of sentence 1; word 1 appears twice in sentence 3, with an absolute distance of 1 from the beginning of sentence 3.

[0174] Based on the information list corresponding to each target document, the relevant information of each candidate word can be easily determined.

[0175] Step S1033: Based on the importance of each candidate word in each statement and the importance of each query word, determine the similarity between each statement and the query information.

[0176] Here, the target document is searched based on each query term in the information to be queried, and at least one relevant statement is obtained.

[0177] In practical applications, based on each query term in the information to be queried, at least one relevant statement containing each query term can be determined from the information list corresponding to each target document.

[0178] In some implementations, a query vector corresponding to the information to be queried is constructed based on the importance (i.e., TF-IDF value) of each query term; and a statement vector corresponding to each relevant statement is constructed based on the importance (i.e., TF-IDF value) of each candidate word in each relevant statement. Furthermore, when determining the statement vector corresponding to a relevant statement, for a candidate word that is the same as any query term in the information to be queried, the TF-IDF value of that candidate word is used as its word vector; for a candidate word that is different from any query term in the information to be queried, the word vector of that candidate word is set to 0.

[0179] Then, the cosine similarity between the query vector of the information to be queried and the statement vector of each related statement is calculated to determine the similarity of each related statement.

[0180] In some embodiments, if the length of the query vector corresponding to the information to be queried is different from the length of the statement vector corresponding to the related statement, the shorter one is padded with zeros so that the two vectors have the same length.

[0181] Step S1034: Determine at least one statement with the highest similarity to the information to be queried as the target statement.

[0182] Here, the target statement can be determined from at least one related statement based on the similarity of each related statement, or the top N most similar statements from at least one related statement can be determined as the target statement.

[0183] In this embodiment, the target document is first determined based on document-level semantic retrieval, and then the target sentence search is only required in the target document. Therefore, the search space can be significantly reduced, and there is no need to store a large number of sentence vectors, thereby saving computational and storage resources.

[0184] Below, in conjunction with Figure 5 This application provides a detailed description of an application embodiment of the document retrieval method provided in this embodiment, specifically the retrieval model training process.

[0185] Reference Figure 5 The process of training the retrieval model in the document retrieval method provided in this application embodiment can be implemented through the following steps S501 to S505:

[0186] Step S501: Obtain the document set; then, execute steps S502 and S503 respectively.

[0187] Step S502: Perform clustering on the document set to obtain a document identifier set; then, proceed to step S504.

[0188] Here, the document set is clustered to obtain the document identifier corresponding to each document in the document set, and the set of document identifiers corresponding to each document is used as the document identifier set.

[0189] The detailed process of clustering the document set to obtain the document identifier set can be found in the detailed description above in conjunction with steps S104 to S107, and will not be repeated here.

[0190] Step S503: Determine the document representation corresponding to each document in the document set; then, execute step S504.

[0191] Here, the method for determining the document representation corresponding to each document can be found in the detailed description above in conjunction with step S301, and will not be repeated here.

[0192] Step S504: Pre-train the neural network based on the document identifier tag and the corresponding document representation for each document to obtain the pre-trained retrieval model; then, execute step S505.

[0193] Here, the detailed process of pre-training the neural network to obtain the pre-trained retrieval model can be found in the detailed description above in conjunction with steps S3011 to S3013, and will not be repeated here.

[0194] Step S505: Fine-tune the pre-trained retrieval model to obtain the fine-tuned retrieval model.

[0195] Here, the detailed process of fine-tuning the pre-trained retrieval model to obtain the fine-tuned retrieval model can be found in the detailed description above in conjunction with steps S3021 to S3023, and will not be repeated here.

[0196] Below, in conjunction with Figure 6 This application provides a detailed description of an example of a document retrieval method that utilizes a retrieval model for document retrieval.

[0197] The document retrieval method provided in this application embodiment utilizes a retrieval model to perform document retrieval, which can be achieved by combining steps S601 to S604 as follows:

[0198] Step S601: Obtain the query statement; then, execute step S602.

[0199] Step S602: Using the fine-tuned retrieval model, determine multiple target document identifiers corresponding to the query statement from the document identifier set; then, execute step S603.

[0200] Here, the detailed process of determining multiple target document identifiers corresponding to the query statement from the document identifier set can be found in the detailed description above in conjunction with steps S1021 to S1022, and will not be repeated here.

[0201] Step S603: Based on multiple target document identifiers, determine multiple target documents; then, proceed to step S604.

[0202] Here, the detailed process of determining multiple target documents based on multiple target document identifiers can be found in the detailed description above in conjunction with steps 103 and S1031, and will not be repeated here.

[0203] Step S604: Determine at least one target statement corresponding to the information to be queried from multiple target documents.

[0204] Here, the detailed process of determining at least one target statement corresponding to the information to be queried from multiple target documents can be found in the detailed description above in conjunction with steps S1032 to S1034, and will not be repeated here.

[0205] Based on the foregoing embodiments, this application also provides a retrieval device, which includes the included units and the modules included in each unit, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0206] Figure 7 This is a schematic diagram of the composition structure of a retrieval device provided in an embodiment of this application, as shown below. Figure 6 As shown, the data processing device 700 includes: an acquisition module 710, a first determination module 720, and a second determination module 730, wherein:

[0207] Module 710 is used to obtain the information to be queried;

[0208] The first determining module 720 is used to determine at least one target document identifier corresponding to the query information from a set of document identifiers based on semantic analysis of the query information; wherein each document identifier in the set of document identifiers has a semantic mapping relationship with the corresponding document;

[0209] The second determining module 730 is used to determine, based on the mapping relationship, at least one target statement that matches the query information from the document corresponding to the target document identifier.

[0210] In some embodiments, the first determining module 720 is configured to:

[0211] Based on the semantic analysis of the information to be queried, the first semantic information is obtained;

[0212] Identify at least one target document identifier in the document identifier set that matches the first semantic information.

[0213] In some embodiments, the apparatus further includes a third determining module, configured to:

[0214] Each document in the document set is clustered to obtain at least one cluster; wherein the second semantic information of each document in each cluster satisfies a specific condition;

[0215] According to the numbering rules, determine the number of each cluster.

[0216] For each cluster, a first document identifier is determined for each document in the cluster based on the cluster number;

[0217] The set of first document identifiers for each document in the document set is taken as the document identifier set.

[0218] In some embodiments, the third determining module is further configured to:

[0219] If the number of documents in the cluster is not greater than the number threshold, for each document in the cluster, determine the document number, and determine the first document identifier of the document based on the document number and the cluster number;

[0220] If the number of documents in the cluster exceeds the threshold, each document in the cluster is clustered to obtain at least one sub-cluster. Based on the numbering rules and the number of the cluster, the number of each sub-cluster is determined. For each sub-cluster, based on the number of the sub-cluster, the first document identifier of each document in the sub-cluster is determined.

[0221] In some embodiments, the apparatus further includes a retrieval model training module, configured to:

[0222] The neural network is pre-trained using a document set and the document identifier tag corresponding to each document in the document set to obtain a pre-trained retrieval model; the document identifier tag corresponding to each document represents the document identifier corresponding to the document in the document identifier set.

[0223] The retrieval model is fine-tuned using a query information sample set and the document identifier tag corresponding to each query information sample in the query information sample set, to obtain the fine-tuned retrieval model; the document identifier tag corresponding to the query information sample represents the document identifier corresponding to the query information sample in the document identifier set.

[0224] The first determining module is used for:

[0225] Using the finely tuned retrieval model, based on semantic analysis of the information to be queried, at least one target document identifier corresponding to the information to be queried is determined from the document identifier set.

[0226] In some embodiments, the retrieval model training module is further configured to:

[0227] Using a neural network, based on semantic analysis of each document in the document set, at least one predicted document identifier is determined for each document; wherein the at least one predicted document identifier belongs to the document identifier set.

[0228] A first loss is determined based on the document identifier tag of each document and at least one corresponding predicted document identifier;

[0229] Based on the first loss, the network parameters of the neural network are updated to obtain the pre-trained retrieval model.

[0230] In some embodiments, the retrieval model training module is further configured to:

[0231] Using the pre-trained retrieval model, based on semantic analysis of each query information sample in the query information sample set, at least one predicted document identifier is determined for each query information sample; wherein, the at least one predicted document identifier belongs to the document identifier set;

[0232] The second loss is determined based on the document identifier label corresponding to each of the query information samples and at least one corresponding predicted document identifier;

[0233] Based on the second loss, the model parameters of the pre-trained retrieval model are updated to obtain the fine-tuned retrieval model.

[0234] In some embodiments, the first determining module 720 is configured to: perform semantic analysis on the information to be queried to obtain first semantic information; determine, according to a preset search method, a plurality of document identifiers with the highest probability of matching the first semantic information from the document identifier set, and determine the plurality of document identifiers as the target document identifier;

[0235] The second determining module 730 is configured to: determine, based on the mapping relationship, a target document corresponding to the target document identifier from the document set; determine the importance of each candidate word in each statement of the target document and the importance of each query word in the query information; determine the similarity between each statement and the query information based on the importance of each candidate word in each statement and the importance of each query word; and determine at least one statement with the highest similarity to the query information as the target statement.

[0236] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0237] It should be noted that, in the embodiments of this application, if the above-described retrieval method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0238] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0239] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0240] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0241] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0242] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0243] It should be noted that, Figure 8 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this application, such as... Figure 8 As shown, the hardware entity of the computer device 800 includes: a processor 801, a communication interface 802, and a memory 803, wherein:

[0244] Processor 801 typically controls the overall operation of computer device 800.

[0245] The communication interface 802 enables computer devices to communicate with other terminals or servers over a network.

[0246] The memory 803 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 801 and various modules in the computer device 800. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 801, the communication interface 802, and the memory 803 can be performed via bus 804.

[0247] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0248] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0249] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0250] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0251] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0252] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0253] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0254] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A retrieval method, the method comprising: Obtain the information to be queried; Based on semantic analysis of the information to be queried, at least one target document identifier corresponding to the information to be queried is determined from the document identifier set; wherein, each document identifier in the document identifier set has a semantic mapping relationship with the corresponding document; the multiple document identifiers in the document identifier set are obtained by clustering the multiple documents based on the similarity of the semantic vectors between the multiple documents in the document set; each document identifier corresponds to a specified document among the multiple documents. Based on the mapping relationship, at least one target statement that matches the query information is determined from the document corresponding to the target document identifier.

2. The method of claim 1, wherein, The step of determining at least one target document identifier corresponding to the query information from the document identifier set based on semantic analysis of the query information includes: Based on the semantic analysis of the information to be queried, the first semantic information is obtained; Identify at least one target document identifier in the document identifier set that matches the first semantic information.

3. The method of claim 1, wherein, The method further includes: Each document in the document set is clustered to obtain at least one cluster; wherein the second semantic information of each document in each cluster satisfies a specific condition; According to the numbering rules, determine the number of each cluster. For each cluster, a first document identifier is determined for each document in the cluster based on the cluster number; The set of first document identifiers for each document in the document set is taken as the document identifier set.

4. The method of claim 3, wherein, The step of determining the first document identifier for each document in the cluster based on the cluster number includes at least one of the following: If the number of documents in the cluster is not greater than the number threshold, for each document in the cluster, determine the document number, and determine the first document identifier of the document based on the document number and the cluster number; If the number of documents in the cluster exceeds the threshold, each document in the cluster is clustered to obtain at least one sub-cluster. Based on the numbering rules and the number of the cluster, the number of each sub-cluster is determined. For each sub-cluster, based on the number of the sub-cluster, the first document identifier of each document in the sub-cluster is determined.

5. The method of any one of claims 1 to 4, wherein, The method further includes: The neural network is pre-trained using a document set and the document identifier tag corresponding to each document in the document set to obtain a pre-trained retrieval model; the document identifier tag corresponding to each document represents the document identifier corresponding to the document in the document identifier set. The retrieval model is fine-tuned using a query information sample set and the document identifier tag corresponding to each query information sample in the query information sample set, to obtain the fine-tuned retrieval model; the document identifier tag corresponding to the query information sample represents the document identifier corresponding to the query information sample in the document identifier set. The step of determining at least one target document identifier corresponding to the query information from the document identifier set based on semantic analysis of the query information includes: Using the finely tuned retrieval model, based on semantic analysis of the information to be queried, at least one target document identifier corresponding to the information to be queried is determined from the document identifier set.

6. The method of claim 5, wherein, The step of pre-training the neural network using a document set and document identifier tags corresponding to each document in the document set to obtain a pre-trained retrieval model includes: Using a neural network, based on semantic analysis of each document in the document set, at least one predicted document identifier is determined for each document; wherein the at least one predicted document identifier belongs to the document identifier set. A first loss is determined based on the document identifier tag of each document and at least one corresponding predicted document identifier; Based on the first loss, the network parameters of the neural network are updated to obtain the pre-trained retrieval model.

7. The method according to claim 5, wherein, The step of fine-tuning the pre-trained retrieval model using a query information sample set and the document identifier tags corresponding to each query information sample in the query information sample set to obtain the fine-tuned retrieval model includes: Using the pre-trained retrieval model, based on semantic analysis of each query information sample in the query information sample set, at least one predicted document identifier is determined for each query information sample; wherein, the at least one predicted document identifier belongs to the document identifier set; The second loss is determined based on the document identifier label corresponding to each of the query information samples and at least one corresponding predicted document identifier; Based on the second loss, the model parameters of the pre-trained retrieval model are updated to obtain the fine-tuned retrieval model.

8. The method according to any one of claims 1 to 5, wherein, The step of determining at least one target document identifier corresponding to the query information from a set of document identifiers based on semantic analysis of the query information includes: performing semantic analysis on the query information to obtain first semantic information; determining multiple document identifiers with the highest matching probability to the first semantic information from the set of document identifiers according to a preset search method, and determining the multiple document identifiers as the target document identifier; The step of determining at least one target statement matching the query information from the documents corresponding to the target document identifier based on the mapping relationship includes: determining the target document corresponding to the target document identifier from the document set based on the mapping relationship; determining the importance of each candidate word in each statement in the target document and the importance of each query word in the query information; determining the similarity between each statement and the query information based on the importance of each candidate word in each statement and the importance of each query word; and determining at least one statement with the highest similarity to the query information as the target statement.

9. A retrieval device, the device comprising: The retrieval module is used to retrieve the information to be queried. The first determining module is used to determine at least one target document identifier corresponding to the query information from a document identifier set based on semantic analysis of the query information; wherein each document identifier in the document identifier set has a semantic mapping relationship with its corresponding document; the multiple document identifiers in the document identifier set are obtained by clustering the multiple documents based on the similarity of their semantic vectors; each document identifier corresponds to a specified document among the multiple documents; The second determining module is used to determine, based on the mapping relationship, at least one target statement that matches the query information from the document corresponding to the target document identifier.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any one of claims 1 to 8.

Images