Semantic retrieval method and device, equipment, medium and product

Abstract

The invention discloses a semantic retrieval method and device, equipment, a medium and a product, and the method comprises the steps: obtaining a query statement and a pre-trained semantic search model which comprises an input layer, a presentation layer and a matching layer; carrying out global modeling on the context of the query statement based on a full-word mask technology of an input layer, and determining a text vector; processing the text vector based on a gating mechanism of a presentation layer to obtain a text global feature vector; and determining target corpus information matched with the text global feature vector in a pre-established reply data corpus based on the matching layer, and taking the target corpus information as a reply result of the reply query statement. Semantic information in a query statement is effectively captured through the input layer, a gating mechanism is introduced to further model text sequence information, richer semantic representation is obtained, and target corpus information is matched through the matching layer. The understanding and characterization capability of the model on the whole query statement is improved, and the accuracy of the query result is improved.

Description

Technical Field

[0001] The present invention relates to the field of information retrieval technology, and in particular to a semantic retrieval method, device, equipment, medium and product. Background Art

[0002] For the entire banking industry, the intelligent customer service system can provide non-real-time consultation around the clock, which reduces user waiting time, solves standardized customer inquiries, and also reduces the workload of manual customer service. The introduction of intelligent customer service technology helps reduce banks' operating costs and significantly improves banking customer service experience and business processing efficiency. How to effectively retrieve corresponding information based on the semantics of customer query content is the key content of intelligent customer service technology.

[0003] Intelligent customer service semantic retrieval technology based on natural language processing: The core of semantic retrieval technology based on natural language processing is to convert natural language into semantic representation, and then use semantic similarity calculation technology to finally achieve semantic matching and search of text. In the intelligent customer service technology of banks, natural language processing technology is used to understand the questions, queries and needs raised by customers. This technology mainly uses traditional methods such as rule-based, statistical and machine learning, and needs to combine word segmentation, part-of-speech tagging, named entity recognition and other methods to extract the semantic information of the text, and then achieve text matching.

[0004] This technology relies on artificially constructed rules and features, and requires human participation in feature extraction and selection. Therefore, it is to some extent limited by human language knowledge and experience. Summary of the invention

[0005] The present invention provides a semantic retrieval method, device, equipment, medium and product to realize automatic retrieval of semantics.

[0006] According to one aspect of the present invention, a semantic retrieval method is provided, comprising:

[0007] Obtaining a query statement and a pre-trained semantic search model, wherein the semantic search model includes an input layer, a representation layer, and a matching layer;

[0008] Performing global modeling on the context of the query statement based on the full-word masking technology of the input layer to determine a text vector;

[0009] Processing the text vector based on the gating mechanism of the representation layer to obtain a global feature vector of the text;

[0010] Based on the matching layer, target corpus information matching the global feature vector of the text is determined in a pre-established response data corpus, and the target corpus information is used as a response result for responding to the query statement.

[0011] According to a second aspect of the present invention, there is provided a semantic search device, comprising:

[0012] An information acquisition module, used to acquire query statements and a pre-trained semantic search model, wherein the semantic search model includes an input layer, a representation layer, and a matching layer;

[0013] A first determination module is used to perform global modeling on the context of the query sentence based on the full-word masking technology of the input layer to determine a text vector;

[0014] A second determination module is used to process the text vector based on the gating mechanism of the representation layer to obtain a text global feature vector;

[0015] A result determination module is used to determine target corpus information that matches the global feature vector of the text in a pre-established response data corpus based on the matching layer, and use the target corpus information as a response result for responding to the query statement.

[0016] According to a third aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the semantic retrieval method described in any embodiment of the present invention.

[0020] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the semantic retrieval method described in any embodiment of the present invention when executed.

[0021] According to a fifth aspect of the present invention, an embodiment of the present invention further provides a computer program product, the computer program product comprising a computer program, and the computer program implements the semantic retrieval method of any embodiment of the present invention when executed by a processor.

[0022] The technical solution of the embodiment of the present invention is to obtain a query statement and a pre-trained semantic search model, wherein the semantic search model includes an input layer, a representation layer, and a matching layer; the context of the query statement is globally modeled based on the full-word masking technology of the input layer to determine the text vector; the text vector is processed based on the gating mechanism of the representation layer to obtain the text global feature vector; the target corpus information matching the text global feature vector is determined in the pre-established reply data corpus based on the matching layer, and the target corpus information is used as the reply result of the reply query statement. The semantic information in the query statement is effectively captured through the input layer, and a number of fixed-length vectors are output. The text sequence information is further modeled through the representation layer that introduces the gating mechanism to obtain a richer text global feature vector, and then the target corpus information is matched based on the text global feature vector through the matching layer. The model's ability to understand and represent the entire query statement is improved, thereby improving the accuracy of the query results.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flowchart of a semantic retrieval method provided according to Embodiment 1 of the present invention;

[0026] Figure 2 is a flowchart of a semantic retrieval method provided according to Embodiment 2 of the present invention;

[0027] Figure 3 is a structural schematic diagram of a semantic retrieval device provided according to Embodiment 3 of the present invention;

[0028] Figure 4 It is a schematic diagram of the structure of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 A flowchart of a semantic search method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of automatic semantic search of query content. The method can be executed by a semantic search device. The semantic search method can be implemented in the form of hardware and / or software. The semantic search method can be configured in an electronic device. Figure 1 As shown, the method includes:

[0033] S110, obtaining a query statement and a pre-trained semantic search model, wherein the semantic search model includes an input layer, a representation layer, and a matching layer.

[0034] In this embodiment, the query statement can be understood as a statement input by the user for business function operation query, such as how to check the balance, how to transfer money, etc. The semantic search model can be understood as a model for semantic matching and search. The input layer can be understood as a layer that converts text into vectors. The representation layer can be understood as a layer that captures the semantics and context information between query texts. The matching layer can be understood as a statement that is used to determine the sentence that matches the query statement in the pre-established response data corpus.

[0035] Specifically, the processor can obtain the query statement input by the user and the pre-trained semantic search model stored in the storage medium.

[0036] S120. Perform global modeling on the context of the query statement based on the full-word masking technology of the input layer to determine the text vector.

[0037] In this embodiment, the whole word masking technology (WWM) can be understood as an algorithm that promotes the model's in-depth understanding of vocabulary and context by masking the entire vocabulary, thereby improving the model's performance on various tasks. Context is used to characterize the relationship between sentence contexts. Text vectors can be understood as text converted into vectors.

[0038] Specifically, the processor can first segment the query sentence, split the query sentence into words, randomly mask the words through the whole word masking technology, obtain the masked words and replace them in the original query sentence, and input the overall query sentence containing the masked words into the input layer, and globally model the context of the overall query sentence containing the masked words based on the self-attention mechanism and Transformer structure of the input layer, determine the masked words, map texts of different lengths into vector representations of fixed length, and obtain text vectors.

[0039] S130 , processing the text vector based on the gating mechanism of the representation layer to obtain a global feature vector of the text.

[0040] In this embodiment, the gating mechanism is a gated recurrent unit (GRU), which is a neural network structure for processing sequence data and is particularly suitable for tasks such as natural language processing (NLP) and time series prediction. The global feature vector of the text can be understood as the vector result processed by the gating mechanism.

[0041] Specifically, the processor can input the text vector into the representation layer and process the text vector based on the gating mechanism of the representation layer. The gating mechanism can dynamically adjust the state of the model according to the text vector. It effectively processes the text vector through the reset gate and update gate mechanism, so that it can better understand the user's query intention, thereby providing a more accurate response and obtaining a global feature vector of the text.

[0042] S140. Determine target corpus information that matches the global feature vector of the text in a pre-established response data corpus based on the matching layer, and use the target corpus information as a response result for the response query statement.

[0043] In this embodiment, the reply data corpus can be understood as a pre-stored corpus for reply, and each reply sentence corresponds to a standard query sentence. The target corpus information can be understood as information for replying to the query sentence.

[0044] Specifically, the processor can calculate the similarity between each reply data corpus and the global feature vector of the text in a pre-established reply data corpus based on the matching layer, take the one with the largest similarity as the target corpus information that matches the global feature vector of the text, and take the target corpus information as the reply result of the reply query statement.

[0045] The technical solution of the embodiment of the present invention is to obtain a query statement and a pre-trained semantic search model, wherein the semantic search model includes an input layer, a representation layer, and a matching layer; the context of the query statement is globally modeled based on the full-word masking technology of the input layer to determine the text vector; the text vector is processed based on the gating mechanism of the representation layer to obtain the text global feature vector; the target corpus information matching the text global feature vector is determined in the pre-established reply data corpus based on the matching layer, and the target corpus information is used as the reply result of the reply query statement. The semantic information in the query statement is effectively captured through the input layer, and a number of fixed-length vectors are output. The text sequence information is further modeled through the representation layer that introduces the gating mechanism to obtain a richer text global feature vector, and then the target corpus information is matched based on the text global feature vector through the matching layer. The model's ability to understand and represent the entire query statement is improved, thereby improving the accuracy of the query results.

[0046] As a first optional embodiment of the first embodiment, based on the above embodiment, the training steps of the pre-trained semantic search model include:

[0047] Obtain a sample set to be trained and an initial semantic search model; construct a masked language modeling task based on the samples to be trained in the sample set and the full-word masking technology; perform masked vocabulary prediction on the masked language modeling task based on the initial input layer of the initial semantic search model to determine the first output vector; process the first output vector based on the initial representation layer and the initial matching layer of the initial semantic search model to determine the model output vector; determine the loss function based on the cross entropy loss function, the model output vector and the real labels in the sample set to be trained; adjust the parameters of the initial semantic search model based on the loss function, and perform iterative training until the training end conditions are met to obtain a pre-trained semantic search model.

[0048] In this embodiment, the sample set to be trained can be understood as a set of labeled samples for training. The initial semantic search model can be understood as an untrained semantic search model. The samples to be trained can be understood as labeled samples to be trained. The Masked Language Modeling task (MLM) can be understood as a task of performing random masking and mask prediction. The first output vector can be understood as a vector output by the initial input layer. The initial input layer is an untrained input layer. The initial representation layer is an untrained representation layer. The initial matching layer is an untrained matching layer. The model output vector can be understood as the matching probability result output by the model. The cross entropy loss function can be understood as a loss function algorithm used to characterize the difference between the model output and the true label. The true label can be understood as the true value corresponding to the sample used in this training. The loss function can be understood as a function used to characterize the difference between the output result of this model and the true label. The training end condition can be that the number of iterative training is reached or the loss function reaches a set threshold.

[0049] Specifically, the processor can obtain a set of samples to be trained and an initial semantic search model. The processor can divide the training samples by vocabulary, decompose the training samples into vocabulary, and randomly select several vocabulary words to replace with mask tags through the whole word masking technology, so that the model predicts the masked words according to the query context. For example, the entire vocabulary can be replaced with the [MASK] tag to construct a masked language modeling task, and introduce a dynamic masking strategy in the MLM task, so that the masked position is not fixed but dynamically generated. The dynamic masking mechanism dynamically selects a certain proportion of words to be masked for each training sample, and then randomly replaces a certain proportion of Chinese vocabulary. Therefore, in each training, the sample will randomly select vocabulary to be masked. Compared with the traditional RoBERTa's MLM task, in which the masking operation is divided in units of single words, this method can better retain the complete information of Chinese words, thereby improving the model's understanding and representation capabilities of the entire word. The processor can perform mask vocabulary prediction on the masked tokens in the masked language modeling task according to the initial input layer of the initial semantic search model to determine the first output vector; the first output vector is processed based on the initial representation layer and the initial matching layer of the initial semantic search model to determine the model output vector. This model uses a cross entropy loss function. Minimizing the cross entropy loss is equivalent to maximizing the conditional probability of the correct label appearing. The loss function is determined according to the cross entropy loss function combined with the model output vector and the real label in the sample set to be trained; the parameters of the initial semantic search model are adjusted according to the loss function, and iterative training is performed until the training end condition is met to obtain a pre-trained semantic search model.

[0050] Exemplarily, the calculation formula of the cross entropy loss function can be:

[0051]

[0052] Among them, D + is the positive example in the training sample, Q is the model output vector, and Λ is all the training parameters of the model.

[0053] Exemplarily, in order to facilitate understanding of the masked vocabulary prediction process in the present invention, a specific example is used as a demonstration. The input sample to be trained is "reasons for failed loan applications". The third input word "failure" is replaced with the [MASK] symbol. [MASK] is encoded as a word vector through the embedding layer of the initial input layer. The output of the [MASK] position is recorded as a vector, which contains context information and can be used to predict the masked words. The remaining words are also encoded into word vectors of the same dimension through the embedding layer. The word vector is input as a feature into the softmax classifier of the initial input layer. The classifier outputs a probability distribution. Since each word in the dictionary has its corresponding probability value, the words in the specific customer query sentence that are masked can be determined.

[0054] Embodiment 2

[0055] Figure 2 This is a flowchart of a semantic search method provided in Embodiment 2 of the present invention. This embodiment is a refinement of the above embodiment. Figure 2 As shown, the method includes:

[0056] S201. Obtain query statements and pre-trained semantic search models.

[0057] S202: Split the query statement into query terms.

[0058] In this embodiment, the query words can be understood as the result of dividing the query sentence into words.

[0059] Specifically, the processor may divide the query statement into units of words by using the whole-word masking technology to obtain the query words included in the query statement.

[0060] For example, taking "how to check balance" and "how to transfer money" as examples, after being split, the query terms are "how" to "check" "balance" and "how" to "transfer money" respectively.

[0061] S203, randomly masking the query words based on the full-word masking technology of the input layer to obtain masked words, and replacing the original query words to obtain an intermediate sentence.

[0062] In this embodiment, the masked words can be understood as words that are blocked, and the intermediate sentences can be understood as sentences replaced with the masked words.

[0063] Specifically, the processor can randomly select one or more words from the query vocabulary for masking based on the full-word masking technology of the input layer, replace the words with mask marks to obtain masked words, and replace them in the corresponding positions in the original query vocabulary to obtain an intermediate sentence.

[0064] S204, globally modeling the context of the intermediate sentence based on the input layer to determine the text vector, the input layer includes an embedding layer, an encoding layer and a preset classifier.

[0065] In this embodiment, the embedding layer can be understood as a layer for performing vector conversion. The encoding layer can use a Transformer encoder to convert the text into a continuous vector representation. The preset classifier can be understood as an algorithm for determining the probability distribution of words in a preset dictionary that are similar to the masked words, for example, a softmax classifier.

[0066] Specifically, the processor can input the intermediate sentence into the input layer, convert the sentence into a vector form through the embedding layer, encoder and preset classifier in the input layer, determine the vocabulary corresponding to the mask vocabulary, and then add it to the original vector to determine the text vector.

[0067] Furthermore, the embedding layer includes a word embedding layer, a paragraph embedding layer, and a position embedding layer. Accordingly, based on the input layer, the context of the intermediate sentence is globally modeled, and the steps of determining the text vector can be refined as follows:

[0068] Based on the preset special character group and word embedding layer, the intermediate sentence is converted into a word vector containing context information, wherein the masked word is converted into a masked vector; according to the paragraph embedding layer, the intermediate sentence is converted into a paragraph vector; according to the position embedding layer, the position vector of each word in the intermediate sentence is determined; each word vector is input into the encoding layer for encoding to obtain an encoding vector; each encoding vector is input into a preset classifier to determine the prediction vector corresponding to the mask vector to obtain a final word vector; and a text vector is obtained by adding the final word vector, the position vector and the paragraph vector.

[0069] In this embodiment, the preset special character group can be understood as a combination of characters used to distinguish different sentences, for example, it can include two special characters [CLS] and [SEP], where [SEP] is used to segment the input sentence, and [CLS] is located at the front of the entire sentence. This symbol can integrate the semantics of the two sentences before and after [SEP]. The word embedding layer can be understood as an algorithm for converting discrete vocabulary into word vector representations in a continuous vector space. The paragraph embedding layer can be understood as an algorithm for converting an entire paragraph or document into a vector representation of a fixed size. The position embedding layer can be understood as an algorithm for the position vector of the position corresponding to each element in the vector. The encoding vector can be understood as a word vector of the same dimension.

[0070] Specifically, the query statement can be distinguished by adding a preset special character group, and the intermediate statement can be converted into a word vector containing context information through a word embedding layer. In the word vector, all normal words contained in the input and the preset special characters in the preset special character group are embedded, wherein the masked word is converted into a masked vector. The processor can convert the intermediate statement into a paragraph vector according to the paragraph embedding layer, and in the paragraph vector, different encoding methods are used to separate the two sentences, for example, the word vectors belonging to the first sentence all use the A encoding method, and the word vectors belonging to the second sentence all use the B encoding method. According to the position embedding layer, the position vector of each word in the intermediate statement is determined, and the position of each word vector in the overall statement is determined by the position vector. Each word vector is input into the encoding layer for encoding, and an encoding vector of the same dimension is output; each encoding vector is input into a preset classifier, and a probability distribution is output by the preset classifier to determine the probability value corresponding to each word in the preset dictionary, and the word with the largest probability value is determined from it, which is the prediction vector corresponding to the mask vector, and the final word vector is obtained. The text vector is obtained by adding the final word vector, the position vector and the paragraph vector.

[0071] S205 : Acquire the previous moment hidden state and the previous moment candidate hidden state determined by the update gate of the gating mechanism of the presentation layer at the previous moment.

[0072] In this embodiment, the previous moment can be understood as the moment of the last processing through the gating mechanism, for example, it can be a query statement including three sentences, and the moment corresponding to the previous sentence of the sentence currently being processed. The update gate is an algorithm used to control whether information needs to be paid attention to and determine which historical data context information is meaningful for the current customer semantic search query. The hidden state at the previous moment can be understood as the output state of the hidden layer at the previous moment. It carries historical information and has a direct impact on the output of the current moment and the hidden state of the next moment. The candidate state at the previous moment can be understood as the candidate hidden state at the current moment. The reset gate is used to control whether the calculation depends on the state of the previous moment.

[0073] Specifically, the processor may obtain the previous moment hidden state and the previous moment candidate hidden state determined by the update gate in the gating mechanism of the representation layer at the previous moment based on the global feature vector of the text at the previous moment.

[0074] S206: Determine a correlation vector between the text vector and the historical query vector based on the hidden state at the previous moment, a preset activation function, and a reset gate of a gating mechanism of the representation layer.

[0075] In this embodiment, the preset activation function may be a Sigmoid function. The historical query vector may be understood as the vector of the last query. The correlation vector may be understood as a vector used to characterize the correlation between the text vector and the historical query vector. The reset gate may be understood as an algorithm used to control whether information is forgotten and to determine the correlation between the text vector input by the customer at the current moment and the historical query vector, so as to better understand the customer's query.

[0076] Specifically, the processor may determine the correlation vector between the text vector and the historical query vector based on the hidden state at the previous moment and a preset activation function through a reset gate of the gating mechanism of the representation layer.

[0077] Exemplarily, the update gate and reset gate are calculated as follows:

[0078] r t =σ(x t W xr +h t-1 W hr +b r )

[0079] z t =σ(x t W xz +h t-1 W hz +b z )

[0080] Among them, W matrix is ​​the weight matrix, x t is the input of the current time step, h t-1 is the hidden state of the previous time step, and b represents the bias term. The activation function of the reset gate and the update gate is the Sigmoid function. Because the value range of this activation function is (0,1), the value range can be used to control whether the flow of information is transmitted backward. 0 means forgetting, and 1 means keeping.

[0081] S207: Update the candidate hidden state at the previous moment according to the correlation vector to obtain the candidate hidden state at the current moment.

[0082] In this embodiment, the candidate hidden state at the current moment may be understood as a candidate hidden state determined at the current moment.

[0083] Specifically, the hidden state at the previous moment can be updated by the correlation vector determined at the current moment. Specifically, the candidate hidden state can be calculated by the following formula:

[0084]

[0085] Among them, tanh represents the hyperbolic tangent activation function.

[0086] S208. Perform a weighted combination of the previous moment hidden state and the current moment candidate hidden state according to the update gate of the gating mechanism to obtain the current moment hidden state.

[0087] Specifically, the hidden state at the previous moment and the candidate hidden state at the current moment may be weightedly combined according to the update gate of the gating mechanism to obtain the hidden state at the current moment.

[0088]

[0089] By introducing the concept of gates, GRU has better memory and selective updating capabilities when processing sequence data, so that it can better capture long-term dependencies. At the same time, by learning to control the weights of the gates, it effectively solves the problems of gradient disappearance and gradient explosion.

[0090] S209: Determine the global feature vector of the text according to the correlation vector.

[0091] Specifically, the prediction result at the current time step can be determined through the correlation vector and the gating mechanism, which is the global feature vector of the text.

[0092] S210. For each corpus vector in the pre-established response data corpus, determine the cosine similarity between the corpus vector and the global feature vector of the text through a matching layer.

[0093] In this embodiment, the corpus vector can be understood as the query corpus stored in the reply data corpus. Since there may be different ways of expressing a word, one or more corpus vectors can be stored in the reply data corpus for a type of query. Cosine similarity can be understood as a way of reflecting the similarity between two vectors through a numerical value.

[0094] Specifically, the goal of the matching layer is to calculate the semantic similarity between the query and each corpus vector in the response data corpus, and use the cosine similarity calculation method to measure the similarity between two texts. The feature vectors corresponding to similar texts are closer in space. The higher the cosine similarity score, the more similar the two texts are. Conversely, it means they are not similar. Finally, the feature vector that is most similar to the query text, that is, the closest in space, is returned. The formula is as follows:

[0095]

[0096] Among them, y Q is the global feature vector of the text output by the representation layer, Represents the i-th corpus vector in the reply data corpus.

[0097] S211. Process each cosine similarity according to a soft maximum function to determine the similarity probability between the query statement and each corpus vector.

[0098] In this embodiment, the softmax() function can be understood as a normalized exponential function. The similarity probability can be understood as a function for representing the degree of similarity between the corpus vector and the query sentence in a probability form.

[0099] Specifically, each cosine similarity can be normalized by a soft maximum function to convert the output of the model into a probability form and maximize the probability of the positive document appearing. The formula is as follows:

[0100]

[0101] Among them, γ is the smoothing factor of Softmax.

[0102] S212. The corpus vector with the largest similarity probability and the corresponding reply information in the reply data corpus are used as target corpus information.

[0103] Specifically, the model may use the corpus vector with the highest similarity probability and the answer information to the question corresponding to the corpus vector as the target corpus information.

[0104] S213. Use the target corpus information as a response result to the query statement.

[0105] Specifically, the processor may use the target corpus information as a response result to the query statement and feed it back to the sender of the query statement to answer the corresponding query question.

[0106] The technical solution of the embodiment of the present invention randomly masks the query vocabulary through the input layer to obtain the masked vocabulary, which can better retain the complete information of the vocabulary, thereby improving the model's understanding and representation ability of the entire vocabulary. The input layer embeds, encodes and predicts the masked vocabulary of the intermediate sentence after replacing the masked vocabulary to obtain a text vector, realizes a unified text representation, and texts of different lengths are mapped to vector representations of fixed lengths. The relationship between the text vector and the historical query vector is determined by the gating mechanism of the representation layer to obtain a text global feature vector, better capture the semantics and contextual information between the customer query text, and better understand the customer's query intention through the representation layer, thereby providing a more accurate response, improving the model's understanding of text semantics and contextual information, and more accurately measuring the similarity between the query text and the historical query data, thereby achieving more accurate intelligent customer service semantic search. The matching target corpus information is determined in the reply data corpus through the matching layer to obtain the final modified reply result. The accuracy and semantic expression ability of text matching in the intelligent customer service scenario are improved through more accurate vector representation, so that the model can understand the query intention more accurately, improve the accuracy of the query, and provide a better experience for users.

[0107] Embodiment 3

[0108] Figure 3 This is a schematic diagram of the structure of a semantic search device provided by Embodiment 3 of the present invention. Figure 3 As shown, the device includes: an information acquisition module 31, a first determination module 32, a second determination module 33 and a result determination module 34. Among them,

[0109] An information acquisition module 31 is used to acquire a query statement and a pre-trained semantic search model, wherein the semantic search model includes an input layer, a representation layer, and a matching layer;

[0110] A first determination module 32, configured to perform global modeling on the context of the query sentence based on the full-word masking technology of the input layer to determine a text vector;

[0111] A second determination module 33 is used to process the text vector based on the gating mechanism of the representation layer to obtain a text global feature vector;

[0112] The result determination module 34 is used to determine the target corpus information matching the global feature vector of the text in the pre-established response data corpus based on the matching layer, and use the target corpus information as the response result for responding to the query statement.

[0113] The technical solution of the embodiment of the present invention is to obtain a query statement and a pre-trained semantic search model, wherein the semantic search model includes an input layer, a representation layer, and a matching layer; the context of the query statement is globally modeled based on the full-word masking technology of the input layer to determine the text vector; the text vector is processed based on the gating mechanism of the representation layer to obtain the text global feature vector; the target corpus information matching the text global feature vector is determined in the pre-established reply data corpus based on the matching layer, and the target corpus information is used as the reply result of the reply query statement. The semantic information in the query statement is effectively captured through the input layer, and a number of fixed-length vectors are output. The text sequence information is further modeled through the representation layer that introduces the gating mechanism to obtain a richer text global feature vector, and then the target corpus information is matched based on the text global feature vector through the matching layer. The model's ability to understand and represent the entire query statement is improved, thereby improving the accuracy of the query results.

[0114] Further, the first determining module 32 includes:

[0115] A vocabulary splitting unit, used for splitting the query statement into query words;

[0116] A sentence determination unit, configured to randomly mask the query words based on the full-word masking technology of the input layer to obtain masked words, and replace the original query words to obtain an intermediate sentence;

[0117] A vector determination unit is used to perform global modeling on the context of the intermediate sentence based on the input layer to determine the text vector, wherein the input layer includes an embedding layer, an encoding layer and a preset classifier.

[0118] The embedding layer includes a word embedding layer, a paragraph embedding layer and a position embedding layer. Accordingly, the vector determination unit is specifically used for:

[0119] Based on a preset special character group and the word embedding layer, converting the intermediate sentence into a word vector containing context information, wherein the mask vocabulary is converted into a mask vector;

[0120] According to the paragraph embedding layer, converting the intermediate sentence into a paragraph vector;

[0121] Determining, according to the position embedding layer, a position vector of each word in the intermediate sentence;

[0122] Input each of the word vectors into the encoding layer for encoding to obtain an encoding vector;

[0123] Input each of the encoding vectors into the preset classifier, determine the prediction vector corresponding to the mask vector, and obtain the final word vector;

[0124] A text vector is obtained by adding the final word vector, the position vector and the paragraph vector.

[0125] Furthermore, the second determining module 33 is specifically configured to:

[0126] Acquire the previous moment hidden state and the previous moment candidate hidden state determined by the update gate of the gating mechanism of the representation layer at the previous moment;

[0127] Determine a correlation vector between the text vector and the historical query vector based on the previous hidden state, a preset activation function, and a reset gate of the gating mechanism of the representation layer;

[0128] The candidate hidden state at the previous moment is updated according to the correlation vector to obtain the candidate hidden state at the current moment;

[0129] Performing a weighted combination of the previous moment hidden state and the current moment candidate hidden state according to the update gate of the gating mechanism to obtain the current moment hidden state;

[0130] A global feature vector of the text is determined according to the correlation vector.

[0131] Furthermore, the result determination module 34 is specifically used for:

[0132] For each corpus vector in the pre-established answer data corpus, determining the cosine similarity between the corpus vector and the text global feature vector through the matching layer;

[0133] Processing each of the cosine similarities according to a soft maximum function to determine a similarity probability between the query statement and each of the corpus vectors;

[0134] The corpus vector with the largest similarity probability and the corresponding reply information in the reply data corpus are used as target corpus information;

[0135] The target corpus information is used as a response result to the query statement.

[0136] Optionally, the device further comprises:

[0137] Model training module, used to obtain the sample set to be trained and the initial semantic search model;

[0138] Constructing a masked language modeling task according to the training samples in the training sample set and the full-word masking technology;

[0139] According to the initial input layer of the initial semantic search model, performing masked vocabulary prediction on the masked language modeling task to determine a first output vector;

[0140] Processing the first output vector based on the initial representation layer and the initial matching layer of the initial semantic search model to determine a model output vector;

[0141] Determine a loss function according to the cross entropy loss function, the model output vector and the real label in the sample set to be trained;

[0142] The parameters of the initial semantic search model are adjusted according to the loss function, and iterative training is performed until a training end condition is met to obtain a pre-trained semantic search model.

[0143] The semantic search device provided in the embodiment of the present invention can execute the semantic search method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0144] Embodiment 4

[0145] Figure 4A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0146] like Figure 4 As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0147] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0148] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 executes the various methods and processes described above, such as the semantic retrieval method.

[0149] In some embodiments, the semantic search method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the semantic search method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the semantic search method in any other appropriate manner (e.g., by means of firmware).

[0150] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0151] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0152] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0154] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0155] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0156] In one embodiment, the embodiment of the present invention further includes a computer program product, the computer program product includes a computer program, and the computer program implements the semantic retrieval method of any embodiment of the present invention when executed by a processor.

[0157] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0158] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0159] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A semantic retrieval method, characterized in that: include: Obtaining a query statement and a pre-trained semantic search model, wherein the semantic search model includes an input layer, a representation layer, and a matching layer; Performing global modeling on the context of the query statement based on the full-word masking technology of the input layer to determine a text vector; Processing the text vector based on the gating mechanism of the representation layer to obtain a global feature vector of the text; Based on the matching layer, target corpus information matching the global feature vector of the text is determined in a pre-established response data corpus, and the target corpus information is used as a response result for responding to the query statement.

2. The method according to claim 1, characterized in that The whole word masking technology based on the input layer performs global modeling on the context of the query sentence to determine the text vector, including: Splitting the query statement into query terms; The query words are randomly masked based on the full-word masking technology of the input layer to obtain masked words, and the original query words are replaced to obtain an intermediate sentence; The context of the intermediate sentence is globally modeled based on the input layer to determine a text vector, wherein the input layer includes an embedding layer, an encoding layer, and a preset classifier.

3. The method according to claim 2, characterized in that The embedding layer includes a word embedding layer, a paragraph embedding layer and a position embedding layer. Accordingly, the context of the intermediate sentence is globally modeled based on the input layer to determine the text vector, including: Based on a preset special character group and the word embedding layer, converting the intermediate sentence into a word vector containing context information, wherein the mask vocabulary is converted into a mask vector; According to the paragraph embedding layer, converting the intermediate sentence into a paragraph vector; Determining, according to the position embedding layer, a position vector of each word in the intermediate sentence; Input each of the word vectors into the encoding layer for encoding to obtain an encoding vector; Input each of the encoding vectors into the preset classifier, determine the prediction vector corresponding to the mask vector, and obtain the final word vector; A text vector is obtained by adding the final word vector, the position vector and the paragraph vector.

4. The method according to claim 1, characterized in that: The gating mechanism based on the representation layer processes the text vector to obtain a global feature vector of the text, including: Acquire the previous moment hidden state and the previous moment candidate hidden state determined by the update gate of the gating mechanism of the representation layer at the previous moment; Determine a correlation vector between the text vector and the historical query vector based on the previous hidden state, a preset activation function, and a reset gate of the gating mechanism of the representation layer; The candidate hidden state at the previous moment is updated according to the correlation vector to obtain the candidate hidden state at the current moment; Performing a weighted combination of the previous moment hidden state and the current moment candidate hidden state according to the update gate of the gating mechanism to obtain the current moment hidden state; A global feature vector of the text is determined according to the correlation vector.

5. The method according to claim 1, characterized in that: The step of determining target corpus information matching the global feature vector of the text in a pre-established response data corpus based on the matching layer, and using the target corpus information as a response result for responding to the query statement, comprises: For each corpus vector in the pre-established answer data corpus, determining the cosine similarity between the corpus vector and the text global feature vector through the matching layer; Processing each of the cosine similarities according to a soft maximum function to determine a similarity probability between the query statement and each of the corpus vectors; The corpus vector with the largest similarity probability and the corresponding reply information in the reply data corpus are used as target corpus information; The target corpus information is used as a response result to the query statement.

6. The method according to claim 1, characterized in that The training steps of the pre-trained semantic search model include: Obtain the sample set to be trained and the initial semantic search model; Constructing a masked language modeling task according to the training samples in the training sample set and the full-word masking technology; According to the initial input layer of the initial semantic search model, performing masked vocabulary prediction on the masked language modeling task to determine a first output vector; Processing the first output vector based on the initial representation layer and the initial matching layer of the initial semantic search model to determine a model output vector; Determine a loss function according to the cross entropy loss function, the model output vector and the real label in the sample set to be trained; The parameters of the initial semantic search model are adjusted according to the loss function, and iterative training is performed until a training end condition is met to obtain a pre-trained semantic search model.

7. A semantic search device, characterized in that: include: An information acquisition module, used to acquire query statements and a pre-trained semantic search model, wherein the semantic search model includes an input layer, a representation layer, and a matching layer; A first determination module is used to perform global modeling on the context of the query sentence based on the full-word masking technology of the input layer to determine a text vector; A second determination module is used to process the text vector based on the gating mechanism of the representation layer to obtain a text global feature vector; A result determination module is used to determine target corpus information that matches the global feature vector of the text in a pre-established response data corpus based on the matching layer, and use the target corpus information as a response result for responding to the query statement.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the semantic retrieval method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the semantic retrieval method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the semantic retrieval method according to any one of claims 1 to 6.

Images