Sentence-document vectorization transform network training method and device, equipment, medium and product

Abstract

The invention provides a sentence-document vectorization transformer network training method and device, equipment, a medium and a product. The method comprises the following steps: acquiring a sentence-document pair; wherein the sentence-document pair comprises a sentence and a document; inputting the sentence into a transformer network to obtain a first sentence embedding vector; the document is input into a transformer network, and at least one first document embedding vector is obtained; wherein the transform network comprises an encoder and a decoder, and the decoder is connected with the transform network; generating a first similarity parameter based on the first sentence embedding vector and the first document embedding vector; and generating a loss value based on the first similarity parameter, and updating the transformer network based on the loss value. According to the method and the device, meanings expressed by sentences and documents can be fully understood, and the training effect of the model is improved.

Description

Technical Field

[0001] This application relates to the field of information retrieval technology, and in particular to a method, apparatus, device, medium, and product for training a sentence-document vectorization transformer network. Background Technology

[0002] Information retrieval is an essential field in daily life; for example, people use search engines to search for unknown questions and acquire new knowledge. With the rapid development of deep learning, information retrieval has also made significant progress. Vectorization, which transforms text into multi-dimensional vectors, is widely used in information retrieval because it can better understand the semantics of text within a context.

[0003] In related technologies, devices can understand sentences and documents in natural language, but they cannot fully understand the meaning expressed in sentences and documents, resulting in poor model training performance. Summary of the Invention

[0004] This application provides a sentence-document vectorization transformer network training method, apparatus, device, medium, and product that can accurately express the semantics of sentences and documents, thereby improving the model training effect.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a method for training a sentence-document vectorization transformer network, the method comprising:

[0007] Obtain sentence-document pairs; wherein, the sentence-document pair includes a sentence and a document;

[0008] The sentence is input into a transformer network to obtain a first sentence embedding vector; and the document is input into a transformer network to obtain at least one first document embedding vector; wherein the transformer network includes an encoder and a decoder;

[0009] A first similarity parameter is generated based on the first sentence embedding vector and the first document embedding vector.

[0010] A loss value is generated based on the first similarity parameter, and the transformer network is updated based on the loss value.

[0011] This application provides a sentence-document vectorization method, including:

[0012] Obtain the target document embedding vector corresponding to the query sentence and at least one target document; wherein, the target document embedding vector is a vector obtained by inputting the target document into a transformer network, and the transformer network includes an encoder and a decoder;

[0013] The query sentence is input into the transformer network to obtain the target sentence embedding vector;

[0014] Based on the target sentence embedding vector and the target document embedding vector corresponding to at least one of the target documents, a third similarity parameter is generated for each target document; wherein, the third similarity parameter is used to indicate the degree of association between the target document and the query sentence;

[0015] Based on at least one of the target documents and the third similarity parameter corresponding to each target document, a response is generated; wherein the response includes a response to the query sentence.

[0016] This application provides a sentence-document vectorization transformer network training device, including:

[0017] The first acquisition module is used to acquire sentence-document pairs; wherein, the sentence-document pair includes a sentence and a document;

[0018] A first obtaining module is configured to input the sentence into a transformer network to obtain a first sentence embedding vector; and to input the document into a transformer network to obtain at least one first document embedding vector; wherein the transformer network includes an encoder and a decoder;

[0019] The first generation module is used to generate a first similarity parameter based on the first sentence embedding vector and the first document embedding vector;

[0020] The loss module is used to generate a loss value based on the first similarity parameter and update the transformer network based on the loss value.

[0021] This application provides a sentence-document vectorization device, including:

[0022] The second acquisition module is used to acquire a query sentence and at least one target document embedding vector; wherein, the target document embedding vector is a vector obtained by inputting the target document into a transformer network, and the transformer network includes an encoder and a decoder;

[0023] The second module is used to input the query sentence into the transformer network to obtain the target sentence embedding vector;

[0024] The second generation module is used to generate a third similarity parameter for each target document based on the target sentence embedding vector and the target document embedding vector corresponding to at least one target document; wherein the third similarity parameter is used to indicate the degree of association between the target document and the query sentence;

[0025] The third generation module is used to generate response content based on at least one of the target documents and the third similarity parameter corresponding to each of the target documents; wherein the response content includes a response to the query sentence.

[0026] This application provides an electronic device, including:

[0027] processor;

[0028] Memory used to store computer programs or instructions;

[0029] The processor executes the computer program or instructions to implement the method provided in the embodiments of this application.

[0030] This application provides a computer-readable storage medium storing a computer program or instructions thereon, which, when executed by a processor, implements the method provided in this application.

[0031] This application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method provided in this application.

[0032] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0033] By inputting sentence-document pairs into a transformer network, the first sentence embedding vector and the first document embedding vector of the document can be obtained. The first sentence embedding vector is obtained by vectorizing the sentence, and the first document embedding vector is obtained by vectorizing the document. Electronic devices can fully understand the meaning expressed by the sentence and the document respectively through the first sentence embedding vector and the first document embedding vector.

[0034] Furthermore, taking a sentence as an example, the encoder and decoder in the transformer network are used to process the sentence together to obtain the first sentence embedding vector. This first sentence embedding vector contains richer contextual information and can more accurately express the semantics of the sentence. Thus, this disclosure integrates the advantages of the encoder and decoder, and uses more neurons to generate vectors rich in the semantics of the sentence and document respectively, making the network's fitting ability stronger, enabling electronic devices to more fully understand the meaning expressed by the sentence and document, and also improving the training effect of the transformer network. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0036] Figure 1 This is a schematic diagram of a Sentence-BERT model in related technologies.

[0037] Figure 2 This is a schematic diagram of a GPT network in related technologies.

[0038] Figure 3 This is a flowchart illustrating a sentence-document vectorization transformer network training method according to an exemplary embodiment.

[0039] Figure 4 This is a schematic diagram illustrating a first encoder and a first decoder according to an exemplary embodiment.

[0040] Figure 5 This is a schematic diagram illustrating a second encoder and a second decoder according to an exemplary embodiment.

[0041] Figure 6 This is a schematic diagram of a transformer network according to an exemplary embodiment.

[0042] Figure 7 This is a flowchart illustrating a sentence-document vectorization method according to an exemplary embodiment.

[0043] Figure 8 This is a schematic diagram illustrating the structure of a sentence-document vectorization transformer network training device according to an exemplary embodiment.

[0044] Figure 9 This is a schematic diagram illustrating the structure of a sentence-document vectorization device according to an exemplary embodiment.

[0045] Figure 10This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0048] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0049] 1) Pre-trained Language Models: Pre-trained Language Models (PLMs) are a core technology in the field of Natural Language Processing (NLP). They refer to models that acquire general language representation capabilities (such as syntax, semantics, and contextual relationships) through self-supervised learning on large-scale unlabeled text data, and then fine-tune the model to adapt it to downstream tasks.

[0050] 2) Transformer Network Architecture: This is a deep learning architecture based entirely on self-attention, first proposed in the 2017 paper "Attention Is All You Need". It revolutionized the field of natural language processing through parallel computation and global dependency modeling capabilities. The original Transformer model consists of stacked encoders and decoders (e.g., in the original model, the encoder and decoder each contain 6 layers).

[0051] 3) Sentence Vectorization: This is the process of converting sentences or phrases in natural language into fixed-dimensional numerical vectors, enabling electronic devices to understand the semantic, syntactic, and contextual information of sentences through mathematical operations. This technology is a core foundation of natural language processing, supporting tasks such as machine translation, sentiment analysis, and text classification.

[0052] 4) Document vectorization refers to the process of converting an entire document into a numerical vector representation, enabling textual information to be processed by machine learning models. Sentence vectorization represents only the semantics of a single sentence, while document vectorization represents the entire document. Because documents contain more sentences, more complex paragraph structures, longer contextual text, and more topics, representation is more difficult. Its core is to generate high-quality long document representations, thereby improving the results of document analysis, recognition, and classification. This method is particularly suitable for scenarios that require processing long texts, such as academic papers, reports, and news articles.

[0053] 5) BERT (Bidirectional Encoder Representations from Transformers) network: This is a pre-trained language model composed of multiple encoders, proposed in 2018, and is a milestone in the field of natural language processing. Its core feature is that it learns deep semantic representations of words through bidirectional context training, significantly improving the performance of various NLP tasks.

[0054] 6) GPT (Generative Pre-trained Transformer) network: This is a pre-trained natural language processing model that uses only the decoder part and consists of multiple stacked layers. It was first proposed in 2018. Its core feature is that it learns language rules through pre-training on large-scale unlabeled text, and then fine-tunes it to adapt to specific tasks, achieving various NLP functions such as text generation, translation, and question answering.

[0055] 7) [CLS] tag: This is a reserved character in pre-trained language models such as BERT, used to identify the beginning of a sentence.

[0056] 8) [EOS] tag: This is a reserved character in pre-trained language models such as GPT, used to identify the end of a sentence.

[0057] Document vectorization is a type of text vectorization. The development of text vectorization technology has evolved from static word vectors to context-aware dynamic representations, and then to sentence-level semantic modeling. Four methods represent significant milestones in this process: One-hot encoding, Word2Vec, BERT-based methods, and GPT-based methods.

[0058] (1) One-hot encoding: This is a technique for converting discrete categorical variables into binary vectors, and it is one of the earliest naive algorithms for representing words as vectors. Its core idea is to map each category to a unique binary vector, where only one element is "1" (activation state), and the rest are "0", thus eliminating spurious order relationships between categories and facilitating algorithm processing. Suppose there are three colors; the three colors after one-hot encoding are: red → [1, 0, 0], green → [0, 1, 0], and blue → [0, 0, 1]. However, this vectorization method suffers from dimensionality explosion. For example, if there are 20,000 words, the one-hot encoded vectors will also have 20,000 elements, resulting in extremely high dimensionality and sparsity, meaning the encoded vectors cannot fully represent the meaning of each word.

[0059] (2) Word2Vec: A word vector model that learns distributed representations of words through shallow neural networks, making semantically similar words close in distance in the vector space. Dimensionality reduction and dense vectors: Representing words as 100-300 dimensional dense vectors solves the high-dimensional sparsity problem of One-Hot encoding. Word vectors represented by Word2Vec support vector operations, such as "king - man + woman ≈ queen". However, a drawback is that the vector of the same word is fixed in different contexts. It cannot handle polysemous words when synonyms have different meanings in the same context; for example, "apple" can refer to a fruit or a brand name.

[0060] (3) BERT-based methods: These are methods based on the Transformer Encoder network. With the emergence of BERT, deep learning entered the era of pre-trained language models. Among them, the representative model is Sentence-BERT, which is a sentence-level semantic modeling model. The Sentence-BERT model uses BERT as the base model. Through the Siamese Network, a dual-tower BERT structure with shared parameters, it independently encodes two sentences to generate sentence vectors, and then calculates the cosine similarity to solve the sentence semantic similarity problem.

[0061] The Sentence-BERT model is a classic vectorization model based on the BERT model. It uses the BERT network as a neural network structure to generate sentence vectors.

[0062] Figure 1 This is a schematic diagram of a Sentence-BERT model in related technologies, such as... Figure 1As shown, the sentence is "I love you China". The character [CLS] is added to the beginning of the sentence and the character [SEP] is added to the end of the sentence. The sentence with the added characters "[CLS]I love you China[SEP]" is input into BERT. The vector at the first position of the last layer vector sequence of BERT, which is the vector at the [CLS] position, is taken and then the embedding vector corresponding to the sentence is obtained through a fully connected network.

[0063] However, the BERT network is a mask model, but its mask matrix is ​​a low-rank matrix, which results in weak expressive power of the embedding vector corresponding to the sentence.

[0064] (4) GPT-based method: The method is based on the transformer decoder structure, and its base model is GPT. The GPT network can be used as a neural network structure to generate sentence vectors.

[0065] Figure 2 This is a schematic diagram of a GPT network in related technologies, such as... Figure 2 As shown, the sentence is "I love you China". The character "[EOS]" is added to the end of the sentence. The sentence with the added character "I love you China [EOS]" is input into BERT. After learning using the GPT network, the vector at the last position of the last layer vector sequence of GPT, which is the vector at the "[EOS]" position, is taken and then the embedding vector corresponding to the sentence is generated through a fully connected network.

[0066] However, the context in the GPT network is a unidirectional structure that can only be generated from left to right, resulting in less contextual information. The disadvantage is that this unidirectional attention cannot fully capture the dependencies of distant words in long texts, leading to weak expressive power of the embedding vectors corresponding to sentences.

[0067] This application provides a sentence-document vectorization transformer network training method, apparatus, device, medium, and product, which can solve the problem of weak embedding vector representation ability of sentences. The sentence-document vectorization transformer network training method provided in this application is described below.

[0068] Figure 3 This is a flowchart illustrating a sentence-document vectorization transformer network training method according to an exemplary embodiment, such as... Figure 3 As shown, it includes:

[0069] Step 301: Obtain sentence-document pairs; where a sentence-document pair includes a sentence and a document.

[0070] In this disclosure, the electronic devices include, but are not limited to, laptops, tablets, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), vehicle terminals, servers, etc.

[0071] A sentence can be a complete sentence at the purely grammatical level, or it can be a semantically independent sentence. For example, a sentence can be a phrase entered by the user, such as "I love you, China". The language of the sentence can be any natural language, such as Chinese, English, French, etc.

[0072] A document can refer to a piece of text, a written work, etc. For example, a document can be an online article, a reply in a comment section, an encyclopedia entry, etc. A document can also be an essay, poem, novel, script, etc. The language of a document can be any natural language, for example, Chinese, English, French, etc.

[0073] The training set may include at least one sentence-document pair. The sentence and document in the sentence-document pair can be positive or negative examples of each other. For example, if the sentence is "I like swimming", and the document is used to make users like swimming, the document includes an introduction to the history, techniques, advantages, etc. of swimming, then the sentence and document are positive examples of each other. If the document is used to make users dislike swimming, the document includes the disadvantages of swimming, then the sentence and document are negative examples of each other.

[0074] The training set typically includes multiple sentence-document pairs, among which there are sentence-document pairs where the sentence and document are positive examples of each other, and sentence-document pairs where the sentence and document are negative examples of each other. This enriches the data in the training set and improves the model's generalization ability, accuracy, and robustness.

[0075] Step 302: Input the sentence into the transformer network to obtain the first sentence embedding vector; and input the document into the transformer network to obtain at least one first document embedding vector; wherein the transformer network includes an encoder and a decoder.

[0076] In this embodiment of the disclosure, the embedding vector is a semantically meaningful numerical vector obtained by converting discrete, non-numerical sentences or documents, which is used to solve the problem that discrete symbols cannot be computed by the model.

[0077] Transformer is a neural network architecture based on a self-attention mechanism. The core of the self-attention mechanism is to capture semantic relationships within and between sequences, thereby enabling the analysis of natural language. The core structure of the transformer network consists of an encoder and a decoder, which are connected. The encoder outputs a sequence of context vectors containing all the semantic information of the input, and the decoder processes the sequence of context vectors containing all the semantic information of the input.

[0078] After the sentence is input into the transformer network, the encoder and decoder work together to obtain the first sentence embedding vector. Compared with related technologies that only generate the final embedding vector of the sentence through the embedding layer or encoder, this disclosure uses the encoder and decoder to obtain the final embedding vector of the sentence, so that the first sentence embedding vector has a richer semantic representation.

[0079] After the document is input into the transformer network, the encoder and decoder work together to obtain the first document embedding vector. Compared with related technologies that only generate the final embedding vector of the document through the embedding layer or encoder, this disclosure uses the encoder and decoder to obtain the final embedding vector of the document, so that the first document embedding vector has a richer semantic representation.

[0080] Step 303: Generate the first similarity parameter based on the first sentence embedding vector and the first document embedding vector.

[0081] In this embodiment of the disclosure, the first similarity parameter can be used to indicate the degree of association between a document and a sentence. The electronic device can calculate the first similarity parameter between the sentence and the document based on a similarity calculation function. The calculation method for the first similarity parameter can be varied; for example, the similarity calculation function includes matrix calculation, etc. This disclosure does not limit the function used to calculate the similarity.

[0082] Step 304: Generate a loss value based on the first similarity parameter, and update the transformer network based on the loss value.

[0083] In this embodiment of the disclosure, a loss value can be generated based on the loss function and the first similarity parameter. The loss function is a core metric used during training to measure the difference between the model's predictions and the actual results. Its role is to provide direction for optimizing model parameters. Generally, the smaller the loss value, the more accurate the prediction and the more convergent the model.

[0084] For example, loss functions may include the Info Noise-Contrastive Estimation (InfoNCE) loss function, the cross-entropy loss function, etc.

[0085] In this disclosure, steps 301 to 304 can be executed cyclically. When executing step 301, sentence-document pairs can be obtained sequentially according to the order of sentence-document pairs in the training set, so as to train the transformer network based on at least one sentence-document pair until the transformer network converges, then the loop ends and the trained transformer network is obtained.

[0086] In this disclosure, the transformer network includes an encoder and a decoder, which can jointly utilize the encoder and decoder to obtain a first sentence embedding vector and a first document embedding vector. This allows both the first sentence embedding vector and the first document embedding vector to have richer contextual information, thereby combining the advantages of the encoder and decoder. By utilizing more neurons to generate embedding vectors that represent the semantics of sentences and documents, the transformer network has a stronger fitting ability, enabling the embedding vectors to more fully understand the meaning expressed by sentences and documents, and also improving the training effect of the transformer network.

[0087] In some embodiments, the encoder includes a first encoder, and the decoder includes a first decoder; inputting the sentence into the transformer network to obtain a first sentence embedding vector includes steps 3021 to 3024, wherein:

[0088] Step 3021: Insert a multi-perspective tag at the beginning of the sentence and use the sentence with the inserted multi-perspective tag as the first inserted sentence.

[0089] In this embodiment of the disclosure, step 3021 represents preprocessing the sentence, and the preprocessing includes inserting multi-view tags at the beginning of the sentence. In addition, the preprocessing may also include basic text cleaning, keyword extraction, etc. Basic text cleaning includes at least one of the following: removing irrelevant noise, word segmentation, sentence segmentation, etc. For example, the sentence is "swimming* in the pool", which includes the meaningless character "*". When performing basic text cleaning on the sentence, the meaningless character "*" will be removed.

[0090] The beginning position of a sentence can refer to the position before the first character of the sentence. The multi-view tag is not [CLS], but can be [MVL] or [ABC], etc. The multi-view tag is used to identify the beginning position of the sentence. The multi-view tag is different from the classification tag [CLS] used as the beginning of the sentence in related technologies.

[0091] For example, the sentence "I love you, China" can begin before "I," with the multi-view tag "[MVL]". The sentence after inserting the multi-view tag would be "[MVL]I love you, China". Of course, besides inserting the multi-view tag at the beginning of the sentence, a sentence-end tag can also be inserted at the end. For example, with the sentence-end tag "[SEP]", the sentence after inserting both the multi-view tag and the sentence-end tag would be "[MVL]I love you, China[SEP]". Segmenting the first inserted sentence yields multiple characters: "[MVL]", "I", "love", "you", "China", and "[SEP]", with the first character being "[MVL".

[0092] When inserting tags into a sentence, multi-view tags [MVL] need to be inserted. Separator characters [SEP] and padding characters [PAD] can also be inserted as needed. Through the self-attention mechanism of the transformer, [MVL] can aggregate the context information of the entire first inserted sentence. Using [MVL] as the sentence beginning, compared with using [CLS] as the sentence beginning in related technologies, can avoid misidentification by the program.

[0093] Step 3022: Input the first inserted sentence into the first encoder to obtain the second sentence embedding vector; wherein, the second sentence embedding vector includes the first tag embedding vector corresponding to the multi-view tag.

[0094] In this embodiment of the disclosure, the first encoder can process the first inserted sentence and output a second sentence embedding vector. The second sentence embedding vector includes the context vector sequence of the first inserted sentence. For example, the second sentence embedding vector includes the first tag embedding vector corresponding to "[MVL]", and also includes embedding vectors corresponding to other characters. The number of layers of the first encoder and the first decoder can be the same or different. For example, the number of layers of the first encoder can be greater than the number of layers of the first decoder.

[0095] The second sentence embedding vector is the embedding vector of the sentence obtained through the encoder. The first encoder supports bidirectional contextual modeling, which can capture the contextualized semantic information in the sentence, solve the problem of polysemy, and more accurately reflect the true semantics of the sentence, rather than a simple concatenation of isolated words. Furthermore, through the self-attention mechanism of the transformer, it can capture the attention weight between any two characters, so that the second sentence embedding vector can more comprehensively reflect the semantic structure of the sentence.

[0096] Step 3023: Input the second sentence embedding vector and the obtained first sentence input content into the first decoder to obtain the third sentence embedding vector corresponding to the last character of the first sentence; wherein, the last character of the first sentence is the last character included in the first sentence input content.

[0097] In this embodiment of the disclosure, the first sentence input content is related to the first inserted sentence. For example, the first sentence input content can be the first inserted sentence, or it can be a multi-view tag of the first inserted sentence. Alternatively, the first sentence input content can be an instruction used to indicate the first inserted sentence or the multi-view tag. For example, the first sentence input content can be a first instruction used to indicate the first inserted sentence; or the first sentence input content can be a second instruction used to indicate the multi-view tag, thereby using the first sentence input content to determine the content input to the first decoder.

[0098] The first decoder is connected to the first encoder. The first decoder can receive the second sentence embedding vector sent by the first encoder. The first decoder also includes a port for receiving the first sentence input content, which can be content input by the user or content automatically selected by the electronic device.

[0099] To enhance the semantics of the second sentence embedding vector, the second sentence embedding vector and the first sentence input are fed into the first decoder. The mask self-attention of the first decoder is unidirectional and can generate a sequence step by step in an autoregressive manner. Thus, the first decoder can be used to enhance the expressive power of the embedding vector corresponding to the sentence, and obtain the third sentence embedding vector.

[0100] Step 3024: Generate the first sentence embedding vector based solely on the third sentence embedding vector; or, generate the first sentence embedding vector based on the third sentence embedding vector and the first label embedding vector; or, generate the first sentence embedding vector based on the third sentence embedding vector and the second sentence embedding vector.

[0101] In this embodiment of the disclosure, the first sentence embedding vector may be obtained based solely on the output of the first decoder; or, the first sentence embedding vector may be obtained based on the outputs of the first decoder and the first encoder.

[0102] Figure 4 This is a schematic diagram illustrating a first encoder and a first decoder according to an exemplary embodiment, as shown below. Figure 4 As shown, a vector can represent the embedding vector of characters in the first inserted sentence, excluding [MVL]. The first inserted sentence is "[MVL]I love you China[SEP]", and the first instruction is "Generate the vector of the sentence:", where [EOS] represents the last character [SEP] in the first inserted sentence; when the input content of the first sentence is a multi-view tag, [EOS] represents the multi-view tag [MVL].

[0103] The first encoder has M1 layers, and the first decoder has N1 layers. M1 is an integer greater than or equal to 1, and N1 is an integer greater than or equal to 1, with M1 greater than N1. That is, the number of layers in the first encoder can be greater than the number of layers in the first decoder. By using the first encoder with more layers and the first decoder with fewer layers, the first sentence embedding vector of the sentence is obtained.

[0104] As a preferred embodiment, the number of layers in the first encoder can be 6, 8, 12, etc., and the number of layers in the first decoder is 1. The number of layers in the first encoder and the number of layers in the first decoder are not equal. This disclosure provides an asymmetric transformer network. By inputting a sentence into the asymmetric transformer network, a first sentence embedding vector with more semantic representation can be obtained.

[0105] Furthermore, the symmetric transformer network in the related technology includes encoders and decoders with the same number of layers, for example, the encoder and decoder both have 6 layers. Therefore, the asymmetric transformer network in this disclosure has a first decoder with fewer layers compared to the symmetric transformer network in the related technology, which reduces the storage space and network parameters required by the first decoder.

[0106] It should be noted that the transformer network in this disclosure, unless otherwise specified or defined, refers to the transformer network in step 302 above, and the same applies to encoders and decoders.

[0107] like Figure 4 As shown, the first encoder can transmit the second sentence embedding vector to the first decoder. The first decoder can have two input ports, one of which is used to receive the second sentence embedding vector and the other is used to receive the first sentence input content.

[0108] Figure 4 The dashed lines in the diagram can represent that when generating the first sentence embedding vector, the first sentence embedding vector can be generated based solely on the third sentence embedding vector output by the first decoder; or, the first sentence embedding vector can be generated based on the third sentence embedding vector output by the first decoder and the first tag embedding vector output by the first encoder; or, the first sentence embedding vector can be generated based on the third sentence embedding vector output by the first decoder and the second sentence embedding vector output by the first encoder.

[0109] In some embodiments, generating a first sentence embedding vector based solely on a third sentence embedding vector includes: step 4011, inputting the third sentence embedding vector into a feedforward neural network to obtain a first sentence embedding vector.

[0110] In this embodiment of the disclosure, the first encoder is connected to a feedforward neural network, or the first encoder includes a feedforward neural network. The feedforward neural network (FNN) includes an input layer, a hidden layer, and an output layer. The input layer is used to receive a third sentence embedding vector and send the third sentence embedding vector to the hidden layer.

[0111] Hidden layers, located between the input and output layers, are the core computational layers of a FNN; there can be one or more hidden layers. The number of hidden layers and neurons are key hyperparameters of the transformer network and need to be adjusted according to the task.

[0112] Each hidden layer contains neurons. These neurons perform a weighted sum of the outputs from the previous layer, and then apply a non-linear transformation using the hidden layer's activation function to extract complex features from the input vector. The output layer, based on the outputs of the hidden layers and the first output, obtains the first sentence embedding vector.

[0113] like Figure 4 As shown, Figure 4 The dashed lines in the diagram represent the absence of a connection between the first encoder and the embedding vector. This means that when generating the final embedding vector of a sentence, the first sentence embedding vector can be obtained using only the output of the first decoder and not the output of the first encoder. This eliminates the need to compute multiple vectors, reducing processing steps. The third sentence embedding vector can be directly input into the FNN, accelerating the generation rate of the first sentence embedding vector.

[0114] In some embodiments, a first sentence embedding vector is generated based on a third sentence embedding vector and a first tag embedding vector, including steps 4021 to 4022, wherein:

[0115] Step 4021: Generate the first average embedding vector based on the third sentence embedding vector and the first label embedding vector.

[0116] In this embodiment of the disclosure, before inputting the third sentence embedding vector and the first label embedding vector into the FNN, the two embedding vectors are first processed to obtain a first average embedding vector. The processing of the two embedding vectors includes averaging, weighted averaging, etc.

[0117] Step 4022: Input the first average embedding vector into the feedforward neural network to obtain the first sentence embedding vector.

[0118] In this embodiment of the disclosure, such as Figure 4 As shown, Figure 4The dashed lines in the diagram can represent the existence of a connection between the first encoder and the embedding vector. That is, before inputting the third sentence embedding vector and the first label embedding vector into the FNN, the average of the two embedding vectors is calculated to obtain the first average embedding vector. Thus, when obtaining the first sentence embedding vector, the first label embedding vector of [MVL] output by the first encoder and the third sentence embedding vector output by the decoder are needed to make the output first sentence embedding vector have a richer semantic representation.

[0119] In some embodiments, step 4021 includes: step 40211, calculating the average value of the third sentence embedding vector and the first tag embedding vector to generate a first average embedding vector.

[0120] In this embodiment of the disclosure, such as Figure 4 As shown, the last layer of the first encoder outputs the first tag embedding vector, and the last layer of the first decoder outputs the third sentence embedding vector. Both vectors have the same dimension; for example, the third sentence embedding vector may have a dimension of 1*512, 1*768, or 1*2048. The average of the third sentence embedding vector and the first tag embedding vector is taken, and this average is used as the first average embedding vector.

[0121] If the first inserted sentence is “[MVL]I love you China[SEP]”, the first label embedding vector is the embedding vector corresponding to [MVL] output from the last layer of the first encoder. The average value between the first label embedding vector corresponding to [MVL] and the third sentence embedding vector is calculated, and this average value is used as the first average embedding vector.

[0122] In some embodiments, a first sentence embedding vector is generated based on a third sentence embedding vector and a second sentence embedding vector, including steps 4031 and 4032, wherein:

[0123] Step 4031: Generate a second average embedding vector based on the third sentence embedding vector and the second sentence embedding vector;

[0124] Step 4032: Input the second average embedding vector into the feedforward neural network to obtain the first sentence embedding vector.

[0125] In this embodiment of the disclosure, the second sentence embedding vector may include embedding vectors corresponding to multiple characters respectively, referring to steps 4011, 4021 to 4022 above, and in combination with... Figure 4 , Figure 4The dashed lines in the diagram can represent the existence of a connection between the first encoder and the embedding vector. That is, before inputting the third sentence embedding vector and the second sentence embedding vector into the FNN, the average of multiple embedding vectors is calculated to obtain the second average embedding vector. Thus, when obtaining the first sentence embedding vector, the second sentence embedding vector output by the first encoder and the third sentence embedding vector output by the decoder are needed to make the output first sentence embedding vector have a richer semantic representation.

[0126] In some embodiments, step 4031 includes: step 40311, calculating the average value of the third sentence embedding vector and the second sentence embedding vector to generate a second average embedding vector.

[0127] In this embodiment of the disclosure, such as Figure 4 As shown, the last layer of the first encoder outputs the second sentence embedding vector, and the last layer of the first decoder outputs the third sentence embedding vector. For example, the first inserted sentence is "[MVL]I love you China[SEP]". The second sentence embedding vector includes the sub-embedding vectors corresponding to the six characters "[MVL]", "I", "love", "you", "China" and "[SEP]". The average value of the sub-embedding vectors corresponding to the six characters and the third sentence embedding vector is calculated, and this average value is used as the second average embedding vector.

[0128] The electronic device may execute steps 4011, 4021 to 4022, or 4031 to 4032 randomly, or, after the user has set the steps to be executed, the electronic device may also execute the steps according to the user's settings.

[0129] In some embodiments, the first sentence input is the first inserted sentence; or, the first sentence input is a multi-view tag.

[0130] In this embodiment of the disclosure, the first sentence input content can be a first inserted sentence that is directly input, or it can be a first input instruction that can indicate the first inserted sentence.

[0131] For example, if the first inserted sentence is “[MVL]I love you China[SEP]”, the third sentence embedding vector output by the last layer of the first decoder is the embedding vector corresponding to [SEP]. Thus, by inputting the embedding vectors of each character output by the first encoder into the first decoder, and by inputting the first inserted sentence into the first decoder, the last layer of the first decoder can output the embedding vector corresponding to [SEP].

[0132] The first sentence input can be a multi-view tag that is entered directly, or it can be a second command that can instruct the multi-view tag.

[0133] For example, if the first inserted sentence is “[MVL]I love you China[SEP]”, the third sentence embedding vector output by the last layer of the first decoder is the embedding vector corresponding to [MVL]. Thus, by inputting the embedding vectors of each character output by the first encoder into the first decoder, and by inputting [MVL] into the first decoder, the last layer of the first decoder can output the embedding vector corresponding to [MVL].

[0134] In some embodiments, the encoder includes a second encoder, and the decoder includes a second decoder; inputting the document into the transformer network to obtain at least one first document embedding vector includes steps 3025 to 3028, wherein:

[0135] Step 3025: Insert a multi-view tag at the beginning of the document and use the document with the inserted multi-view tag as the first inserted document; wherein, the number of multi-view tags is a preset number.

[0136] In this embodiment of the disclosure, step 3025 indicates that the document is preprocessed, and the preprocessing includes inserting a preset number of multi-view tags at the beginning position of the document. The preset number can be K, where K is an integer greater than or equal to 1. For example, if K is 9, then 9 [MVL] characters are inserted at the beginning position of the document. The number K can be an arbitrarily set value, or it can be a value determined through experimentation, etc.

[0137] In addition, preprocessing may include basic text cleaning, keyword extraction, etc., which will not be elaborated here.

[0138] The starting position of a document can refer to the position before the first character of the document. Multi-view tags can be [MVL], etc., to identify the starting position of a document. Multi-view tags are different from the classification tags [CLS] used as the beginning of sentences in related technologies.

[0139] Of course, when inserting tags into a document, K multi-view tags [MVL] need to be inserted. Separator characters [SEP], padding characters [PAD], etc. can also be inserted as needed. By utilizing the self-attention mechanism of the transformer, [MVL] can aggregate the context information of the entire first inserted document. Moreover, using [MVL] as the beginning of a sentence, compared to using [CLS] as the beginning of a sentence in related technologies, can avoid misidentification by the program.

[0140] Step 3026: Input the first inserted document into the second encoder to obtain the second document embedding vector; wherein, the second document embedding vector includes the second tag embedding vector corresponding to the multi-view tags.

[0141] In this embodiment of the disclosure, a first encoder and a first decoder are used to process sentences, and a second encoder and a second decoder are used to process documents. The number of layers in the first encoder and the second encoder may be the same or different, and the number of layers in the first decoder and the second decoder may be the same or different. For example, the number of layers in the second encoder may be greater than the number of layers in the second decoder.

[0142] The first encoder can process the first inserted document and output a second document embedding vector. The second document embedding vector includes the context vector sequence of the first inserted document. For example, the second document embedding vector includes the second tag embedding vector corresponding to "[MVL]", and also includes the embedding vectors corresponding to other characters in the first inserted document.

[0143] The second document embedding vector is the embedding vector of the document obtained through the encoder. The second encoder supports bidirectional contextual modeling, which can capture the contextualized semantic information in the document, solve the problem of polysemy, and more accurately reflect the true semantics of the document, rather than a simple concatenation of isolated words. Furthermore, through the self-attention mechanism of the transformer, it can capture the attention weight between any two characters, so that the second document embedding vector can more comprehensively reflect the semantic structure of the document.

[0144] Step 3027: Input the second document embedding vector and the obtained first document input content into the second decoder to obtain the third document embedding vector.

[0145] In this embodiment of the disclosure, the first document input content is related to the first inserted document. For example, the first document input content can be the first inserted document, or it can be K multi-view tags of the first inserted document. Alternatively, the first document input content can be an instruction used to indicate the first inserted document or the K multi-view tags. For example, the first document input content can be a third instruction used to indicate the first inserted document; or the first document input content can be a fourth instruction used to indicate the K multi-view tags, thereby determining the content input to the second decoder through the first document input content.

[0146] The second decoder is connected to the second encoder. The second decoder can receive the second document embedding vector sent by the second encoder. The second decoder also includes a port for receiving the first document input content, which can be content entered by the user or content automatically selected by the electronic device.

[0147] To increase the semantics in the second document embedding vector of the document, the second document embedding vector and the first document input content are input into the second decoder. The masked self-attention of the second decoder is unidirectional and can generate a sequence step by step in an autoregressive manner, so that the second decoder can be used to enhance the expressive ability of the embedding vector corresponding to the document, and a third document embedding vector is obtained.

[0148] Step 3028, generate at least one first document embedding vector only based on the third document embedding vector; or, generate multiple first document embedding vectors based on the third document embedding vector and the second label embedding vector; or, generate multiple first document embedding vectors based on the third document embedding vector and the second document embedding vector.

[0149] In the embodiments of the present disclosure, the first document embedding vector can be obtained only based on the output of the second decoder; or, the first document embedding vector can be obtained based on the outputs of the second decoder and the second encoder.

[0150] Figure 5 is a schematic diagram of a second encoder and a second decoder shown according to an exemplary embodiment, as Figure 5 shown, the vector can represent the embedding vector of the characters in the first inserted document except for K [MVL]s and [SEP]. The first inserted document is "[MVL]1[MVL]2...[MVL]K China is...[SEP]", and the third instruction is "generate the vector of the document:".

[0151] The number of layers of the second encoder is M2, and the number of layers of the second decoder is N2. M2 is an integer greater than or equal to 1, N2 is an integer greater than or equal to 1, and M2 is greater than N2. That is, the number of layers of the second encoder can be greater than the number of layers of the second decoder. By using the second encoder with more layers and the second decoder with fewer layers, the first document embedding vector of the document is obtained.

[0152] As a preferred embodiment, the number of layers of the second encoder can also be 6, 8, 12, etc., and the number of layers of the second decoder is 1. The number of layers of the second encoder and the second decoder are not equal. In the present disclosure, an asymmetric transformer network is provided, and the document is input into the asymmetric transformer network, so that the first document embedding vector with more semantic representations can be obtained.

[0153] And the symmetric transformer network in the related art includes an encoder and a decoder with the same number of layers. Therefore, the asymmetric transformer network in the present disclosure has a second decoder with fewer layers compared to the symmetric transformer network in the related art, reducing the storage space and network parameters required by the second decoder.

[0154] As Figure 5 shown, the second Encoder may transmit a second document embedding vector to the second Decoder. The second Decoder may have two input ports, one for receiving the second document embedding vector, and the other for receiving the first document input content.

[0155] Figure 5 The dashed line in

[0155] may indicate that when generating the first document embedding vector, the first document embedding vector may be generated only based on the third document embedding vector output by the second Decoder; or, multiple first document embedding vectors may be generated based on the third document embedding vector output by the second Decoder and the second tag embedding vector output by the second Encoder; or, the first document embedding vector may be generated based on the third document embedding vector output by the second Decoder and the second document embedding vector output by the second Encoder.

[0156] In some embodiments, the first document input content includes a first inserted document or a preset number of multi-view tags;

[0157] When the first document input content is the first inserted document, the third document embedding vector is the first last character embedding vector corresponding to the last character of the first document; wherein, the first document last character is the last character included in the first inserted document;

[0158] When the first document input content is a preset number of multi-view tags, the third document embedding vector includes third tag embedding vectors corresponding to the preset number of multi-view tags respectively.

[0159] In the embodiments of the present disclosure, as Figure 5 shown, the first inserted document is "[MVL]1[MVL]2...[MVL]K China is...[SEP]", that is, K [MVL]s and 1 [SEP] are inserted into the document. The K [MVL]s are used to identify the start of the document, and the [SEP] is used to identify the end of the document. The first document last character is [SEP]. After the second Decoder receives the first inserted document input, the first last character embedding vector is the embedding vector of [SEP] output by the last layer of the second Decoder.

[0160] Or, after the second Decoder receives the K [MVL]s, the third tag embedding vector is the embedding vector of [MVL] output by the last layer of the second Decoder, and the last layer of the second Decoder will output third tag embedding vectors corresponding to the K [MVL]s respectively.

[0161] In some embodiments, generating at least one first document embedding vector only based on the third document embedding vector includes steps 5011 to step 5012, wherein:

[0162] Step 5011, in response to the first document input content being the first inserted document, use the first last character embedding vector as the first document embedding vector; or,

[0163] Step 5012: In response to the first document input content being a preset number of multi-view tags, each third tag embedding vector is used as a first document embedding vector; wherein, the number of first document embedding vectors is a preset number.

[0164] In this embodiment of the disclosure, when the first document input content is the first inserted document, the last layer of the second decoder will output the first last character embedding vector corresponding to [SEP]. Since the document contains a lot of information and characters, the first last character embedding vector contains rich contextual representations. Therefore, it is not necessary to input the first last character embedding vector into the FNN; the first last character embedding vector can be directly used as the first document embedding vector.

[0165] When the first document input contains a preset number of multi-view tags, the last layer of the second decoder will output the third tag embedding vector corresponding to each [MVL]; then the third tag embedding vector corresponding to each [MVL] will be directly used as the first document embedding vector, that is, K first document embedding vectors will be generated.

[0166] like Figure 5 As shown, Figure 5 The dashed lines in the diagram represent the absence of a connection between the second encoder and the embedding vector. This means that when generating the final embedding vector of the document, the first document embedding vector can be obtained using only the output of the second decoder and not the output of the second encoder. This eliminates the need to concatenate the embedding vectors output from the last layer of the second decoder, reducing the processing of the embedding vectors and accelerating the generation of the first document embedding vector.

[0167] In some embodiments, multiple first document embedding vectors are generated based on the third document embedding vector and the second tag embedding vector, including steps 5021 to 5022, wherein:

[0168] Step 5021: In response to the first document input content being the first inserted document, concatenate a preset number of second tag embedding vectors and the first last character embedding vector to generate multiple first document embedding vectors; or,

[0169] Step 5022: In response to the first document input content being a preset number of multi-view tags, a preset number of second tag embedding vectors and a preset number of third tag embedding vectors are concatenated to generate multiple first document embedding vectors.

[0170] In this embodiment of the disclosure, when the first document input content is the first inserted document, the last layer of the second decoder will output the first last character embedding vector corresponding to [SEP], and the last layer of the second encoder will output the second tag embedding vector corresponding to each [MVL]. In the first inserted document, the number of [MVL] is K, and the number of second tag embedding vectors is also K.

[0171] We need to concatenate K second-tag embedding vectors and one first-last-character embedding vector, and use the concatenated embedding vector as the first document embedding vector. When concatenating multiple embedding vectors, we can use Contact concatenation, Cartesian product concatenation, or repeated concatenation, etc.

[0172] For example, when using Contact concatenation, the K second tag embedding vectors and the 1 first last character embedding vector can be concatenated according to the matrix rows / columns to obtain K first document embedding vectors. Alternatively, when using Cartesian product concatenation, the K second tag embedding vectors and the 1 first last character embedding vector can be concatenated one by one to obtain K first document embedding vectors. Or, for another example, the K second tag embedding vectors and the 1 first last character embedding vector can be concatenated pairwise to obtain K+1 first document embedding vectors, and so on.

[0173] When the first document input contains a preset number of multi-view tags, the last layer of the second decoder will output the third tag embedding vector corresponding to each of the K [MVL] tags, and the last layer of the second encoder will output the second tag embedding vector corresponding to each of the K [MVL] tags. In the first inserted document, the number of [MVL] tags is K, and the number of the second tag embedding vector and the number of the third tag embedding vector are both K.

[0174] We need to concatenate the K second-label embedding vectors and the K third-label embedding vectors, and use the resulting concatenated embedding vector as the first document embedding vector. When concatenating the K second-label embedding vectors and the K third-label embedding vectors, we can use Contact concatenation, Cartesian product concatenation, repeated concatenation, etc., which will not be elaborated here. The number of the resulting first document embedding vectors can be greater than or equal to K.

[0175] like Figure 5 As shown, Figure 5 The dashed lines in the diagram represent the connection between the second encoder and the embedding vector. That is, when generating the first document embedding vector, the outputs of the second encoder and the second decoder need to be concatenated to obtain multiple first document embedding vectors, so that the first document embedding vector has richer contextual information.

[0176] In some embodiments, multiple first document embedding vectors are generated based on the third document embedding vector and the second document embedding vector, including steps 5031 to 5032, wherein:

[0177] Step 5031: In response to the first document input content being the first inserted document, concatenate the second document embedding vector and the first last character embedding vector to generate multiple first document embedding vectors; or,

[0178] Step 5032: In response to the first document input content being a preset number of multi-view tags, the second document embedding vector and the preset number of third tag embedding vectors are concatenated to generate multiple first document embedding vectors.

[0179] In this embodiment of the disclosure, such as Figure 5 As shown, the last layer of the second encoder outputs the second document embedding vector, and the last layer of the second decoder outputs the first last character embedding vector. For example, if the first inserted document is "[MVL]1[MVL]2……[MVL]KChina is……[SEP]", the second sentence embedding vector includes the embedding vectors corresponding to each character in the first inserted document. The second sentence embedding vectors corresponding to multiple characters and the first last character embedding vector corresponding to [SEP] are concatenated to obtain the first document embedding vector.

[0180] Or, such as Figure 5 As shown, the last layer of the second encoder outputs the second document embedding vector, and the last layer of the second decoder outputs K third tag embedding vectors. For example, if the first inserted document is "[MVL]1[MVL]2……[MVL]KChina is……[SEP]", the second sentence embedding vector includes the embedding vectors corresponding to each character in the first inserted document. The second sentence embedding vectors corresponding to multiple characters and the K third tag embedding vectors corresponding to the K [MVL] are concatenated to obtain the first document embedding vector.

[0181] The electronic device can randomly execute steps 5011, 5021, or 5031 based on the first document input content as the first inserted document, or it can randomly execute steps 5012, 5022, or 5032 based on the first document input content as a preset number of multi-view tags; or, after the user sets the steps to be executed, the electronic device can also execute the steps set by the user.

[0182] In some embodiments, step 303 includes steps 3031 to 3032, wherein:

[0183] Step 3031: Based on the first sentence embedding vector and at least one first document embedding vector, generate a second similarity parameter corresponding to each first document embedding vector.

[0184] In this embodiment of the disclosure, Figure 6 This is a schematic diagram of a transformer network according to an exemplary embodiment, such as... Figure 6 As shown, the transformer includes a neural network, which includes an encoder and a decoder. The first neural network includes a first encoder and a first decoder, and the second neural network includes a second encoder and a second decoder. Inputting a sentence into the first neural network yields one first sentence embedding vector, and inputting a document into the second neural network yields at least one first document embedding vector. Sim represents similarity calculation or a second similarity parameter. Assuming the number of first document embedding vectors is Q, where Q is an integer greater than or equal to 1, then the number of second similarity parameters is also Q. Calculating the Q second similarity parameters yields one first similarity parameter, which in turn generates a loss value.

[0185] For example, if Q is K, meaning the number of first document embedding vectors is K, then for each first document embedding vector, a second similarity parameter is generated, resulting in K second similarity parameters.

[0186] Step 3032: Generate a first similarity parameter based on at least one second similarity parameter.

[0187] In this embodiment of the disclosure, when Q is 1, one second similarity parameter can be used as the first similarity parameter; when Q is an integer greater than 1, the first similarity parameter can be obtained by averaging or weighted averaging the Q second similarity parameters.

[0188] In some embodiments, step 304 includes steps 3041 to 3043, wherein:

[0189] Step 3041: If the document is a positive sample of a sentence, determine the first encoding parameter corresponding to the positive sample based on the first similarity parameter; or, if the document is a negative sample of a sentence, determine the second encoding parameter corresponding to the negative sample based on the first similarity parameter.

[0190] Step 3042: Determine the sample encoding parameters corresponding to the sentence based on the first sentence embedding vector of the sentence;

[0191] Step 3043: Generate a loss value based on the sample encoding parameters and either the first encoding parameter or the second encoding parameter.

[0192] In this embodiment of the disclosure, the document can be a positive or negative sample of a sentence, and the first encoding parameter can be represented as f(x). + The second encoding parameter can be expressed as f(x);- The sample encoding parameter can be expressed as f. T (x).

[0193] The loss function can be used, based on the sample encoding parameters f. T (x), and the first encoding parameter f(x) + ) or the second encoding parameter f(x) - ), generate loss value The loss function can be expressed as the following formula (1).

[0194]

[0195] As shown in formula (1) above, the loss function includes InfoNCE, where D represents the number of samples, referring to the number of sentence-document pairs. τ is a temperature parameter, a hyperparameter used to control the sharpness of the probability distribution. The smaller the value of τ, the sharper the distribution, and the stronger the transformer network's ability to distinguish between positive and negative samples; the larger the value of τ, the smoother the distribution, which can prevent the transformer network from overfitting. In the i-th sentence-document pair, one can be obtained. or That is, in the above formula (1), the i-th sentence-document pair or The loss value equals 0. The network parameters in the transformer network can be updated using the loss value, allowing gradient descent until the transformer network converges.

[0196] Of course, the above formula (1) is an example of a loss function, and this disclosure does not limit the loss function.

[0197] In some embodiments, the number of network layers in the encoder is greater than the number of network layers in the decoder.

[0198] In this embodiment of the disclosure, the transformer network includes an encoder and a decoder with an asymmetric number of network layers, that is, the transformer network is an asymmetric transformer network. Compared with the symmetric transformer network in the related art, the asymmetric transformer network in this disclosure has a decoder with fewer network layers, thereby reducing the number of decoder layers, reducing the network parameters required by the decoder, and reducing the storage space occupied by the transformer network.

[0199] The encoder may include the first encoder and the second encoder mentioned above, and the decoder may include the first decoder and the second decoder. The number of network layers of the first encoder is greater than the number of network layers of the first decoder, and the number of network layers of the second encoder is greater than the number of network layers of the second decoder.

[0200] That is, the encoders for processing documents and sentences may not be the same encoder, and the decoders for processing documents and sentences may not be the same decoder; or, the same encoder and / or the same decoder may be used to process documents and sentences, and this disclosure does not limit the number of encoders and decoders.

[0201] Figure 7 This is a flowchart illustrating a sentence-document vectorization method according to an exemplary embodiment, such as... Figure 7 As shown, the method includes:

[0202] Step 701: Obtain the target document embedding vector corresponding to the query sentence and at least one target document; wherein, the target document embedding vector is a vector obtained by inputting the target document into a transformer network, and the transformer network includes an encoder and a decoder.

[0203] In this embodiment of the disclosure, the query sentence can be a sentence entered by the user, or a sentence obtained through the user's voice input, or a sentence received from other devices (such as the cloud), etc. This disclosure does not limit the source of the query sentence.

[0204] The electronic device can process the query sentence in real time, thereby obtaining the response content of the query sentence as soon as possible and reducing the waiting time; or, the processing queue may include at least one query sentence, and the electronic device can retrieve the query sentences in the processing queue at preset intervals, thereby processing multiple query sentences sequentially.

[0205] The aforementioned transformer network refers to the trained transformer network. The database may include at least one document and a document embedding vector corresponding to each document. For example, the electronic device includes an in-vehicle terminal capable of running the trained transformer network, and also possessing voice acquisition and speech-to-text conversion capabilities, able to convert user-input speech into a sentence and use this sentence as a query sentence. Alternatively, the in-vehicle terminal may also have communication transmission capabilities, capable of sending the query sentence to the cloud, where a transformer network can run; the electronic device includes a cloud component.

[0206] All documents in the database can be used as target documents, allowing the trained transformer network to gain a more accurate understanding of the sentences and provide responses that better match the user's needs—that is, to give precise answers to the query sentences. Alternatively, electronic devices can run models, knowledge graphs, query engines, etc., from related technologies to initially screen all documents in the database based on the query sentence, obtaining documents related to the query sentence, and using these related documents as target documents, thereby reducing the number of target documents.

[0207] Step 702: Input the query sentence into the transformer network to obtain the target sentence embedding vector.

[0208] In this embodiment of the disclosure, an asymmetric transformer network can be used to understand the contextual semantics of a sentence, thereby obtaining the target sentence embedding vector of the query sentence.

[0209] Step 703: Based on the target sentence embedding vector and the target document embedding vector corresponding to at least one target document, generate a third similarity parameter for each target document; wherein, the third similarity parameter is used to indicate the degree of association between the target document and the query sentence.

[0210] In this embodiment of the disclosure, as can be seen from the above content, the similarity between the query sentence and the target document can be calculated through similarity calculation, thereby obtaining the degree of association between the query sentence and the target document. For example, the target document can be further filtered through a third similarity parameter to delete target documents with a low degree of association.

[0211] Step 704: Generate response content based on at least one target document and a third similarity parameter corresponding to each target document; wherein the response content includes a response to the query sentence.

[0212] In this embodiment of the disclosure, the response content can be the original words in the target document, or it can be the content after extracting the viewpoints of the target document.

[0213] For example, if the query is "recommend several textbooks", the electronic device can sort multiple third similarity parameters in descending order; the titles of the target documents corresponding to the top-ranked third similarity parameters can form the response content.

[0214] For example, if the query is "What kind of place is Beijing?", multiple target documents may contain information about Beijing's weather, transportation, history, etc. The system can combine the content of multiple target documents to form a response, which can include information about Beijing's weather, transportation, history, etc., providing a multi-faceted introduction to Beijing and giving users a more comprehensive understanding of the city.

[0215] The above example uses a database containing at least one document and its corresponding document embedding vector. Alternatively, a user-provided query sentence and target document can be obtained, and a trained transformer network can be used to generate a response to the query sentence. This disclosure does not limit the process of obtaining the target document.

[0216] This disclosure provides a sentence-document vectorization method that inputs a target sentence into a trained transformer network, which includes an encoder and a decoder. By utilizing the collaborative processing of the encoder and decoder, a richer contextual representation of the query sentence can be obtained, making the target sentence embedding vector more consistent with the true semantics of the query sentence. Furthermore, the target document embedding vector of the target document is also obtained through the trained transformer network, thus containing richer contextual information. Then, for each target document embedding vector and the target sentence embedding vector of the query sentence, the response content is obtained, making the response content more in line with the user's needs and improving the accuracy of information retrieval.

[0217] In some embodiments, the encoder includes a first encoder, and the decoder includes a first decoder; inputting the query sentence into the transformer network to obtain the target sentence embedding vector includes steps 7021 to 7024, wherein:

[0218] Step 7021: Insert a multi-perspective tag at the beginning of the query sentence, and use the query sentence with the inserted multi-perspective tag as the second inserted sentence;

[0219] Step 7022: Input the second inserted sentence into the first encoder to obtain the fourth sentence embedding vector; wherein, the fourth sentence embedding vector includes the fourth tag embedding vector corresponding to the multi-view tag;

[0220] Step 7023: Input the four sentence embedding vectors and the obtained second sentence input content into the first decoder to obtain the fifth sentence embedding vector corresponding to the last character of the second sentence; where the last character of the second sentence is the last character included in the second sentence input content;

[0221] Step 7024: Generate the target sentence embedding vector based solely on the fifth sentence embedding vector; or, generate the target sentence embedding vector based on both the fifth sentence embedding vector and the fourth label embedding vector; or, generate the target sentence embedding vector based on both the fifth sentence embedding vector and the fourth sentence embedding vector.

[0222] In some embodiments, generating a target sentence embedding vector based solely on the fifth sentence embedding vector includes: step 7101, inputting the fifth sentence embedding vector into a feedforward neural network to obtain the target sentence embedding vector.

[0223] In some embodiments, a target sentence embedding vector is generated based on the fifth sentence embedding vector and the fourth tag embedding vector, including steps 7111 to 7112, wherein:

[0224] Step 7111: Generate the third average embedding vector based on the fifth sentence embedding vector and the fourth label embedding vector;

[0225] Step 7112: Input the third average embedding vector into the feedforward neural network to obtain the target sentence embedding vector.

[0226] In some embodiments, a target sentence embedding vector is generated based on the fifth sentence embedding vector and the fourth sentence embedding vector, including steps 7121 to 7122, wherein:

[0227] Step 7121: Generate a fourth average embedding vector based on the fifth sentence embedding vector and the fourth sentence embedding vector;

[0228] Step 7122: Input the fourth average embedding vector into the feedforward neural network to obtain the target sentence embedding vector.

[0229] In some embodiments, step 7121 includes: step 71211, calculating the average value of the fifth sentence embedding vector and the fourth sentence embedding vector to generate a fourth average embedding vector.

[0230] In some embodiments, the second sentence input is the second inserted sentence; or, the second sentence input is a multi-view tag.

[0231] In some embodiments, the electronic device may use a trained transformer network to generate document embedding vectors for each document in the database, the encoder including a second encoder, and the decoder including a second decoder; the method further includes steps 7201 to 7204, wherein:

[0232] Step 7201: For each target document, insert a multi-view tag at the beginning of the target document, and use the target document with the inserted multi-view tag as the second insertion document; wherein, the number of multi-view tags is a preset number;

[0233] Step 7202: Input the second inserted document into the second encoder to obtain the fourth document embedding vector; wherein, the fourth document embedding vector includes the fifth tag embedding vector corresponding to the multi-view tags;

[0234] Step 7203: Input the fourth document embedding vector and the obtained second document input content into the second decoder to obtain the fifth document embedding vector;

[0235] Step 7204: Generate at least one target document embedding vector based solely on the fifth document embedding vector; or, generate multiple target document embedding vectors based on the fifth document embedding vector and the fifth tag embedding vector; or, generate multiple target document embedding vectors based on the fifth document embedding vector and the fourth document embedding vector.

[0236] In some embodiments, the second document input content includes a second inserted document or a preset number of multi-view tags;

[0237] When the second document input content is the second inserted document, the fifth document embedding vector is the second last character embedding vector corresponding to the last character of the second document; where the last character of the second document is the last character included in the second inserted document;

[0238] When the second document input contains a preset number of multi-view tags, the fifth document embedding vector includes the sixth tag embedding vector corresponding to the preset number of multi-view tags.

[0239] In some embodiments, at least one target document embedding vector is generated based solely on the fifth document embedding vector, including steps 7301 to 7302, wherein:

[0240] Step 7301, in response to the second document input content being the second inserted document, use the second last character embedding vector as the target document embedding vector; or,

[0241] Step 7302: In response to the second document input content being a preset number of multi-view tags, each sixth tag embedding vector is used as the target document embedding vector; wherein, the number of target document embedding vectors is a preset number.

[0242] In some embodiments, multiple target document embedding vectors are generated based on the fifth document embedding vector and the fifth tag embedding vector, including steps 7311 to 7312, wherein:

[0243] Step 7311: In response to the second document input content being the second inserted document, concatenate a preset number of fifth tag embedding vectors and second last character embedding vectors to generate multiple target document embedding vectors; or,

[0244] Step 7312: In response to the second document input content being a preset number of multi-view tags, a preset number of fifth tag embedding vectors and a preset number of sixth tag embedding vectors are concatenated to generate multiple target document embedding vectors.

[0245] In some embodiments, multiple target document embedding vectors are generated based on the fifth document embedding vector and the fourth document embedding vector, including steps 7321 to 7322, wherein:

[0246] Step 7321: In response to the second document input being the second inserted document, concatenate the fifth document embedding vector and the second last character embedding vector to generate multiple target document embedding vectors; or,

[0247] Step 7322: In response to the second document input content being a preset number of multi-view tags, the fourth document embedding vector and the preset number of sixth tag embedding vectors are concatenated to generate multiple target document embedding vectors.

[0248] In some embodiments, step 703 includes steps 7031 to 7032, wherein:

[0249] Step 7031: For each target document, generate a fourth similarity parameter corresponding to each target document embedding vector based on the target sentence embedding vector and at least one target document embedding vector;

[0250] Step 7032: Generate a third similarity parameter based on at least one fourth similarity parameter.

[0251] In some embodiments, the number of network layers in the encoder is greater than the number of network layers in the decoder.

[0252] In this embodiment of the disclosure, each step in the sentence-document vectorization transformer network training method has been described, and the explanation of the above steps will not be repeated here.

[0253] In this embodiment of the disclosure, Figure 8 This is a schematic diagram illustrating the structure of a sentence-document vectorization transformer network training device according to an exemplary embodiment, such as... Figure 8 As shown, the device 800 includes:

[0254] The first acquisition module 801 is used to acquire sentence-document pairs; wherein, a sentence-document pair includes a sentence and a document;

[0255] The first obtaining module 802 is used to input a sentence into a transformer network to obtain a first sentence embedding vector; and to input a document into a transformer network to obtain at least one first document embedding vector; wherein the transformer network includes an encoder and a decoder;

[0256] The first generation module 803 is used to generate a first similarity parameter based on the first sentence embedding vector and the first document embedding vector;

[0257] The loss module 804 is used to generate a loss value based on the first similarity parameter and update the transformer network based on the loss value.

[0258] In some embodiments, the encoder includes a first encoder, and the decoder includes a first decoder; the first obtaining module 802 is configured to:

[0259] Insert a multi-perspective tag at the beginning of the sentence and use the sentence with the inserted multi-perspective tag as the first sentence to insert;

[0260] The first inserted sentence is input into the first encoder to obtain the second sentence embedding vector; wherein, the second sentence embedding vector includes the first tag embedding vector corresponding to the multi-view tag;

[0261] The second sentence embedding vector and the obtained first sentence input content are input into the first decoder to obtain the third sentence embedding vector corresponding to the last character of the first sentence; where the last character of the first sentence is the last character included in the first sentence input content;

[0262] The first sentence embedding vector is generated based solely on the third sentence embedding vector; or, the first sentence embedding vector is generated based on both the third sentence embedding vector and the first label embedding vector; or, the first sentence embedding vector is generated based on both the third sentence embedding vector and the second sentence embedding vector.

[0263] In some embodiments, the first obtaining module 802 is configured to: input the third sentence embedding vector into a feedforward neural network to obtain the first sentence embedding vector.

[0264] In some embodiments, the first obtaining module 802 is configured to: generate a first average embedding vector based on the third sentence embedding vector and the first label embedding vector; and input the first average embedding vector into a feedforward neural network to obtain the first sentence embedding vector.

[0265] In some embodiments, the first obtaining module 802 is configured to: generate a second average embedding vector based on the third sentence embedding vector and the second sentence embedding vector; and input the second average embedding vector into a feedforward neural network to obtain a first sentence embedding vector.

[0266] In some embodiments, the first obtaining module 802 is used to: calculate the average value of the third sentence embedding vector and the second sentence embedding vector to generate a second average embedding vector.

[0267] In some embodiments, the first sentence input is the first inserted sentence; or, the first sentence input is a multi-view tag.

[0268] In some embodiments, the encoder includes a second encoder, and the decoder includes a second decoder; the first obtaining module 802 is configured to:

[0269] Insert a multi-view tag at the beginning of the document and use the document with the inserted multi-view tag as the first inserted document; the number of multi-view tags is a preset number;

[0270] The first inserted document is input into the second encoder to obtain the second document embedding vector; wherein, the second document embedding vector includes the second tag embedding vector corresponding to the multi-view tags;

[0271] The second document embedding vector and the obtained first document input content are input into the second decoder to obtain the third document embedding vector;

[0272] At least one first document embedding vector is generated based solely on the third document embedding vector; or, multiple first document embedding vectors are generated based on the third document embedding vector and the second tag embedding vector; or, multiple first document embedding vectors are generated based on the third document embedding vector and the second document embedding vector.

[0273] In some embodiments, the first document input content includes a first inserted document or a preset number of multi-view tags;

[0274] When the first document input content is the first inserted document, the third document embedding vector is the first last character embedding vector corresponding to the last character of the first document; where the last character of the first document is the last character included in the first inserted document;

[0275] When the first document input contains a preset number of multi-view tags, the third document embedding vector includes the third tag embedding vectors corresponding to the preset number of multi-view tags.

[0276] In some embodiments, the first receiving module 802 is configured to:

[0277] In response to the first document input being the first inserted document, the embedding vector of the first last character is used as the first document embedding vector; or...

[0278] In response to the first document input content being a preset number of multi-view tags, each third tag embedding vector is used as the first document embedding vector; wherein, the number of first document embedding vectors is a preset number.

[0279] In some embodiments, the first receiving module 802 is configured to:

[0280] In response to the first document input being the first inserted document, a preset number of second tag embedding vectors and the first last character embedding vector are concatenated to generate multiple first document embedding vectors; or...

[0281] In response to the first document input content being a preset number of multi-view tags, a preset number of second tag embedding vectors and a preset number of third tag embedding vectors are concatenated to generate multiple first document embedding vectors.

[0282] In some embodiments, the first receiving module 802 is configured to:

[0283] In response to the first document input being the first inserted document, the second document embedding vector and the first last character embedding vector are concatenated to generate multiple first document embedding vectors; or...

[0284] In response to the first document input being a preset number of multi-view tags, the second document embedding vector and a preset number of third tag embedding vectors are concatenated to generate multiple first document embedding vectors.

[0285] In some embodiments, the first generation module 803 is configured to:

[0286] Based on the first sentence embedding vector and at least one first document embedding vector, generate a second similarity parameter corresponding to each first document embedding vector;

[0287] A first similarity parameter is generated based on at least one second similarity parameter.

[0288] In some embodiments, the loss module 804 is used for:

[0289] When the document is a positive sample of a sentence, the first encoding parameter corresponding to the positive sample is determined based on the first similarity parameter; or, when the document is a negative sample of a sentence, the second encoding parameter corresponding to the negative sample is determined based on the first similarity parameter.

[0290] Based on the first sentence embedding vector of the sentence, determine the sample encoding parameters corresponding to the sentence;

[0291] The loss value is generated based on the sample encoding parameters, as well as either the first encoding parameter or the second encoding parameter.

[0292] In some embodiments, the number of network layers in the encoder is greater than the number of network layers in the decoder.

[0293] In this embodiment, the explanation of each module can be found in the explanation of each step in the above sentence-document vectorization transformer network training method, and will not be repeated here.

[0294] Figure 9 This is a schematic diagram illustrating the structure of a sentence-document vectorization device according to an exemplary embodiment, such as... Figure 9 As shown, the device 900 includes:

[0295] The second acquisition module 901 is used to acquire the query sentence and at least one target document embedding vector; wherein, the target document embedding vector is a vector obtained by inputting the target document into a transformer network, and the transformer network includes an encoder and a decoder;

[0296] The second module 902 is used to input the query sentence into the transformer network to obtain the target sentence embedding vector;

[0297] The second generation module 903 is used to generate a third similarity parameter for each target document based on the target sentence embedding vector and the target document embedding vector corresponding to at least one target document; wherein, the third similarity parameter is used to indicate the degree of association between the target document and the query sentence;

[0298] The third generation module 904 is used to generate response content based on at least one target document and a third similarity parameter corresponding to each target document; wherein the response content includes a response to the query sentence.

[0299] In some embodiments, the encoder includes a first encoder, and the decoder includes a first decoder; the second obtaining module 902 is configured to:

[0300] Insert a multi-perspective tag at the beginning of the query sentence, and use the query sentence with the inserted multi-perspective tag as the second inserted sentence;

[0301] The second inserted sentence is input into the first encoder to obtain the fourth sentence embedding vector; wherein, the fourth sentence embedding vector includes the fourth label embedding vector corresponding to the multi-view label;

[0302] The four sentence embedding vectors and the obtained second sentence input content are input into the first decoder to obtain the fifth sentence embedding vector corresponding to the last character of the second sentence; where the last character of the second sentence is the last character included in the second sentence input content;

[0303] Generate the target sentence embedding vector based solely on the fifth sentence embedding vector; or, generate the target sentence embedding vector based on both the fifth sentence embedding vector and the fourth label embedding vector; or, generate the target sentence embedding vector based on both the fifth sentence embedding vector and the fourth sentence embedding vector.

[0304] In some embodiments, the second obtaining module 902 is used to: input the fifth sentence embedding vector into a feedforward neural network to obtain the target sentence embedding vector.

[0305] In some embodiments, the second obtaining module 902 is configured to:

[0306] A third average embedding vector is generated based on the fifth sentence embedding vector and the fourth label embedding vector;

[0307] The third average embedding vector is input into the feedforward neural network to obtain the target sentence embedding vector.

[0308] In some embodiments, the second obtaining module 902 is configured to:

[0309] A fourth average embedding vector is generated based on the fifth sentence embedding vector and the fourth sentence embedding vector;

[0310] The fourth average embedding vector is input into the feedforward neural network to obtain the target sentence embedding vector.

[0311] In some embodiments, the second obtaining module 902 is configured to: calculate the average value of the fifth sentence embedding vector and the fourth sentence embedding vector to generate a fourth average embedding vector.

[0312] In some embodiments, the second sentence input is the second inserted sentence; or, the second sentence input is a multi-view tag.

[0313] In some embodiments, the encoder includes a second encoder, and the decoder includes a second decoder; the apparatus further includes a third obtaining module, for:

[0314] For each target document, insert a multi-view tag at the beginning of the target document, and use the target document with the inserted multi-view tag as the second insertion document; the number of multi-view tags is a preset number;

[0315] The second inserted document is input into the second encoder to obtain the fourth document embedding vector; wherein, the fourth document embedding vector includes the fifth tag embedding vector corresponding to the multi-view tags;

[0316] The fourth document embedding vector and the obtained second document input content are input into the second decoder to obtain the fifth document embedding vector;

[0317] Generate at least one target document embedding vector based solely on the fifth document embedding vector; or, generate multiple target document embedding vectors based on the fifth document embedding vector and the fifth tag embedding vector; or, generate multiple target document embedding vectors based on the fifth document embedding vector and the fourth document embedding vector.

[0318] In some embodiments, the second document input content includes a second inserted document or a preset number of multi-view tags;

[0319] When the second document input content is the second inserted document, the fifth document embedding vector is the second last character embedding vector corresponding to the last character of the second document; where the last character of the second document is the last character included in the second inserted document;

[0320] When the second document input contains a preset number of multi-view tags, the fifth document embedding vector includes the sixth tag embedding vector corresponding to the preset number of multi-view tags.

[0321] In some embodiments, the third obtaining module is configured to:

[0322] In response to the second document input being the second inserted document, the embedding vector of the second last character is used as the embedding vector of the target document; or...

[0323] In response to the second document input being a preset number of multi-view tags, each sixth tag embedding vector is used as the target document embedding vector; wherein, the number of target document embedding vectors is a preset number.

[0324] In some embodiments, the third obtaining module is configured to:

[0325] In response to the second document input being the second inserted document, a preset number of fifth tag embedding vectors and second last character embedding vectors are concatenated to generate multiple target document embedding vectors; or...

[0326] In response to the second document input being a preset number of multi-view tags, a preset number of fifth tag embedding vectors and a preset number of sixth tag embedding vectors are concatenated to generate multiple target document embedding vectors.

[0327] In some embodiments, the third obtaining module is configured to:

[0328] In response to the second document input being the second inserted document, the fifth document embedding vector and the second last character embedding vector are concatenated to generate multiple target document embedding vectors; or...

[0329] In response to the second document input being a preset number of multi-view tags, the fourth document embedding vector and a preset number of sixth tag embedding vectors are concatenated to generate multiple target document embedding vectors.

[0330] In some embodiments, the second generation module 903 is configured to:

[0331] For each target document, a fourth similarity parameter is generated based on the target sentence embedding vector and at least one target document embedding vector;

[0332] A third similarity parameter is generated based on at least one fourth similarity parameter.

[0333] In some embodiments, the number of network layers in the encoder is greater than the number of network layers in the decoder.

[0334] In this embodiment of the disclosure, the explanation of each module can be referred to the explanation of each step in the sentence-document vectorization method above, and will not be repeated here.

[0335] Figure 10 This is a schematic diagram illustrating the structure of an electronic device 1000 according to an exemplary embodiment. For example, the electronic device 1000 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0336] Reference Figure 10 The electronic device 1000 may include one or more of the following components: processing component 1002, memory 1004, power supply component 1006, multimedia component 1008, audio component 1010, input / output (I / O) interface 1012, sensor component 1014, and communication component 1016.

[0337] Processing component 1002 typically controls the overall operation of electronic device 1000, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 1002 may include one or more processors 1020 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1002 may include one or more modules to facilitate interaction between processing component 1002 and other components. For example, processing component 1002 may include a multimedia module to facilitate interaction between multimedia component 1008 and processing component 1002.

[0338] Memory 1004 is configured to store various types of data to support operation on electronic device 1000. Examples of such data include at least one of the following: instructions for any application or method operating on electronic device 1000, contact data, phonebook data, messages, pictures, and videos. Memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0339] Power supply component 1006 provides power to various components of electronic device 1000. Power supply component 1006 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1000.

[0340] Multimedia component 1008 includes a screen that provides an output interface between electronic device 1000 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1008 includes a front-facing camera and / or a rear-facing camera. When electronic device 1000 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0341] Audio component 1010 is configured to output and / or input audio signals. For example, audio component 1010 includes a microphone (MIC) configured to receive external audio signals when electronic device 1000 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1004 or transmitted via communication component 1016. In some embodiments, audio component 1010 also includes a speaker for outputting audio signals.

[0342] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0343] Sensor assembly 1014 includes one or more sensors for providing state assessments of various aspects of electronic device 1000. For example, sensor assembly 1014 may detect the on / off state of electronic device 1000, the relative positioning of components (e.g., the display and keypad of electronic device 1000), changes in position of electronic device 1000 or one of its components, the presence or absence of user contact with electronic device 1000, orientation or acceleration / deceleration of electronic device 1000, and temperature changes of electronic device 1000. Sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1014 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 1014 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetometer, a pressure sensor, and a temperature sensor.

[0344] Communication component 1016 is configured to facilitate wired or wireless communication between electronic device 1000 and other devices. Electronic device 1000 can access wireless networks based on communication standards, such as Wi-Fi, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 1016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1016 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), Bluetooth (BT), and other technologies.

[0345] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0346] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including executable instructions or a computer program, which can be executed by the processor 1020 of the electronic device 1000 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0347] A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is able to perform any of the sentence-document vectorization transformer network training methods or sentence-document vectorization methods described in the embodiments of this disclosure.

[0348] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the electronic device to perform any of the sentence-document vectorization transformer network training methods or sentence-document vectorization methods described above in this disclosure.

[0349] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0350] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for training a sentence-document vectorization transformer network, characterized in that, The method includes: Obtain sentence-document pairs; wherein, the sentence-document pair includes a sentence and a document; The sentence is input into a transformer network to obtain a first sentence embedding vector; and the document is input into a transformer network to obtain at least one first document embedding vector; wherein the transformer network includes an encoder and a decoder; A first similarity parameter is generated based on the first sentence embedding vector and the first document embedding vector. A loss value is generated based on the first similarity parameter, and the transformer network is updated based on the loss value.

2. The method according to claim 1, characterized in that, The encoder includes a first encoder, and the decoder includes a first decoder; the step of inputting the sentence into the transformer network to obtain the first sentence embedding vector includes: Insert a multi-view tag at the beginning of the sentence, and use the sentence with the inserted multi-view tag as the first inserted sentence; The first inserted sentence is input into the first encoder to obtain the second sentence embedding vector; wherein, the second sentence embedding vector includes the first tag embedding vector corresponding to the multi-view tag; The second sentence embedding vector and the obtained first sentence input content are input into the first decoder to obtain the third sentence embedding vector corresponding to the last character of the first sentence; wherein, the last character of the first sentence is the last character included in the first sentence input content; The first sentence embedding vector is generated based solely on the third sentence embedding vector; or, the first sentence embedding vector is generated based on the third sentence embedding vector and the first tag embedding vector; or, the first sentence embedding vector is generated based on the third sentence embedding vector and the second sentence embedding vector.

3. The method according to claim 2, characterized in that, The step of generating the first sentence embedding vector based solely on the third sentence embedding vector includes: The third sentence embedding vector is input into a feedforward neural network to obtain the first sentence embedding vector.

4. The method according to claim 2, characterized in that, The step of generating the first sentence embedding vector based on the third sentence embedding vector and the first tag embedding vector includes: A first average embedding vector is generated based on the third sentence embedding vector and the first tag embedding vector; The first average embedding vector is input into a feedforward neural network to obtain the first sentence embedding vector.

5. The method according to claim 2, characterized in that, The step of generating the first sentence embedding vector based on the third sentence embedding vector and the second sentence embedding vector includes: A second average embedding vector is generated based on the third sentence embedding vector and the second sentence embedding vector; The second average embedding vector is input into the feedforward neural network to obtain the first sentence embedding vector.

6. The method according to claim 5, characterized in that, The step of generating a second average embedding vector based on the third sentence embedding vector and the second sentence embedding vector includes: The average value of the third sentence embedding vector and the second sentence embedding vector is calculated to generate the second average embedding vector.

7. The method according to any one of claims 2 to 6, characterized in that, The first sentence input content is the first inserted sentence; or, The first sentence input content is the multi-view tag.

8. The method according to any one of claims 1 to 6, characterized in that, The encoder includes a second encoder, and the decoder includes a second decoder; the step of inputting the document into a transformer network to obtain at least one first document embedding vector includes: Insert a multi-view tag at the beginning of the document, and use the document with the inserted multi-view tag as the first inserted document; wherein the number of multi-view tags is a preset number; The first inserted document is input into the second encoder to obtain a second document embedding vector; wherein, the second document embedding vector includes the second tag embedding vector corresponding to the multi-view tag; The second document embedding vector and the obtained first document input content are input into the second decoder to obtain the third document embedding vector; At least one first document embedding vector is generated based solely on the third document embedding vector; or, multiple first document embedding vectors are generated based on the third document embedding vector and the second tag embedding vector; or, multiple first document embedding vectors are generated based on the third document embedding vector and the second document embedding vector.

9. The method according to claim 8, characterized in that, The first document input content includes the first inserted document or a preset number of multi-view tags; When the first document input content is the first inserted document, the third document embedding vector is the first last character embedding vector corresponding to the last character of the first document; wherein, the first last character of the first document is the last character included in the first inserted document; When the first document input content is the preset number of multi-view tags, the third document embedding vector includes the third tag embedding vectors corresponding to the preset number of multi-view tags respectively.

10. The method according to claim 9, characterized in that, The step of generating at least one first document embedding vector based solely on the third document embedding vector includes: In response to the first document input content being the first inserted document, the first last character embedding vector is used as the first document embedding vector; or... In response to the first document input content being the preset number of multi-view tags, each of the third tag embedding vectors is used as the first document embedding vector; wherein, the number of the first document embedding vectors is the preset number.

11. The method according to claim 9, characterized in that, The process of generating multiple first document embedding vectors based on the third document embedding vector and the second tag embedding vector includes: In response to the first document input content being the first inserted document, a preset number of second tag embedding vectors and the first last character embedding vector are concatenated to generate multiple first document embedding vectors; or... In response to the first document input content being the preset number of multi-view tags, the preset number of second tag embedding vectors and the preset number of third tag embedding vectors are concatenated to generate multiple first document embedding vectors.

12. The method according to claim 9, characterized in that, The generation of multiple first document embedding vectors based on the third document embedding vector and the second document embedding vector includes: In response to the first document input content being the first inserted document, the second document embedding vector and the first last character embedding vector are concatenated to generate multiple first document embedding vectors; or... In response to the first document input content being the preset number of multi-view tags, the second document embedding vector and the preset number of third tag embedding vectors are concatenated to generate multiple first document embedding vectors.

13. The method according to any one of claims 1 to 6, characterized in that, The step of generating a first similarity parameter based on the first sentence embedding vector and the first document embedding vector includes: Based on the first sentence embedding vector and at least one first document embedding vector, a second similarity parameter is generated for each first document embedding vector. The first similarity parameter is generated based on at least one of the second similarity parameters.

14. The method according to any one of claims 1 to 6, characterized in that, The step of generating a loss value based on the first similarity parameter includes: If the document is a positive sample of the sentence, a first encoding parameter corresponding to the positive sample is determined based on the first similarity parameter; or, if the document is a negative sample of the sentence, a second encoding parameter corresponding to the negative sample is determined based on the first similarity parameter. Based on the first sentence embedding vector of the sentence, determine the sample encoding parameters corresponding to the sentence; The loss value is generated based on the sample encoding parameters and either the first encoding parameter or the second encoding parameter.

15. The method according to any one of claims 1 to 6, characterized in that, The number of network layers in the encoder is greater than the number of network layers in the decoder.

16. A sentence-document vectorization method, characterized in that, The method includes: Obtain the target document embedding vector corresponding to the query sentence and at least one target document; wherein, the target document embedding vector is a vector obtained by inputting the target document into a transformer network, and the transformer network includes an encoder and a decoder; The query sentence is input into the transformer network to obtain the target sentence embedding vector; Based on the target sentence embedding vector and the target document embedding vector corresponding to at least one of the target documents, a third similarity parameter is generated for each target document; wherein, the third similarity parameter is used to indicate the degree of association between the target document and the query sentence; Based on at least one of the target documents and the third similarity parameter corresponding to each target document, a response is generated; wherein the response includes a response to the query sentence.

17. A sentence-document vectorization transformer network training device, characterized in that, The device includes: The first acquisition module is used to acquire sentence-document pairs; wherein, the sentence-document pair includes a sentence and a document; A first obtaining module is configured to input the sentence into a transformer network to obtain a first sentence embedding vector; and to input the document into a transformer network to obtain at least one first document embedding vector; wherein the transformer network includes an encoder and a decoder; The first generation module is used to generate a first similarity parameter based on the first sentence embedding vector and the first document embedding vector; The loss module is used to generate a loss value based on the first similarity parameter and update the transformer network based on the loss value.

18. A sentence-document vectorization device, characterized in that, The device includes: The second acquisition module is used to acquire a query sentence and at least one target document embedding vector; wherein, the target document embedding vector is a vector obtained by inputting the target document into a transformer network, and the transformer network includes an encoder and a decoder; The second module is used to input the query sentence into the transformer network to obtain the target sentence embedding vector; The second generation module is used to generate a third similarity parameter for each target document based on the target sentence embedding vector and the target document embedding vector corresponding to at least one target document; wherein the third similarity parameter is used to indicate the degree of association between the target document and the query sentence; The third generation module is used to generate response content based on at least one of the target documents and the third similarity parameter corresponding to each of the target documents; wherein the response content includes a response to the query sentence.

19. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the method according to any one of claims 1 to 15 or claim 16.

20. A computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 15 or claim 16 are implemented.

21. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 15 or claim 16.

Images