Method and device for training embedded model

Abstract

A method and apparatus for training an embedding model, the embedding model comprising a plurality of network layer sets corresponding to a plurality of text granularities, a plurality of network layers of each network layer set comprising an attention module, the method comprising: determining a first lexical sequence of a first input text and a mask matrix corresponding to each text granularity, the mask matrix indicates that the first network layer only pays attention to the content of the last lexical element position in each first granularity unit in the input sequence of the first network layer, and the first granularity units are obtained by dividing the first lexical element sequence according to the first text granularity; obtaining an attention value matrix corresponding to an attention module of each network layer based on the first lexical element sequence through the plurality of network layer sets; and determining the attention loss value of each network layer by utilizing the product of the attention value matrix corresponding to each network layer and the complementary matrix of the mask matrix of each network layer, and training to obtain an embedded model simulating a human understanding language structure by taking the minimization of the attention loss value of each network layer as a target.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a method and apparatus for training an embedded model. Background Technology

[0002] In recent years, Retrieval Augmented Generation (RAG) has become an important application area of ​​Large Language Modeling (LLM). The basic working principle of RAG is to use an embedding model to encode a query and a set of existing documents, generating text embeddings for both the query and each document. Based on these embeddings, the relevance between the query and each document is compared, and then documents strongly related to the query are retrieved to assist the Large Language Model in responding to the query.

[0003] In the above process, the embedding model is a crucial component, and the accuracy of its text embeddings for documents and queries significantly impacts the overall performance of the RAG system. Therefore, there is an urgent need for an embedding model that can generate highly accurate text embeddings to improve the performance of the RAG system. Summary of the Invention

[0004] This specification provides one or more embodiments of a method and apparatus for training an embedding model to obtain an embedding model that mimics human understanding of language structure, and then generates highly accurate text embeddings for input text based on this type of embedding model.

[0005] According to a first aspect, a method for training an embedding model is provided, wherein the embedding model includes a set of multiple network layers corresponding to multiple text granularities, each set of network layers including several network layers, and each network layer including an attention module; the method includes:

[0006] A first word sequence of the first input text and its corresponding mask matrices for each text granularity are determined. The first mask matrix corresponds to the first network layer of the first text granularity. The first mask matrix is ​​used to indicate that the first network layer only focuses on the content of the last word position in each first granularity unit in its input sequence. The input sequence of the first network layer is obtained based on the first word sequence. The first granularity unit is obtained by dividing the first word sequence with reference to the first text granularity.

[0007] By using multiple sets of network layers corresponding to multiple text granularities, and based on the first word sequence, the attention value matrix corresponding to the attention module of each network layer is obtained;

[0008] The attention loss value of each network layer is determined by multiplying the attention value matrix and the complement of its mask matrix.

[0009] The embedding model is trained with the objective of minimizing the attention loss of each network layer.

[0010] According to a second aspect, an apparatus for training an embedding model is provided, wherein the embedding model includes a plurality of network layer sets corresponding to a plurality of text granularities, each network layer set including a plurality of network layers, each network layer including an attention module, the apparatus comprising:

[0011] The first determining module is configured to determine the first word sequence of the first input text and its corresponding mask matrices for each text granularity, wherein the first mask matrix corresponds to the first network layer of the first text granularity, and the first mask matrix is ​​used to indicate that the first network layer only focuses on the content of the last word position in each first granularity unit in its input sequence, the input sequence of the first network layer is obtained based on the first word sequence, and the first granularity unit is obtained by dividing the first word sequence with reference to the first text granularity;

[0012] The module is configured to obtain the attention value matrix corresponding to the attention module of each network layer based on the first word sequence through multiple network layer sets corresponding to multiple text granularities;

[0013] The second determining module is configured to determine the attention loss value of each network layer by multiplying the attention value matrix and the complement of the mask matrix corresponding to each network layer.

[0014] The training module is configured to train the embedding model with the objective of at least minimizing the attention loss of each network layer.

[0015] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first aspect.

[0016] According to a fourth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in the first aspect.

[0017] According to the method and apparatus provided in the embodiments of this specification, the correspondence between each set of network layers in the embedding model and each text granularity is determined. One set of network layers includes several network layers, and each network layer includes an attention module. During the training of the embedding model, the first word sequence of the first input text and its corresponding mask matrices for each text granularity are determined. The first mask matrix corresponds to the first network layer of the first text granularity. The first mask matrix is ​​used to indicate that the first network layer only focuses on the content of the last word position in each first granularity unit in its input sequence. The input sequence of the first network layer is obtained based on the first word sequence, and the first granularity unit is obtained by dividing the first word sequence with reference to the first text granularity. Through multiple sets of network layers corresponding to multiple text granularities, the attention value matrix corresponding to the attention module of each network layer is obtained based on the first word sequence. Then, the attention loss value of each network layer is determined by multiplying the attention value matrix corresponding to each network layer and the complement of its mask matrix. The embedding model is trained with the goal of minimizing the attention loss value of each network layer.

[0018] Through the above process, based on the complement matrix of each mask matrix corresponding to each text granularity of the first input text, and the attention value matrix obtained by each network layer based on the first word sequence, the attention loss value of each network layer is constructed. By applying the loss to the attention value, the network layer in the supervised embedding model corresponding to each text granularity is guided to only pay attention to the content of the last word position in the corresponding text granularity unit in its input sequence. Each text granularity is determined based on the way humans understand language structure. At least the goal is to minimize the attention loss value of each network layer, that is, to minimize the attention value of the content of each word position in the attention value matrix of each network layer to the content of the unattended word position. The embedding model is trained so that the embedding model learns to imitate the hierarchical (i.e., different text granularities) understanding of language structure of humans and to process the input text in the way that humans understand language structure. This can result in a text embedding that is more adapted to human language understanding and has higher accuracy. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1A This is a schematic diagram illustrating the implementation framework of one embodiment disclosed in this specification;

[0021] Figure 1B This is a schematic diagram of another implementation framework of one embodiment disclosed in this specification.

[0022] Figure 2 A flowchart illustrating a method for training an embedding model provided in an embodiment;

[0023] Figure 3 A schematic diagram of an intra-batch loss calculation scenario provided for an embodiment;

[0024] Figure 4 A schematic block diagram of an apparatus for training an embedding model provided in an embodiment. Detailed Implementation

[0025] The technical solutions of the embodiments of this specification will now be described in detail with reference to the accompanying drawings.

[0026] This specification discloses a method and apparatus for training an embedding model. The application scenarios and technical concepts of the method are first introduced below:

[0027] As mentioned earlier, the accuracy of the text embeddings (i.e., vectorized representations of text) generated by the embedding model for documents and queries has a significant impact on the performance of the entire RAG system. Therefore, there is an urgent need for an embedding model that can generate highly accurate text embeddings to improve the performance of the RAG system.

[0028] In view of this, the inventors propose a method for training an embedding model. In this method, referencing how humans understand language (text) structure, such as the understanding of multi-level (i.e., multi-text granularity) structures of words-phrases-sentences (or words-phrases-sentences-paragraphs), the embedding model is trained under supervision using each text in the training samples and mask matrices corresponding to different text granularities constructed based on the text structure (text granularity) of each text in the training samples. This training aims to obtain an embedding model that learns to mimic the hierarchical understanding of human language, thereby achieving high-quality text embeddings based on this type of embedding model and providing a foundation for improving the accuracy of downstream task execution results.

[0029] In some possible examples, the embedding model can be a large language model based on the Transformer architecture.

[0030] Figure 1A The diagram illustrates an implementation scenario according to an embodiment disclosed in this specification. In this implementation scenario, the embedding model includes L layers of network layers arranged serially, each network layer including an attention module. The L layers are divided into multiple network layer sets, each network layer set including several network layers, and different network layer sets correspond to different text granularities.

[0031] In this specific example, assume there are four text granularities: text granularity 1 corresponds to the first *a* layers of the embedding model; text granularity 2 corresponds to layers *a+1* to *a+b* (a total of *b* layers); text granularity 3 corresponds to layers *a+b+1* to *a+b+c* (a total of *c* layers); and text granularity 4 corresponds to layers *a+b+c+1* to *a+b+c+d* (a total of *d* layers). *a*, *b*, *c*, and *d* are all positive integers, and their sum is L. The values ​​of *a*, *b*, *c*, and *d* can be the same or different.

[0032] In some possible examples, to better align with how humans understand language structure, text granularity 1 is smaller than text granularity 2, text granularity 2 is smaller than text granularity 3, and text granularity 3 is smaller than text granularity 4. Referring to how humans understand language (text) structure and how embedding models understand text, text granularity 1, text granularity 2, text granularity 3, and text granularity 4 can be respectively: sub-word granularity, word granularity, phrase granularity, and sentence granularity.

[0033] During training, mask matrices corresponding to different text granularities are used to guide the corresponding network layers (e.g., the mask matrix corresponding to text granularity 1 corresponds to a network layer of text granularity 1) to focus only on the content of the last word position in the corresponding text granularity unit of their input sequence. This means guiding the attention values ​​of each word position in the attention value matrix of each network layer to approach 0 for the content of unattended word positions (i.e., non-last word positions in the corresponding text granularity unit of the input sequence). This guides the embedding model to learn human understanding of language (text) structure, first focusing on understanding the information within each small text granularity unit (e.g., sub-word granularity, word granularity), and then gradually focusing on understanding the aggregated information within larger text granularity units (e.g., phrase granularity, sentence granularity). The corresponding text granularity units in the input sequence of each network layer are obtained by pre-dividing the input text into word sequences according to the corresponding text granularity.

[0034] Specifically, during the training process, the first word sequence of the first input text X1 is determined. And the mask matrices corresponding to each text granularity of the first input text (i.e., the mask matrices corresponding to text granularity 1). Mask matrix corresponding to text granularity 2 Mask matrix corresponding to text granularity 3 and the mask matrix corresponding to text granularity 4 The first mask matrix corresponds to the first network layer at the first text granularity. The first mask matrix is ​​used to instruct the first network layer to only pay attention to the content of the last word position in each first granularity unit in its input sequence. That is, it instructs the attention value of each word position in the attention value matrix of the first network layer (calculated by the attention module) to move closer to 0 to the attention value of the content of the unattended word positions (i.e., the non-last word positions in each first granularity unit in the input sequence). The input sequence of the first network layer is obtained based on the first word sequence, and the first granularity unit is obtained by dividing the first word sequence with reference to the first text granularity.

[0035] In this system, there is a one-to-one correspondence between the word positions in the input sequence of each network layer and the word positions in the first word sequence. The s-th word position in the input sequence of any network layer corresponds to the s-th word position in the first word sequence. For example, the input sequence of the first network layer is obtained by processing the first word sequence through the embedding layer of the embedding model; the input sequences of each non-first network layer (e.g., network layer l) are obtained by processing the input sequence of network layer l-1, which is the layer preceding network layer l-1.

[0036] By using multiple network layer sets corresponding to multiple text granularities, and based on the first word sequence, the attention value matrix A corresponding to the attention module of each network layer is obtained; the attention loss value of each network layer is determined by multiplying the attention value matrix corresponding to each network layer and the complement of its mask matrix; the embedding model is trained with the goal of minimizing the attention loss value of each network layer, that is, minimizing the attention value of the content of each word position in the attention value matrix of each network layer to the content of the unattended word position.

[0037] The complement of the mask matrix is ​​the matrix obtained by subtracting the mask matrix element by element from the matrix of all ones.

[0038] like Figure 1B As shown, assuming the first word sequence of the first input text X1 includes 6 words, i.e., represented as {t1,t2,t3,t4,t5,t6}; when the text granularity 1 is the sub-word granularity, the corresponding text granularity unit includes one word corresponding to the sub-word (i.e., one word).

[0039] Through the mask matrix corresponding to the sub-word granularity of the first input text, such as Figure 1B As shown, this can guide the network layer corresponding to sub-word granularity to focus on the content of each word position. Accordingly, in the network layer corresponding to sub-word granularity, when calculating the aggregation information of the content of each word position, the content of each word position can focus on (aggregate) the content of itself and the content of the word positions before it.

[0040] When text granularity 2 is word granularity, a corresponding text granularity unit (called a word granularity unit) includes one or more sub-words (these one or more sub-words form a word) and their corresponding lexical units (i.e., one or more lexical units). Assuming that in the first lexical unit sequence {t1,t2,t3,t4,t5,t6}, among the 6 lexical units, t1 itself is a word granularity unit (t1 itself is the last lexical unit position of the word granularity unit), t2 and t3 form a word granularity unit (where t3 is the last lexical unit position of the word granularity unit), t4 and t5 form a word granularity unit (where t5 is the last lexical unit position of the word granularity unit), and t6 is a special lexical unit (e.g., the eos lexical unit), which itself forms a word granularity unit (t6 is the last lexical unit position of the word granularity unit).

[0041] Using the word-level mask matrix of the first input text, such as Figure 1B As shown, this can guide the network layer corresponding to word granularity to focus on the content of the last word element position in each word granularity unit. Correspondingly, in the network layer corresponding to word granularity, when calculating the aggregation information of the content at each word element position, the content of the last word element position in each word granularity unit can be focused on (i.e., aggregated) to the content of the last word element positions in itself and the word granularity units preceding it. For example... Figure 1B As shown, the content at position t1 only aggregates itself; the content at position t3 aggregates the content at position t1 and the content at position t3 itself; the content at position t5 aggregates the content at position t1, the content at position t3, and the content at position t5; the content at position t6 aggregates the content at position t1, the content at position t3, the content at position t5, and the content at position t6.

[0042] When the text granularity is phrase granularity, a corresponding text granularity unit (called a phrase granularity unit) includes one or more sub-words (which together form a phrase) and their corresponding lexical units (i.e., one or more lexical units). For example, in the first lexical unit sequence {t1,t2,t3,t4,t5,t6}, among the six lexical units, t1, t2, and t3 form a phrase granularity unit (where t3 is the last lexical unit position), t4 and t5 form a phrase granularity unit (where t5 is the last lexical unit position), and t6 itself is a phrase granularity unit (where t6 itself is the last lexical unit position).

[0043] Using the mask matrix corresponding to the phrase granularity of the first input text, such as Figure 1BAs shown, network layers corresponding to phrase granularity can be guided to focus on the content of the last word position in each phrase granularity unit. Accordingly, in the network layers corresponding to phrase granularity, when calculating the aggregation information of the content at each word position, the content of the last word position in each phrase granularity unit can be focused on (i.e., aggregated) to the content of the last word positions in itself and the phrase granularity units preceding it. For example... Figure 1B As shown, the content at position t3 is aggregated with the content at position t3 itself; the content at position t5 is aggregated with the content at position t3 and the content at position t5; the content at position t6 is aggregated with the content at position t3, the content at position t5, and the content at position t6.

[0044] When the text granularity is sentence granularity, a corresponding text granularity unit (called a sentence granularity unit) includes one or more sub-words (which together form a sentence) and their corresponding lexical units (i.e., one or more lexical units). Assume that for the first lexical unit sequence {t1,t2,t3,t4,t5,t6}, among the six lexical units, t1, t2, t3, t4, and t5 form a sentence granularity unit (where t5 is the last lexical unit position), and t6 itself is a sentence granularity unit (where t6 itself is the last lexical unit position).

[0045] Using the sentence-level mask matrix of the first input text, such as Figure 1B As shown, network layers corresponding to phrase granularity can be guided to focus on the content of the last word position in each sentence granularity unit. Correspondingly, in network layers corresponding to sentence granularity, when calculating the aggregation information of the content at each word position, the content of the last word position in each sentence granularity unit can be considered (i.e., aggregated) to the content of the last word positions in itself and the preceding sentence granularity units. For example... Figure 1B The content at position t5 only aggregates the content at position t5; the content at position t6 aggregates the content at positions t5 and t6.

[0046] In the above process, the attention loss value of each network layer is constructed based on the complement matrix of the mask matrix corresponding to each text granularity of the first input text and the attention value matrix obtained by each network layer based on the first word sequence. By applying the loss to the attention value, the network layer in the supervised embedding model corresponding to each text granularity is guided to only pay attention to the content of the last word position in the corresponding text granularity unit in its input sequence. Each text granularity is determined based on the way humans understand language structure. The goal is to minimize the attention loss value of each network layer, that is, to minimize the attention value of the content of each word position in the attention value matrix of each network layer to the content of the unattended word position. The embedding model is trained so that the embedding model learns to imitate the hierarchical (i.e., different text granularities) understanding of language structure of humans and to process the input text in the same way as humans understand language structure. This can result in a text embedding that is more adapted to human language understanding and has higher accuracy.

[0047] The method for training an embedding model provided in this specification will be described in detail below with reference to specific embodiments.

[0048] The following section will introduce the process of constructing the training samples required for training the embedding model.

[0049] The process of constructing training samples required for training the embedding model includes: collecting data for each task type from several specified data sources (i.e., the subsequent initial data); then, based on the task characteristics of the initial data for each task type, determining the training samples corresponding to each task type to obtain data in a format that conforms to the format required for training the embedding model, forming a specified training set for training the embedding model. Understandably, for an embedding model, each training sample generally includes: a sample query (text), and the corresponding positive sample document and at least one negative sample document.

[0050] Among them, the specified data sources can be various open source data sources in related technologies.

[0051] In some examples, to ensure the diversity of data in the training set of the embedding model, the aforementioned task types may include, but are not limited to: retrieval tasks, natural language reasoning tasks, summary generation tasks, similar text identification tasks, classification tasks, and clustering tasks.

[0052] In some examples, the initial data for each task type can be categorized into three main types based on the characteristics of the initial data: retrieval format data, classification format data, and clustering format data. The retrieval format data can include initial data for retrieval tasks, natural language inference tasks, summary generation tasks, and similar text identification tasks. The classification format data can include initial data for binary classification tasks. The clustering format data can include initial data for clustering tasks and initial data for multi-class classification tasks.

[0053] The process of constructing training samples for training embedding models based on the three different data formats mentioned above will be described below.

[0054] On the one hand, for data in the retrieval format class, the process of constructing training samples for training the embedding model based on this type of data is as follows:

[0055] For each initial data corresponding to the original retrieval task, each initial data includes: the query and its corresponding tag query results; at this time, for any initial data in this type of initial data, the original query can be used as the sample query, and the original tag query results can be used as the sample positive document corresponding to the sample query.

[0056] For the initial data corresponding to the original natural language reasoning task, each initial data includes: reasoning premise data and its corresponding necessary assumption data, neutral assumption data, and contradictory assumption data. In this case, for any initial data in this category, the reasoning premise data can be used as a sample query, the necessary assumption data as the corresponding positive sample document, and the neutral and contradictory assumption data as the corresponding negative sample document.

[0057] For each initial data point corresponding to the summary generation task, each initial data point includes: the original text (i.e., the text for which a summary needs to be generated) and its corresponding summary text. In this case, for any initial data point in this type of initial data, the summary text can be used as a sample query, and the original text can be used as the corresponding sample document for the sample query.

[0058] For each initial data set corresponding to the similar text determination task, each initial data set includes two texts (e.g., text A and text B) whose text similarity exceeds a specified text similarity threshold. In this case, for any initial data set, either text (e.g., text A) can be used as a sample query, and the other text (e.g., text B) can be used as the corresponding sample document for that sample query. The text similarity threshold can be set according to actual needs.

[0059] To ensure data diversity, for initial data of retrieval format (and initial data of clustering format), when constructing training samples based on this type of initial data, the number of negative documents in each training sample constructed based on this type of initial data is set to a specified number (e.g., 24 or 7). This specified number can be set according to actual needs.

[0060] Given the above, for the initial data of the retrieval format type, it is considered that the original initial data may not include data that can be used as sample queries for corresponding negative documents, or the number of sample negative documents that can be used as sample queries is small.

[0061] This specification provides a process for mining negative documents corresponding to sample queries, in order to determine a specified number of negative documents corresponding to sample queries constructed based on data of a retrieval format class. Specifically, this mining process includes:

[0062] By using a specified embedding generation model, each specified sample query and each document in a specified document pool are encoded to obtain the embeddings of each specified sample query and each document in the specified document pool.

[0063] Then, for any specified sample query z in each specified sample query, based on the similarity between the embedding of the specified sample query z and the embedding of each document in the specified document pool, a preset number (100 for example) of documents most relevant to the specified sample query z are recalled from the specified document pool as the relevant documents of the specified sample query z.

[0064] Next, for a specified sample query z, the top 5 most relevant documents (the number can be set according to a preset limit) are removed from its 100 relevant documents, resulting in 95 remaining relevant documents. Then, from these remaining 95 relevant documents, multiple relevant documents that meet the following conditions are selected as the sample negative documents corresponding to the specified sample query z. Removing the top 5 most relevant documents avoids using false negative documents as the sample negative documents corresponding to the specified sample query z.

[0065] The condition includes: the similarity between the relevant document and the specified sample query z is less than a first similarity threshold and less than a specified percentage of the similarity between the specified sample query z and its corresponding positive sample document. For example, the first similarity threshold can be set according to actual needs, such as 0.8, and the specified percentage can be set according to actual needs and the first similarity threshold, such as 95%.

[0066] If the number of relevant documents filtered by the above conditions exceeds the specified number of negative documents corresponding to the specified sample query z (or exceeds the difference between the specified number and the number of existing negative documents for the specified sample query z), then the corresponding number of relevant documents with the highest similarity to the specified sample query z can be filtered again from the relevant documents filtered by the above conditions, and used as the negative documents corresponding to the specified sample query z.

[0067] The above method yields a specified number of negative documents corresponding to each specified sample query. These specified sample queries are constructed based on data from a retrieval format class.

[0068] Correspondingly, for a sample query belonging to a specified task type (i.e., a sample query constructed based on data of a retrieval format class), its corresponding negative sample documents include: documents in a specified document pool whose similarity to the sample query meets preset conditions. The preset conditions include: the similarity to the sample query is less than a first similarity threshold and less than a specified percentage of the similarity between the sample query and its corresponding positive sample document.

[0069] At this point, for each specified task type categorized into the aforementioned retrieval format classes, a sample query, its corresponding positive sample document, and multiple negative sample documents constitute a training sample for training the embedding model. This process is repeated to obtain multiple training samples corresponding to each specified task type categorized into the aforementioned retrieval format classes.

[0070] Using the above method, for each specified sample query, negative sample documents that are not directly related to the specified sample query but are easily mistaken for related can be mined from the specified document pool. At this time, such negative sample documents can be called the hard negative samples corresponding to the specified sample query. Through such negative sample documents, the accuracy of the text embedding generated by the embedding model can be better guaranteed.

[0071] On the other hand, for data in a categorical format, the process of constructing training samples for training the embedding model based on this type of data is as follows:

[0072] For each initial data set corresponding to a binary classification task, each initial data set includes: the data to be classified and its corresponding class label (called the positive class label) and the corresponding class label (called the negative class label). Then, for any initial data set in this class, the data to be classified can be used as a sample query, the positive class label of the data to be classified can be used as the corresponding positive document, and the corresponding negative class label can be used as the corresponding negative document. Specifically, for data in the classification format class, the sample query in the constructed training samples corresponds to one negative document. Thus, for a binary classification task, a sample query, its corresponding positive document, and its corresponding negative document constitute a training sample for training the embedding model. This process is repeated to obtain multiple training samples for the binary classification task.

[0073] On the other hand, for clustered data, the process of constructing training samples for training the embedding model based on this type of data is as follows:

[0074] For the initial data corresponding to the original clustering task, each initial data includes multiple input texts. In this case, for any initial data, any one of the multiple input texts *x* is used as a sample query, and any text from other texts belonging to the same cluster as *x* is used as the corresponding positive sample document. A specified number of texts randomly selected from the other clusters are used as multiple negative sample documents corresponding to the sample query. At this point, a sample query, its corresponding positive sample document, and multiple negative sample documents constitute a training sample for training the embedding model. This process is repeated to obtain multiple training samples corresponding to the original clustering task.

[0075] For each initial data set corresponding to the original multi-class classification task, each initial data set includes: an input text and its corresponding classification category. Then, for any initial data set, the input text is used as a sample query, and any text from other texts belonging to the same classification category as the input text is used as the corresponding positive document for that sample query; a specified number of texts randomly selected from other classification categories are used as multiple negative documents for that sample query. At this point, a sample query, its corresponding positive document, and multiple negative documents constitute a training sample for training the embedding model. This process is repeated to obtain multiple training samples for the original multi-class classification task.

[0076] By using the above method, multiple training samples corresponding to each task type can be obtained to obtain a specified training set.

[0077] In some possible examples, multiple training samples corresponding to each task type belonging to the retrieval format class can be stored as a training subset (and assigned a format label), multiple training samples corresponding to each task type belonging to the classification format class can be stored as a training subset (and assigned a format label), and multiple training samples corresponding to each task type belonging to the clustering format class can be stored as a training subset (and assigned a format label). This categorized storage of training subsets corresponding to different format classes facilitates the ordered training of the embedding model. The aforementioned three training subsets constitute the aforementioned specified training set.

[0078] Subsequently, in order for the embedding model to learn how humans understand language (text) hierarchically, appropriate guidance information needs to be set according to the text structure of each text in each training sample, and a specific loss function is set to encourage the embedding model to learn to understand language (text) according to the hierarchical structure of different text granularity levels (e.g., word-phrase-sentence) during the training process.

[0079] Specifically, for each text in each training sample (i.e., the sample query and its corresponding positive and negative documents), at least based on the text structure of each text, a suitable mask matrix M corresponding to the granularity of each text needs to be set. In order to combine the mask matrix M corresponding to the granularity of each training sample and each text in it, the embedding model is trained to guide several network layers in the embedding model corresponding to the granularity of each text to aggregate information from the text at the corresponding granularity level (text granularity unit).

[0080] Understandably, the principle of constructing a mask matrix M corresponding to each text granularity for any text in each training sample of the aforementioned specified training set is similar. The following section uses any text (e.g., text Y) from any training sample as an example to introduce the process of constructing the mask matrix M corresponding to each text granularity.

[0081] The following example illustrates the process of using an embedding model that includes network layer sets corresponding to sub-word granularity, word granularity, phrase granularity, and sentence granularity. The process includes:

[0082] For a text Y, the sub-word information of text Y is determined using the tokenizer of the embedding model (or the tokenizer of other large language models) that needs to be trained. For example, the sub-word information may include the sub-word sequence of text Y. Based on the sub-word sequence of text Y, the token sequence of text Y can be obtained through the vocabulary corresponding to the embedding model.

[0083] The token sequence of text Y includes: each token corresponding to each sub-word in the sub-word sequence of text Y, and a special token at the last position. This special token is either an end-of-sequence (eos) token or a token added to the end of the original token sequence of text Y.

[0084] In some possible examples, the token sequence of text Y can be stored in a specified training set, corresponding to text Y, so that during the training of the embedding model, the token sequence of text Y can be directly sampled when the training sample containing text Y is used to train the embedding model.

[0085] For text Y, a specified segmenter (e.g., Spacy) is used to determine the word information, phrase information, and sentence information of text Y. For example, word information may include a sequence of words in text Y, where a single word may include one or more sub-words (or lexical units); phrase information may include a sequence of phrases in text Y, where a single phrase may include one or more sub-words (or lexical units), and may also be referred to as a single phrase comprising one or more words; sentence information may include a sequence of sentences in text Y, where a single sentence may include one or more sub-words (or lexical units), and may also be referred to as a single sentence comprising one or more words (or comprising one or more phrases).

[0086] Then, using the sub-word information of the text Y determined above, based on the word sequence of the text Y, the text granularity unit sequence corresponding to the sub-word granularity of the text Y is determined (one text granularity unit corresponds to one sub-word, that is, one word); the text granularity unit sequence corresponding to the sub-word granularity is subsequently called the sub-word granularity unit sequence.

[0087] Using the word information of the text Y determined above, and based on the word sequence of the text Y, determine the text granularity unit sequence of the text Y corresponding to the word granularity (where one text granularity unit corresponds to one word, that is, one or more words that make up a word); the text granularity unit sequence corresponding to the word granularity is subsequently referred to as the word granularity unit sequence.

[0088] Using the phrase information of the text Y determined above, and based on the word sequence of the text Y, determine the text granularity unit sequence of the text Y corresponding to the phrase granularity (where one text granularity unit corresponds to one phrase, that is, one or more words that make up a phrase); the text granularity unit sequence corresponding to the phrase granularity is hereby referred to as the phrase granularity unit sequence.

[0089] Using the sentence information of the text Y determined above, and based on the word sequence of the text Y, the text granularity unit sequence corresponding to the sentence granularity of the text Y is determined (one text granularity unit corresponds to one sentence, that is, one or more words that make up a sentence). The text granularity unit sequence corresponding to the sentence granularity is referred to as the sentence granularity unit sequence.

[0090] Thus, the sub-word granularity unit sequence, word granularity unit sequence, phrase granularity unit sequence, and sentence granularity unit sequence of text Y are obtained. Among them, the special word element at the last position in the word element sequence of text Y is set as a separate granularity unit in the sub-word granularity unit sequence, word granularity unit sequence, phrase granularity unit sequence, and sentence granularity unit sequence.

[0091] Then, based on the following attention conditions, a mask matrix corresponding to the sub-word granularity of text Y is constructed according to the sub-word granularity unit sequence, word granularity unit sequence, phrase granularity unit sequence, and sentence granularity unit sequence of text Y, respectively. Mask matrix corresponding to word granularity Mask matrix corresponding to phrase granularity and the mask matrix corresponding to sentence granularity

[0092] The aforementioned criteria for attention include:

[0093] For several network layers corresponding to sub-word granularity, it is necessary to ensure that the content at each word position in the word sequence (i.e., the word) can pay attention to the content of the word itself and all word positions before it in the word sequence;

[0094] For several network layers corresponding to word granularity, it is necessary to completely mask the content of all word positions in each word (i.e., word granularity unit) in its input sequence (determined based on the word sequence), except for the content of the last word position; that is, for each word (i.e., word granularity unit) in the input sequence of each network layer corresponding to word granularity, the content of the word positions other than the content of the last word position should no longer be considered by the content of the word positions after it.

[0095] For several network layers corresponding to phrase granularity, it is necessary to mask all word positions in each phrase (i.e., phrase granularity unit) in its input sequence (determined based on the word sequence), except for the content of the last word position; that is, for each phrase (i.e., phrase granularity unit) in the input sequence of each network layer corresponding to phrase granularity, the content of word positions other than the content of the last word position should no longer be considered by the content of word positions after it.

[0096] For several network layers corresponding to sentence granularity, it is necessary to mask all word positions in each sentence (i.e., sentence granularity unit) in its input sequence (determined based on the word sequence), except for the content of the last word position; that is, for each sentence (i.e., sentence granularity unit) of each network layer corresponding to sentence granularity, the content of word positions other than the content of the last word position should no longer be considered by the content of word positions after it.

[0097] In the embodiments of this specification, a specific loss function can be set to apply the loss to the attention value calculated by the attention module of each network layer of the embedded model, so that each network layer corresponding to each text granularity only focuses on the content of the last word position in the corresponding text granularity unit in its input sequence (for example, the network layer corresponding to the word granularity only focuses on the content of the last word position in each word granularity unit of its input sequence).

[0098] Specifically, by setting a specific loss function, for each network layer corresponding to each text granularity (e.g., any network layer l), combined with the mask matrix of each text (each text in the training sample) corresponding to the same text granularity as network layer l, it indicates which word positions in the input sequence of network layer l need to be focused on (e.g., the content of the last word position in each word granularity unit in the input sequence), and determines which word positions need to be masked (i.e., not focused on) (e.g., the content of non-last word positions in each word granularity unit in the input sequence). Subsequently, for each word whose content at a given word position needs to be masked (hereinafter referred to as masked word), a specified type of loss (e.g., mean squared error (MSE) loss) is applied to the attention values ​​of all other words (content at the given position) in the input sequence for this type of masked word (content at the given position). This causes the attention values ​​of all other words (content at the given position) in the input sequence to approach 0, thus encouraging network layer l to disregard these masked word (content at the given position), thereby enabling the embedding model to learn the hierarchical understanding of language (text) by humans.

[0099] Accordingly, for text Y, taking any text granularity n as an example, the mask matrix of text Y corresponding to text granularity n is... Each element in the matrix corresponds one-to-one with the attention value in the attention value matrix obtained by the attention module of the network layer corresponding to text granularity n, based on the word sequence of text Y. That is, the mask matrix of text Y corresponding to text granularity n. The element at the i-th row and j-th column of the mask matrix corresponds to the attention value at the i-th row and j-th column of the corresponding attention value matrix (i.e., the attention value matrix obtained by the attention module of the network layer corresponding to text granularity n based on the word sequence of text Y). If i is less than or equal to j, and the two word positions corresponding to the attention value at the i-th row and j-th column of the corresponding attention value matrix are both the last word position of the corresponding text granularity unit (i.e., the unit obtained based on text granularity n) in the input sequence of the corresponding network layer (obtained based on the word sequence of text Y), then the mask matrix... The element at the i-th row and j-th column position has a value of 1, otherwise it has a value of 0.

[0100] For example, mask matrix Given a binary mask matrix, for a sequence of terms in a text Y of dimension m (or o) (i.e., the sequence of terms contains m (or o) terms), its corresponding mask matrix is... Let G be an m x m (or o x o) dimensional binary mask matrix. Assume that the sequence of tokens in text Y, divided according to text granularity n, is represented as a sequence of text granularity units, G. n =[g1,g2,……g R ], where any text granularity unit g r Including one or more consecutive lexical units in the lexical sequence of text Y, g r Represented as Among them, each g r,c Represents a word. The mask matrix of text Y corresponding to text granularity n. The element at any position (e.g., the element at the i-th row and j-th column) The value of ) can be represented by, for example, the following formula:

[0101]

[0102] in, and This indicates that the attention value at the i-th row and j-th column of the attention value matrix corresponding to the network layer with text granularity n (i.e., the attention module of the network layer with text granularity n) corresponds to two word positions: text granularity unit g in the text granularity unit sequence (the input sequence of the attention module of the network layer with text granularity n). p The last word (position) and text granularity unit g q The last word (position) and the text granularity unit g q g is the text granularity unit p The subsequent text granularity unit.

[0103] As is known, rows in the attention value matrix represent the positions of keys (Keys) obtained from the content at each word position in the input sequence of the corresponding attention module (via the subsequent attention value matrix A). l The formula can be determined), and the column represents the position of the query (derived based on the word positions) obtained from the content at each word position in the input sequence of the corresponding attention module. Accordingly, This represents the Query (i.e., the Query at position j) obtained from the content at the word position j in the input sequence based on the corresponding attention module in the corresponding attention value matrix, and the Key (i.e., the Key at position i) obtained from the content at the word position i in the input sequence based on the corresponding attention module.

[0104] Continuing with the previous example, let's take text Y as the first input text X1 for illustration. For the aforementioned first word sequence {t1,t2,t3,t4,t5,t6}, which includes 6 words, the mask matrix corresponding to the sub-word granularity of its first input text X1 is... It can be represented as follows:

[0105]

[0106] The first input text X1 has a mask matrix corresponding to the word granularity. It can be represented as follows:

[0107]

[0108] The mask matrix corresponding to the phrase granularity of its first input text X1 It can be represented as follows:

[0109]

[0110] The first input text X1 has a mask matrix corresponding to the sentence granularity. It can be represented as follows:

[0111]

[0112] Using the above method, a mask matrix corresponding to each text granularity in each training sample of the specified training set is obtained. Then, the embedding model is trained using each text in each training sample and its corresponding mask matrix at each text granularity.

[0113] The process of training the embedding model is described below. Figure 2 A flowchart illustrating a method for training an embedding model according to one embodiment of this specification is shown. This method is executed via an electronic device, which can be any device, equipment, platform, device cluster, etc., possessing computing and processing capabilities.

[0114] The embedding model to be trained includes a set of network layers arranged sequentially, corresponding to multiple text granularities. Each set of network layers includes several network layers, and each network layer includes an attention module. In some possible examples, the multiple text granularities include at least two of the following: sub-word granularity, word granularity, phrase granularity, sentence granularity, and paragraph granularity.

[0115] In some possible examples, the embedded model is a Transformer-based model, where each network layer includes an attention module and a feedforward neural network module. In each network layer, the output of its attention module is the input of its feedforward neural network module; the output of the feedforward neural network module in the l-th layer is the input of the attention module in the (l+1)-th layer. The structure of the attention module and feedforward neural network module in each network layer can be found in the structure of Transformer-based models in related technologies.

[0116] In some possible examples, an embedding layer (as shown in Figure 1) may be included before the multiple network layers of the embedding model. This layer is used to vectorize each word in the input word sequence, mapping each word to a vector of fixed dimensions.

[0117] In some possible examples, during the training of the embedding model, the electronic device can launch a data loader that can launch one or more processes. Each process independently samples training samples from the aforementioned specified training set during each batch of model iterations, and then processes each text in the sampled training samples through the embedding model and constructs a corresponding loss for each text. Then, the electronic device combines the losses of each text constructed by all processes to train the embedding model.

[0118] In each batch of model iterations, each process samples training samples from only one of the aforementioned training subsets when sampling training samples from the specified training set. That is, during each batch of model iterations, the training samples sampled by one process belong to the same format class (e.g., all are retrieval format training samples, or all are classification format training samples, or all are clustering format training samples), while the training samples sampled by different processes can belong to different format classes or the same format class.

[0119] In some implementations, when sampling training samples of the retrieval format class and the clustering format class, any process can randomly select a first number of negative documents from a specified number of negative documents in that class of training samples for training the embedding model, thereby reducing computational cost to some extent. This first number can be less than the specified number. For example, the specified number is 24, and the first number can be 7.

[0120] In the above example, starting multiple processes for the data loader can improve the training efficiency of the embedding model. Furthermore, by starting multiple independent processes that sample and process training samples from the aforementioned specified training set, the above settings can ensure that the data source within the same batch of the same process is the same, avoiding mutual interference between data of different formats.

[0121] Understandably, the processing of training samples is the same in each of the multiple processes. The following section will introduce the process of training the embedding model from the perspective of any process E in the multiple processes.

[0122] During the training of the embedding model, such as Figure 2 As shown, the method includes the following steps S210-S250:

[0123] In step S210, the first word sequence of the first input text and its corresponding mask matrices for each text granularity are determined. The first mask matrix corresponds to the first network layer of the first text granularity. The first mask matrix is ​​used to indicate that the first network layer only focuses on the content of the last word position in each first granularity unit in its input sequence. The input sequence of the first network layer is obtained based on the first word sequence. The first granularity unit is obtained by dividing the first word sequence with reference to the first text granularity.

[0124] In some examples, the first word sequence of the first input text can be obtained directly from a specified training set by the electronic device (process E in the process); or it can be obtained by the electronic device after obtaining the first input text, by segmenting the first input text into words through the word segmenter of the embedding model, and combining it with the vocabulary of the embedding model.

[0125] The mask matrices corresponding to each text granularity of the first input text can be obtained by the electronic device (process E) from a specified storage space, or they can be constructed based on the first word sequence in the aforementioned method of constructing mask matrices.

[0126] The element at the i-th row and j-th column of the first mask matrix corresponds to the attention value at the i-th row and j-th column of the first attention value matrix corresponding to the first network layer. If i is less than or equal to j, and the two word positions corresponding to the attention value at the i-th row and j-th column of the first attention value matrix are both the last word position of the first text granularity unit in the input sequence of the first network layer, then the value of the element at the i-th row and j-th column of the first mask matrix is ​​1; otherwise, it is 0.

[0127] The first input text can be any text from the first sample query and its corresponding first sample positive document and first sample negative document. The first sample query and its corresponding first sample positive document and first sample negative document can belong to any training sample in the aforementioned specified training set.

[0128] In some possible examples, if the first input text (i.e. the first sample query) is a sample query belonging to a specified task type, then its corresponding negative sample document includes: a document in a specified document pool whose similarity to the sample query meets a preset condition. The preset condition includes at least: the similarity to the sample query is less than a first similarity threshold and less than a specified percentage of the similarity between the sample query and its corresponding positive sample document.

[0129] The specified task type includes at least one of the following task types: retrieval task, natural language reasoning task, summary generation task, and similar text identification task.

[0130] Next, in step S220, attention value matrices corresponding to the attention modules of each network layer are obtained based on the first word sequence, using multiple network layer sets corresponding to multiple text granularities.

[0131] In this step, after the electronic device (process E) obtains the first word sequence, it first inputs the first word sequence into the embedding layer of the embedding model to obtain the vectorized first word sequence; then, through multiple network layer sets corresponding to multiple text granularities, it obtains the attention value matrix corresponding to the attention module of each network layer based on the vectorized first word sequence.

[0132] Understandably, the vectorized first word sequence can be input into the first network layer of the embedding module. This allows the first network layer 1 to process the vectorized first word sequence based on its attention module 1, obtaining the attention value matrix and output of its attention module. Then, the first network layer 1 processes the output of its attention module 1 based on its feedforward neural network module 1, obtaining its output as well. Subsequently, the output of the feedforward neural network module 1 of the first network layer 1 is used as the input to the attention module 2 of the second network layer 2, and so on, to obtain the attention value matrix for each network layer's attention module. Additionally, the output of the last network layer (the feedforward neural network module) can also be obtained from the embedding model.

[0133] For example, for any network layer l, the attention value matrix A corresponding to its attention module... l It can be represented as:

[0134]

[0135] Among them, Q l This represents the query Q-matrix computed by the attention module of network layer l (based on its input sequence). This represents the transpose of the key K matrix computed by the attention module of network layer l (based on its input sequence); The dimension of the K matrix calculated by the attention module of network layer l is represented by , and softmax() represents the normalization function.

[0136] Next, in step S230, the attention loss value of each network layer is determined by multiplying the attention value matrix corresponding to each network layer and the complement of its mask matrix. The complement of the mask matrix is ​​obtained by subtracting the mask matrix element-wise from the matrix consisting of all ones.

[0137] Among them, the mask matrix corresponding to any network layer l in each network layer is the mask matrix of the first input text with the same text granularity as the network layer l.

[0138] In this step, the electronic device (process E) can calculate the modulus of the product of the attention value matrix and the complement of the mask matrix corresponding to any network layer l in each network layer (called the first modulus), and then calculate the ratio of the first modulus to the modulus of the first mask matrix, and use this ratio as the attention loss value of network layer l.

[0139] Here, the product of the attention value matrix and the complement of the mask matrix corresponding to network layer l refers to the result of element-wise multiplication of the attention value matrix and the complement of the mask matrix corresponding to network layer l.

[0140] Wherein, the attention loss value of network layer l This can be expressed by the following formula:

[0141]

[0142] Among them, A l1 This represents the attention value matrix corresponding to network layer l. This represents the mask matrix corresponding to network layer l. The complement matrix, Here, assuming the dimension of the first word sequence is m, the corresponding "1" represents a vector of dimension m in which all elements are 1. T This represents the transpose of a dimension m vector whose elements are all 1s. A represents l1 and Element-by-element multiplication; This indicates the calculation of the modulus.

[0143] This allows us to obtain the attention loss values ​​for each network layer.

[0144] Taking network layer l as an example corresponding to word granularity, continuing from the previous example, the first word sequence of the first input text X1 includes 6 words, represented as {t1,t2,t3,t4,t5,t6}, where t1, t2, and t3 form a phrase granularity unit (where t3 is the last word position of the phrase granularity unit), t4 and t5 form a phrase granularity unit (where t5 is the last word position of the phrase granularity unit), and t6 itself is a phrase granularity unit (where t6 itself is the last word position of the phrase granularity unit).

[0145] At this time, the mask matrix corresponding to network layer l complement matrix And the attention value matrix A corresponding to network layer l l1 They are represented as follows:

[0146]

[0147] Where, any a ij This represents the attention value in the i-th row and j-th column.

[0148] Attention loss value of network layer l It can be represented as:

[0149]

[0150] This allows us to obtain the attention loss values ​​for each network layer.

[0151] Then, in step S240, the embedding model is trained with the goal of minimizing the attention loss value of each network layer. In this step, the mean (or sum) of the attention loss values ​​of all network layers corresponding to the first input text can be calculated. Then, the embedding model is trained with the goal of minimizing the mean (or sum) of the attention loss values ​​of all network layers corresponding to the first input text. For example, backpropagation can be used to calculate the model gradient using the mean (or sum) of the attention loss values ​​of all network layers corresponding to the first input text. Then, with the goal of minimizing the mean (or sum) of the attention loss values ​​of all network layers corresponding to the first input text, the updated values ​​of the parameters of each network layer of the embedding model are calculated using the model gradient and the current values ​​of the parameters of each network layer of the embedding model. Based on the updated values ​​of the parameters of each network layer, the parameters of each network layer of the embedding model are adjusted, i.e., the embedding model is trained.

[0152] The mean of the attention loss values ​​of all network layers corresponding to the first input text 1 can be expressed by the following formula:

[0153]

[0154] Among them, L attn denoted as the mean of the attention loss values ​​of all network layers corresponding to the first input text, and L represents the number of network layers in the embedding model.

[0155] In some possible examples, the first input text is: the first sample query and any text in the first sample positive document and the first sample negative document;

[0156] Step S240 may include the following steps 11-12:

[0157] In step 11, a comprehensive attention loss value is determined based on the total first attention loss value of the first sample query, the total second attention loss value of the first sample positive document, and the total third attention loss value of the first sample negative document. The comprehensive attention loss value is positively correlated with the total first attention loss value, the total second attention loss value, and the total third attention loss value, respectively. The total first attention loss value is determined based on the attention loss values ​​of each network layer corresponding to the first sample query, the total second attention loss value is determined based on the attention loss values ​​of each network layer corresponding to the first sample positive document, and the total third attention loss value is determined based on the attention loss values ​​of each network layer corresponding to the first sample negative document.

[0158] In some implementations, the first total attention loss can be determined based on the average (or sum) of the attention loss values ​​of all network layers corresponding to the first sample query. The second total attention loss can be determined based on the average (or sum) of the attention loss values ​​of all network layers corresponding to the sample positive document. The third total attention loss can be determined based on the average (or sum) of the attention loss values ​​of all network layers corresponding to all sample negative documents.

[0159] In this step, the comprehensive attention loss value can be determined based on the average of the first total attention loss value, the second total attention loss value, and the third total attention loss value (hereinafter referred to as the mean attention loss value).

[0160] Then, in step 12, the embedding model is trained with the goal of minimizing the overall attention loss.

[0161] In some possible examples, to better ensure the accuracy and quality of the text embeddings generated by the embedding model, the text embedding of the first input text can be determined from the output of the last network layer of the embedding model. That is, the text embeddings of a sample query and its corresponding positive and negative documents are obtained respectively. Then, based on the hard negative sample loss function, the hard negative sample loss value corresponding to the first sample query is determined by using the text embeddings of the first sample query and its corresponding positive and negative documents. Next, the embedding model is trained with the goal of minimizing the comprehensive attention loss value and minimizing the hard negative sample loss value corresponding to the first sample query.

[0162] Wherein, the hard-to-bear sample loss value L corresponds to the first sample query. hard It can be expressed by the following formula:

[0163]

[0164] Where τ1 represents the preset first temperature parameter, used to adjust the degree of attention the embedding model pays to the sample negative document; the smaller the value, the more attention is paid to the corresponding sample negative document; q1 represents the first sample query. This indicates that the first sample query corresponds to the sample document. This represents the u-th negative document among the U sample negative documents corresponding to the first sample query; s(,) represents the similarity function used to calculate the similarity between two text embeddings.

[0165] In some possible examples, to better ensure the accuracy and quality of the text embeddings generated by the embedding model, the in-batch loss and / or hard-to-negative sample loss can be combined with the comprehensive attention loss value to jointly train the embedding model.

[0166] In some possible implementations, step S240 may include the following steps 21-24:

[0167] In step 21, a comprehensive attention loss value is determined based on the total first attention loss value of the first sample query, the total second attention loss value of the first sample positive document, and the total third attention loss value of the first sample negative document. The comprehensive attention loss value is positively correlated with the total first attention loss value, the total second attention loss value, and the total third attention loss value, respectively. The total first attention loss value is determined based on the attention loss values ​​of each network layer corresponding to the first sample query, the total second attention loss value is determined based on the attention loss values ​​of each network layer corresponding to the first sample positive document, and the total third attention loss value is determined based on the attention loss values ​​of each network layer corresponding to the first sample negative document.

[0168] The implementation principle of step 21 is similar to that of step 11 mentioned above. The implementation process can be found in the implementation process of step 11 mentioned above, and will not be repeated here.

[0169] In step 22, the text embedding of the first input text is determined from the output of the last layer of the embedding model. Understandably, the last lexical element in the first lexical sequence is either an end-of-sequence (eos) element or a special element added to the original input lexical sequence. After the embedding model processes the first lexical sequence of the first input text, the content of the last lexical element has aggregated the information of each lexical element in the first lexical sequence. Therefore, the content (representation) of the last lexical element can be determined from the output of the last layer of the embedding model as the text embedding of the first input text.

[0170] In step 23, the first loss value is determined by using the text embeddings of the first sample query, the first sample positive document, and the first sample negative document.

[0171] In this step, based on the batch loss function and / or the hard-to-negative sample loss function, the first loss value is determined by utilizing the text embeddings of the first sample query, the first sample positive document, and the first sample negative document.

[0172] In some possible examples, where the training sample to which the first sample query belongs belongs to the retrieval format class, it is necessary to calculate the hard-to-batch loss and the in-batch loss for the first sample query. Accordingly, in step 23, including steps 231-233:

[0173] In step 231, if the first sample query corresponds to the first format label, based on the intra-batch loss function, the intra-batch loss value corresponding to the first sample query is determined by using the text embeddings of the first sample query and the second sample document, as well as the text embedding of the second sample document corresponding to the second sample query, wherein the second sample query is data within the same batch as the first sample query.

[0174] In this step, if the first sample query corresponds to the first format label, it indicates that the training sample to which the first sample query belongs belongs to the training sample of the retrieval format class. Accordingly, it is necessary to calculate the intra-batch loss value corresponding to the first sample query based on the intra-batch loss function. Wherein, the intra-batch loss value L corresponding to the first sample query... in-batch It can be expressed by the following formula:

[0175]

[0176] Where τ2 represents a preset second temperature parameter used to adjust the degree of attention the embedding model pays to the positive documents of the samples; the smaller the value, the more attention is paid to the corresponding positive documents. B represents the batch size (i.e., the number of training samples in the batch). This represents the b-th sample document in this batch.

[0177] The corresponding batch loss value L was obtained by querying the first sample. in-batch By treating the positive document pairs of other sample queries (second sample queries) within the same batch as "free negative documents, i.e., negative samples" for the first sample query, memory consumption during embedding model training can be saved, and sample utilization and model training speed can be improved. The above process ensures that the text embedding of the first sample query is as similar as possible to the text embedding of its own first sample positive document, and as dissimilar as possible to the text embeddings of the positive documents of other sample queries.

[0178] Understandably, the second sample query in the same batch as the first sample query can include one or more. The second sample query includes sample queries from the training samples sampled by process E within this batch, as well as sample queries from the training samples sampled by other processes within this batch.

[0179] In some possible examples, if there are also sample queries in the second sample query whose training samples belong to the retrieval format class (i.e., there are second sample queries corresponding to the first format label), it is also necessary to calculate the intra-batch loss value corresponding to each second sample query corresponding to the first format label based on the intra-batch loss function; then, by combining the intra-batch loss value corresponding to the first sample query and the average of the intra-batch loss values ​​corresponding to each second sample query corresponding to the first format label, the total intra-batch loss value of this batch is determined.

[0180] like Figure 3 As shown, the electronic device initiates two processes to jointly train the embedding model. Process E samples two training samples (both of which are training samples of the retrieval format class) within this batch, namely training sample 1 (e.g., ...). Figure 3 This includes the first sample query q1 and the first sample document. and three first sample negative documents Training sample 2 (e.g.) Figure 3 This includes the second sample query q2, and the second sample positive document. and three second-sample negative documents Process F samples two training samples in this batch (neither of which are training samples of the retrieval format class), namely training sample 3 (e.g. Figure 3 This includes the second sample query q3, and the second sample positive document. and three second-sample negative documents Training sample 4 (e.g.) Figure 3 This includes the second sample query q4, and the second sample positive document. and three first sample negative documents ).

[0181] like Figure 3 As shown, it is necessary to use the first sample query q1 and the first sample positive document based on the in-batch loss function. Their respective text embeddings, and the second sample positive document corresponding to the second sample query q2. Text embedding, second sample query q3 corresponding to the second sample positive document Text embedding and the second sample positive document corresponding to the second sample query q4 The text embedding is used to determine the in-batch loss value corresponding to the first sample query q1; and it is necessary to use the second sample query q2 and the second sample positive document based on the in-batch loss function. Their respective text embeddings, and the first sample positive document corresponding to the first sample query q1. Text embedding, second sample query q3 corresponding to the second sample positive document Text embedding and the second sample positive document corresponding to the second sample query q4 The text embedding is used to determine the in-batch loss value corresponding to the second sample query q2.

[0182] Subsequently, the total intra-batch loss value of this batch is determined by averaging the intra-batch loss values ​​corresponding to q1 from the first sample query and q2 from the second sample query.

[0183] Then, in step 232, based on the hard-to-bear sample loss function, the hard-to-bear sample loss value corresponding to the first sample query is determined using the text embeddings of the first sample query and its corresponding positive and negative sample documents. The implementation process of step 232 can be found in the aforementioned process for determining the hard-to-bear sample loss value corresponding to the first sample query, and will not be repeated here.

[0184] Understandably, the electronic device (process E) also needs to query other samples in this batch, calculate the hard-to-bear sample loss value corresponding to other sample queries based on the hard-to-bear sample loss function; then, based on the hard-to-bear sample loss value corresponding to the first sample query and the hard-to-bear sample loss value corresponding to other sample queries, calculate the corresponding total hard-to-bear sample loss value.

[0185] Next, in step 233, a first loss value is determined based on the hard-to-bear sample loss value and the intra-batch loss value corresponding to the first sample query. In this step, the hard-to-bear sample loss value and the intra-batch loss value corresponding to the first sample query can be weighted and summed to obtain the first loss value. The first loss value is positively correlated with both the hard-to-bear sample loss value and the intra-batch loss value corresponding to the first sample query.

[0186] In some possible examples, the first loss value can be obtained by weighted summation of the total loss value within the batch and the total loss value of the difficult-to-bear samples mentioned above.

[0187] In the above example, the intra-batch loss is constructed only for the first format label (i.e., the training samples of the retrieval format class), which can better improve the quality of the text embedding obtained by the embedding model (i.e. improve the model training effect) and reduce the training cost to a certain extent.

[0188] In some other possible examples, the in-batch loss can also be calculated for training samples of each format class to jointly train the embedding model.

[0189] In some possible examples, if the first sample query corresponds to a non-first format label, that is, the training to which the first sample query belongs does not belong to the retrieval format class, then step 23 may include step 234:

[0190] In step 234, if the first sample query corresponds to a label that is not in the first format, based on the hard-to-bear sample loss function, the hard-to-bear sample loss value corresponding to the first sample query is constructed using the text embeddings of the first sample query and its corresponding positive and negative sample documents. The implementation process of step 234 can be found in the aforementioned implementation process for determining the hard-to-bear sample loss value corresponding to the first sample query, and will not be repeated here.

[0191] Then, in step 24, the embedding model is trained with the goal of minimizing the comprehensive attention loss value and minimizing the first loss value.

[0192] In this step, we can first calculate the product of the comprehensive attention loss value and its first weight, and the sum of the products of the first loss value and its second weight to obtain the total loss value; then, with the goal of minimizing this total loss value, we adjust the parameters of each network layer of the embedding model to train the embedding model.

[0193] In some possible examples, the electronic device can obtain the total loss values ​​calculated by each process for each training sample in the same batch, and then train the embedding model with the goal of minimizing the sum of all the total loss values.

[0194] The above only describes the process of training an embedding model using a single batch of training samples. Multiple batches of training samples can be used to perform multiple rounds of training iterations on the embedding model until a preset convergence condition is met, resulting in a fully trained embedding model. Subsequently, inference services are provided based on the trained embedding model.

[0195] The aforementioned preset convergence conditions may include, but are not limited to: the total loss value being lower than a specified threshold, the number of training iterations exceeding a specified number, and the training duration exceeding a specified duration.

[0196] During the inference phase, the target text to be inferred is obtained, and the target word sequence of the target text is input into the trained embedding model. The embedding model can obtain the target text embedding of the inferred target text based on the target word sequence through its network layers corresponding to multiple text granularities (which is the content corresponding to a word position in the output of the embedding model).

[0197] In a subsequent possible example, the embedding model can be followed by a trained linear module (e.g., consisting of sequentially set linear layers and activation functions) to perform downstream tasks using the target text embedding obtained from the embedding model. These downstream tasks can include, but are not limited to: text classification (e.g., sentiment classification and intent classification), text retrieval (recalling the most relevant documents from a large number of candidate paragraphs for a target text), text clustering (e.g., clustering papers by topic for a target text), etc.

[0198] In this embodiment, the mask matrices corresponding to each text granularity of the first input text are used to guide the network layers in the supervised embedding model to focus only on the content of the last word position in the corresponding text granularity unit in their input sequence. The text granularity is determined based on how humans understand language structure, with the goal of at least minimizing the attention loss of each network layer. The embedding model is trained so that it learns to imitate the hierarchical (i.e., different text granularities) understanding of language structure by humans and processes the input text in the same way as humans understand language structure, so as to obtain a text embedding that is more adapted to human language understanding and has higher accuracy.

[0199] Furthermore, based on the native causal attention of the embedding model, by constraining the internal attention flow of the model, it guides it to form a human-like hierarchical text perspective, focusing on low-level sub-words and words, and gradually aggregating information to high-level large-granularity units such as phrases and sentences. By directly applying mean squared error (MSE) loss to the model's attention values, the embedding model is trained so that during subsequent inference, there is no need to add an external mask matrix to the input inference text. The trained embedding module can autonomously master these hierarchical understandings of text structure, thus retaining the speed advantage of the underlying model in the inference stage without additional computational overhead.

[0200] The foregoing description describes specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments, and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily need to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0201] Corresponding to the above method embodiments, this specification provides an apparatus 400 for training an embedding model. The embedding model includes multiple sets of network layers corresponding to multiple text granularities, arranged sequentially. Each set of network layers includes several network layers, and each network layer includes an attention module. A schematic block diagram of the apparatus is shown below. Figure 4 As shown, it includes:

[0202] The first determining module 410 is configured to determine the first word sequence of the first input text and its corresponding mask matrices for each text granularity, wherein the first mask matrix corresponds to the first network layer of the first text granularity, and the first mask matrix is ​​used to indicate that the first network layer only focuses on the content of the last word position in each first granularity unit in its input sequence, the input sequence of the first network layer is obtained based on the first word sequence, and the first granularity unit is obtained by dividing the first word sequence with reference to the first text granularity;

[0203] The module 420 is configured to obtain the attention value matrix corresponding to the attention module of each network layer based on the first word sequence through multiple network layer sets corresponding to multiple text granularities;

[0204] The second determining module 430 is configured to determine the attention loss value of each network layer by using the product of the attention value matrix and the complement of the mask matrix corresponding to each network layer.

[0205] Training module 440 is configured to train the embedding model with the objective of at least minimizing the attention loss of each network layer.

[0206] In some possible examples, the text granularity includes at least two of the following: sub-word granularity, word granularity, phrase granularity, sentence granularity, and paragraph granularity.

[0207] In some possible examples, the element at the i-th row and j-th column position of the first mask matrix corresponds to the attention value at the i-th row and j-th column position of the first attention value matrix corresponding to the first network layer. If i is less than or equal to j, and the two word positions corresponding to the attention value at the i-th row and j-th column position of the first attention value matrix are both the last word position of the first text granularity unit in the input sequence of the first network layer, then the value of the element at the i-th row and j-th column position of the first mask matrix is ​​1; otherwise, it is 0.

[0208] In some possible examples, if the first input text is a sample query belonging to a specified task type, then its corresponding negative sample document includes: documents in a specified document pool whose similarity to the sample query meets a preset condition, the preset condition including at least: the similarity to the sample query is less than a first similarity threshold and less than a specified percentage of the similarity between the sample query and its corresponding positive sample document.

[0209] In some possible examples, the specified task type includes at least one of the following types: retrieval task, natural language inference task, summary generation task, and similar text identification task.

[0210] In some possible examples, the first input text is: any text in the first sample query and its corresponding first sample positive document and first sample negative document;

[0211] The training module 440 is specifically configured to determine a comprehensive attention loss value based on the total first attention loss value of the first sample query, the total second attention loss value of the first sample positive document, and the total third attention loss value of the first sample negative document. The comprehensive attention loss value is positively correlated with the total first attention loss value, the total second attention loss value, and the total third attention loss value. The total first attention loss value is determined based on the attention loss values ​​of each network layer corresponding to the first sample query, the total second attention loss value is determined based on the attention loss values ​​of each network layer corresponding to the first sample positive document, and the total third attention loss value is determined based on the attention loss values ​​of each network layer corresponding to the first sample negative document.

[0212] The embedding model is trained with the goal of minimizing the overall attention loss value.

[0213] In some possible examples, a third determining module (not shown in the figure) is also included, configured to determine the text embedding of the first input text from the output of the last network layer of the embedding model;

[0214] The fourth determining module (not shown in the figure) is configured to determine the first loss value by utilizing the text embeddings of the first sample query, the first sample positive document, and the first sample negative document.

[0215] The training module 440 is specifically configured to train the embedding model with the goal of minimizing the comprehensive attention loss value and minimizing the first loss value.

[0216] In some possible examples, the fourth determining module is specifically configured to determine the intra-batch loss value corresponding to the first sample query if the first sample query corresponds to the first format label, based on the intra-batch loss function, using the text embeddings of the first sample query and the first sample document respectively, and the text embedding of the second sample document corresponding to the second sample query, wherein the second sample query is data within the same batch as the first sample query.

[0217] Based on the hard-to-bear sample loss function, the hard-to-bear sample loss value corresponding to the first sample query is determined by utilizing the text embeddings of the first sample query, the first sample positive document, and the first sample negative document.

[0218] The first loss value is determined based on the loss value of the difficult-to-bear sample corresponding to the first sample query and the loss value within the batch.

[0219] In some possible examples, the fourth determining module is specifically configured to, if the first sample query corresponds to a label that is not in the first format, construct the hard-to-bear sample loss value corresponding to the first sample query based on the hard-to-bear sample loss formula, using the text embeddings of the first sample query and its corresponding positive and negative sample documents.

[0220] Based on the loss value of the hard-to-bear sample corresponding to the first sample query, the first loss value is determined.

[0221] In some possible examples, the first sample query and its corresponding positive and negative sample documents belong to a specified training set; it also includes:

[0222] The acquisition module (not shown in the figure) is configured to acquire initial data for each task type from multiple data sources before determining the first word sequence of the first input text and its corresponding mask matrices for each text granularity;

[0223] The fifth determination module (not shown in the figure) is configured to determine the training samples corresponding to each task type based on the task characteristics of the initial data of each task type, so as to obtain a specified training set. Each training sample includes a sample query and its corresponding positive sample document and at least one negative sample document.

[0224] The above-described apparatus embodiments correspond to the method embodiments, and detailed descriptions can be found in the description of the method embodiments section, which will not be repeated here. The apparatus embodiments are derived based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments; detailed descriptions can be found in the corresponding method embodiments.

[0225] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method for training an embedding model provided in this specification.

[0226] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method for training an embedding model provided in this specification.

[0227] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for storage media and computing devices are basically similar to the method embodiments, so they are described more simply; relevant parts can be referred to the descriptions of the method embodiments.

[0228] In this specification, the Large Language Model (LLM) may also be referred to simply as the Large Model. A Large Language Model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large Language Models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. Large Language Models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, giving them excellent generalization capabilities. The core capabilities of Large Language Models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the Large Language Model to generate specific outputs by designing prompts. Cue words can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of large language models. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0229] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0230] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025, public link: https: / / doi.org / 10.48550 / arXiv.2303.18223), and will not be repeated here.

[0231] Those skilled in the art will recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0232] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training an embedding model, wherein, The embedding model includes multiple sets of network layers corresponding to multiple text granularities, set sequentially. Each set of network layers includes several network layers, and each network layer includes an attention module. The method includes: A first word sequence of the first input text and its corresponding mask matrices for each text granularity are determined. The first mask matrix corresponds to the first network layer of the first text granularity. The first mask matrix is ​​used to indicate that the first network layer only focuses on the content of the last word position in each first granularity unit in its input sequence. The input sequence of the first network layer is obtained based on the first word sequence. The first granularity unit is obtained by dividing the first word sequence with reference to the first text granularity. By using multiple sets of network layers corresponding to multiple text granularities, and based on the first word sequence, the attention value matrix corresponding to the attention module of each network layer is obtained; The attention loss value of each network layer is determined by multiplying the attention value matrix and the complement of its mask matrix. The embedding model is trained with the objective of minimizing the attention loss of each network layer.

2. The method as described in claim 1, wherein, The text granularity includes at least two of the following: sub-word granularity, word granularity, phrase granularity, sentence granularity, and paragraph granularity.

3. The method as described in claim 1, wherein, The element at the i-th row and j-th column of the first mask matrix corresponds to the attention value at the i-th row and j-th column of the first attention value matrix corresponding to the first network layer. If i is less than or equal to j, and the two word positions corresponding to the attention value at the i-th row and j-th column of the first attention value matrix are both the last word position of the first text granularity unit in the input sequence of the first network layer, then the value of the element at the i-th row and j-th column of the first mask matrix is ​​1; otherwise, it is 0.

4. The method of claim 1, wherein, If the first input text is a sample query belonging to a specified task type, then its corresponding negative sample document includes: documents in the specified document pool whose similarity to the sample query meets preset conditions. The preset conditions include at least: the similarity to the sample query is less than a first similarity threshold and less than a specified percentage of the similarity between the sample query and its corresponding positive sample document.

5. The method of claim 4, wherein, The specified task type includes at least one of the following types: retrieval task, natural language reasoning task, summary generation task, and similar text identification task.

6. The method according to any one of claims 1-5, wherein, The first input text is: any text from the first sample query and its corresponding first sample positive document and first sample negative document; Training the embedding model includes: A comprehensive attention loss value is determined based on the first total attention loss value of the first sample query, the second total attention loss value of the first sample positive document, and the third total attention loss value of the first sample negative document. The comprehensive attention loss value is positively correlated with the first total attention loss value, the second total attention loss value, and the third total attention loss value. The first total attention loss value is determined based on the attention loss values ​​of each network layer corresponding to the first sample query, the second total attention loss value is determined based on the attention loss values ​​of each network layer corresponding to the first sample positive document, and the third total attention loss value is determined based on the attention loss values ​​of each network layer corresponding to the first sample negative document. The embedding model is trained with the goal of minimizing the overall attention loss value.

7. The method of claim 6, further comprising: The text embedding of the first input text is determined from the output of the last network layer of the embedding model; The first loss value is determined by using the text embeddings of the first sample query, the first sample positive document, and the first sample negative document. Training the embedding model includes: The embedding model is trained with the goal of minimizing the comprehensive attention loss value and minimizing the first loss value.

8. The method of claim 6, wherein, Determining the first loss value includes: If the first sample query corresponds to the first format label, based on the intra-batch loss function, the intra-batch loss value corresponding to the first sample query is determined by utilizing the text embeddings of the first sample query and the first sample document, as well as the text embedding of the second sample document corresponding to the second sample query, wherein the second sample query is the data within the same batch as the first sample query; Based on the hard-to-bear sample loss function, the hard-to-bear sample loss value corresponding to the first sample query is determined by utilizing the text embeddings of the first sample query, the first sample positive document, and the first sample negative document. The first loss value is determined based on the loss value of the difficult-to-bear sample corresponding to the first sample query and the loss value within the batch.

9. The method of claim 6, wherein, Determining the first loss value includes: If the first sample query corresponds to a label that is not in the first format, based on the hard-to-bear sample loss formula, the hard-to-bear sample loss value corresponding to the first sample query is constructed by using the text embedding of the first sample query and its corresponding positive and negative sample documents. Based on the loss value of the hard-to-bear sample corresponding to the first sample query, the first loss value is determined.

10. The method as described in claim 6, wherein the first sample query and its corresponding positive and negative sample documents belong to a specified training set; Before determining the first word sequence of the first input text and its corresponding mask matrices for each text granularity, the method further includes: Obtain initial data for each task type from multiple data sources; Based on the characteristics of the initial data for each task type, the training samples corresponding to each task type are determined to obtain a specified training set. Each training sample includes a sample query and its corresponding positive sample document and at least one negative sample document.

11. An apparatus for training an embedding model, wherein, The embedding model includes multiple sets of network layers corresponding to multiple text granularities, configured sequentially. Each set of network layers includes several network layers, and each network layer includes an attention module. The device includes: The first determining module is configured to determine the first word sequence of the first input text and its corresponding mask matrices for each text granularity, wherein the first mask matrix corresponds to the first network layer of the first text granularity, and the first mask matrix is ​​used to indicate that the first network layer only focuses on the content of the last word position in each first granularity unit in its input sequence, the input sequence of the first network layer is obtained based on the first word sequence, and the first granularity unit is obtained by dividing the first word sequence with reference to the first text granularity; The module is configured to obtain the attention value matrix corresponding to the attention module of each network layer based on the first word sequence through multiple network layer sets corresponding to multiple text granularities; The second determining module is configured to determine the attention loss value of each network layer by multiplying the attention value matrix and the complement of the mask matrix corresponding to each network layer. The training module is configured to train the embedding model with the objective of at least minimizing the attention loss of each network layer.

12. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-10.

Images