Training method and device of embedded model based on large model, equipment and medium

By using multiple coding sublayers in the embedded model to generate predicted feature vectors and calculate the loss value for training, the problem of low encoding accuracy of the embedded model in the vertical field is solved, and the coding accuracy and retrieval accuracy of the model are improved.

CN118917372APending Publication Date: 2024-11-08HISENSE GRP HLDG CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410865861.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-28
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the prior art, the encoding accuracy of embedded models in the vertical field is low during training, which affects the accuracy of retrieval.

Method used

Using the embedded model training method based on large models, the sample texts of a specific field are obtained for encoding, and the predicted feature vector is generated using multiple encoding sublayers, and the loss value is calculated based on the predicted feature vector and the standard feature vector for model training, and the different levels are fused. semantic features to improve coding accuracy.

Benefits of technology

It improves the encoding accuracy of the embedded model, enhances the text feature representation ability in specific fields, and improves the retrieval accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917372A_ABST
    Figure CN118917372A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of natural language processing, in particular to a training method and device of an embedded model based on a large model, equipment and a medium. A sample text in a specific field is input into an encoder of an initial embedding model to be encoded, the encoder comprises a plurality of encoding sub-layers, when the encoder performs encoding, a first feature vector output by each encoding sub-layer in the encoder is obtained, and according to a weight corresponding to each encoding sub-layer and the first feature vector, a first feature vector output by each encoding sub-layer in the encoder is obtained; the prediction feature vector of the sample text is determined, and the prediction feature vector is determined by fusing the first feature vectors of different coding sub-layers, so that the prediction feature vector is fused with semantic features of different levels, the coding accuracy is improved, and the coding efficiency is improved. The first loss value is determined according to the predicted feature vector and the standard feature vector of the sample text, and the embedding model obtained by training the initial embedding model according to the first loss value has higher accuracy during coding.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a training method, device, equipment and medium for an embedding model based on a large model. Background Art

[0002] In the engineering field of Retrieval-augmented Generation (RAG) models, open source embedding models are often used for construction, such as BGE, M3E (Multimodal Multitask Meta-Embedding), BCE and other models. This method depends on the size and quality of the corpus during training. Since the model training process is based on general sample data, it may not necessarily obtain good accuracy when embedding in vertical fields. At the same time, in RAG knowledge retrieval, the accuracy of retrieval is heavily dependent on the accuracy of the embedding encoding.

[0003] Therefore, how to improve the accuracy of Embedding model encoding has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a training method, device, equipment and medium for an embedding model based on a large model, so as to solve the problem of low accuracy of embedding model encoding in the prior art.

[0005] In a first aspect, the present application provides a training method for an embedding model based on a large model, wherein the initial embedding model includes an encoder, the encoder includes multiple encoding sublayers, and the method includes:

[0006] Obtaining a training set that is pre-saved for a specific field, wherein the training set includes sample texts;

[0007] Inputting the sample text into the encoder for encoding, and obtaining a first feature vector output by each encoding sublayer in the encoder;

[0008] Determining a predicted feature vector of the sample text according to the weight corresponding to each encoding sublayer and the first feature vector;

[0009] A first loss value is determined according to the predicted feature vector and the standard feature vector of the sample text, and the initial embedding model is trained according to the first loss value to obtain an embedding model.

[0010] In a second aspect, the present application provides a training device for an embedding model based on a large model, wherein the initial embedding model includes an encoder, the encoder includes multiple encoding sublayers, and the device includes:

[0011] An acquisition module, used to acquire a training set saved in advance for a specific field, wherein the training set includes sample texts;

[0012] An encoding module, used for inputting the sample text into the encoder for encoding, and obtaining a first feature vector output by each encoding sublayer in the encoder;

[0013] A determination module, used to determine the predicted feature vector of the sample text according to the weight corresponding to each encoding sublayer and the first feature vector;

[0014] A training module is used to determine a first loss value according to the predicted feature vector and the standard feature vector of the sample text, and train the initial embedding model according to the first loss value to obtain an embedding model.

[0015] In a third aspect, the present application also provides an electronic device, comprising a processor, wherein the processor is used to implement the steps of any of the above-mentioned methods for training an embedding model based on a large model when executing a computer program stored in a memory.

[0016] In a fourth aspect, the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for training an embedding model based on a large model.

[0017] Because in an embodiment of the present application, when training the embedding model, the sample text of the specific field is input into the encoder of the initial embedding model for encoding, and the encoder includes multiple encoding sublayers. When the encoder is encoding, the first feature vector output by each encoding sublayer in the encoder is obtained, and the predicted feature vector of the sample text is determined according to the weight and the first feature vector corresponding to each encoding sublayer. Since the predicted feature vector is determined by fusing the first feature vectors of different encoding sublayers, the predicted feature vector is a fusion of semantic features of different levels, which improves the encoding accuracy, and the first loss value is determined according to the predicted feature vector and the standard feature vector of the sample text. The embedding model obtained by training the initial embedding model according to the first loss value has a higher accuracy rate during encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1A flowchart of a training process of an embedding model based on a large model provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of an encoder structure provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of a model input provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of the structure of an embedded model provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of training an embedding model provided in an embodiment of the present application;

[0024] Figure 6 A similarity diagram provided for an embodiment of the present application;

[0025] Figure 7 A schematic diagram of a fine-tuning training architecture provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of a fine-tuning training architecture provided in an embodiment of the present application;

[0027] Fig. 9 A schematic diagram of similar text determination provided in an embodiment of the present application;

[0028] Fig.10 A schematic diagram of similar text determination provided in an embodiment of the present application;

[0029] Fig.11 A schematic diagram of generating a question and summary text provided in an embodiment of the present application;

[0030] Fig.12 A schematic diagram of generating a reply message provided in an embodiment of the present application;

[0031] Fig.13 A schematic diagram of the structure of a training device for an embedded model based on a large model provided in an embodiment of the present application;

[0032] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0034] The embodiments of the present application provide a training method, apparatus, device and medium for an embedding model based on a large model. In the method, the initial embedding model includes an encoder, which includes multiple encoding sublayers. During training, a training set saved in advance for a specific field is obtained, and the training set includes sample text; the sample text is input into the encoder for encoding, and a first feature vector output by each encoding sublayer in the encoder is obtained; the predicted feature vector of the sample text is determined according to the weight corresponding to each encoding sublayer and the first feature vector; the first loss value is determined according to the predicted feature vector and the standard feature vector of the sample text, and the initial embedding model is trained according to the first loss value to obtain the embedding model.

[0035] Figure 1 A flowchart of a training process of an embedding model based on a large model provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the process includes the following steps:

[0036] S101: Acquire a training set that is pre-saved for a specific field, wherein the training set includes sample texts.

[0037] The large-model-based embedding model training method provided in the present application is applied to an electronic device, which may be a server, a PC, a mobile terminal, etc.

[0038] In order to obtain an embedding model with higher accuracy, in an embodiment of the present application, a training set is pre-configured, and the initial embedding model is trained according to the sample text included in the training set to obtain the embedding model. In order for the trained embedding model to have better encoding capabilities in a specific field, in an embodiment of the present application, the pre-configured training set can be pre-configured for the specific field. In other words, the sample texts saved in the training set all belong to the same field. Optionally, the specific field can be the field of education, the field of transportation, the field of mobile phones, the field of computers, etc., and technicians in this field can configure it as needed, and the embodiment of the present application does not limit this.

[0039] S102: Input the sample text into the encoder for encoding, obtain a first feature vector output by each encoding sublayer in the encoder, and determine a predicted feature vector of the sample text according to a weight corresponding to each encoding sublayer and the first feature vector.

[0040] In order to train the initial embedding model, in an embodiment of the present application, after obtaining the training set, the sample text can be input into an encoder for encoding. The encoder is used to generate a feature vector of the sample text. Exemplarily, the encoder can be a BERT (Bidirectional Encoder Representations from Transformers) model. When encoding, the encoder can randomly mask the sample text, such as performing a 30% mask, and the encoder then predicts the masked part to obtain a vector representation of the sample text.

[0041] In an embodiment of the present application, the encoder may include multiple encoding sublayers, each encoding sublayer can be regarded as an encoder, for example, the encoder of the embedded model is composed of a 12-layer Transformer Encoder.

[0042] The feature vector obtained by the conventional BERT model is often a 768-dimensional feature vector determined by CLS encoding (Classification Encoding) or by taking the average of each word encoding. However, through analysis, in the field of image feature extraction, the feature information obtained by the fusion of the deep and shallow layers of the network is more conducive to improving the accuracy of model detection, and can better focus on the shallow information of the image, such as lines, shapes, etc. Correspondingly, in the field of semantic feature extraction of text, the understanding of each layer of text in the model should also be different. Therefore, in an embodiment of the present application, during the encoding process, the first feature vector output by each encoding sub-layer in the encoder can be obtained.

[0043] For a model, the deeper the model, the stronger the semantic information it extracts, and the more important it is for the semantic feature representation of the text. However, the shallow network may pay more attention to the detailed information such as the part of speech of each word. Therefore, although the design should pay more attention to the deep features, it should also be combined with the shallow detailed information, especially for the feature representation of the text. Based on this, after obtaining the first feature vector output by each encoding sublayer, the predicted feature vector of the sample text can be determined according to the weight corresponding to each sublayer and the first feature vector. In an embodiment of the present application, the weight corresponding to each encoding sublayer can be the same or different.

[0044] Figure 2 A schematic diagram of an encoder structure provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the encoder includes 6 coding sublayers. Figure 2Each DECODER_2 shown on the left side (left and right in the figure) represents each encoding sublayer. First, input the sample text into the first encoding sublayer to obtain the first feature vector output by the first encoding sublayer; then input the feature vector into the second encoding sublayer to obtain the first feature vector output by the second encoding sublayer; then input the first feature vector into the third encoding sublayer to obtain the first feature vector output by the third encoding sublayer, and so on, until the first feature vector output by the last encoding sublayer is obtained. Figure 2 The figure composed of five circles in the figure represents the first eigenvector. After obtaining each first eigenvector, the predicted eigenvector of the sample text can be determined according to the weight corresponding to each encoding sublayer and the first eigenvector. Figure 2 The circle diagram on the right side (left and right in the diagram) represents the predicted feature vector. In the embodiment of the present application, the weight corresponding to the shallow coding sublayer is small, and the weight corresponding to the deeper coding sublayer is large. Figure 2 As shown, the weight corresponding to the first coding sublayer is 0.00001, the weight corresponding to the second coding sublayer is 0.0001, the weight corresponding to the third coding sublayer is 0.001, the weight corresponding to the fourth coding sublayer is 0.01, the weight corresponding to the fifth coding sublayer is 0.1, and the weight corresponding to the sixth coding sublayer is 1. For ease of understanding, the predicted feature vector of the sample text can be determined based on the following formula:

[0045] x=r n-2 x1+r n-3 x2+…+rx n-1 +x n

[0046] Where x represents the predicted feature vector, n represents the number of coding sublayers included in the encoder, x1 represents the first coding sublayer, x2 represents the second coding sublayer, and x n represents the nth coding sublayer, r represents the weight coefficient, and those skilled in the art can set the value as needed, for example, r=0.1.

[0047] S103: Determine a first loss value according to the predicted feature vector and the standard feature vector of the sample text, and train the initial embedding model according to the first loss value to obtain an embedding model.

[0048] After determining the predicted feature vector of the sample text, a first loss value may be determined to train the initial embedding model according to the first loss value. When determining the first loss value, it may be determined according to the predicted feature vector and the standard feature vector of the sample text. In order to facilitate model understanding, when the sample text is input into the encoder, the sample text may be converted into a word encoding. Figure 3A schematic diagram of a model input provided in an embodiment of the present application, such as Figure 3 The input of the model shown may include three parts: word encoding, context encoding and position encoding, wherein the word encoding is determined based on each character in the sample text, the context encoding is used to identify whether the corresponding characters belong to the same sentence, and the position encoding is used to identify the position of the corresponding characters in the sample text. The feature vector obtained by the encoding conversion can be used as a standard feature vector. The standard feature vector is then masked, and the encoder processes the masked standard feature vector to obtain a predicted feature vector. Thus, the first loss value is determined based on the deviation between the predicted feature vector and the standard feature vector. It should be noted that how to determine the predicted feature vector and the standard feature vector of the sample text, and how to determine the first loss value are prior art and will not be repeated here.

[0049] After the first loss value is determined, the initial embedding model can be trained according to the first loss value to obtain the embedding model, and the obtained embedding model can be used to determine any text feature vector.

[0050] Because in an embodiment of the present application, when training the embedding model, the sample text of the specific field is input into the encoder of the initial embedding model for encoding, and the encoder includes multiple encoding sublayers. When the encoder is encoding, the first feature vector output by each encoding sublayer in the encoder is obtained, and the predicted feature vector of the sample text is determined according to the weight and the first feature vector corresponding to each encoding sublayer. Since the predicted feature vector is determined by fusing the first feature vectors of different encoding sublayers, the predicted feature vector is a fusion of semantic features of different levels, which improves the encoding accuracy, and the first loss value is determined according to the predicted feature vector and the standard feature vector of the sample text. The embedding model obtained by training the initial embedding model according to the first loss value has a higher accuracy rate during encoding.

[0051] In order to further improve the accuracy of the embedded model obtained by training, on the basis of the above embodiment, in the embodiment of the present application, the initial embedded model also includes a decoder, and after determining the predicted feature vector of the sample text and before training the initial embedded model according to the first loss value, the method further includes:

[0052] Inputting the predicted feature vector into the decoder for decoding to obtain the predicted text output by the decoder;

[0053] Determine a second loss value according to the predicted text and the sample text;

[0054] The step of training the initial embedding model according to the first loss value to obtain the embedding model comprises:

[0055] The initial embedding model is trained according to the first loss value and the second loss value to obtain the embedding model.

[0056] In order to further improve the accuracy of the trained embedding model, the initial embedding model can also include a decoder during model training. The decoder is responsible for using the vector to restore the original sample text. The decoder consists of only one layer of Transformer decoder. In the decoder part, the sentence is further noised, and the mask ratio can be 70%. The complexity of the task is increased by increasing the mask ratio. The decoder has a simple structure, with only a single layer of Transformer, and the mask rate of the input sentence is very high, which makes decoding challenging and forces the encoder to generate high-quality prediction feature vectors.

[0057] In an embodiment of the present application, the predicted feature vector determined by the decoder can be input into the decoder for decoding, thereby obtaining the predicted text output by the decoder. The second loss value is determined based on the predicted text and the sample text. After the second loss value is determined, the initial embedding model can be trained based on the sum of the first loss value and the second loss value to obtain the embedding model.

[0058] Specifically, the sum of the first loss value and the second loss value can be determined as the target loss value, so as to train the initial embedding model according to the target loss value. Of course, the product value, quotient value, etc. of the first loss value and the second loss value can also be determined as the target loss value, and those skilled in the art can set it as needed.

[0059] In the embodiment of the present application, the training task of the entire model is: in the Encoder stage, the masked part is reconstructed, that is, Masked Language Modeling (MLM), and in the Decoder stage, the entire sample text is reconstructed. The overall loss function is:

[0060] Loss = L encode +L encode

[0061] Among them, Loss represents the target loss value, based on which the initial embedding model can be trained; L encode represents the first loss value, that is, the loss value of the encoder; L encode Represents the second loss value, that is, the loss value of the decoder.

[0062] The embodiment of the present application is improved based on the RetroMAE structure. The starting point of the improvement is to enrich the vector information of the text. The hidden state vector of the CLS token as a semantic representation can also be enriched based on different layers, taking into account the encoding information of each layer and the token information of the last layer. Among them, the RetroMAE structure is a new retrieval-oriented pre-training paradigm based on Masked Auto-Encoder (MAE). The training process of the embedding model is explained in conjunction with a specific embodiment. Figure 4 A schematic diagram of the structure of an embedded model provided in an embodiment of the present application is shown in FIG. Figure 4 As shown in Figure 2, the embedding model consists of two parts: Figure 4 The left side (left and right in the diagram) is the encoder, and the right side (left and right in the diagram) is the decoder. The encoder is used to generate a predicted feature vector, and the decoder is responsible for using the vector to restore the original sample text. In this embodiment of the present application, the sample text is "ABCDEFG". In the Encoder stage, the sample text is masked by 30%, and the predicted feature vector is obtained through the Encoder. The predicted feature vector is determined based on the word encoding (token) of each word output by the Encoder and the first feature vector output by each coding sublayer. In other words, the predicted feature vector is determined based on the deeply fused CLS and word token vectors. After obtaining the predicted feature vector, the predicted feature vector is input into the Decoder part. In the Decoder part, the sample text is further noise-enhanced, and the mask ratio is 70%. The complexity of the task is increased by increasing the mask ratio, such as Figure 4 As shown, the sample text of the mask is [M][M]C[M]EF[M].

[0063] The training process of the embedding model is described below in conjunction with another specific embodiment. The storage format of the training set can be unlabeled text fragments, such as:

[0064] [{

[0065] "text": "fragment"

[0066] }]

[0067] During the training phase, training can be performed based on the stored text dataset, so that the resulting embedding model has powerful representation capabilities. Figure 5 A training diagram of an embedding model provided in an embodiment of the present application, such as Figure 5 As shown, the text in the training set is input into the initial embedding model Model to obtain the output data of the model. The structure of the Model can be Figure 4The structure shown in .

[0068] In order to further improve the accuracy of the embedded model encoding, based on the above embodiments, in the embodiment of the present application, the method further includes:

[0069] Acquire any sample text in the training set as a first fine-tuned text, a first similar text corresponding to the first fine-tuned text, and a first non-similar text corresponding to the first fine-tuned text;

[0070] Inputting the first fine-tuned text, the first similar text, and the first non-similar text into the embedding model respectively to obtain each second feature vector;

[0071] Determine a third loss value according to a first similarity between the first fine-tuned text and the second feature vector corresponding to the first similar text, and a second similarity between the first fine-tuned sample and the second feature vector corresponding to the first non-similar text;

[0072] The embedding model is fine-tuned according to the third loss value.

[0073] In order to further improve the accuracy of the embedding model encoding, in an embodiment of the present application, the trained embedding model can also be fine-tuned. During the fine-tuning training, any sample text can be obtained in the training set as the first fine-tuning text, the first similar text corresponding to the first fine-tuning text, and the first non-similar text corresponding to the first fine-tuning text. In other words, in order to enhance the generalization of the model, there are some similar texts in the texts saved in the training set. The similar text of any fine-tuning text can be one or more, and similarly, the non-similar text can be one or more. Among them, similar text can be understood as a positive sample of the fine-tuning text, and non-similar text can be understood as a negative sample of the fine-tuning text.

[0074] Subsequently, comparative learning can be performed based on the first fine-tuned text, the first similar text, and the first non-similar text. In the comparative learning process, the larger the batch size, the more negative samples will be seen in a batch, so the model will learn more accurately. Therefore, the batch size setting can be increased in this training phase. Batchsize is an important parameter that determines the amount of data used in each training iteration.

[0075] The first fine-tuned text, the first similar text, and the first non-similar text obtained are respectively input into the embedding model to obtain each second feature vector. In the embodiment of the present application, the embedding model for fine-tuning training only includes the encoder at this time. Therefore, the obtained second feature vector is a feature vector comprehensively determined according to the weights corresponding to each encoding sublayer and the output feature vector.

[0076] After obtaining each second feature vector, the first similarity between the first fine-tuned text and the second feature vector corresponding to the first similar text can be determined, that is, the first similarity between the first fine-tuned sample and the positive sample can be determined. At the same time, the second similarity between the first fine-tuned sample and the second feature vector corresponding to the first non-similar text is determined, that is, the second similarity between the first fine-tuned sample and the negative sample is determined. After determining the first similarity and the second similarity, the third loss value can be determined based on the first similarity and the second similarity, so that the obtained embedding model can be fine-tuned and trained based on the third loss value.

[0077] Specifically, the third loss value can be calculated based on the following formula:

[0078]

[0079] Wherein, p represents the first fine-tuning text; q represents the first similar text; q' represents the first non-similar text; T represents temperature; Represents the similarity between the first fine-tuned text and the second feature vector corresponding to the first similar text; represents the similarity between the second feature vectors corresponding to the first fine-tuned text and the first non-similar text; min·· represents the expected calculated third loss value to be as small as possible; Q' represents the set of text pairs consisting of the first fine-tuned text and each first non-similar text.

[0080] According to the above formula, the numerator represents the similarity between the first fine-tuned text and the positive sample, and the denominator represents the similarity between the first fine-tuned text and the negative sample. Therefore, it can be seen that the greater the similarity of the same category, the smaller the similarity of different categories, and the smaller the loss value. Therefore, the goal of the above formula is to make the numerator as large as possible and the denominator as small as possible.

[0081] Combine the following Figure 6 Explain the construction of positive and negative sample pairs. Figure 6 A similarity diagram provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, T1, T2, T3, ..., T N , I1, I2, I3, …, I N, respectively represent the second feature vectors obtained after different texts are processed by the embedding model. I1T1 represents the similarity between the second feature vector I1 and the second feature vector T1, I2T1 represents the similarity between the second feature vector I2 and the second feature vector T1, and so on. The similarity may be cosine similarity. Figure 6 The cosine similarity of the main diagonal is as large as possible, and the cosine similarity of other places is as small as possible. The diagonal is the two matching sentence pairs, which constitute a positive sample pair. For example, the first fine-tuned text and the first similar text can be used as a positive sample pair. The non-diagonal constitutes a negative sample pair, which is two dissimilar sentence pairs. For example, the first fine-tuned text and the first dissimilar text can be used as a negative sample pair.

[0082] The following is an explanation of the fine-tuning training process of the embedding model with reference to a specific embodiment. Figure 7 A schematic diagram of a fine-tuning training architecture provided in an embodiment of the present application, such as Figure 7 As shown in the figure, during the fine-tuning training process, in order to improve the efficiency of model training, two embedding models can be used to process the text. Figure 7 The structures and parameters of the two models shown in are the same. Figure 7 The model on the left side (left and right in the figure) is used to determine the second feature vector of the Text, and the Text can be understood as the first fine-tuning sample described in the above embodiment. Figure 7 The model on the right side (left and right in the figure) is used to determine the second feature vector of Page, which can be understood as the first similar text and the first non-similar text in the above embodiments. After obtaining the second feature vector output by each model, comparative learning can be performed based on each second feature vector obtained.

[0083] In order to further improve the accuracy of the embedded model encoding, based on the above embodiments, in the embodiment of the present application, the method further includes:

[0084] Obtain any sample in the training set as a second fine-tuning text, a second similar text corresponding to the second fine-tuning sample, and a second non-similar text corresponding to the second fine-tuning text;

[0085] Inputting the second fine-tuned text, the second similar text, and the second non-similar text into the embedding model respectively to obtain each third feature vector;

[0086] Determining a third similarity of a third feature vector between the second fine-tuned sample and the second similar text, and a fourth similarity of a third feature vector between the second fine-tuned sample and the second non-similar text;

[0087] Determining a fourth loss value according to a difference between the third similarity and the fourth similarity and an activation function;

[0088] Fine-tune the embedding model according to the fourth loss value.

[0089] In order to further improve the accuracy of the embedding model encoding, in an embodiment of the present application, the trained embedding model can also be fine-tuned with respect to a specific task. During the fine-tuning training process, any sample text can be obtained in the training set as the second fine-tuning text, the second similar text corresponding to the second fine-tuning text, and the second non-similar text corresponding to the second fine-tuning text. The acquired embedding model can be subsequently trained based on the acquired second fine-tuning text, the second similar text, and the second non-similar text. In an embodiment of the present application, the trained embedding model can be a model obtained through pre-training, or it can be a model obtained by performing general fine-tuning training on the pre-trained model again.

[0090] When the model is trained, the second fine-tuning text, the second similar text, and the second non-similar text can be respectively input into the embedding model to obtain each third feature vector. In the embodiment of the present application, the embedding model for fine-tuning training only includes the encoder at this time. Therefore, the third feature vector obtained is a feature vector comprehensively determined according to the weights corresponding to each encoding sublayer and the output feature vector.

[0091] After obtaining each third feature vector, the third similarity between the third feature vectors corresponding to the second fine-tuned text and the second similar text can be determined, that is, the third similarity between the second fine-tuned sample and the positive sample can be determined. At the same time, the fourth similarity between the third feature vectors corresponding to the second fine-tuned sample and the second non-similar text can be determined, that is, the fourth similarity between the second fine-tuned sample and the negative sample can be determined.

[0092] After the third similarity and the fourth similarity are determined, the fourth loss value can be determined by the difference between the third similarity and the fourth similarity and the activation function, so that the obtained embedding model can be fine-tuned according to the fourth loss value.

[0093] Specifically, the fourth loss value can be calculated based on the following formula:

[0094] sim_pos=cosine(text,positive)

[0095] sim_neg=cosine(text,negative_text)

[0096] sim_diff=sigmoid(sim_pos-neg_pos)

[0097] loss = log(sim_diff)

[0098] Among them, sim_pos represents the third similarity; text represents the third feature vector of the second fine-tuned text; positive represents the third feature vector of the second similar text; cosine(text,positive) represents the cosine similarity between the third feature vector of the second fine-tuned text and the third feature vector of the second similar text; sim_neg represents the fourth similarity; negative_text represents the third feature vector of the second non-similar text; cosine(text,negative_text) represents the cosine similarity between the third feature vector of the second fine-tuned sample and the third feature vector of the second non-similar text; sigmoid() represents the activation function; sim_diff represents the mapped value; loss represents the fourth loss value.

[0099] The loss function provided in the embodiment of the present application can make the model converge more easily and greatly improve the model effect.

[0100] In addition, since the positive and negative samples between texts can be determined in this application, this part of the training belongs to labeled training and can be regarded as a common classification task. In the embodiment of this application, the cross entropy loss can also be used for training.

[0101] In an embodiment of the present application, the third similarity or fourth similarity of the third feature vector between the second fine-tuning sample and the second similar text or the second non-similar text can be determined. Thereby, the probability value corresponding to the classification of "similar" and the probability value corresponding to the classification of "dissimilar" are determined based on the third similarity or the fourth similarity. Since whether the corresponding texts are similar is known in advance, that is, the standard classification of the corresponding texts is known, therefore, in an embodiment of the present application, the fourth loss value can be determined based on the probability value corresponding to the standard classification. Specifically, the fourth loss value can be calculated with reference to the following formula:

[0102] L model =-p t log(p t )

[0103] Among them, L model represents the fourth loss value; p t Represents the probability value corresponding to the standard classification.

[0104] p t The calculation method is as follows:

[0105]

[0106] Among them, y=1 represents the corresponding probability value when the predicted classification result is the standard classification.

[0107] The following is an explanation of the fine-tuning training process of the embedding model with reference to a specific embodiment. Figure 8 A schematic diagram of a fine-tuning training architecture provided in an embodiment of the present application, such as Figure 8 As shown in the figure, during the fine-tuning training process, in order to improve the efficiency of model training, two embedding models can be used to process the text. Figure 8 The structures and parameters of the two models shown in are the same. Figure 8 The model on the left side (left and right in the figure) is used to determine the third feature vector of the text, and the text can be understood as the second fine-tuning sample described in the above embodiment. Figure 8 The model on the right side (left and right in the figure) is used to determine the second feature vectors of Pos and Neg, where Pos can be understood as the second similar text in the above embodiments, and Neg can be understood as the second non-similar text in the above embodiments. After obtaining the third feature vector output by each model, the fourth loss value can be calculated based on each third feature vector obtained and the calculation formula of the fourth loss value provided in the above embodiments, so as to perform fine-tuning training according to the fourth loss value.

[0108] In order to further improve the accuracy of embedding model encoding, based on the above embodiments, in the embodiment of the present application, the process of determining a similar text corresponding to any fine-tuning text includes:

[0109] Acquire similar text pairs generated based on the large model, and determine the similar text pairs containing the fine-tuned text as target text pairs;

[0110] Determine a fourth feature vector of each text of the target text pair based on other pre-trained embedding models;

[0111] According to the fifth similarity between each fourth feature vector and a preset threshold, it is determined whether the non-fine-tuning text in the target text pair is a similar text to the fine-tuning text.

[0112] In the embodiment of the present application, when determining similar texts corresponding to any fine-tuned text, similar text pairs generated based on the large model can be obtained, and the similar text pairs containing the fine-tuned text can be determined as target text pairs. The non-fine-tuned texts included in the target text pairs are all texts related to the fine-tuned text.

[0113] After the target text pair is determined, the fourth feature vector of each text in the target text pair can be determined based on other pre-trained embedding models, so as to determine whether the non-fine-tuning text in the target text pair is a similar text to the fine-tuning text based on the fifth similarity between each fourth feature vector and a preset threshold. In an embodiment of the present application, the fourth feature vector of each text can be determined multiple times based on multiple other embedding models, so as to perform multiple fifth similarity calculations, and then determine multiple times whether the non-fine-tuning text in the target text pair is a similar text to the fine-tuning text. In other words, multiple other embedding models are used to make multiple judgments.

[0114] Specifically, when the fine-tuning text is question 1, the determined target text pair can be question 1-summary, question 1-answer, question 1-fragment, etc. The fourth feature vector of each text in the target text pair can be determined based on the M3E (Moka Massive Mixed Embedding), BGE, and Text2Vec models. The fifth similarity between the corresponding fourth feature vectors in each target text pair is calculated, and then according to the fifth similarity and the preset threshold, it is determined whether the non-fine-tuning text in the target text pair is a similar text to the fine-tuning text. Among them, the preset threshold can be any decimal, such as 0.7.

[0115] Fig. 9 A similar text determination schematic diagram provided in an embodiment of the present application, such as Fig. 9 As shown, for any text pair, the fourth feature vector of each text included in the text pair is determined based on three other embedding models, namely, M3E, BGE, and Text2Vec models. Hard voting is performed after the fourth feature vector is determined. In the hard voting stage, for each other embedding model, the fifth similarity between the fourth feature vectors output by the other embedding model can be determined, and it can be determined whether the fifth similarity is greater than a preset threshold. If so, it is determined that the texts in the text pair are similar texts to each other.

[0116] When any two of the three other models determine that the texts in the corresponding text pair are similar texts to each other, it can be determined that the texts in the text pair are indeed similar.

[0117] In order to further improve the accuracy of embedding model encoding, on the basis of the above embodiments, in the embodiment of the present application, after determining the similar text pair containing the fine-tuning text as the target text pair, and before determining each fourth feature vector of the target text pair based on other pre-trained embedding models, the method further includes:

[0118] Based on the fourth largest model, determine whether the texts in the target text pair are similar. If so, continue to perform subsequent steps based on other pre-trained embedding models to determine each fourth feature vector of the target text pair.

[0119] Since the model training is in the final stage, the quality requirements for the data in the training set are higher. Therefore, in the embodiment of the present application, a more demanding method can be used to determine the sample text. After determining the target text pair, before determining each fourth eigenvector, it is also possible to determine whether the texts in the target text pair are similar based on the fourth largest model. If so, continue to perform the subsequent steps of determining each fourth eigenvector of the target text pair based on other embedded models completed in advance. In other words, for any target text pair, first use the prompt text method based on the fourth largest model to determine whether the two belong to similar texts. Only after the fourth largest model is verified, it is further determined based on other embedded models.

[0120] Fig.10 A schematic diagram of similar text determination is provided for an embodiment of the present application. In the embodiment of the present application, a text pair may include a question-summary; a question-answer; a question-fragment; a summary-answer; a summary-fragment; an answer-fragment. For any text pair, a large model judgment is performed based on three other embedding models, namely, M3E, BGE, Text2Vec model and the fourth large model, to determine the fourth feature vector of each text included in the text pair. After the fourth feature vector is determined, a hard vote is performed. In the hard voting stage, for each other embedding model, the fifth similarity between the fourth feature vectors output by the other embedding model can be determined, and it can be determined whether the fifth similarity is greater than a preset threshold. If so, it is determined that the texts in the text pair are similar texts to each other. At the same time, the result of the large model judgment is obtained. If the verification results of the above four models are all passed, it can be determined that the texts in the text pair are indeed similar.

[0121] In addition, in the embodiment of the present application, when determining the non-similar text of any fine-tuned text, the bge, m3e, and text2vec basic models can be used to vectorize all texts and calculate the similarity between the fine-tuned text and each text. Texts with a similarity range of 0.3 to 0.6 are determined as non-similar texts, i.e., negative samples. Samples with a similarity range of 0.6 to 0.7, or texts that are considered relevant by the large model but have a similarity of less than 0.5 are determined as difficult samples, which can also be used as non-similar texts.

[0122] In order to further improve the accuracy of the embedded model encoding, on the basis of the above embodiments, in the embodiment of the present application, the process of determining the similar text pairs generated based on the large model includes:

[0123] Generate associated text corresponding to the text segment obtained based on the large model, where the associated text includes at least one of the question of the text segment, the overview text of the text segment, and the reply information corresponding to the question of the text segment;

[0124] Randomly select any two texts from the text segment and the associated text as a pair of similar texts.

[0125] In the embodiments of the present application, when generating a pair of similar texts, associated text corresponding to the text segment obtained based on the large model can be generated. The associated text includes at least one of the question of the text segment, the overview text of the text segment, and the reply information corresponding to the question of the text segment. Among them, the text segment can be a text segment obtained by segmenting a pre-collected document. Specifically, the langchain framework can be used to perform fragment cutting on PDF and txt, and the markDown document can be segmented according to the title. The langchain framework is a programming framework that helps to use large language models (LLMs) in applications.

[0126] Specifically, when generating the question of the text segment based on the large model, the segmented text segment and a pre-designed prompt text can be input into the large model, and the large model is allowed to generate the corresponding question according to the text segment. The large model can be the Chatgpt (Chat Generative Pre-trained Transformer) large model. Among them, the input data of the large model can be:

[0127] '<Instruction>You are a Q&A assistant. Your current task is to generate corresponding questions based on the known information. If no question can be extracted from it, say "No corresponding question can be generated based on the known information",'

[0128] 'Do not fabricate components for the known fragment information, and the generation of questions should use Chinese. < / Instruction>\n'

[0129] '<Known information>{{context}}< / Known information>\n'

[0130] '<Question>{{question}}< / Question>\n'.

[0131] When generating the overview text of the text segment based on the large model, the input data of the large model can be:

[0132] '<Instruction>You are a Q&A assistant. Your current task is to generate a summary based on the known information using your expertise. If no summary can be obtained from it, say "No corresponding summary can be generated based on the known information",'

[0133] 'It is not allowed to generate fabricated components based on the fragments. Please use Chinese for the answers.< / Instruction>\n'

[0134] '<Known Information>{{context}}< / Known Information>\n'

[0135] '<Abstract>{{abstrast}}< / Abstract>\n'.

[0136] Fig.11 A schematic diagram for generating a problem and an overview text provided by an embodiment of the present application is as follows Fig.11 shown. When generating the problem and the abstract of the article fragment, that is, the segmented text segment, based on the large model, the pre-designed prompt words (i.e., prompt text) and the article fragment can be input into the GPT large model to obtain the problem and the abstract of the article fragment. Among them, the abstract is the overview text described in the above embodiment.

[0137] After generating the problem of the text segment, it is also possible to deliberately generate the response information corresponding to the problem of the text segment based on the large model. At this time, the input data of the large model can be:

[0138] '<Instruction>You are a Q&A assistant. Your current task is to answer questions based on the known information using your expertise. If you cannot obtain the answer from it, please say "The question cannot be answered based on the known information",'

[0139] 'It is not allowed to add fabricated components to the answer. Please use Chinese for the answer.< / Instruction>\n'

[0140] '<Known Information>{{top-k_context}}< / Known Information>\n'

[0141] '<Answer>{{answer}}< / Answer>\n'

[0142] The large model will reply to the corpus question based on the above input data.

[0143] Fig.12 A schematic diagram for generating response information provided by an embodiment of the present application is as follows Fig.12 shown. When generating the response information of the problem of the article fragment, that is, the segmented text segment, based on the large model, the problem and the article fragment can be input into the GPT large model to obtain the response information of the problem of the article fragment.

[0144] After determining the associated text corresponding to the text segment, any two texts can be randomly selected from the text segment and the associated text as a similar text pair. For example, the similar text pair can be problem - overview text; problem - response information; problem - text segment; overview text - response information; overview text - text segment; response information - text segment.

[0145] Based on the same inventive concept, the embodiment of the present application also provides a training device for an embedded model based on a large model. Fig.13 A schematic diagram of the structure of a training device for an embedding model based on a large model provided in an embodiment of the present application, wherein the initial embedding model includes an encoder, the encoder includes multiple encoding sublayers, and the device includes:

[0146] An acquisition module 1301 is used to acquire a training set that is pre-stored for a specific field, wherein the training set includes sample texts;

[0147] An encoding module 1302 is used to input the sample text into the encoder for encoding, and obtain a first feature vector output by each encoding sublayer in the encoder;

[0148] A determination module 1303, configured to determine a predicted feature vector of the sample text according to the weight corresponding to each coding sublayer and the first feature vector;

[0149] The training module 1304 is used to determine a first loss value according to the predicted feature vector and the standard feature vector of the sample text, and train the initial embedding model according to the first loss value to obtain an embedding model.

[0150] In a possible implementation, the initial embedding model further includes a decoder, and the device further includes:

[0151] A decoding module 1305, configured to input the predicted feature vector into the decoder for decoding, and obtain a predicted text output by the decoder;

[0152] The training module 1304 is further used to determine a second loss value based on the predicted text and the sample text; and train the initial embedding model based on the first loss value and the second loss value to obtain the embedding model.

[0153] In a possible implementation, the acquisition module 1301 is further configured to acquire any sample text in the training set as a first fine-tuned text, a first similar text corresponding to the first fine-tuned text, and a first non-similar text corresponding to the first fine-tuned text;

[0154] The encoding module 1302 is further used to input the first fine-tuning text, the first similar text and the first non-similar text into the embedding model respectively to obtain each second feature vector;

[0155] The training module 1304 is also used to determine a third loss value based on a first similarity between the second feature vector corresponding to the first fine-tuning text and the first similar text, and a second similarity between the first fine-tuning sample and the second feature vector corresponding to the first non-similar text; and fine-tune the embedding model based on the third loss value.

[0156] In a possible implementation, the acquisition module 1301 is further configured to acquire any sample in the training set as the second fine-tuning text, the second similar text corresponding to the second fine-tuning sample, and the second non-similar text corresponding to the second fine-tuning text;

[0157] The encoding module 1302 is further used to input the second fine-tuning text, the second similar text and the second non-similar text into the embedding model respectively to obtain each third feature vector;

[0158] The training module 1304 is also used to determine the third similarity of the third feature vector between the second fine-tuning sample and the second similar text, and the fourth similarity of the third feature vector between the second fine-tuning sample and the second non-similar text; determine a fourth loss value based on the difference between the third similarity and the fourth similarity, and an activation function; and fine-tune the embedding model based on the fourth loss value.

[0159] In a possible implementation, the determination module 1303 is also used to obtain similar text pairs generated based on the large model, and determine the similar text pair containing the fine-tuning text as the target text pair; based on other pre-trained embedding models, determine the fourth feature vector of each text of the target text pair; and determine whether the non-fine-tuning text in the target text pair is a similar text of the fine-tuning text based on the fifth similarity between each fourth feature vector and a preset threshold.

[0160] In a possible implementation, the determination module 1303 is also used to determine whether the texts in the target text pair are similar based on the fourth largest model. If so, continue to execute subsequent steps based on other pre-trained embedding models to determine each fourth feature vector of the target text pair.

[0161] In a possible implementation, the determination module 1303 is also used to generate associated text corresponding to the acquired text segment based on the large model, and the associated text includes at least one of the question of the text segment, the summary text of the text segment, and the answer information corresponding to the question of the text segment; and randomly select any two texts from the text segment and the associated text as a similar text pair.

[0162] Based on the same inventive concept, an embodiment of the present application further provides an electronic device, Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.14 As shown, it includes: a processor 1401, a communication interface 1402, a memory 1403 and a communication bus 1404, wherein the processor 1401, the communication interface 1402, and the memory 1403 communicate with each other through the communication bus 1404;

[0163] The memory 1403 stores a computer program. When the program is executed by the processor 1401, the processor 1401 performs the following steps:

[0164] Obtaining a training set that is pre-saved for a specific field, wherein the training set includes sample texts;

[0165] Inputting the sample text into the encoder for encoding, and obtaining a first feature vector output by each encoding sublayer in the encoder;

[0166] Determining a predicted feature vector of the sample text according to the weight corresponding to each encoding sublayer and the first feature vector;

[0167] A first loss value is determined according to the predicted feature vector and the standard feature vector of the sample text, and the initial embedding model is trained according to the first loss value to obtain an embedding model.

[0168] In a possible implementation, the initial embedding model further includes a decoder, and after determining the predicted feature vector of the sample text and before training the initial embedding model according to the first loss value, the method further includes:

[0169] Inputting the predicted feature vector into the decoder for decoding to obtain the predicted text output by the decoder;

[0170] Determine a second loss value according to the predicted text and the sample text;

[0171] The step of training the initial embedding model according to the first loss value to obtain the embedding model comprises:

[0172] The initial embedding model is trained according to the first loss value and the second loss value to obtain the embedding model.

[0173] In a possible implementation, the method further includes:

[0174] Acquire any sample text in the training set as a first fine-tuned text, a first similar text corresponding to the first fine-tuned text, and a first non-similar text corresponding to the first fine-tuned text;

[0175] Inputting the first fine-tuned text, the first similar text, and the first non-similar text into the embedding model respectively to obtain each second feature vector;

[0176] Determine a third loss value according to a first similarity between the first fine-tuned text and the second feature vector corresponding to the first similar text, and a second similarity between the first fine-tuned sample and the second feature vector corresponding to the first non-similar text;

[0177] The embedding model is fine-tuned according to the third loss value.

[0178] In a possible implementation, the method further includes:

[0179] Obtain any sample in the training set as a second fine-tuning text, a second similar text corresponding to the second fine-tuning sample, and a second non-similar text corresponding to the second fine-tuning text;

[0180] Inputting the second fine-tuned text, the second similar text, and the second non-similar text into the embedding model respectively to obtain each third feature vector;

[0181] Determining a third similarity of a third feature vector between the second fine-tuned sample and the second similar text, and a fourth similarity of a third feature vector between the second fine-tuned sample and the second non-similar text;

[0182] Determining a fourth loss value according to a difference between the third similarity and the fourth similarity and an activation function;

[0183] Fine-tune the embedding model according to the fourth loss value.

[0184] In a possible implementation, the process of determining similar text corresponding to any fine-tuning text includes:

[0185] Acquire similar text pairs generated based on the large model, and determine the similar text pairs containing the fine-tuned text as target text pairs;

[0186] Determine a fourth feature vector of each text of the target text pair based on other pre-trained embedding models;

[0187] According to the fifth similarity between each fourth feature vector and a preset threshold, it is determined whether the non-fine-tuning text in the target text pair is a similar text to the fine-tuning text.

[0188] In a possible implementation, after determining the similar text pair including the fine-tuning text as the target text pair, and before determining each fourth feature vector of the target text pair based on other pre-trained embedding models, the method further includes:

[0189] Based on the fourth model, determine whether the texts in the target text pair are similar. If so, continue to perform subsequent steps based on other pre-trained embedding models to determine each fourth feature vector of the target text pair.

[0190] In a possible implementation, the process of determining similar text pairs generated based on the large model includes:

[0191] Generate associated text corresponding to the acquired text segment based on the large model, wherein the associated text includes at least one of a question of the text segment, a summary text of the text segment, and answer information corresponding to the question of the text segment;

[0192] Randomly select any two texts from the text segment and the associated text as similar text pairs.

[0193] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 1402 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0194] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0195] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, in which a computer program executable by a processor is stored. When the program runs on the processor, the processor implements the following steps when executing:

[0196] Obtaining a training set that is pre-saved for a specific field, wherein the training set includes sample texts;

[0197] Inputting the sample text into the encoder for encoding, and obtaining a first feature vector output by each encoding sublayer in the encoder;

[0198] Determining a predicted feature vector of the sample text according to the weight corresponding to each encoding sublayer and the first feature vector;

[0199] A first loss value is determined according to the predicted feature vector and the standard feature vector of the sample text, and the initial embedding model is trained according to the first loss value to obtain an embedding model.

[0200] In a possible implementation, the initial embedding model further includes a decoder, and after determining the predicted feature vector of the sample text and before training the initial embedding model according to the first loss value, the method further includes:

[0201] Inputting the predicted feature vector into the decoder for decoding to obtain the predicted text output by the decoder;

[0202] Determine a second loss value according to the predicted text and the sample text;

[0203] The step of training the initial embedding model according to the first loss value to obtain the embedding model comprises:

[0204] The initial embedding model is trained according to the first loss value and the second loss value to obtain the embedding model.

[0205] In a possible implementation, the method further includes:

[0206] Acquire any sample text in the training set as a first fine-tuned text, a first similar text corresponding to the first fine-tuned text, and a first non-similar text corresponding to the first fine-tuned text;

[0207] Inputting the first fine-tuned text, the first similar text, and the first non-similar text into the embedding model respectively to obtain each second feature vector;

[0208] Determine a third loss value according to a first similarity between the first fine-tuned text and the second feature vector corresponding to the first similar text, and a second similarity between the first fine-tuned sample and the second feature vector corresponding to the first non-similar text;

[0209] The embedding model is fine-tuned according to the third loss value.

[0210] In a possible implementation, the method further includes:

[0211] Obtain any sample in the training set as a second fine-tuning text, a second similar text corresponding to the second fine-tuning sample, and a second non-similar text corresponding to the second fine-tuning text;

[0212] Inputting the second fine-tuned text, the second similar text, and the second non-similar text into the embedding model respectively to obtain each third feature vector;

[0213] Determining a third similarity of a third feature vector between the second fine-tuned sample and the second similar text, and a fourth similarity of a third feature vector between the second fine-tuned sample and the second non-similar text;

[0214] Determining a fourth loss value according to a difference between the third similarity and the fourth similarity and an activation function;

[0215] Fine-tune the embedding model according to the fourth loss value.

[0216] In a possible implementation, the process of determining similar text corresponding to any fine-tuning text includes:

[0217] Acquire similar text pairs generated based on the large model, and determine the similar text pairs containing the fine-tuned text as target text pairs;

[0218] Determine a fourth feature vector of each text of the target text pair based on other pre-trained embedding models;

[0219] According to the fifth similarity between each fourth feature vector and a preset threshold, it is determined whether the non-fine-tuning text in the target text pair is a similar text to the fine-tuning text.

[0220] In a possible implementation, after determining the similar text pair including the fine-tuning text as the target text pair, and before determining each fourth feature vector of the target text pair based on other pre-trained embedding models, the method further includes:

[0221] Based on the fourth largest model, determine whether the texts in the target text pair are similar. If so, continue to perform subsequent steps based on other pre-trained embedding models to determine each fourth feature vector of the target text pair.

[0222] In a possible implementation, the process of determining similar text pairs generated based on the large model includes:

[0223] Generate associated text corresponding to the acquired text segment based on the large model, wherein the associated text includes at least one of a question of the text segment, a summary text of the text segment, and answer information corresponding to the question of the text segment;

[0224] Randomly select any two texts from the text segment and the associated text as similar text pairs.

[0225] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0226] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0227] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0228] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0229] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A training method for embedding models based on large models, It is characterized in that The initial embedding model includes an encoder, the encoder includes a plurality of encoding sub-layers, and the method includes: Obtaining a training set that is pre-saved for a specific field, wherein the training set includes sample texts; Inputting the sample text into the encoder for encoding, and obtaining a first feature vector output by each encoding sublayer in the encoder; Determining a predicted feature vector of the sample text according to the weight corresponding to each encoding sublayer and the first feature vector; A first loss value is determined according to the predicted feature vector and the standard feature vector of the sample text, and the initial embedding model is trained according to the first loss value to obtain an embedding model.

2. The method according to claim 1, It is characterized in that The initial embedding model further includes a decoder. After determining the predicted feature vector of the sample text and before training the initial embedding model according to the first loss value, the method further includes: Inputting the predicted feature vector into the decoder for decoding to obtain the predicted text output by the decoder; Determine a second loss value according to the predicted text and the sample text; The step of training the initial embedding model according to the first loss value to obtain the embedding model comprises: The initial embedding model is trained according to the first loss value and the second loss value to obtain the embedding model.

3. The method according to claim 1, It is characterized in that The method further comprises: Acquire any sample text in the training set as a first fine-tuned text, a first similar text corresponding to the first fine-tuned text, and a first non-similar text corresponding to the first fine-tuned text; Inputting the first fine-tuned text, the first similar text, and the first non-similar text into the embedding model respectively to obtain each second feature vector; Determine a third loss value according to a first similarity between the first fine-tuned text and the second feature vector corresponding to the first similar text, and a second similarity between the first fine-tuned sample and the second feature vector corresponding to the first non-similar text; The embedding model is fine-tuned according to the third loss value.

4. The method according to claim 1, It is characterized in that The method further comprises: Obtain any sample in the training set as a second fine-tuning text, a second similar text corresponding to the second fine-tuning sample, and a second non-similar text corresponding to the second fine-tuning text; Inputting the second fine-tuned text, the second similar text, and the second non-similar text into the embedding model respectively to obtain each third feature vector; Determining a third similarity of a third feature vector between the second fine-tuned sample and the second similar text, and a fourth similarity of a third feature vector between the second fine-tuned sample and the second non-similar text; Determining a fourth loss value according to a difference between the third similarity and the fourth similarity and an activation function; Fine-tune the embedding model according to the fourth loss value.

5. The method according to claim 3 or 4, It is characterized in that The process of determining similar text corresponding to any fine-tuned text includes: Acquire similar text pairs generated based on the large model, and determine the similar text pairs containing the fine-tuned text as target text pairs; Determine a fourth feature vector of each text of the target text pair based on other pre-trained embedding models; According to the fifth similarity between each fourth feature vector and a preset threshold, it is determined whether the non-fine-tuning text in the target text pair is a similar text to the fine-tuning text.

6. The method according to claim 5, It is characterized in that After determining the similar text pair including the fine-tuning text as the target text pair, and before determining each fourth feature vector of the target text pair based on other pre-trained embedding models, the method further includes: Based on the fourth largest model, determine whether the texts in the target text pair are similar. If so, continue to perform subsequent steps based on other pre-trained embedding models to determine each fourth feature vector of the target text pair.

7. The method according to claim 5, It is characterized in that The process of determining similar text pairs generated based on the large model includes: Generate associated text corresponding to the acquired text segment based on the large model, wherein the associated text includes at least one of a question of the text segment, a summary text of the text segment, and answer information corresponding to the question of the text segment; Randomly select any two texts from the text segment and the associated text as similar text pairs.

8. A training device for an embedding model based on a large model, It is characterized in that The initial embedding model includes an encoder, the encoder includes a plurality of encoding sub-layers, and the apparatus includes: An acquisition module, used to acquire a training set saved in advance for a specific field, wherein the training set includes sample texts; An encoding module, used for inputting the sample text into the encoder for encoding, and obtaining a first feature vector output by each encoding sublayer in the encoder; A determination module, used to determine the predicted feature vector of the sample text according to the weight corresponding to each encoding sublayer and the first feature vector; A training module is used to determine a first loss value according to the predicted feature vector and the standard feature vector of the sample text, and train the initial embedding model according to the first loss value to obtain an embedding model.

9. An electronic device, It is characterized in that The electronic device comprises a processor, and the processor is used to implement the steps of the training method of the embedding model based on the large model as described in any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium, It is characterized in that It stores a computer program, which, when executed by a processor, implements the steps of the training method of the large-model-based embedding model as described in any one of claims 1 to 7.

Citation Information

Cited By

  • Question answering method based on large model, and electronic device

    WO2025246204A1