Model training method and device, data processing method and device and electronic equipment

Abstract

The invention provides a model training method and device, a data processing method and device and electronic equipment. The model training method comprises the steps of determining a first mask text and a second mask text which belong to different languages; processing the training data pair in the first training data set based on the second mask probability to obtain a third mask text and a fourth mask text; inputting the first mask text, the second mask text, the third mask text and the fourth mask text into a model to respectively obtain a first prediction text, a second prediction text and a third prediction text; adjusting weights of a second encoder and a mask language model prediction head based on the first prediction text, the second prediction text, the third prediction text and the training data pair; therefore, texts belonging to different languages can be subjected to semantic null alignment in the languages, and when users belonging to different languages perform retrieval, accurate retrieval results can be obtained without translating the to-be-retrieved texts to the language attribution types corresponding to the retrieval library in advance.

Description

Technical Field

[0001] This disclosure relates to the field of large model technology, and in particular to a model training method, apparatus, data processing method, apparatus and electronic equipment. Background Technology

[0002] In related technologies, most retrieval models are trained on corpora of a single language (or language affiliation). During inference, they usually search for similar texts in the same language affiliation database based on the retrieval text of a single language affiliation type, resulting in poor performance of cross-language text retrieval. Summary of the Invention

[0003] This disclosure provides a model training method, apparatus, data processing method, apparatus, and electronic device to at least solve the above-mentioned technical problems existing in the prior art.

[0004] According to a first aspect of this disclosure, a model training method is provided, the model including a first encoder, a second encoder, a cross-decoder, and a masked language model prediction head, the method comprising: Determine a first training dataset that includes multiple training data pairs; The training data pairs of the first training dataset are processed based on the first mask probability to obtain first mask text and second mask text with different language attribution types. The first mask text is input into the first encoder to obtain the first encoded vector and the first mask text vector; the second mask text is input into the second encoder to obtain the second encoded vector and the second mask text vector. Based on the second mask probability, the training data pairs in the first training dataset are processed to obtain third and fourth mask texts with different language attribution types; One of the first and second encoded vectors, as well as one of the third and fourth masked texts, are input into the cross-decoder to obtain the third masked text vector; The first masked text vector is input into the masked language model prediction head to obtain the first predicted text; the second masked text vector is input into the masked language model prediction head to obtain the second predicted text. The third masked text vector is input into the masked language model prediction head, and the output of the masked language model prediction head is the third predicted text; Based on the first predicted text, the second predicted text, the third predicted text, and the training data pair, adjust the weights of the second encoder and the prediction head of the masked language model.

[0005] In the above scheme, the probability of the first mask is less than the probability of the second mask; The language type of text that the first encoder can process is different from the language type of text that the second encoder can process. The language type of the first masked text is the same as the language type of the text that the first encoder can process; The language type of the second masked text is the same as the language type of the text that the second encoder can process; Each training data pair includes two training texts that belong to different languages ​​but have the same semantics.

[0006] In the above scheme, the One of the first and second encoded vectors, along with one of the third and fourth masked texts, is input into the cross-decoder to obtain the third masked text vector, which includes: The first encoded vector and the third masked text are concatenated to obtain the first concatenated text; The second encoded vector and the fourth masked text are concatenated to obtain the second concatenated text. The encoding vector portion in the first concatenated text and the second concatenated text are either replaced or not replaced, and the mask text portion in the first concatenated text and the second concatenated text is either replaced or not replaced, to obtain the input text of the cross-decoder; The input text is fed into the cross-decoder to obtain the third mask text vector.

[0007] In the above scheme, adjusting the weights of the second encoder and the masked language model prediction head based on the first predicted text, the second predicted text, and the training data pair includes: A first loss is determined based on a first training text that has the same language affiliation type as the first predicted text and the training data pair; wherein, the first predicted text includes predictions of texts in the first training text that are masked based on a first masking probability; A second loss is determined based on a second training text in the training data pair that has the same language affiliation as the second predicted text; wherein the second predicted text includes predictions of text in the second training text that are masked based on a first masking probability; The third loss is determined based on the third predicted text and the training data; wherein the third predicted text includes the prediction of the text masked based on the second mask probability in the first training text or the second training text. Based on the first loss, the second loss, and the third loss, the first training loss is determined; The weights of the second encoder and the masked language model prediction head are adjusted based on the first training loss.

[0008] In the above scheme, after adjusting the weights of the second encoder and the masked language model prediction head based on the first predicted text, the second predicted text, and the training data pair, the method further includes: Determine a second training dataset, which includes at least one training sample, each sample including training text, first related text and multiple second related texts; Based on the language affiliation type of each sample, including the training text, the first related text, and multiple second related texts, the training text, the first related text, and multiple second related texts are input into the first encoder or the second encoder to obtain the training text features corresponding to the training text, the first related features corresponding to the first related text, and the second related features corresponding to each second related text. The weights of the second encoder are adjusted based on the training text features, the first relevant features, and all the second relevant features.

[0009] In each sample, the correlation between the first related text and the training text is greater than a first threshold; the correlation between the second related text and the training text is less than a second threshold.

[0010] In the above scheme, adjusting the weights of the second encoder based on the training text features, the first relevant features, and all the second relevant features includes: The first relevance is determined based on the features of the training text and the first relevance features; The second relevance is determined based on the training text features, the first relevance feature, and all the second relevance features; The second training loss is determined based on the first and second correlations; The weights of the second encoder are adjusted based on the second training loss.

[0011] According to a second aspect of this disclosure, a data processing method is provided, comprising a first encoder and a second encoder based on a model trained using the above-described model training method, the method comprising: Determine the language category of the data to be retrieved; If the language affiliation type of the data to be retrieved is a first language affiliation type, then the data to be retrieved is transmitted to the first encoder; If the language affiliation type of the data to be retrieved is the second language affiliation type, then the data to be retrieved is transmitted to the second encoder; Based on the embedding value output by the first encoder or the embedding value output by the second encoder, determine the matching data corresponding to the data to be retrieved; The first encoder and the second encoder process different language affiliation types.

[0012] According to a third aspect of this disclosure, a model training apparatus is provided, the apparatus comprising: The first determining unit is used to determine a first training dataset that includes multiple training data pairs; The first masking unit is used to process training data pairs of the first training dataset based on the first masking probability to obtain first masked text and second masked text with different language affiliation types. The encoding unit is used to input the first mask text into the first encoder to obtain a first encoded vector and a first mask text vector; and to input the second mask text into the second encoder to obtain a second encoded vector and a second mask text vector. The second masking unit is used to process the training data pairs in the first training dataset based on the second masking probability to obtain the third masking text and the fourth masking text with different language affiliation types. The input unit is used to input one of the first encoding vector and the second encoding vector, as well as one of the third mask text and the fourth mask text, into the cross-decoder to obtain the third mask text vector; The prediction unit is used to input the first masked text vector into the masked language model prediction head to obtain the first predicted text; input the second masked text vector into the masked language model prediction head to obtain the second predicted text; and input the third masked text vector into the masked language model prediction head, wherein the output of the masked language model prediction head is the third predicted text. The first adjustment unit is used to adjust the weights of the second encoder and the masked language model prediction head based on the first predicted text, the second predicted text, the third predicted text and the training data pair.

[0013] According to a fourth aspect of this disclosure, a data processing apparatus is provided, comprising a first encoder and a second encoder based on a model trained using the above-described model training method, the apparatus comprising: Language attribution unit, used to determine the language attribution type of the data to be retrieved; The first transmission unit is configured to transmit the data to be retrieved to the first encoder in response to the language affiliation type of the data to be retrieved being a first language affiliation type. The second transmission unit is configured to transmit the data to be retrieved to the second encoder in response to the language affiliation type of the data to be retrieved being the second language affiliation type. The retrieval unit is used to determine the matching data corresponding to the data to be retrieved based on the embedding value output by the first encoder or the embedding value output by the second encoder. The first encoder and the second encoder process different language affiliation types.

[0014] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods of this disclosure.

[0015] The model training method disclosed herein, through A first training dataset comprising multiple training data pairs is determined; the training data pairs in the first training dataset are processed based on a first mask probability to obtain first masked text and second masked text with different language attribution types; the first masked text is input into a first encoder to obtain a first encoded vector and a first masked text vector; the second masked text is input into a second encoder to obtain a second encoded vector and a second masked text vector; the training data pairs in the first training dataset are processed based on a second mask probability to obtain third masked text and fourth masked text with different language attribution types; the first encoded vector and the second masked text vector are then processed together. One of the code vectors, along with one of the third and fourth masked texts, is input into the cross-decoder to obtain the third masked text vector. The first masked text vector is then input into the masked language model prediction head to obtain the first predicted text. The second masked text vector is input into the masked language model prediction head to obtain the second predicted text. The third masked text vector is then input into the masked language model prediction head, and the output of the masked language model prediction head is the third predicted text. Based on the first, second, and third predicted texts and the training data pairs, the weights of the second encoder and the masked language model prediction head are adjusted. This allows texts belonging to different languages ​​to be semantically aligned, improving cross-language text retrieval capabilities.

[0016] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0017] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0018] Figure 1 This illustration shows a first optional flowchart of the model training method provided in an embodiment of the present disclosure; Figure 2 A schematic diagram of a first structural representation of the model provided in this disclosure is shown; Figure 3 A schematic diagram of a second optional process for the model training method provided in this embodiment of the present disclosure is shown; Figure 4 A schematic diagram of a third optional process for the model training method provided in this disclosure embodiment is shown; Figure 5 A second structural schematic diagram of the model provided in this disclosure embodiment is shown; Figure 6 A schematic diagram of the data flow of the model training method provided in an embodiment of this disclosure is shown; Figure 7 A schematic diagram of an optional flow of the data processing method provided in an embodiment of this disclosure is shown; Figure 8 A data diagram illustrating the data processing method is shown. Figure 9 A schematic diagram of an optional structure of the model training apparatus provided in an embodiment of this disclosure is shown; Figure 10 A schematic diagram of an optional structure of the data processing apparatus provided in an embodiment of this disclosure is shown; Figure 11 A schematic diagram of the composition structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0019] To make the objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0020] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0021] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0022] Unless otherwise defined, all technical and scientific terms used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used in this disclosure is for the purpose of describing embodiments of this disclosure only and is not intended to be limiting of this disclosure.

[0023] It should be understood that in the various embodiments of this disclosure, the sequence number of each implementation process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0024] Before providing a further detailed description of the embodiments of this disclosure, the nouns and terms involved in the embodiments of this disclosure will be explained, and the nouns and terms involved in the embodiments of this disclosure shall be interpreted as follows.

[0025] Bidirectional Encoder Representations from Transformers (BERT)

[0026] Generative Pre-trained Transformer (GPT) model.

[0027] Text-to-Text Transfer Transformer, T5.

[0028] Masked Language Model (MLM).

[0029] Next Token Prediction (NTP) is a training task in natural language processing.

[0030] Sequence-to-sequence prediction: seq2seq.

[0031] Masked Auto Encoder (MAE)

[0032] A novel pre-training paradigm for retrieval based on masked autoencoder: Retrieval of Masked Auto Encoder (RetroMAE).

[0033] In related technologies, the basic pre-trained models are unsupervised pre-trained models represented by BERT, GPT, T5, etc. They mainly use MLM, NSP, NTP, seq2seq learning methods to aggregate contextual information to obtain the embedding values ​​of identifiers or tokens and sentences. The drawback is that they lack the ability to represent sentence-level embedding values.

[0034] Self-contrastive learning, represented by SimCSE, is a type of model that learns semantic representations and matching abilities through self-contrast. Its characteristics include using dropout and other methods to augment data and construct similar pairings. It is an unsupervised learning method, so it can learn directly from massive amounts of corpus. Its disadvantage is that it is relatively dependent on the data augmentation method, and the positive and negative pairings constructed are relatively simple.

[0035] Masked Auto-Encoders, represented by RetroMAE, are also unsupervised pre-trained models. Their characteristic is that they construct self-supervised learning in the form of auto-encoders, avoiding the drawbacks of using data augmentation. The disadvantage is that most existing models are learned from single-language corpora, resulting in poor performance for cross-language text matching.

[0036] In view of the problems existing in the related technologies, the present disclosure provides a model training method, apparatus, data processing method and apparatus to at least solve some or all of the above-mentioned technical problems.

[0037] Figure 1 A schematic diagram of a first alternative process for a model training method provided in an embodiment of this disclosure is shown, and the steps will be described accordingly.

[0038] Figure 2 A schematic diagram of a first structure of the model provided in an embodiment of this disclosure is shown.

[0039] like Figure 2 As shown, the model includes a first encoder (Sourse Encoder), a second encoder (Target Encoder), a cross decoder, and a masked language model prediction head (MLP Prediction Head).

[0040] The first encoder receives and processes text belonging to a first language type; the second encoder receives and processes text belonging to a second language type; the cross-decoder receives the encoded vectors of text belonging to the first and second languages ​​with the same semantics, as well as text belonging to different languages ​​with the same semantics. It then substitutes or does not substitute the encoded vectors, and substitutes or does not substitute the text to obtain the input text for the cross-decoder. This input text is then decoded using positional encoding to obtain a third masked text vector. Thus, the cross-decoder can reconstruct text based on the encoded vector of the first language type and the fourth masked text region of the second language type, or based on the encoded vector of the second language type and the fourth masked text region of the first language type, or based on the encoded vector of the first language type and the fourth masked text region of the first language type, or based on the encoded vector of the second language type and the fourth masked text region of the second language type. Utilizing the cross-encoder's understanding of vectors and text belonging to different languages, the second encoder and the masked language model prediction head are trained.

[0041] The masked language model prediction head is used to receive the outputs of the first encoder, the second encoder, and the cross-decoder, respectively, and output the predicted text of the first language attribution type and the predicted text of the second language attribution type.

[0042] During training, a cross-decoder is used to decode and reconstruct the corresponding text across languages ​​using the encoded information of the text. For example, the vector representation of a text sentence is used to reconstruct the token of the corresponding English text, and vice versa. By introducing this cross-decoding learning task, the language gap is bridged, allowing the outputs of the first and second encoders to be aligned in the cross-language semantic space, making the distance between texts belonging to different languages ​​similar. Specific model training methods include: Step S101: Determine a first training dataset that includes multiple training data pairs.

[0043] In some embodiments, the first training dataset includes multiple training data pairs, each training data pair including two training texts that have different language affiliations but the same semantics.

[0044] Step S102: Process the training data pairs of the first training dataset based on the first mask probability to obtain first mask text and second mask text with different language affiliation types.

[0045] In some embodiments, the carrier implementing the model training method (hereinafter referred to as the carrier) performs the same operation on each training data pair in the first training dataset.

[0046] In some embodiments, the carrier processes the first training text in the first training dataset based on a first mask probability, where the probability of processing each character, word (or token) in the first training text is the first mask probability; if any token is processed, the carrier replaces the arbitrary token with a mask marker (such as a Mask). Figure 5 The first training text can be processed by either replacing any token with any word in the first vocabulary, or not processing any token at all. The text obtained after processing all tokens in the first training text is the first masked text. The first vocabulary can be set in advance according to inference requirements, and the language classification of the characters or words in the first vocabulary is the same as the language classification of the first training text.

[0047] In some embodiments, the carrier processes the second training text in the first training dataset based on a first mask probability, where the probability of processing each token in the second training text is the first mask probability; if any token is processed, the carrier replaces the arbitrary token with a mask marker (such as a Mask). Figure 5 The text obtained after processing all tokens in the second training text is the second masked text. The second vocabulary can be pre-set according to inference requirements, and the language classification of characters or words in the second vocabulary is the same as the language classification of the second training text.

[0048] It should be noted that the position or meaning of the masked tokens in the first training text is independent of the position or meaning of the masked tokens in the second training text; there is no necessary connection between the two. For example, if the x-th token in the first training text is masked, and the y-th token in the second training text is also masked, x and y are independent and can be the same or different. Similarly, the x-th token and the y-th token are independent and their semantics can be the same or different.

[0049] The carrier can be computer programs, electronic circuits, databases, mobile applications, electronic devices, cloud computing platforms, distributed systems, artificial intelligence frameworks, mathematical models, automation tools, and microcontrollers, etc., which are software or hardware capable of implementing algorithms and methods.

[0050] Step S103: Input the first mask text into the first encoder to obtain the first encoding vector and the first mask text vector; input the second mask text into the second encoder to obtain the second encoding vector and the second mask text vector.

[0051] In some embodiments, the language type of the text that the first encoder can process is different from the language type of the text that the second encoder can process; the language type of the first masked text is the same as the language type of the text that the first encoder can process; and the language type of the second masked text is the same as the language type of the text that the second encoder can process. For example, if the first encoder can process text with a language type of Chinese, then the second encoder can process text with a language type other than Chinese.

[0052] In some embodiments, the carrier inputs the first masked text into the first encoder to obtain a first encoded vector corresponding to the first masked text, and a text vector corresponding to each character in the first masked text (including masked characters and unmasked text characters), wherein the text vector corresponding to a masked character is a masked text vector. Before inputting the first masked text into the first encoder, the carrier may also determine the position of each masked character based on the first position encoded vector, and determine the first masked text vector corresponding to all masked characters based on the position of each masked character.

[0053] In some embodiments, the carrier inputs the second masked text into the second encoder to obtain a second encoding vector corresponding to the second masked text, and a text vector corresponding to each character in the second masked text (including masked characters and unmasked text characters). Before inputting the second masked text into the second encoder, the carrier can also determine the position of each masked character based on the second position encoding vector, and determine the second masked text vector corresponding to all masked characters based on the position of each masked character.

[0054] Step S104: Process the training data pairs in the first training dataset based on the second mask probability to obtain the third mask text and the fourth mask text.

[0055] In some embodiments, the carrier performs the same operation on each training data pair in the first training dataset based on a second mask probability.

[0056] In some embodiments, the carrier processes the first training text of the training data pair based on the second mask probability to obtain the third mask text; and processes the second training text of the training data pair based on the second mask probability to obtain the fourth mask text. The method by which the carrier obtains the third and fourth mask texts based on the second mask probability is the same as the method by which the carrier obtains the first and second mask texts based on the first mask probability (i.e., as described in step S102), and will not be repeated here. It should be noted that the first mask probability is less than the second mask probability.

[0057] Step S105: Input one of the first encoding vector and the second encoding vector, as well as one of the third mask text and the fourth mask text, into the cross-decoder to obtain the third mask text vector.

[0058] In some embodiments, the carrier concatenates a first encoded vector and a third mask text to obtain a first concatenated text; concatenates a second encoded vector and a fourth mask text to obtain a second concatenated text; replaces or does not replace the encoded vector portion in the first and second concatenated texts, and replaces or does not replace the mask text portion in the first and second concatenated texts to obtain the input text for the cross-decoder; the input text is input into the cross-decoder to obtain a third mask text vector.

[0059] The third mask text vector includes the mask text portion of the input text and the vector corresponding to the mask characters.

[0060] Step S106: Input the first masked text vector into the masked language model prediction head to obtain the first predicted text; input the second masked text vector into the masked language model prediction head to obtain the second predicted text.

[0061] In some embodiments, the carrier inputs the first masked text vector into the masked language model prediction head to obtain the predicted text of the masked characters in the first masked text, i.e., the first predicted text.

[0062] In some embodiments, the carrier inputs the second masked text vector into the prediction head of the masked language model to obtain the predicted text of the masked characters in the second masked text, i.e., the second predicted text.

[0063] Step S107: Input the third masked text vector into the masked language model prediction head, and the output of the masked language model prediction head is the third predicted text.

[0064] In some embodiments, the mask language model can be viewed as a classifier that determines the type of multiple tokens obtained by the cross-decoder.

[0065] Step S108: Based on the first predicted text, the second predicted text, the third predicted text, and the training data pair, adjust the weights of the second encoder and the masked language model prediction head.

[0066] In some embodiments, the carrier may determine a first loss based on a first training text in a pair of training data that has the same language affiliation as the first predicted text; wherein the first predicted text includes predictions of text in the first training text that are masked based on a first mask probability.

[0067] In some embodiments, the carrier may determine a second loss based on a second training text in a training data pair that has the same language affiliation type as the second predicted text; wherein the second predicted text includes predictions of text in the second training text that are masked based on a first mask probability.

[0068] In some embodiments, the carrier can determine a third loss based on a third predicted text and training data; wherein the third predicted text includes predictions of text masked based on a second mask probability in a first training text or a second training text; a first training loss is determined based on the first loss, the second loss, and the third loss; and the weights of the second encoder and the masked language model prediction head are adjusted based on the first training loss. The purpose of training is to make the model's input the same as the model's input, that is, after processing by the model's first encoder, second encoder, and cross-decoder, the model can reconstruct the input text.

[0069] The first predicted text and the first training text have the same language classification; the second predicted text and the second training text have the same language classification.

[0070] In some alternative embodiments, the first loss, the second loss, and the third loss can be determined based on cross-entropy, and the first loss, the second loss, and the third loss can be weighted and summed to obtain the first training loss.

[0071] Thus, the data processing method provided in this embodiment enables texts belonging to different languages ​​to be semantically aligned, allowing users of different languages ​​to perform searches without having to translate the text to be searched into the language category corresponding to the search database in advance, making it easier for users to use; in addition, when faced with databases of multiple language categories (such as search databases, data databases, etc.), the search purpose can be achieved through the familiar language, maximizing the utilization of data in the database.

[0072] Figure 3 A second alternative flowchart of the model training method provided in this disclosure embodiment is shown, and will be described step by step.

[0073] Step S301: Perform the first stage of training on the model.

[0074] In some embodiments, steps S101 to S106 constitute the first stage of model training, the purpose of which is to align the outputs of the first encoder and the second encoder in the cross-linguistic semantic space, so that texts belonging to different languages ​​are close in space. The specific training method of the first stage will not be described in detail.

[0075] In some embodiments, steps S302 to S304 constitute the second stage of model training. Since the ultimate goal of the model is to retrieve text that is similar to (or matches) the text to be retrieved, the purpose of the second stage of training is to increase the matching degree of at least two similar texts and decrease the matching degree of at least two dissimilar texts. Specific training methods include: Step S302: Determine the second training dataset.

[0076] In some embodiments, the second training dataset includes at least one training sample, and each training sample includes training text, a first related text, and multiple second related texts. In each training sample, the relevance between the first related text and the training text is greater than a first threshold; the relevance between the second related text and the training text is less than a second threshold. The relevance can be cosine similarity, etc., and the first and second thresholds can be set according to actual needs. The first related text is related to the training text, and the second related text is unrelated to the training text.

[0077] In some embodiments, each training sample includes training text, first related text, and multiple second related texts that belong to the same language category or to different language categories; the second training dataset includes texts of at least two language categories.

[0078] Step S303: Input the second training dataset into the first encoder and the second encoder to obtain training text features, first related features and at least one second related feature.

[0079] In some embodiments, the carrier performs the same operation on each training sample in the second dataset, that is, the carrier first determines the language attribution type of the training text, the first related text and multiple second related texts in each sample, and inputs the texts (training text, first related text and multiple second related texts) included in the sample into the first encoder or the second encoder that matches the language attribution type according to the language attribution type, so as to obtain the corresponding training text features, the first related features and at least one second related feature.

[0080] In practice, based on the language affiliation type of each sample, including the training text, the first related text, and multiple second related texts, the training text, the first related text, and multiple second related texts are input into the first encoder or the second encoder to obtain the training text features corresponding to the training text, the first related features corresponding to the first related text, and the second related features corresponding to each second related text.

[0081] Specifically, the carrier performs the following operations on each training sample: determining the language classification of the training text in the training sample, and inputting the training text into a first encoder or a second encoder with the same language classification; for example, if the training text belongs to the first language classification, it is input into the first encoder; if the training text belongs to the second language classification, it is input into the second encoder. The output of the first encoder or the second encoder relative to the training text is the training text feature.

[0082] The carrier determines the language affiliation type of the first relevant feature and at least one second relevant feature according to the above operation, and inputs them into the first encoder or the second encoder with the same language affiliation type to obtain the first relevant feature and at least one second relevant feature.

[0083] Step S304: Adjust the weights of the second encoder based on the training text features, the first relevant features, and all the second relevant features.

[0084] In some embodiments, the purpose of training the carrier is to make the similarity between the training text and the first related text greater than the similarity between the training text and the second related text. Therefore, in determining the objective function, the carrier determines a first relevance based on the features of the training text and the first related features; determines a second relevance based on the features of the training text, the first related features, and all the second related features; determines a second training loss based on the first relevance and the second relevance; and adjusts the weights of the second encoder based on the second training loss.

[0085] In practice, the second training loss of the model can be determined based on the first relevance and the sum of the first relevance and all second relevances. The weights of the second encoder can then be adjusted based on this second training loss. Optionally, a temperature coefficient can be introduced during the determination of the second training loss to control its smoothness or steepness, thereby improving the model's ability to distinguish between the first and second relevance texts.

[0086] Thus, the data processing method provided in this embodiment enables texts belonging to different languages ​​to be semantically aligned, allowing users of different languages ​​to perform searches without needing to translate the text to be searched into the language category corresponding to the search database in advance, making it easier for users. Furthermore, by training the model based on the second training dataset, the matching degree of at least two similar texts is increased, and the matching degree of at least two dissimilar texts is decreased, improving the model's search capability. When facing databases with multiple language categories (such as search databases, information databases, etc.), the search purpose can be achieved through the familiar language, maximizing the utilization of data in the database.

[0087] Figure 4A schematic diagram of a third optional process for the model training method provided in this disclosure is shown, and the steps will be described accordingly.

[0088] Step S401: Perform the first stage of training on the model.

[0089] Figure 5 A second structural schematic diagram of the model provided in the embodiments of this disclosure is shown.

[0090] In some embodiments, the carrier trains the model described in the present disclosure embodiments using the learning architecture of the RetroMAE model.

[0091] In some embodiments, conventional models typically target text belonging to only one language type and usually employ only one encoder to process the input text. However, this disclosure targets text belonging to multiple language types, therefore introducing a second encoder (Target Encoder) as a cross-language adaptation module to represent text belonging to different language types. For example, the first encoder (Sourse Encoder) targets Chinese representations, while the second encoder targets non-Chinese representations. Non-Chinese representations may include English, Japanese, Thai, German, French, Spanish, Russian, Arabic, Korean, Portuguese, Indonesian, etc.

[0092] In practice, the first encoder and the second encoder can choose different language versions of the same open-source model architecture. For example, the Source Encoder can choose gte-base-zh and the Target Encoder can choose gte-base-en.

[0093] In some embodiments, the model is trained in the first stage using a first training dataset. The first training dataset includes training data pairs of bilingual parallel corpora, and vector representations of encoders corresponding to text inputs of different language types, such as a Source Encoder for Chinese text input and a Target Encoder for English text input. The first training dataset can be an open-source parallel corpus dataset such as translation2019 or wiki_translate, with a quantity exceeding 5 million.

[0094] In some embodiments, the model further includes a cross-decoder that can decode and reconstruct cross-language parallel texts using the encoded information of the text. For example, it can reconstruct the token of the corresponding English text using the vector representation of the Chinese text, and vice versa. By introducing this cross-decoding learning task, the gap between language affiliation types is broken down, allowing the two decoders to align in the semantic spaces of different language affiliation types.

[0095] In some alternative embodiments, the carrier performs masking processing on the text input to the first encoder or the second encoder based on a first masking probability. The first masking probability can refer to the masking probability of the encoder in the RetroMAE model, such as 30%, or any value set according to actual needs. The carrier performs masking processing on the text input to the cross-decoder based on a second masking probability. The second masking probability can refer to the masking probability of the decoder in the RetroMAE model, such as any value between 50% and 70%, or any value set according to actual needs.

[0096] In some embodiments, during training, the weights of the first encoder are frozen, the cross-entropy loss is calculated based on the model's input and output, and the weights of the second encoder and the masked language model prediction head are adjusted based on the cross-entropy loss.

[0097] In some embodiments, the number of vectors output by the first encoder is the number of characters included in the first masked text plus 1, including the first encoded vector (i.e., sentence vector) corresponding to the entire first masked text, and multiple text vectors (i.e., character vectors, vocabulary vectors, or masked character vectors), the number of text vectors being the same as the number of characters included in the first masked text.

[0098] The first encoded vector, as a possible output of the cross-decoder, is concatenated with the third or fourth masked text to obtain the input text, which is then input into the cross-decoder. Among the multiple text vectors, the masked text vector corresponding to the masked character is input into the masked language model prediction head, which predicts the first predicted text.

[0099] The vector output by the second encoder and the subsequent process are similar to those of the first encoder, and will not be repeated here.

[0100] Step S402: Perform the second stage of training on the model.

[0101] Figure 6 A schematic diagram of the data flow of the model training method provided in an embodiment of this disclosure is shown.

[0102] In some embodiments, such as Figure 6 As shown, the second stage of training is fine-tuning training, which involves the first encoder, the second encoder, and the masked language model prediction head; the weights of the second encoder are adjusted during the training process.

[0103] In some embodiments, the training samples included in the second training dataset may be<q,d+,{d1,d2,...dN}> , where q represents the training text (the text to be queried), d+ represents the text (or document) related to it, i.e. the first relevant text, and di represents the text (or document) unrelated to q, i.e. the second relevant text.

[0104] During fine-tuning, the objective function is InfoNCE Loss:

[0105] in, Using the temperature coefficient as an example, the objective function forces the first relevant text d+ to achieve a higher matching score (or relevance) compared to the second relevant text di and the training text.

[0106] In practice, the second training dataset uses a bilingual mixed corpus, that is, q, d+, and di are randomly sampled from two language categories. q, d+, and di may be the same language category or different language categories. The model needs to distinguish the correlation between q and d+ and between q and di during the training process of the second training dataset.

[0107] In some embodiments, the carrier uses the Source Encoder and Target Encoder obtained in step S301 to obtain the text of the language type corresponding to the training samples, and freezes the weights of the Source Encoder, only adjusting the weights of the Target Encoder.

[0108] In some alternative embodiments, such as Figure 5 As shown, the carrier can be equipped with a language recognition module (langid gating) before the first encoder and the second encoder, which is used to assign the input text to the first encoder or the second encoder according to the language affiliation.

[0109] In some alternative embodiments, the second training dataset can use open-source datasets such as MMARCO and wikilingua, with a sample size of over 600,000.

[0110] Thus, the data processing method provided in this embodiment enables texts belonging to different languages ​​to be semantically aligned, allowing users of different languages ​​to perform searches without needing to translate the text to be searched into the language category corresponding to the search database in advance, making it easier for users. Furthermore, by training the model based on the second training dataset, the matching degree of at least two similar texts is increased, and the matching degree of at least two dissimilar texts is decreased, improving the model's search capability. When facing databases with multiple language categories (such as search databases, information databases, etc.), the search purpose can be achieved through the familiar language, maximizing the utilization of data in the database.

[0111] Figure 7 This illustration shows an optional flowchart of the data processing method provided in an embodiment of the present disclosure. Figure 8A data diagram illustrating the data processing method is shown, and the steps will be explained accordingly.

[0112] Step S601: Determine the language category of the data to be retrieved.

[0113] In some embodiments, such as Figure 8 As shown, the input data to be retrieved is first classified based on the language recognition module to determine its language type.

[0114] In step S602, in response to the language affiliation type of the data to be retrieved being the first language affiliation type, the data to be retrieved is transmitted to the first encoder.

[0115] Among them, the language type of the text that the first encoder can process is the first language type.

[0116] In step S603, in response to the language affiliation type of the data to be retrieved being the second language affiliation type, the data to be retrieved is transmitted to the second encoder.

[0117] The second encoder can process text whose language affiliation type is the second language affiliation type.

[0118] Step S604: Based on the embedding value output by the first encoder or the embedding value output by the second encoder, determine the matching data corresponding to the data to be retrieved.

[0119] In some embodiments, the database is searched based on the embedding value output by the first encoder or the second encoder to determine the embedding value in the database that has the highest similarity (or matching degree) with the embedding value output by the first encoder or the second encoder. This value is then identified as the matching embedding value corresponding to the data to be retrieved. The matching embedding value is then mapped from the vector space to the discrete symbol space to obtain the matching data corresponding to the data to be retrieved.

[0120] Thus, through the data processing method provided in this embodiment, users with different language affiliations do not need to translate the text to be searched into the language affiliation corresponding to the search database in advance when conducting searches, which is convenient for users; when facing databases with multiple language affiliations (such as search databases, data databases, etc.), they can achieve their search objectives through their familiar language, thereby maximizing the utilization of data in the database.

[0121] Figure 9 A schematic diagram of an optional structure of the model training apparatus provided in an embodiment of this disclosure is shown, and the details will be described in terms of each part.

[0122] In some embodiments, the model training apparatus includes a first determining unit, a first masking unit, an encoding unit, a second masking unit, an input unit, a prediction unit, and a first adjustment unit.

[0123] The first determining unit is used to determine a first training dataset that includes multiple training data pairs; The first masking unit is used to process training data pairs of the first training dataset based on the first masking probability to obtain first masked text and second masked text with different language affiliation types. The encoding unit is used to input the first mask text into the first encoder to obtain a first encoded vector and a first mask text vector; and to input the second mask text into the second encoder to obtain a second encoded vector and a second mask text vector. The second masking unit is used to process the training data pairs in the first training dataset based on the second masking probability to obtain the third masking text and the fourth masking text with different language affiliation types. The input unit is used to input one of the first encoding vector and the second encoding vector, as well as one of the third mask text and the fourth mask text, into the cross-decoder to obtain the third mask text vector; The prediction unit is used to input the first masked text vector into the masked language model prediction head to obtain the first predicted text; input the second masked text vector into the masked language model prediction head to obtain the second predicted text; and input the third masked text vector into the masked language model prediction head, wherein the output of the masked language model prediction head is the third predicted text. The first adjustment unit is used to adjust the weights of the second encoder and the masked language model prediction head based on the first predicted text, the second predicted text, the third predicted text and the training data pair.

[0124] In some embodiments, the first mask probability is less than the second mask probability; the language classification of the text that the first encoder can process is different from the language classification of the text that the second encoder can process; the language classification of the first mask text is the same as the language classification of the text that the first encoder can process; the language classification of the second mask text is the same as the language classification of the text that the second encoder can process; each training data pair includes two training texts with different language classifications but the same semantics.

[0125] The input unit is specifically used to concatenate the first encoding vector and the third mask text to obtain the first concatenated text; The second encoded vector and the fourth masked text are concatenated to obtain the second concatenated text. The encoding vector portion in the first concatenated text and the second concatenated text are either replaced or not replaced, and the mask text portion in the first concatenated text and the second concatenated text is either replaced or not replaced, to obtain the input text of the cross-decoder; The input text is fed into the cross-decoder to obtain the third mask text vector.

[0126] A first loss is determined based on a first training text that has the same language affiliation type as the first predicted text and the training data pair; wherein, the first predicted text includes predictions of texts in the first training text that are masked based on a first masking probability; A second loss is determined based on a second training text in the training data pair that has the same language affiliation as the second predicted text; wherein the second predicted text includes predictions of text in the second training text that are masked based on a first masking probability; The third loss is determined based on the third predicted text and the training data; wherein the third predicted text includes the prediction of the text masked based on the second mask probability in the first training text or the second training text. Based on the first loss, the second loss, and the third loss, the first training loss is determined; The weights of the second encoder and the masked language model prediction head are adjusted based on the first training loss.

[0127] The first determining unit is further configured to determine a second training dataset, the second training dataset including at least one training sample, each sample including training text, first related text and multiple second related texts; The input unit is further configured to input the training text, the first related text, and the multiple second related texts into the first encoder or the second encoder based on the language affiliation type of each sample, including the training text, the first related text, and the multiple second related texts, to obtain the training text features corresponding to the training text, the first related features corresponding to the first related text, and the second related features corresponding to each second related text; In some alternative embodiments, the model training apparatus may further include a second adjustment unit.

[0128] The second adjustment unit is used to adjust the weights of the second encoder based on the training text features, the first relevant features, and all the second relevant features.

[0129] In each sample, the correlation between the first related text and the training text is greater than a first threshold; the correlation between the second related text and the training text is less than a second threshold.

[0130] The second adjustment unit is specifically used to determine the first correlation based on the training text features and the first correlation features; The second relevance is determined based on the training text features, the first relevance feature, and all the second relevance features; The second training loss is determined based on the first and second correlations; The weights of the second encoder are adjusted based on the second training loss.

[0131] Figure 10 A schematic diagram of an optional structure of the data processing apparatus provided in an embodiment of this disclosure is shown, and the details will be described in terms of each part.

[0132] In some embodiments, the data processing apparatus includes a language attribution unit, a first transmission unit, a second transmission unit, and a retrieval unit.

[0133] Language attribution unit, used to determine the language attribution type of the data to be retrieved; The first transmission unit is configured to transmit the data to be retrieved to the first encoder in response to the language affiliation type of the data to be retrieved being a first language affiliation type. The second transmission unit is configured to transmit the data to be retrieved to the second encoder in response to the language affiliation type of the data to be retrieved being the second language affiliation type. The retrieval unit is used to determine the matching data corresponding to the data to be retrieved based on the embedding value output by the first encoder or the embedding value output by the second encoder. The first encoder and the second encoder process different language affiliation types.

[0134] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0135] Figure 11 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0136] like Figure 11 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0137] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0138] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as data processing methods or model training methods. For example, in some embodiments, the data processing methods or model training methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data processing methods or model training methods described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform a data processing method or a model training method by any other suitable means (e.g., by means of firmware).

[0139] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0140] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0141] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0143] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0144] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0145] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0146] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0147] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A model training method, characterized in that, The model includes a first encoder, a second encoder, a cross-decoder, and a masked language model prediction head; the method includes: Determine a first training dataset that includes multiple training data pairs; The training data pairs of the first training dataset are processed based on the first mask probability to obtain first mask text and second mask text with different language attribution types. The first mask text is input into the first encoder to obtain the first encoded vector and the first mask text vector; the second mask text is input into the second encoder to obtain the second encoded vector and the second mask text vector. Based on the second mask probability, the training data pairs in the first training dataset are processed to obtain third and fourth mask texts with different language attribution types; One of the first and second encoded vectors, as well as one of the third and fourth masked texts, are input into the cross-decoder to obtain the third masked text vector; The first masked text vector is input into the masked language model prediction head to obtain the first predicted text; the second masked text vector is input into the masked language model prediction head to obtain the second predicted text. The third masked text vector is input into the masked language model prediction head, and the output of the masked language model prediction head is the third predicted text; Based on the first predicted text, the second predicted text, the third predicted text, and the training data pair, adjust the weights of the second encoder and the prediction head of the masked language model.

2. The method according to claim 1, characterized in that, The probability of the first mask is less than the probability of the second mask; The language type of text that the first encoder can process is different from the language type of text that the second encoder can process. The language type of the first masked text is the same as the language type of the text that the first encoder can process; The language type of the second masked text is the same as the language type of the text that the second encoder can process; Each training data pair includes two training texts that belong to different languages ​​but have the same semantics.

3. The method according to claim 1, characterized in that, The step of inputting one of the first and second encoded vectors, and one of the third and fourth masked texts, into the cross-decoder to obtain the third masked text vector includes: The first encoded vector and the third masked text are concatenated to obtain the first concatenated text; The second encoded vector and the fourth masked text are concatenated to obtain the second concatenated text. The encoding vector portion in the first concatenated text and the second concatenated text are either replaced or not replaced, and the mask text portion in the first concatenated text and the second concatenated text is either replaced or not replaced, to obtain the input text of the cross-decoder; The input text is fed into the cross-decoder to obtain the third mask text vector.

4. The method according to claim 1, characterized in that, The step of adjusting the weights of the second encoder and the masked language model prediction head based on the first predicted text, the second predicted text, and the training data pair includes: A first loss is determined based on a first training text that has the same language affiliation type as the first predicted text and the training data pair; wherein, the first predicted text includes predictions of texts in the first training text that are masked based on a first masking probability; A second loss is determined based on a second training text in the training data pair that has the same language affiliation as the second predicted text; wherein the second predicted text includes predictions of text in the second training text that are masked based on a first masking probability; The third loss is determined based on the third predicted text and the training data; wherein the third predicted text includes the prediction of the text masked based on the second mask probability in the first training text or the second training text. Based on the first loss, the second loss, and the third loss, the first training loss is determined; The weights of the second encoder and the masked language model prediction head are adjusted based on the first training loss.

5. The method according to claim 1, characterized in that, After adjusting the weights of the second encoder and the masked language model prediction head based on the first predicted text, the second predicted text, and the training data pair, the method further includes: Determine a second training dataset, which includes at least one training sample, each sample including training text, first related text and multiple second related texts; Based on the language affiliation type of each sample, including the training text, the first related text, and multiple second related texts, the training text, the first related text, and multiple second related texts are input into the first encoder or the second encoder to obtain the training text features corresponding to the training text, the first related features corresponding to the first related text, and the second related features corresponding to each second related text. The weights of the second encoder are adjusted based on the training text features, the first relevant features, and all the second relevant features; In each sample, the correlation between the first related text and the training text is greater than a first threshold; the correlation between the second related text and the training text is less than a second threshold.

6. The method according to claim 5, characterized in that, The step of adjusting the weights of the second encoder based on training text features, first relevant features, and all second relevant features includes: The first relevance is determined based on the features of the training text and the first relevance features; The second relevance is determined based on the training text features, the first relevance feature, and all the second relevance features; The second training loss is determined based on the first and second correlations; The weights of the second encoder are adjusted based on the second training loss.

7. A data processing method, characterized in that, The method, implemented based on the first encoder and second encoder of the model trained according to claims 1 to 6, includes: Determine the language category of the data to be retrieved; If the language affiliation type of the data to be retrieved is a first language affiliation type, then the data to be retrieved is transmitted to the first encoder; If the language affiliation type of the data to be retrieved is the second language affiliation type, then the data to be retrieved is transmitted to the second encoder; Based on the embedding value output by the first encoder or the embedding value output by the second encoder, determine the matching data corresponding to the data to be retrieved; The first encoder and the second encoder process different language affiliation types.

8. A model training device, characterized in that, The device includes: The first determining unit is used to determine a first training dataset that includes multiple training data pairs; The first masking unit is used to process training data pairs of the first training dataset based on the first masking probability to obtain first masked text and second masked text with different language affiliation types. The encoding unit is used to input the first mask text into the first encoder to obtain a first encoded vector and a first mask text vector; and to input the second mask text into the second encoder to obtain a second encoded vector and a second mask text vector. The second masking unit is used to process the training data pairs in the first training dataset based on the second masking probability to obtain the third masking text and the fourth masking text with different language affiliation types. The input unit is used to input one of the first encoding vector and the second encoding vector, as well as one of the third mask text and the fourth mask text, into the cross-decoder to obtain the third mask text vector; The prediction unit is used to input the first masked text vector into the masked language model prediction head to obtain the first predicted text; input the second masked text vector into the masked language model prediction head to obtain the second predicted text; and input the third masked text vector into the masked language model prediction head, wherein the output of the masked language model prediction head is the third predicted text. The first adjustment unit is used to adjust the weights of the second encoder and the masked language model prediction head based on the first predicted text, the second predicted text, the third predicted text and the training data pair.

9. A data processing apparatus, characterized in that, The device comprises a first encoder and a second encoder based on the model trained according to claims 1 to 6, and the device includes: Language attribution unit, used to determine the language attribution type of the data to be retrieved; The first transmission unit is configured to transmit the data to be retrieved to the first encoder in response to the language affiliation type of the data to be retrieved being a first language affiliation type. The second transmission unit is configured to transmit the data to be retrieved to the second encoder in response to the language affiliation type of the data to be retrieved being the second language affiliation type. The retrieval unit is used to determine the matching data corresponding to the data to be retrieved based on the embedding value output by the first encoder or the embedding value output by the second encoder. The first encoder and the second encoder process different language affiliation types.

10. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6; Alternatively, the method described in claim 7 may be performed.

Images