Text embedding training method, program instruction and related device

Abstract

The embodiment of the invention provides a text embedding training method, a program instruction and a related device. Wherein the electronic equipment obtains a pre-training model, and the pre-training model is obtained by pre-training a to-be-trained model through a pre-training text of a target domain; selecting a text pair from a fine tuning dataset, the fine tuning dataset originating from the target domain; fine tuning is conducted on the pre-training model through the text pair and a fine tuning instruction, and the fine tuning instruction is used for indicating the pre-training model to judge whether semantics of the text pair are similar or not; and if the pre-training model after fine tuning does not meet the preset convergence condition, returning to the step of selecting the text pair from the fine tuning data set until the preset convergence condition is met, and obtaining the text embedding model. Therefore, the pre-training model is obtained by utilizing the pre-training text of the target domain, and the pre-training model is subjected to micro-panning by utilizing a specific fine tuning method, so that the text embedding model suitable for the target domain is obtained.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and more specifically, to a text embedding training method, program instructions, and related devices. Background Art

[0002] As the business continues to develop and the documents in specific business areas are improved, the number of relevant documents accumulated by the enterprise is increasing, and the accumulated business knowledge areas are becoming more and more complex. Nowadays, when it is necessary to process and understand the knowledge in specific business areas in natural language, such as information retrieval, knowledge question and answer, text matching, text classification, etc., it is necessary to encode the text, that is, text embedding operation.

[0003] In practice, it is found that traditional and general-domain text embedding models have problems such as low efficiency, prone to errors, and difficulty in generalization when embedding files in specific business domains. Summary of the invention

[0004] In order to overcome at least one of the deficiencies in the prior art, the present application provides a text embedding training method, program instructions and related devices, specifically including:

[0005] In a first aspect, the present application provides a text embedding training method, the method comprising:

[0006] Obtaining a pre-trained model, wherein the pre-trained model is obtained by pre-training a model to be trained using pre-trained texts in a target field;

[0007] Selecting text pairs from a fine-tuning dataset, wherein the fine-tuning dataset is derived from the target domain;

[0008] Fine-tuning the pre-trained model using the text pair and the fine-tuning instruction, wherein the fine-tuning instruction is used to instruct the pre-trained model to determine whether the semantics of the text pair are similar;

[0009] If the fine-tuned pre-trained model does not meet the preset convergence condition, the process returns to selecting text pairs from the fine-tuning dataset until the preset convergence condition is met to obtain a text embedding model.

[0010] In conjunction with the optional implementation manner of the first aspect, the fine-tuning dataset includes a positive sample set and a negative sample set with semantically different positive samples in the positive sample set. After fine-tuning the pre-trained model by using the text pair and the fine-tuning instruction, the method further includes:

[0011] Generate new negative samples according to the text pairs;

[0012] The new negative sample is added to the negative sample set.

[0013] In conjunction with the optional implementation manner of the first aspect, before selecting the text pair from the fine-tuning dataset, the method further includes:

[0014] Acquire a positive sample set and an initial negative sample set that is semantically different from the positive samples in the positive sample set;

[0015] Randomly sampling the initial negative sample set to obtain a random negative sample set;

[0016] Sampling the initial negative sample set using the BM25 algorithm to obtain a BM25 negative sample set;

[0017] The fine-tuning dataset is obtained according to the positive sample set, the random negative sample set and the BM25 negative sample set.

[0018] In combination with the optional implementation of the first aspect, the pre-trained model has an automatic encoding network architecture.

[0019] In combination with the optional implementation manner of the first aspect, the automatic encoding network architecture of the model to be trained includes an encoder and a decoder, and the obtaining of the pre-trained model includes:

[0020] Get pre-trained text;

[0021] Masking the pre-trained text at a first masking ratio to obtain a first masked text;

[0022] Masking the pre-trained text at a second masking ratio to obtain a second masked text, wherein the second masking ratio is greater than the first masking ratio;

[0023] Processing the first masked text by the encoder to obtain a text encoding of the first masked text;

[0024] Inputting the text code of the second masked text and the text code of the first masked text into the decoder to obtain a reconstructed text at the masked position;

[0025] updating the model to be trained according to the difference between the reconstructed text and the actual text at the masked position;

[0026] If the updated model to be trained does not meet the preset convergence condition, the process returns to processing the first masked text through the encoder to obtain the text encoding of the first masked text, until the preset convergence condition is met, thereby obtaining the pre-trained model.

[0027] In combination with the optional implementation of the first aspect, the encoder is a BERT model, and the decoder is a decoding layer of a Transformer model.

[0028] In a second aspect, the present application further provides a text embedding training device, the device comprising:

[0029] A pre-training module, used to obtain a pre-training model, wherein the pre-training model is obtained by pre-training a model to be trained using pre-training texts in a target field;

[0030] A model fine-tuning module, configured to select text pairs from a fine-tuning dataset, wherein the fine-tuning dataset is derived from the target domain;

[0031] Fine-tuning the pre-trained model using the text pair and the fine-tuning instruction, wherein the fine-tuning instruction is used to instruct the pre-trained model to determine whether the semantics of the text pair are similar;

[0032] If the fine-tuned pre-trained model does not meet the preset convergence condition, the process returns to selecting text pairs from the fine-tuning dataset until the preset convergence condition is met to obtain a text embedding model.

[0033] In conjunction with the optional implementation manner of the second aspect, the fine-tuning dataset includes a positive sample set and a negative sample set with semantically different positive samples in the positive sample set, and the model fine-tuning module is further used to:

[0034] Generate new negative samples according to the text pairs;

[0035] The new negative sample is added to the negative sample set.

[0036] In conjunction with the optional implementation manner of the second aspect, before selecting the text pairs from the fine-tuning dataset, the model fine-tuning module is further used to:

[0037] Acquire a positive sample set and an initial negative sample set that is semantically different from the positive samples in the positive sample set;

[0038] Randomly sampling the initial negative sample set to obtain a random negative sample set;

[0039] Sampling the initial negative sample set using the BM25 algorithm to obtain a BM25 negative sample set;

[0040] The fine-tuning dataset is obtained according to the positive sample set, the random negative sample set and the BM25 negative sample set.

[0041] In combination with the optional implementation of the second aspect, the pre-trained model has an automatic encoding network architecture.

[0042] In conjunction with the optional implementation manner of the second aspect, the automatic encoding network architecture of the model to be trained includes an encoder and a decoder, and the pre-training module is specifically used for:

[0043] Get pre-trained text;

[0044] Masking the pre-trained text at a first masking ratio to obtain a first masked text;

[0045] Masking the pre-trained text at a second masking ratio to obtain a second masked text, wherein the second masking ratio is greater than the first masking ratio;

[0046] Processing the first masked text by the encoder to obtain a text encoding of the first masked text;

[0047] Inputting the text code of the second masked text and the text code of the first masked text into the decoder to obtain a reconstructed text at the masked position;

[0048] updating the model to be trained according to the difference between the reconstructed text and the actual text at the masked position;

[0049] If the updated model to be trained does not meet the preset convergence condition, the process returns to processing the first masked text through the encoder to obtain the text encoding of the first masked text, until the preset convergence condition is met, thereby obtaining the pre-trained model.

[0050] In combination with the optional implementation of the second aspect, the encoder is a BERT model, and the decoder is a decoding layer of a Transformer model.

[0051] In a third aspect, the present application also provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the text embedding training method is implemented.

[0052] In a fourth aspect, the present application also provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the text embedding training method is implemented.

[0053] In a fifth aspect, the present application also provides a program instruction, which implements the text embedding training method when executed by a processor.

[0054] Compared with the prior art, this application has the following beneficial effects:

[0055] The present embodiment provides a text embedding training method, program instructions and related devices. Among them, the electronic device obtains a pre-trained model, wherein the pre-trained model is obtained by pre-training the model to be trained through the pre-trained text of the target field; selects text pairs from the fine-tuning data set, wherein the fine-tuning data set originates from the target field; fine-tunes the pre-trained model through the text pairs and the fine-tuning instructions, wherein the fine-tuning instructions are used to instruct the pre-trained model to determine whether the semantics of the text pairs are similar; if the fine-tuned pre-trained model does not meet the preset convergence conditions, it returns to selecting text pairs from the fine-tuning data set until the preset convergence conditions are met, and the text embedding model is obtained. In this way, the pre-trained model is obtained using the pre-trained text of the target field, and the pre-trained model is fine-tuned using a unique fine-tuning method to obtain a text embedding model suitable for the target field. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 A flowchart of a text embedding training method provided in an embodiment of the present application;

[0058] Figure 2 An example flow chart provided for an embodiment of the present application;

[0059] Figure 3 A schematic diagram of the structure of a text embedding training device provided in an embodiment of the present application;

[0060] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0061] Icons: 11-pre-training module; 12-model fine-tuning module; 21-memory; 22-processor; 23-communication unit; 24-system bus. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0063] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0064] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0065] In the description of the present application, it should be noted that the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. In addition, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0066] Based on the above statement, as introduced in the background technology, traditional and general-domain text embedding models have problems such as low efficiency, prone to errors, and difficulty in generalization when performing embedding operations on documents in specific business domains.

[0067] For example, take the enterprise-specific business knowledge content as an example. Use mainstream Word2Vec, Glove, Sentence-bert, Simcse and other algorithm models, combine business knowledge texts to annotate texts according to the mask model, fine-tune training based on the data set, and then use the evaluation data for evaluation, and finally put it online. However, in practice, it was found that Word2Vec and Glove have lower model complexity than Transformer models, so the knowledge expression ability is insufficient; Sentence-Bert is simply based on NLI (Natural Language Inference) data set training, the task is relatively simple, and it is difficult to meet practical applications; Simcse uses contrastive learning, which requires a large number of negative samples and relies on similarity measurement functions. Therefore, for specific fields, it is difficult to achieve applicability by simply fine-tuning the existing model.

[0068] Based on the discovery of the above technical problems, the inventors have proposed the following technical solutions to solve or improve the above problems through creative work. It should be noted that the defects in the solutions in the above prior art are the results obtained by the inventors after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in the embodiments of the present application for the above problems below should all be the contributions made by the inventors to the present application in the process of invention and creation, and should not be understood as technical contents known to those skilled in the art.

[0069] In view of this, the present embodiment provides a text embedding training method. In the method, an electronic device obtains a pre-trained model, wherein the pre-trained model is obtained by pre-training a model to be trained through pre-training texts in the target field; a text pair is selected from a fine-tuning data set, wherein the fine-tuning data set originates from the target field; the pre-trained model is fine-tuned through the text pair and the fine-tuning instruction, wherein the fine-tuning instruction is used to instruct the pre-trained model to determine whether the semantics of the text pair are similar; if the fine-tuned pre-trained model does not meet the preset convergence condition, it returns to selecting a text pair from the fine-tuning data set until the preset convergence condition is met, and a text embedding model is obtained. In this way, the pre-trained model is obtained using the pre-trained text in the target field, and the pre-trained model is fine-tuned using a unique fine-tuning method to obtain a text embedding model suitable for the target field.

[0070] It should be noted that the electronic device implementing the text embedding training method may be, but is not limited to, a mobile terminal, a tablet computer, a laptop computer, a desktop computer, and a server that can provide sufficient computing power. The server may be a single server or a server group. The server group may be centralized or distributed (for example, the server may be a distributed system). In some embodiments, the server may be local or remote relative to the user terminal. In some embodiments, the server may be implemented on a cloud platform; as an example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud (Community Cloud), a distributed cloud, an inter-cloud (Inter-Cloud), a multi-cloud (Multi-Cloud), etc., or any combination thereof. In some embodiments, the server may be implemented on an electronic device having one or more components.

[0071] To make the solution provided by this embodiment clearer, the following uses a server as an electronic device and combines Figure 1 Each step of the method is described in detail. However, it should be understood that the operations of the flowchart may not be implemented in order, and steps without logical contextual relationships may be reversed in order or implemented simultaneously. In addition, those skilled in the art may add one or more other operations to the flowchart, or remove one or more operations from the flowchart under the guidance of the content of this application. Figure 1 As shown, the method includes:

[0072] S1, obtain the pre-trained model.

[0073] Among them, the pre-trained model is obtained by pre-training the model to be trained through the pre-training text of the target field. In this embodiment, the pre-trained model has an automatic encoding network architecture. Specifically, the automatic encoding network architecture of the model to be trained includes an encoder and a decoder. Exemplarily, the encoder is a BERT model, and the decoder is a decoding layer of a Transformer model. Among them, the BERT model specifically has 24 layers of 1024-dimensional hidden layers and 16 attention heads; the Transformer model specifically includes 1 layer of 1024-dimensional hidden layer and 16 attention heads. The following is a detailed description of the pre-training process of the model to be trained:

[0074] S1-1, obtain pre-training text.

[0075] For example, assuming that the target domain is a specific business of an enterprise, historical enterprise-specific business document data can be collected, and the data can be structured into objects through layout analysis and other technologies to form {"title": [AAA], "paragraph": [BBB], "table": [CCC], "header": [DDD], "footer": [EEE]}, and the structured data can be used as the original text for model training and evaluation. Then, the original text is analyzed, and the headers, footers, tables, etc. in the original text that do not contain much text semantic information are filtered out to obtain the pre-trained text.

[0076] Based on the pre-trained text obtained in the above implementation, step S1 further includes:

[0077] S1-2, masking the pre-trained text at a first masking ratio to obtain a first masked text.

[0078] In this regard, in this embodiment, the masked language model task can be combined to randomly mask each text in the pre-trained text at a ratio of 15-30% to obtain the first masked text. Of course, the above masking ratio is only an example provided by this embodiment and should not be regarded as a limitation of this embodiment.

[0079] S1-3, masking the pre-trained text at a second masking ratio to obtain a second masked text.

[0080] Among them, the second masking ratio is greater than the first masking ratio. In this regard, continuing with the above-mentioned 15-30% ratio as an example, each paragraph of the pre-trained text can be randomly masked at a ratio of 50%-70% to obtain the second masked text. In this way, by introducing an asymmetric model structure and differentiated masking ratios, the encoder is forced to learn higher-quality sentence vectors, thereby improving the performance of the model in tasks such as retrieval, sorting, sentence similarity, classification, and clustering.

[0081] S1-4, processing the first masked text through an encoder to obtain a text encoding of the first masked text.

[0082] S1-4, input the text encoding of the second masked text and the text encoding of the first masked text into the decoder to obtain the reconstructed text of the masked position.

[0083] S1-5, updating the model to be trained according to the difference between the reconstructed text and the actual text at the masked position.

[0084] It can be understood that in the pre-training task, the model to be trained uses the contextual information of the masked position to predict the text at the masked position and compares it with the actual text at the masked position, so that the model to be trained can learn the semantic information of the target domain. In this regard, this embodiment provides the following pre-training learning objective function:

[0085]

[0086] In the formula, Enc represents the encoder, Dec represents the decoder, Code pre-trained text and masked text separately. Indicates the encoding of masked text.

[0087] S1-6, determine whether the updated model to be trained meets the preset convergence condition. If not, return to S1-4 until the preset convergence condition is met to obtain the pre-trained model.

[0088] Based on the pre-trained model obtained in the embodiment of step S1 above, continue to refer to Figure 1 , the text embedding training method provided in this embodiment also includes:

[0089] S2, select text pairs from the fine-tuning dataset.

[0090] The fine-tuning dataset is derived from the target domain. For the fine-tuning dataset, this embodiment provides the following construction method of the fine-tuning dataset:

[0091] The server obtains a positive sample set and an initial negative sample set with semantically different positive samples in the positive sample set; randomly samples the initial negative sample set to obtain a random negative sample set; samples the initial negative sample set through a BM25 algorithm to obtain a BM25 negative sample set; and obtains a fine-tuning data set based on the positive sample set, the random negative sample set, and the BM25 negative sample set.

[0092] To make the above embodiment clearer, the following is an explanation with specific examples. The text paragraphs in the target field are judged to be similar according to the business expert rules, and positive samples are annotated, which include symmetric and asymmetric samples. The symmetric and asymmetric samples are collected as positive sample sets and represented as D + , where the symmetric samples in the positive sample set represent texts with consistent lengths, while the asymmetric sample pairs represent texts with large length differences.

[0093] Then, negative sample annotation is performed. The negative sample annotation process specifically includes: first, samples that are judged to be dissimilar according to the business expert rules are randomly sampled to form a random negative sample set, which is expressed as Then, run the BM25 algorithm to sample negative samples and form the BM25 negative sample set, which is expressed as The random negative sample set and the BM25 negative sample set are combined to form a negative sample set, that is, Finally, the positive sample set and the negative sample set are combined, that is, The data obtained by the union is split into fine-tuning data sets D train , and the validation dataset D test , the ratio between the two can be 7:3. Among them, the fine-tuning dataset is used for model fine-tuning training, and the validation set is used for model performance evaluation.

[0094] Based on the fine-tuning dataset obtained in the above example, text pairs are selected from the dataset to fine-tune the pre-trained model. Therefore, the text embedding training method provided in this embodiment also includes:

[0095] S3, fine-tuning the pre-trained model through text pairs and fine-tuning instructions.

[0096] Among them, the fine-tuning instruction is used to instruct the pre-trained model to determine whether the semantics of the text pair are similar.

[0097] For example, the two texts in the text pair are called text 1 and text 2 respectively. Then the fine-tuning instruction can be "Are [text 1] and [text 2] similar?" It can be understood that the fine-tuning instruction is a verbal prompt used to specify the nature of the fine-tuning task. In contrast, this embodiment also provides a fine-tuning training learning objective function:

[0098]

[0099] Where, symbols q and t represent the two texts in the text pair, t′∈T′ represents a negative sample, and γ represents the temperature coefficient.

[0100] In addition, it should be noted that in order to improve the fine-tuning effect, this embodiment also expands the existing negative sample set. In the specific expansion method, the server generates new negative samples based on the text pair; and adds the new negative samples to the negative sample set. For example, the server also processes each text in the text pair with an approximate nearest neighbor negative sampling model, mines a corresponding hard negative sample, adds it to the fine-tuning data set, and continues to be used to fine-tune the pre-trained model.

[0101] S4, determine whether the fine-tuned pre-trained model meets the preset convergence conditions. If so, return to step S2 until the preset convergence conditions are met to obtain the text embedding model.

[0102] To this end, the pre-trained model is fine-tuned and the encoder part of the pre-trained model is retained to obtain a text embedding model suitable for the target domain.

[0103] To make the above embodiments easier to understand, Figure 2 , the above embodiment is described again with a complete example. Figure 2 As shown, documents in specific business areas are obtained. If documents such as PDF and scanned files are encountered, OCR and other technologies are used to convert the documents into text data, and the text data is structured based on the layout analysis algorithm.

[0104] Then, based on the structured documents, the masked language model is used for style annotation to construct pre-trained text. In addition, similar text pairs and negative samples are annotated based on the structured documents, and the annotated samples are divided into pre-trained text and verification data sets.

[0105] Next, the pre-training model is trained using the constructed pre-training text, and the pre-training model is fine-tuned by constructing fine-tuning instructions using the verification dataset. In addition, the samples are expanded through approximate nearest neighbor negative sampling.

[0106] Finally, the validation dataset is used to evaluate the performance of the fine-tuned model.

[0107] Based on the same inventive concept as the text embedding training method provided in this embodiment, this embodiment also provides related devices, including virtual devices and physical devices. Specifically, this embodiment also provides a text embedding training device. The device includes at least one software function module that can be stored in a memory or solidified in an electronic device in the form of software. The processor in the electronic device is used to execute the executable module stored in the memory. For example, the software function module and computer program included in the text embedding training device. Please refer to Figure 3, functionally speaking, the device may include:

[0108] A pre-training module 11 is used to obtain a pre-training model, wherein the pre-training model is obtained by pre-training a model to be trained using pre-training texts in a target field;

[0109] A model fine-tuning module 12, configured to select text pairs from a fine-tuning dataset, wherein the fine-tuning dataset is derived from a target domain;

[0110] Fine-tune the pre-trained model using text pairs and fine-tuning instructions, wherein the fine-tuning instructions are used to instruct the pre-trained model to determine whether the semantics of the text pairs are similar;

[0111] If the fine-tuned pre-trained model does not meet the preset convergence conditions, the process returns to selecting text pairs from the fine-tuning dataset until the preset convergence conditions are met to obtain a text embedding model.

[0112] In this embodiment, the pre-training module 11 is used to implement Figure 1 In step S1, the model fine-tuning module 12 is used to implement Figure 1 Therefore, for the detailed description of each of the above modules, please refer to the specific implementation of the corresponding step. In addition, in view of the same inventive concept as the above text embedding training, the above modules can also be used to implement other steps or sub-steps of the method, which will not be described in detail in this embodiment.

[0113] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0114] It should also be understood that if the above implementation is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application.

[0115] Therefore, this embodiment also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by the processor, the text embedding training method provided in this embodiment is implemented. Among them, the storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, etc., which can store program codes.

[0116] This embodiment provides an electronic device for implementing a text embedding training method. Figure 4 The electronic device may include a processor 22 and a memory 21. In addition, the memory 21 stores a computer program, and the processor implements the text embedding training method provided in this embodiment by reading and executing the computer program corresponding to the above implementation in the memory 21.

[0117] Continue to see Figure 4 The electronic device further includes a communication unit 23. The memory 21, the processor 22 and the communication unit 23 are electrically connected to each other directly or indirectly through a system bus 24 to achieve data transmission or interaction.

[0118] The memory 21 may be an information recording device based on any electronic, magnetic, optical or other physical principle, used to record execution instructions, data, etc. In some embodiments, the memory 21 may be, but is not limited to, a volatile memory, a non-volatile memory, a storage drive, etc.

[0119] In some embodiments, the volatile memory may be a random access memory (RAM); in some embodiments, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), a flash memory, etc.; in some embodiments, the storage drive may be a disk drive, a solid-state drive, any type of storage disk (such as a CD, a DVD, etc.), or a similar storage medium, or a combination thereof, etc.

[0120] The communication unit 23 is used to send and receive data through a network. In some embodiments, the network may include a wired network, a wireless network, a fiber optic network, a telecommunication network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a wide area network (WAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, or a near field communication (NFC) network, or any combination thereof. In some embodiments, the network may include one or more network access points. For example, the network may include a wired or wireless network access point, such as a base station and / or a network switching node, through which one or more components of the service request processing system may be connected to the network to exchange data and / or information.

[0121] The processor 22 may be an integrated circuit chip having a signal processing capability, and the processor may include one or more processing cores (e.g., a single-core processor or a multi-core processor). By way of example only, the processor may include a central processing unit (CPU), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), or a microprocessor, or any combination thereof.

[0122] Understandably, Figure 4The structure shown is for illustration only. The electronic device may also have Figure 4 More or fewer components than shown, or with Figure 4 Different configurations shown. Figure 4 The components shown may be implemented in hardware, software or a combination thereof.

[0123] In addition, this embodiment also provides a program instruction, and a text embedding training method is implemented when the program instruction is executed by a processor.

[0124] It should be understood that the apparatus and method disclosed in the above-mentioned embodiments can also be implemented in other ways. The apparatus embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A text embedding training method, characterized in that: The method comprises: Obtaining a pre-trained model, wherein the pre-trained model is obtained by pre-training a model to be trained using pre-trained texts in a target field; Selecting text pairs from a fine-tuning dataset, wherein the fine-tuning dataset is derived from the target domain; Fine-tuning the pre-trained model using the text pair and the fine-tuning instruction, wherein the fine-tuning instruction is used to instruct the pre-trained model to determine whether the semantics of the text pair are similar; If the fine-tuned pre-trained model does not meet the preset convergence condition, the process returns to selecting text pairs from the fine-tuning dataset until the preset convergence condition is met to obtain a text embedding model.

2. The text embedding training method according to claim 1, characterized in that The fine-tuning data set includes a positive sample set and a negative sample set with semantically different positive samples in the positive sample set. After fine-tuning the pre-training model by using the text pair and the fine-tuning instruction, the method further includes: Generate new negative samples according to the text pairs; The new negative sample is added to the negative sample set.

3. The text embedding training method according to claim 1, characterized in that: Before selecting text pairs from the fine-tuning dataset, the method further includes: Acquire a positive sample set and an initial negative sample set that is semantically different from the positive samples in the positive sample set; Randomly sampling the initial negative sample set to obtain a random negative sample set; Sampling the initial negative sample set using the BM25 algorithm to obtain a BM25 negative sample set; The fine-tuning dataset is obtained according to the positive sample set, the random negative sample set and the BM25 negative sample set.

4. The text embedding training method according to claim 1, characterized in that The pre-trained model has an autoencoder network architecture.

5. The text embedding training method according to claim 4, characterized in that: The automatic encoding network architecture of the model to be trained includes an encoder and a decoder, and the obtaining of the pre-trained model includes: Get pre-trained text; Masking the pre-trained text at a first masking ratio to obtain a first masked text; Masking the pre-trained text at a second masking ratio to obtain a second masked text, wherein the second masking ratio is greater than the first masking ratio; Processing the first masked text by the encoder to obtain a text encoding of the first masked text; Inputting the text code of the second masked text and the text code of the first masked text into the decoder to obtain a reconstructed text at the masked position; updating the model to be trained according to the difference between the reconstructed text and the actual text at the masked position; If the updated model to be trained does not meet the preset convergence condition, the process returns to processing the first masked text through the encoder to obtain the text encoding of the first masked text, and the pre-trained model is obtained after the preset convergence condition is met.

6. The text embedding training method according to claim 5, characterized in that: The encoder is a BERT model, and the decoder is a decoding layer of a Transformer model.

7. A text embedding training device, characterized in that: The device comprises: A pre-training module, used to obtain a pre-training model, wherein the pre-training model is obtained by pre-training a model to be trained using pre-training texts in a target field; A model fine-tuning module, configured to select text pairs from a fine-tuning dataset, wherein the fine-tuning dataset is derived from the target domain; Fine-tuning the pre-trained model using the text pair and the fine-tuning instruction, wherein the fine-tuning instruction is used to instruct the pre-trained model to determine whether the semantics of the text pair are similar; If the fine-tuned pre-trained model does not meet the preset convergence condition, the process returns to selecting text pairs from the fine-tuning dataset until the preset convergence condition is met to obtain a text embedding model.

8. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the text embedding training method described in any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory stores a computer program, and when the computer program is executed by the processor, the text embedding training method described in any one of claims 1 to 6 is implemented.

10. A program instruction, characterized in that: When the program instructions are executed by a processor, the text embedding training method as described in any one of claims 1-6 is implemented.

Images