Text embedding model training method and related apparatuses

Abstract

The text embedding model training method and the related device provided in the application, the data processing device obtains the paraphrase vector of the target vocabulary and the initial vector of the sample text, inputs the paraphrase vector of the target vocabulary and the initial vector of the sample text into the first neural network model to be trained for training, and obtains a text embedding model having an embedding vector conversion function. Since the sample text includes the target vocabulary, and the paraphrase vector of the target vocabulary is an embedding vector representing the paraphrase information of the target vocabulary, the paraphrase information is used as prior knowledge, so that the trained text embedding model has a better embedding vector conversion effect.

Description

Technical Field

[0001] This application relates to the field of natural language processing, and more specifically, to a method and apparatus for training a text embedding model. Background Technology

[0002] In natural language processing tasks, text corpora cannot be directly used for feature extraction; instead, they need to be converted into numerical values ​​that computers can read. Therefore, an embedding vector transformation technique is proposed to convert text corpora into numerical values ​​that computers can directly process.

[0003] Therefore, the transformation effect of the embedding vector directly affects the subsequent processing effect of the text corpus; currently, the main approach is to improve the structure of the embedding vector transformation model in order to improve the transformation effect of the embedding vector from the algorithm level. Summary of the Invention

[0004] To overcome at least one deficiency in the prior art, one of the objectives of this application is to provide a text embedding model training method and related apparatus, used to train a text embedding model with better embedding vector transformation performance by using the semantic information of words as prior knowledge, specifically including:

[0005] Firstly, this application provides a method for training a text embedding model, the method comprising:

[0006] Obtain the definition vector of the first target word and the initial vector of the sample text, wherein the sample text includes the first target word, and the definition vector of the first target word is an embedding vector representing the definition information of the first target word.

[0007] The definition vector of the first target word and the initial vector of the sample text are input together into the first neural network model to be trained to obtain a text embedding model with embedding vector transformation function.

[0008] Secondly, this application provides a text embedding model training device, the device comprising:

[0009] The vector conversion module is used to obtain the definition vector of the first target word and the initial vector of the sample text, wherein the sample text includes the first target word, and the definition vector of the first target word is an embedding vector representing the definition information of the first target word.

[0010] The model training module is used to input the definition vector of the first target word and the initial vector of the sample text into the first neural network model to be trained for training, so as to obtain a text embedding model with embedding vector transformation function.

[0011] Thirdly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the text embedding model training method described above.

[0012] Fourthly, this application provides a data processing device, which includes a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it implements the text embedding model training method.

[0013] Compared with the prior art, this application has the following beneficial effects:

[0014] In the text embedding model training method and related apparatus provided in this application, the data processing device acquires the definition vector of the first target word and the initial vector of the sample text, and inputs the definition vector of the first target word and the initial vector of the sample text together into the first neural network model to be trained for training, thereby obtaining a text embedding model with embedding vector conversion function; since the sample text includes the first target word, and the definition vector of the first target word is an embedding vector representing the definition information of the first target word, using the definition information as prior knowledge enables the trained text embedding model to have a better conversion effect. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the method flow provided in the embodiments of this application;

[0017] Figure 2 A schematic diagram of the existing BERT input format provided for embodiments of this application;

[0018] Figure 3 A schematic diagram of the initial vector for concatenated text provided in an embodiment of this application;

[0019] Figure 4 A schematic diagram of the Bert input format provided for embodiments of this application;

[0020] Figure 5 This is a schematic diagram of the virtual device structure provided in the embodiments of this application;

[0021] Figure 6 This is a schematic diagram of the data processing device structure provided in an embodiment of this application.

[0022] Icons: 101 - Vector transformation module; 102 - Model training module; 210 - Memory; 220 - Processor; 230 - Communication unit; 240 - System bus. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0024] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0025] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0026] In the description of this application, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0027] As described in the background section, in natural language processing tasks, text corpora cannot be directly used for feature extraction and need to be converted into numerical values ​​that computers can read. Therefore, text embedding technology is introduced to convert text into vector-based embedding vectors. In this embodiment, the embedding vector can be the character vector of each character in the text, or it can be the word vector of each word after the text has been segmented.

[0028] For example, one-hot encoding assigns a number to each word in the corpus. However, for texts with a large number of words, the dimensionality of the encoding results obtained by one-hot encoding is extremely large. Therefore, the word2vec model was proposed, which aims to predict words through Skip-Gram (Continuous skip-gram Model) or CBOW (Continuous Bag-Of-Words) model, thereby realizing the conversion of text into embedding vectors through neural networks.

[0029] However, the word2vec model can only obtain a unique embedding vector for the same word and cannot adaptively adjust it according to the context of the word. For example, in "I like to eat apples" and "I like to use apples," "apple" refers to a type of fruit in the former and a type of mobile phone in the latter. Therefore, the BERT model was proposed, which can output different embedding vectors based on the context of the same word.

[0030] Therefore, current methods primarily focus on improving the structure of text embedding models to enhance the conversion performance of embedding vectors at the algorithmic level. However, research has found that, without optimizing the existing model structure, adding more prior information for the model to learn during embedding vector conversion can also improve the conversion performance of existing models.

[0031] It should be noted that the defects in the solutions in the prior art are all the results of the inventors’ practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the embodiments of this application in the following text should be the inventors’ contributions to this application in the process of invention and creation, and should not be understood as technical content known to those skilled in the art.

[0032] Based on the above inventive concept, this embodiment provides a text embedding model training method for data processing devices, which uses the definition information of the first target word as prior knowledge to train a text embedding model with better embedding vector transformation effect.

[0033] Since a large number of mathematical calculations are required during model training, any electronic device that can provide sufficient computing power for model training can be used as a data processing device.

[0034] For example, the data processing device may be a server. The server may be a single server or a group of servers. The server group may be centralized or distributed (e.g., the servers may be a distributed system). In some embodiments, the server may be local or remote relative to the user terminal. In some embodiments, the server may be implemented on a cloud platform; by way of example only, the cloud platform may include private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, inter-cloud, multi-cloud, etc., or any combination thereof. In some embodiments, the server may be implemented on an electronic device having one or more components.

[0035] Of course, in other embodiments, the data processing device can also be a user terminal. These include, by way of example only, mobile terminals, tablet computers, laptop computers, and desktop computers. In some embodiments, a mobile terminal can include smart home devices, smart mobile devices, etc. In some embodiments, smart home devices can include smart TVs, smart cameras, smart speakers, etc. In some embodiments, smart mobile devices can include smartphones, personal digital assistants (PDAs), gaming devices, navigation devices, etc.

[0036] Based on the above introduction, the following will combine... Figure 1 The training method for this text embedding model is explained in detail, step by step. Figure 1 As shown, the method includes:

[0037] S101, obtain the definition vector of the first target word and the initial vector of the sample text.

[0038] The sample text includes a first target word, and the definition vector of the first target word is the embedding vector of the definition information of the first target word.

[0039] In this embodiment, the number of first target words is not specifically limited; that is, the first target words can be some or all of the sample text. For example, assuming the sample text is "Spring is a suitable season for travel", and its word segmentation results are "spring", "is", "one", "suitable", "travel", and "season", then the first target words in the sample text can be at least one of "spring", "is", "one", "suitable", "travel", and "season".

[0040] It should also be understood that, for each neural network model involved in this embodiment, in order to facilitate the distinction between the input and output of the neural network model, the output vector is called the prediction vector, and the input vector or the vector compared with the prediction vector is called the initial vector.

[0041] Therefore, the initial vector of the sample text needs to meet the input format requirements of the first neural network. In an optional implementation, the first neural network can be a BERT-based network model, in which case the initial vector of the sample text includes word vectors, position vectors, and segment vectors.

[0042] like Figure 2 As shown, for the purpose of explaining character vectors, position vectors, and segment vectors, the sample text in English form, "my dog ​​is cute, he likes palying," is used as an example. However, it should be understood that the implementation scenarios of this embodiment are not limited to this.

[0043] First, the sample text "my dog ​​is cute, he likes playing" needs to be converted to:

[0044] “[CLS]my dog ​​is cute[SEP]he likes paly##ing[SEP]”

[0045] [CLS] is placed at the beginning of the sample text to serve as a header identifier in subsequent classification tasks, while [SEP] is placed between two sentences to serve as a sentence separator.

[0046] Then, each word (or character) in the sample text is converted into a fixed-length vector to obtain the character vector of the sample text.

[0047] Continue as Figure 2 As shown, the vector representation of "[CLS]" is E [CLS] The vector representation of "dog" is E. dog Other words follow the same principle, and will not be repeated in this example.

[0048] Position vector representation encodes the positional information of words into feature vectors, thereby introducing the positional relationships between words in the sample text.

[0049] Continue as Figure 2 As shown, the position information of the first word "[CLS]" is represented by E1, and the position information of the last word "[SEP]" is represented by E. 10 To express.

[0050] Segment vector representation encodes the information of the sentence to which a word belongs into a feature vector.

[0051] Continue as Figure 2 As shown, "[CLS]my dog ​​is cute" is the first sentence. Then, the information of the sentence to which each word belongs is represented by "E". A The second sentence is represented by "[SEP]he likes playing[SEP]". The information of the sentence to which each word belongs is represented by "E". B "to indicate".

[0052] Based on the above introduction regarding the initial vector of the sample text, this embodiment provides the following specific implementation method for obtaining the definition vector of the first target word, namely, step S101 includes:

[0053] S101-4: Based on the first target word, obtain the definition vector of the first target word from the definition dictionary.

[0054] The definition dictionary records definition vectors for multiple words, and this definition dictionary is obtained through a definition vector model. To obtain this definition vector model, this embodiment also trains a second neural network model using the definition texts of sample words. Therefore, step S101 further includes:

[0055] S101-1, Obtain the definition text of the sample words.

[0056] S101-2, input the initial vector of the explanatory text into the second neural network model to be trained to obtain the predicted vector of the sample words.

[0057] S101-3, based on the loss between the predicted vector of the sample words and the initial vector of the sample words, update the second neural network model until the definition vector model is obtained.

[0058] Therefore, in the above steps, the data processing device trains a second neural network model to predict the embedding vector of the sample words corresponding to the explanatory text using the explanatory text, until the second network model meets the preset convergence condition, and then obtains the explanatory vector model.

[0059] Thus, based on this definition vector model, the embedding vectors of multiple words can be further obtained, which can be combined into a definition dictionary. This allows the definition vector of the first target word in the sample text to be obtained by querying the definition dictionary when the first neural network model is trained with sample text.

[0060] Of course, this embodiment does not impose specific limitations on the structure of the second neural network model. When implementing this solution, those skilled in the art can choose according to their needs, such as the Bert structure, the Word2Vec structure, etc.

[0061] In this embodiment, a neural network model with a Bert structure is selected as the second neural network model, and the following implementation manners are provided to train the neural network model with a Bert structure to obtain a paraphrase vector model. When the second neural network model is a neural network model with a Bert structure, step S101-2 includes the following specific implementation manners:

[0062] S101-2-1, obtain a concatenated text according to the paraphrase text of the sample vocabulary.

[0063] The concatenated text includes a first mask of the sample vocabulary and a preprocessed text in which a second target vocabulary in the paraphrase text is replaced with a second mask. The first mask is different from the second mask, and the second mask is the default mask provided by the neural network model with a Bert structure. In this way, since the first mask is different from the second mask, the neural network model with a Bert structure can distinguish the sample vocabulary from the paraphrase text of the sample vocabulary according to the loss between the two during training, so as to achieve the purpose of training the neural network model with a Bert structure to predict the sample vocabulary corresponding to the paraphrase text according to the paraphrase text.

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following takes the sample vocabularies "spring" and "season" as examples for exemplary illustration. The respective paraphrases of "spring" and "season" are shown in the following table:

[0065]

[0066] Taking "spring" in the above table and its paraphrase text "the season between winter and summer, astronomically from the vernal equinox in March to the summer solstice in June" as an example, when the paraphrase vector of the first target vocabulary obtained based on the paraphrase vector model is a combination of single-character vectors, the concatenated text can be expressed as:

[0067] "[CLS]wmask wmask[SEP]winter to summer mask between the seasons, astronomically from the vernal equinox in March to the summer solstice in June"

[0068] That is, in the above concatenated text, the sample vocabulary "spring" is replaced with the mask "wmaskwmask", and the randomly determined "zhi", "jie", and "xue" in the paraphrase text are used as the second target vocabulary and replaced with the default mask "mask" provided by the neural network model with a Bert structure.

[0069] When the semantic vector of the first target word obtained based on the semantic vector model needs to be a word vector, then the semantic text of "spring" "the season between winter and summer, astronomically from the vernal equinox in March to the summer solstice in June" needs to be segmented. Assuming the segmentation result is "winter, to, summer, between, of, season, astronomy, on, is, from, March, of, vernal equinox, to, June, of, summer solstice", then the concatenated text can be represented as:

[0070] "[CLS]wmask[SEP] mask between winter and the summer solstice, astronomically from the spring equinox in March to the summer solstice in June."

[0071] In the above concatenated text, the sample word "spring" is replaced with the mask "wmask," and the randomly selected words "winter," "season," "to," and "of" from the definition text are used as the second target words, which are then replaced with the default mask "mask" provided by the BERT-structured neural network model.

[0072] However, it should be understood that "wmask" and "mask" are merely examples provided to facilitate the explanation of the concept and function of masks, and technicians may use other representations when implementing this solution.

[0073] S101-2-2, Based on the input format of the neural network model with BERT structure, obtain the initial vector of the concatenated text.

[0074] S101-2-3, the initial vector of the concatenated text is used as the initial vector of the explanatory text, and input into the second neural network model to obtain the prediction vectors of the sample words and the prediction vectors of the second target words.

[0075] Based on the prediction vectors of the above sample words and the prediction vectors of the second target word, step S101-3 may include the following specific implementation methods:

[0076] S101-3-1, based on the loss between the predicted vector of the sample words and the initial vector of the sample words, and the word-level classification task loss between the sample words and the second target words, update the second neural network model until the definition vector model is obtained.

[0077] For example, such as Figure 3 As shown, let's continue using the above concatenated text as an example:

[0078] "[CLS]wmask wmask[SEP] is the seasonal mask between winter and summer, astronomically spanning from the spring equinox in March to the summer solstice in June."

[0079] Convert the concatenated text into an initial vector that conforms to the BERT input format:

[0080] "[CLS]wmask…wmask[SEP]t1…mask…t n "

[0081] It should be understood that, for ease of description, the initial vector of the remaining nth word in the concatenated text is represented as t. n For detailed information on BERT input formats, please refer to [link to relevant documentation]. Figure 2 A detailed introduction is not required here.

[0082] Then, the data processing device inputs the initial vector of the concatenated text into the second neural network model with a BERT structure. This allows the second neural network model to output the predicted vectors of the wmask and mask marker positions based on the semantic information of the context within the concatenated text. Figure 3 The predicted vectors "character 1...character n" of the sample words and the predicted vector t of the second target word. i .

[0083] Finally, the data processing device updates the data based on the loss between the predicted vectors of the sample words and the initial vectors of the sample words, and the word-level classification task loss between the sample words and the second target word. Figure 3 The second neural network model in the process.

[0084] In a specific implementation, cosine similarity can be used to measure the loss between the predicted vector and the initial vector of the sample word. The word-level classification task loss between the sample word and the second target word is consistent with the word-level classification task loss in the original BERT Mask-LM, representing the loss between the one-hot encoding and the predicted one-hot encoding of the sample word, and the loss between the one-hot encoding and the predicted one-hot encoding of the second target word. That is, this embodiment, based on the original BERT Mask-LM word-level classification task loss, further introduces the loss in the embedding vector dimension between the predicted vector and the initial vector of the sample word, to achieve a better semantic vector conversion effect in the obtained semantic vector model.

[0085] Thus, the above training process in this example is repeated until the preset convergence condition is met, and the interpretation vector model is obtained.

[0086] It should be understood here that the second network model with the BERT structure includes specific hidden layers. During the nth (n>1)th round of training, the initial vectors of the sample words and the initial vectors of the second target words are actually the weight parameters of the hidden layer after n-1 rounds of updates. Therefore, the initial vectors of the sample words and the second target words do not remain unchanged after initialization, but change with the subsequent training process of the second network model with the BERT structure. For specific update details of the second network model with the BERT structure, please refer to the relevant introduction of BERT, which will not be repeated in this embodiment.

[0087] Based on the above introduction to the semantic vectors of the first target vocabulary and the initial vectors of the sample text, we continue as follows: Figure 1 As shown, the text embedding model training method also includes:

[0088] S102, input the definition vector of the first target word and the initial vector of the sample text into the first neural network model to be trained for training, and obtain a text embedding model with embedding vector transformation function.

[0089] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, it is continued to be assumed that the first neural network model is a neural network model with a BERT structure. Therefore, as Figure 4 As shown, in this embodiment... Figure 2 In addition to the input character vector, position vector, and segment vector shown, the definition vector of the first target word is also input.

[0090] In summary, in the above implementation of the text embedding model training method, the data processing device obtains the definition vector of the first target word and the initial vector of the sample text, and inputs the definition vector of the first target word and the initial vector of the sample text together into the first neural network model to be trained for training, thereby obtaining a text embedding model with embedding vector transformation function. Since the sample text includes the first target word, and the definition vector of the first target word is the embedding vector representing the definition information of the first target word, using the definition information as prior knowledge enables the trained text embedding model to have a better embedding vector transformation effect.

[0091] Based on the same inventive concept as the text embedding model training method, this embodiment also provides a text embedding model training apparatus applied to a data processing device. This text embedding model training apparatus includes at least one software functional module that can be stored in memory or embedded in the operating system (OS) of the data processing device. The processor in the data processing device executes the executable module stored in memory. For example, the software functional module and computer program included in the text embedding model training apparatus. Please refer to... Figure 5 Functionally, text embedding model training devices can include:

[0092] The vector conversion module 101 is used to obtain the definition vector of the first target word and the initial vector of the sample text, wherein the sample text includes the first target word, and the definition vector of the first target word is an embedding vector representing the definition information of the first target word.

[0093] In this embodiment, the vector transformation module 101 is used to implement Figure 1 For a detailed description of step S101, please refer to the detailed description of step S101.

[0094] The model training module 102 is used to input the definition vector of the first target word and the initial vector of the sample text into the first neural network model to be trained for training, so as to obtain a text embedding model with embedding vector transformation function.

[0095] In this embodiment, the model training module 102 is used to implement Figure 1 For a detailed description of step S102, please refer to the detailed description of step S102.

[0096] In addition, it is worth noting that since the text embedding model training method and the text embedding model training device have the same inventive concept, the above vector conversion module 101 and model training module 102 can also be used to implement other steps or sub-steps of the text embedding model training method, which will not be described in detail in this embodiment.

[0097] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0098] It should also be understood that if the above embodiments are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0099] Therefore, this embodiment also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the text embedding model training method provided in this embodiment. The computer-readable storage medium can be any medium capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0100] This embodiment also provides a data processing device. For example... Figure 6 As shown, the data processing device may include a processor 220 and a memory 210. The processor 220 and the memory 210 can communicate via a system bus 240. Furthermore, the memory 210 stores a computer program, and the processor implements the text embedding model training method provided in this embodiment by reading and executing the computer program corresponding to the above embodiments stored in the memory 210.

[0101] Continue as Figure 6 As shown, the data processing device also includes a communication unit. The memory 210, processor 220, and communication unit 230 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0102] The memory 210 can be an information recording device based on any electronic, magnetic, optical, or other physical principles, used to record execution instructions, data, etc. In some embodiments, the memory 210 can be, but is not limited to, volatile memory, non-volatile memory, memory drive, etc.

[0103] For example only, the volatile memory can be random access memory (RAM). The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), flash memory, etc.; the storage drive can be a disk drive, solid-state drive, any type of storage disk (such as optical disc, DVD, etc.), or similar storage media, or a combination thereof.

[0104] The communication unit 230 is used to send and receive data over a network. In some embodiments, the network may include a wired network, a wireless network, a fiber optic network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, or a near field communication (NFC) network, or any combination thereof. In some embodiments, the network may include one or more network access points. For example, the network may include wired or wireless network access points, such as base stations and / or network switching nodes, through which one or more components of the service request processing system can connect to the network to exchange data and / or information.

[0105] The processor 220 may be an integrated circuit chip with signal processing capabilities, and the processor may include one or more processing cores (e.g., a single-core processor or a multi-core processor). By way of example only, the processor described above may include a Central Processing Unit (CPU), an Application-Specific Integrated Circuit (ASIC), an Application-Specific Instruction-set Processor (ASIP), a Graphics Processing Unit (GPU), a Physics Processing Unit (PPU), a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), a controller, a microcontroller unit, a Reduced Instruction Set Computing (RISC) computer, or a microprocessor, or any combination thereof.

[0106] Based on the relevant descriptions in the above embodiments, the following is combined with... Figure 2 The flowchart shown illustrates in detail the text embedding model training method provided in this embodiment. However, it should be understood that the operations in the flowchart may not be implemented in sequence; steps without logical contextual relationships may be reversed or implemented simultaneously. Furthermore, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0107] It should be understood that the apparatus and methods disclosed in the above embodiments can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0108] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for training a text embedding model, characterized in that, The method includes: Obtain the definition text of the sample words; Based on the definition text of the sample words, a concatenated text is obtained, wherein the concatenated text includes a first mask of the sample words and a preprocessed text in which a second target word in the definition text is replaced with a second mask, wherein the first mask is different from the second mask, and the second mask is a default mask provided by a neural network model with a BERT structure. Based on the input format of the neural network model of the BERT structure, the initial vector of the concatenated text is obtained; The initial vector of the concatenated text is used as the initial vector of the explanatory text and input into the second neural network model to obtain the predicted vectors of the sample words and the predicted vectors of the second target words. The second neural network model is updated based on the loss between the predicted vector of the sample words and the initial vector of the sample words until the definition vector model is obtained. Based on the first target word, the definition vector of the first target word is obtained from the definition dictionary, wherein the definition dictionary records the definition vectors of multiple words, and the definition dictionary is obtained through the definition vector model; Obtain the initial vector of the sample text, wherein the sample text includes the first target word, and the definition vector of the first target word is an embedding vector representing the definition information of the first target word. The definition vector of the first target word and the initial vector of the sample text are input together into the first neural network model to be trained to obtain a text embedding model with embedding vector transformation function.

2. The text embedding model training method according to claim 1, characterized in that, The step of updating the second neural network model based on the loss between the predicted vector of the sample words and the initial vector of the sample words until the definition vector model is obtained includes: The second neural network model is updated based on the loss between the predicted vector of the sample word and the initial vector of the sample word, and the word-level classification task loss between the sample word and the second target word, until the semantic vector model is obtained.

3. The text embedding model training method according to claim 1, characterized in that, The first neural network model is a BERT-structured neural network model.

4. A text embedding model training device, characterized in that, The device includes: The vector conversion module is used to obtain the definition vector of the first target word and the initial vector of the sample text, wherein the sample text includes the first target word, and the definition vector of the first target word is an embedding vector representing the definition information of the first target word. The vector conversion module is further specifically used to obtain the definition vector of the first target word from the definition dictionary based on the first target word, wherein the definition dictionary records the definition vectors of multiple words and is obtained through a definition vector model; The model training module is used to input the definition vector of the first target word and the initial vector of the sample text into the first neural network model to be trained for training, so as to obtain a text embedding model with embedding vector transformation function. The text embedding model training device is also used for: Obtain the definition text of the sample words; Based on the definition text of the sample words, a concatenated text is obtained, wherein the concatenated text includes a first mask of the sample words and a preprocessed text in which a second target word in the definition text is replaced with a second mask, wherein the first mask is different from the second mask, and the second mask is a default mask provided by a neural network model with a BERT structure. Based on the input format of the neural network model of the BERT structure, the initial vector of the concatenated text is obtained; The initial vector of the concatenated text is used as the initial vector of the explanatory text and input into the second neural network model to obtain the predicted vectors of the sample words and the predicted vectors of the second target words. The second neural network model is updated based on the loss between the predicted vector of the sample word and the initial vector of the sample word until the semantic vector model is obtained.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the text embedding model training method according to any one of claims 1-3.

6. A data processing device, characterized in that, The data processing device includes a processor and a memory, the memory storing a computer program, which, when executed by the processor, implements the text embedding model training method according to any one of claims 1-3.

Images