Text representation model training method and device, equipment, medium and program product

By using antonyms to input text representation model and adjusting parameters, the problems of low efficiency and error of antonyms processing in the prior art are solved, and more efficient and accurate text representation is achieved.

CN119089188APending Publication Date: 2024-12-06BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310658991.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-05
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

When existing text representation methods deal with antonyms, the separation of rules and models leads to low processing efficiency, and there is error in the antonyms mutual exclusion rules, resulting in dissimilar text representations corresponding to short sentences with high actual semantic similarity.

Method used

By inputting antonyms pairs into text representation model, antonyms encoding vector pairs are generated, and based on the antonyms encoding vector differences and sample text encoding vector differences, the parameters of the text representation model are adjusted, the dependence on rules is reduced, and processing efficiency and accuracy are improved.

Benefits of technology

提高了文本表示模型对于反义词的处理能力,减少了处理效率低和误差,实现了更准确的文本表示。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089188A_ABST
    Figure CN119089188A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a text representation model training method and device, equipment, a medium and a program product. A specific embodiment of the method comprises the following steps: inputting an antonym pair into a text representation model to obtain an antonym coding vector pair; generating antonym loss according to the difference between two antonym coding vectors in the antonym coding vector pair; respectively inputting the sample text and the mask sample text corresponding to the sample text into a text representation model to obtain a sample text coding vector and a mask sample text coding vector; generating a prediction text based on the sample text coding vector, the mask sample text coding vector and a text prediction network; generating a text generation loss according to the difference between the prediction text and the sample text; and adjusting parameters of the text representation model according to the antonym loss and the text generation loss. The implementation mode is related to artificial intelligence, and the antonym processing capacity of the text representation model is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a text representation model training method, apparatus, device, medium, and program product. Background Art

[0002] Text representation refers to the use of natural language processing technology to determine the vector representation of words or short sentences in the semantic space. Text representation is widely used in text retrieval, user opinion recognition, text sentiment analysis and other fields. When dealing with two antonyms, related text representation methods often use a combination of models and rules. That is, on the basis of the semantic representation model, the antonym mutual exclusion rule is used, that is, antonym pairs are sorted out. When antonym pairs appear in two sentences, the two sentences are considered to be dissimilar.

[0003] However, the inventors have found that when the above method is used to represent text, the following technical problems often occur:

[0004] First, the separation of rules and models requires additional rule plug-ins, which results in low processing efficiency.

[0005] Second, there are errors in the antonym mutual exclusion rule. For example, since "the noise is not small" and "the noise is large" contain a pair of antonyms "small" and "large", the text representations corresponding to the two short sentences with high actual semantic similarity will be dissimilar.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0007] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0008] Some embodiments of the present disclosure propose text representation model training methods, devices, equipment, media and program products to solve one or more of the technical problems mentioned in the above background technology section.

[0009] In a first aspect, some embodiments of the present disclosure provide a text representation model training method, the method comprising: inputting an antonym pair into the text representation model to obtain an antonym encoding vector pair; generating an antonym loss based on the difference between two antonym encoding vectors in the antonym encoding vector pair; inputting a sample text and a masked sample text corresponding to the sample text into the text representation model respectively to obtain a sample text encoding vector and a masked sample text encoding vector; generating a predicted text based on the sample text encoding vector, the masked sample text encoding vector and a text prediction network; generating a text generation loss based on the difference between the predicted text and the sample text; and adjusting the parameters of the text representation model based on the antonym loss and the text generation loss.

[0010] In a second aspect, some embodiments of the present disclosure provide a text representation model training device, the device comprising: a vector generation unit, configured to input an antonym pair into the text representation model to obtain an antonym encoding vector pair; an antonym loss generation unit, configured to generate antonym loss according to the difference between two antonym encoding vectors in the antonym encoding vector pair; an input unit, configured to input a sample text and a masked sample text corresponding to the sample text into the text representation model respectively to obtain a sample text encoding vector and a masked sample text encoding vector; a text generation unit, configured to generate a predicted text based on the sample text encoding vector, the masked sample text encoding vector and a text prediction network; a text loss generation unit, configured to generate a text generation loss according to the difference between the predicted text and the sample text; and an adjustment unit, configured to adjust the parameters of the text representation model according to the antonym loss and the text generation loss.

[0011] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0012] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0013] In a fifth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which implements the method described in any implementation manner of the above-mentioned first aspect when executed by a processor.

[0014] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the processing capability of the text representation model for antonyms is improved, which helps to improve the text representation processing efficiency. Specifically, the reason why the relevant text representation methods have low processing efficiency is that they use rule plug-ins for processing. Based on this, the text representation model training method of some embodiments of the present disclosure improves the text representation model's processing capability for antonyms by inputting antonym pairs into the text representation model and constraining the text representation model through antonym loss. Therefore, there is no need to use additional rule plug-ins, thereby improving the text representation processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0016] Figure 1 is a schematic diagram of an application scenario of a text representation model training method according to some embodiments of the present disclosure;

[0017] Figure 2 is a flowchart of some embodiments of the text representation model training method according to the present disclosure;

[0018] Figure 3 An exemplary schematic diagram of generating predictive text is shown;

[0019] Figure 4 is a flowchart of some other embodiments of the text representation model training method according to the present disclosure;

[0020] Figure 5 is a schematic diagram of the structure of some embodiments of the text representation model training device according to the present disclosure;

[0021] Figure 6 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0024] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0025] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0026] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0027] With regard to the collection, storage, and use of user personal information involved in this disclosure, before performing the corresponding operations, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, fulfilling the obligation to inform the personal information subject, and obtaining the authorization and consent of the personal information subject in advance.

[0028] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0029] Figure 1 It is a schematic diagram of an application scenario of a text representation model training method according to some embodiments of the present disclosure.

[0030] like Figure 1 As shown, the executor of the text representation model training method can first obtain a general text representation model. On this basis, the general text representation model is trained using an antonym pair library and a sample text library (such as an e-commerce short sentence text library). Thereby, antonym knowledge is integrated into the text representation model. Thus, a new text representation model can be obtained. It can be understood that the new text representation model incorporates antonym knowledge and the text representation capability is enhanced. Optionally, the new text representation model can be tested using test texts in the e-commerce short sentence text library. After the test passes, downstream applications can be performed, including but not limited to text retrieval, user opinion recognition, text sentiment analysis, etc.

[0031] It is understandable that the execution subject of the text representation model training method can be hardware or various software. When it is hardware, it can be various electronic devices with information processing capabilities, including but not limited to smart phones, tablet computers, e-book readers, laptop computers, desktop computers, etc. When it is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules for providing distributed services, or it can be implemented as a single software or software module. No specific limitation is made here.

[0032] Continue to refer Figure 2 , shows a process 200 of some embodiments of the text representation model training method according to the present disclosure. The text representation model training method comprises the following steps:

[0033] Step 201, input antonym pairs into a text representation model to obtain antonym encoding vector pairs.

[0034] In some embodiments, the execution subject of the text representation model training method can be any electronic device. On this basis, the execution subject can first obtain a pre-collected antonym pair library, wherein each antonym pair includes two semantically opposite words, for example, "slow" and "fast" constitute an antonym pair. On this basis, the antonym pair can be input into the text representation model to obtain an antonym encoding vector pair. Among them, the text representation model can be an artificial neural network for encoding text, such as a Transformer encoder, a recurrent neural network, etc.

[0035] Step 202: Generate antonym loss according to the difference between two antonym encoding vectors in the antonym encoding vector pair.

[0036] In some embodiments, the execution subject may generate antonym loss according to the difference between two antonym encoding vectors in the antonym encoding vector pair. For example, the difference between 1 and the cosine similarity between the two antonym encoding vectors may be determined as the antonym loss.

[0037] Step 203: input the sample text and the masked sample text corresponding to the sample text into the text representation model respectively to obtain the sample text encoding vector and the masked sample text encoding vector.

[0038] In some embodiments, the execution subject may input the sample text and the masked sample text into the text representation model respectively, and obtain the sample text encoding vector and the masked sample text encoding vector respectively. The masked sample text is obtained by masking the sample text. For example, at least one word in the sample text may be randomly masked to obtain the masked sample text. The sample text may be any phrase, sentence, etc.

[0039] In some optional implementations of some embodiments, the masked sample text is obtained by masking the keywords in the sample text. As an example, the keywords can be obtained by identifying the grammatical parse tree or by part-of-speech tagging. In these implementations, by masking the keywords, the model's understanding of key semantics can be improved, and the isolated understanding of antonyms can be avoided. For example, for the text "The noise is actually not big", if the text is represented based on "big" in isolation, the result will be contrary to the actual semantics. Marking and training "not big" as a whole means understanding "not big" as a whole, which helps to improve the model's understanding of key semantics and make the text representation more accurate.

[0040] Step 204, generating predicted text based on the sample text encoding vector, the masked sample text encoding vector and the text prediction network.

[0041] In some embodiments, the above-mentioned execution entity can generate predicted text based on the present text encoding vector, the masked sample text encoding vector and the text prediction network.

[0042] As an example, Figure 3 An exemplary schematic diagram of generating predicted text is shown. Figure 3 As shown, the sample text can be "not much noise", and the masked sample text can be "noise [mask] [mask]". On this basis, the generated sample text encoding vector is V, and the masked sample text encoding vector is V mask . Afterwards, the sample text encoding vector and the masked sample text encoding vector can be fused to obtain a fused text encoding vector. Afterwards, the fused text encoding vector is input into the text prediction network to obtain the predicted text. For example, the sample text encoding vector is V and the masked sample text encoding vector is V mask , fused text encoding vector C = [V, V mask , V*V mask ,|VV mask |]. The text prediction network may be a network for predicting the content of the masked part, such as a Transformer decoder. For example, “not big” may be obtained.

[0043] In some optional implementations of some embodiments, the sample text encoding vector and the masked sample text encoding vector may be concatenated and input into a text prediction network to generate predicted text.

[0044] Step 205: Generate text generation loss according to the difference between the predicted text and the sample text.

[0045] In some embodiments, the execution subject may generate a text generation loss based on the difference between the predicted text and the sample text. For example, the difference between the predicted text and the sample text may be determined using the following loss function and used as the text generation loss:

[0046]

[0047] Among them, k is the number of masks in the mask sample text, P Dec (t i |t <i , C) represents the output of the prediction network for the i-th mask position, that is, the predicted text of the i-th mask position. It can be understood that the prediction network predicts and outputs each mask position one by one, so the generation loss can be the logarithm and negation of the predicted text of each mask position.

[0048] Step 206: Adjust the parameters of the text representation model according to the antonym loss and the text generation loss.

[0049] In some embodiments, the execution subject may adjust the parameters of the text representation model by, for example, back propagation, random gradient descent, etc., based on the sum of the antonym loss and the text generation loss. It is understood that the above steps 201-206 may be repeated until the training end condition is met. As an example, the training end condition may be that the number of iterations reaches a preset threshold, the loss is less than a preset loss threshold, etc.

[0050] Some embodiments of the present disclosure provide a method that improves the text representation model's processing capability for antonyms by inputting antonym pairs into a text representation model and constraining the text representation model through antonym loss. Therefore, there is no need to use additional rule plug-ins, thereby improving the text representation processing efficiency.

[0051] Further references Figure 4 , which shows a process 400 of some other embodiments of the text representation model training method. The process 400 of the text representation model training method includes the following steps:

[0052] Step 401, input antonym pairs into a text representation model to obtain antonym encoding vector pairs.

[0053] Step 402: Generate antonym loss according to the difference between two antonym encoding vectors in the antonym encoding vector pair.

[0054] Step 403: Input the sample text and the masked sample text corresponding to the sample text into the text representation model respectively to obtain the sample text encoding vector and the masked sample text encoding vector.

[0055] Step 404, generating predicted text based on the sample text encoding vector, the masked sample text encoding vector and the text prediction network.

[0056] Step 405: Generate text generation loss according to the difference between the predicted text and the sample text.

[0057] In some embodiments, the specific implementation of the above steps 401-405 and the technical effects thereof can be referred to in Figure 2 The corresponding embodiments 201-205 will not be described in detail here.

[0058] Step 406: Input the sample text in the sample text group into the text representation model twice to obtain a first sample text encoding vector and a second sample text encoding vector corresponding to the sample text.

[0059] In some embodiments, due to random drop-out of the text representation model during the training phase, there is a difference between the first sample text encoding vector and the second sample text encoding vector obtained by inputting the text representation model twice for the same sample text.

[0060] Step 407 , generating a contrast loss by using the first sample text encoding vector and the second sample text encoding vector respectively corresponding to each sample text in the sample text group.

[0061] In some embodiments, as an example, the logarithm of the distance between the first sample text encoding vector and the second sample text encoding vector may be determined as the contrast loss.

[0062] In some optional implementations of some embodiments, for each sample text, the contrast loss of the sample text is generated according to the difference between the first sample text encoding vector corresponding to the sample text and the second sample text encoding vector, and the difference between the first sample text encoding vector corresponding to the sample text and the first sample text encoding vectors corresponding to the remaining sample texts in the sample text group. For example, for the same sample text group (a batch of sample texts), the contrast loss can be generated by the following loss function:

[0063]

[0064] Among them, f(v i , v′ i ) represents the first sample text encoding vector v corresponding to the i-th sample text i and the second sample text encoding vector v′ i The distance between (in radians on the hypersphere). f(v i , v j ) represents the first sample text encoding vector v corresponding to the i-th sample text and the j-th sample text respectively. iand v j The distance between (in radians on the hypersphere). The j-th sample text may be any sample text in the sample text group except the i-th sample text.

[0065] v a and v b The arc angle on the hypersphere can be determined by the following formula:

[0066]

[0067] In these implementations, by considering the difference between the first sample text encoding vector and the second sample text encoding vector corresponding to the sample text, the distance between the text representations of the same sample text can be made closer, and by considering the difference between the first sample text encoding vector corresponding to the sample text and the first sample text encoding vectors corresponding to the remaining sample texts in the sample text group, the distance between the text representations of different sample texts can be made farther, thereby solving the problem of uneven distribution of text representation in the representation space, increasing the distance of non-synonymous samples in the semantic space, and reducing the distance of synonymous samples, so that the representation model is more balanced in the representation of text in a specific field.

[0068] Step 408, adjusting the parameters of the text representation model according to the antonym loss, the text generation loss and the contrast loss.

[0069] In some embodiments, as an example, the above-mentioned execution entity can adjust the parameters of the text representation model according to the weighted sum of antonym loss, text generation loss and contrast loss, wherein the weights of antonym loss and text generation loss are equal, and the contrast loss weight can be less than the weight of antonym loss.

[0070] from Figure 4 It can be seen that Figure 2 Compared with the description of some corresponding embodiments, Figure 4 In some corresponding embodiments, process 400 of the text representation model training method inputs the text representation model twice by sample text and constrains it through contrast loss, thereby reducing the distance between synonymous samples in the representation space, thereby further improving the representation ability of the text representation model.

[0071] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a text representation model training device, and these device embodiments are Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0072] like Figure 5As shown, some embodiments of the text representation model training device 500 include: a vector generation unit 501, configured to input an antonym pair into the text representation model to obtain an antonym encoding vector pair; an antonym loss generation unit 502, configured to generate antonym loss according to the difference between two antonym encoding vectors in the antonym encoding vector pair; an input unit 503, configured to input the sample text and the masked sample text corresponding to the sample text into the text representation model respectively to obtain the sample text encoding vector and the masked sample text encoding vector; a text generation unit 504, configured to generate a predicted text based on the sample text encoding vector, the masked sample text encoding vector and the text prediction network; a text loss generation unit 505, configured to generate a text generation loss according to the difference between the predicted text and the sample text; an adjustment unit 506, configured to adjust the parameters of the text representation model according to the antonym loss and the text generation loss.

[0073] In an optional implementation of some embodiments, the device 500 also includes: a contrast loss generation unit, which is configured to: input the sample text in the sample text group into the text representation model twice to obtain a first sample text encoding vector and a second sample text encoding vector corresponding to the sample text; generate a contrast loss using the first sample text encoding vector and the second sample text encoding vector corresponding to each sample text in the sample text group; and the adjustment unit 506 is further configured to adjust the parameters of the text representation model according to the antonym loss, the text generation loss and the contrast loss.

[0074] In an optional implementation manner of some embodiments, the masked sample text is obtained by masking the keywords in the sample text.

[0075] In an optional implementation of some embodiments, the adjustment unit 506 is further configured to adjust the parameters of the text representation model according to a weighted sum of antonym loss, text generation loss and contrast loss, wherein the weights of antonym loss and text generation loss are equal.

[0076] In an optional implementation of some embodiments, the generation unit 504 is further configured to fuse the sample text encoding vector and the mask sample text encoding vector to obtain a fused text encoding vector; and input the fused text encoding vector into a text prediction network to obtain a predicted text.

[0077] In an optional implementation of some embodiments, the text loss generation unit 505 is further configured to generate, for each sample text, a contrast loss of the sample text according to the difference between the first sample text encoding vector and the second sample text encoding vector corresponding to the sample text, and the difference between the first sample text encoding vector corresponding to the sample text and the first sample text encoding vectors corresponding to the remaining sample texts in the sample text group.

[0078] It is understood that the units described in the device 500 are similar to those described in the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 500 and the units included therein, and will not be described in detail here.

[0079] Reference below Figure 6 , which shows a structural schematic diagram of an electronic device 600 suitable for implementing some embodiments of the present disclosure. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0080] like Figure 6 As shown, the electronic device 600 may include a processing device 601 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0081] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0082] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0083] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0084] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0085] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: inputs the antonym pair into the text representation model to obtain an antonym encoding vector pair; generates antonym loss according to the difference between the two antonym encoding vectors in the antonym encoding vector pair; inputs the sample text and the masked sample text corresponding to the sample text into the text representation model respectively to obtain the sample text encoding vector and the masked sample text encoding vector; generates predicted text based on the sample text encoding vector, the masked sample text encoding vector and the text prediction network; generates text generation loss according to the difference between the predicted text and the sample text; and adjusts the parameters of the text representation model according to the antonym loss and the text generation loss.

[0086] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0087] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0088] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including a vector generation unit, an antonym loss generation unit, an input unit, a text generation unit, a text loss generation unit, and an adjustment unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the vector generation unit may also be described as a "unit for generating antonym encoding vector pairs".

[0089] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0090] Some embodiments of the present disclosure also provide a computer program product, including a computer program, which implements any of the above-mentioned text representation model training methods when executed by a processor.

[0091] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A text representation model training method, comprising: Input the antonym pair into the text representation model to obtain the antonym encoding vector pair; generating antonym loss according to a difference between two antonym encoding vectors in the antonym encoding vector pair; Inputting the sample text and the masked sample text corresponding to the sample text into the text representation model respectively to obtain the sample text encoding vector and the masked sample text encoding vector; Generate predicted text based on the sample text encoding vector, the masked sample text encoding vector and the text prediction network; Generate a text generation loss according to the difference between the predicted text and the sample text; According to the antonym loss and the text generation loss, parameters of the text representation model are adjusted.

2. The method according to claim 1, wherein: Before adjusting the parameters of the text representation model according to the antonym loss and the text generation loss, the method further includes: Inputting a sample text in the sample text group into the text representation model twice to obtain a first sample text encoding vector and a second sample text encoding vector corresponding to the sample text; Generate a contrast loss by using the first sample text encoding vector and the second sample text encoding vector respectively corresponding to each sample text in the sample text group; and The adjusting the parameters of the text representation model according to the antonym loss and the text generation loss includes: Parameters of the text representation model are adjusted according to the antonym loss, the text generation loss and the contrast loss.

3. The method according to claim 1, wherein: The masked sample text is obtained by performing masking processing on the keywords in the sample text.

4. The method according to claim 2, wherein: The adjusting the parameters of the text representation model according to the antonym loss, the text generation loss and the contrast loss comprises: The parameters of the text representation model are adjusted according to a weighted sum of the antonym loss, the text generation loss and the contrast loss, wherein the antonym loss and the text generation loss have equal weights.

5. The method according to claim 1, wherein: The generating of predicted text based on the sample text encoding vector, the masked sample text encoding vector and the text prediction network comprises: Fusing the sample text encoding vector and the mask sample text encoding vector to obtain a fused text encoding vector; The fused text encoding vector is input into the text prediction network to obtain predicted text.

6. The method according to claim 2, wherein: The step of generating contrast loss by using the first sample text encoding vector and the second sample text encoding vector respectively corresponding to each sample text in the sample text group comprises: For each sample text, a contrast loss of the sample text is generated according to the difference between a first sample text encoding vector and a second sample text encoding vector corresponding to the sample text, and the difference between the first sample text encoding vector corresponding to the sample text and the first sample text encoding vectors corresponding to the remaining sample texts in the sample text group.

7. A text representation model training device, comprising: A vector generation unit is configured to input the antonym pair into the text representation model to obtain an antonym encoding vector pair; an antonym loss generating unit, configured to generate an antonym loss according to a difference between two antonym encoding vectors in the antonym encoding vector pair; An input unit, configured to input a sample text and a masked sample text corresponding to the sample text into the text representation model respectively, to obtain a sample text encoding vector and a masked sample text encoding vector; A text generation unit, configured to generate a predicted text based on the sample text encoding vector, the masked sample text encoding vector and a text prediction network; A text loss generating unit, configured to generate a text generation loss according to a difference between the predicted text and the sample text; An adjustment unit is configured to adjust parameters of the text representation model according to the antonym loss and the text generation loss.

8. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.