Training method and device of key phrase generation model, and key phrase generation method and device

Abstract

Embodiments of the present application provide a key phrase generation model training method, a key phrase generation method and device. The training method comprises: obtaining training documents and their corresponding key phrase true values; based on optimal transport theory, assigning the key phrase true values to control codes; inputting the training documents into a generator to make the generator output candidate key phrases; inputting the candidate key phrases into a selector to make the selector output decision labels, the decision labels being used to indicate whether to retain or discard the corresponding candidate key phrases; constructing a first loss function corresponding to the generator and a second loss function corresponding to the selector, and respectively optimizing the generator and the selector according to the first loss function and the second loss function to obtain a target key phrase generation model. The technical solution of the embodiments of the present application can take into account the accuracy and recall rate of key phrase generation, and ensure the quality of key phrase generation.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a training method, a key phrase generation method, and an apparatus for a key phrase generation model. Background Technology

[0002] A keyphrase is a word or phrase that represents or captures the core concepts and themes of a document, article, report, research paper, or other textual material. Keyphrases allow users to grasp the main information of a document without reading the entire document. Current technologies often employ a single model for keyphrase generation (KPG), such as Large Language Models (LLMs). However, relying on a single model for keyphrase generation cannot simultaneously achieve high precision and high recall. Therefore, balancing precision and recall in keyphrase generation while ensuring high-quality keyphrase generation has become a pressing technical challenge. Summary of the Invention

[0003] The embodiments of this application provide a training method, a key phrase generation method, and an apparatus for a key phrase generation model, which can at least to some extent balance the accuracy and recall of key phrase generation and ensure the quality of key phrase generation.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to one aspect of the embodiments of this application, a training method for a key phrase generation model is provided. The key phrase generation model includes a generator and a selector, wherein the generator is based on a Transformer, the decoder of the generator has several control codes, and the selector is constructed based on a large language model.

[0006] The method includes:

[0007] Obtain the training document and the truth values ​​of its corresponding key phrases;

[0008] Based on optimal transmission theory, the truth values ​​of several key phrases are assigned to several control codes as monitoring signals;

[0009] The training document is input into the generator so that the generator outputs a number of candidate key phrases corresponding to the training document;

[0010] A number of candidate key phrases are input into the selector so that the selector outputs a decision label corresponding to each candidate key phrase, the decision label being used to indicate whether to retain or discard the corresponding candidate key phrase;

[0011] A first loss function corresponding to the generator and a second loss function corresponding to the selector are constructed, and the generator and the selector are tuned according to the first loss function and the second loss function respectively to obtain the target key phrase generation model.

[0012] According to one aspect of the embodiments of this application, a key phrase generation method is provided, comprising:

[0013] Get the document to be processed;

[0014] The document to be processed is input into a pre-trained target key phrase generation model, so that the target key phrase generation model outputs at least one target key phrase corresponding to the document to be processed. The target key phrase generation model is trained by the training method described in the above embodiments.

[0015] According to one aspect of the embodiments of this application, a training apparatus for a key phrase generation model is provided. The key phrase generation model includes a generator and a selector, wherein the generator is based on a Transformer, the decoder of the generator has several control codes, and the selector is constructed based on a large language model.

[0016] The device includes:

[0017] The first acquisition module acquires the training documents and the truth values ​​of several corresponding key phrases;

[0018] The allocation module is used to allocate the truth values ​​of several key phrases to several control codes as monitoring signals based on the optimal transmission theory.

[0019] The first input module is used to input the training document into the generator so that the generator outputs a number of candidate key phrases corresponding to the training document;

[0020] The second input module is used to input a number of candidate key phrases into the selector, so that the selector outputs a decision label corresponding to each candidate key phrase, and the decision label is used to indicate whether to retain or discard the corresponding candidate key phrase;

[0021] The first processing module is used to construct a first loss function corresponding to the generator and a second loss function corresponding to the selector, and to fine-tune the generator and the selector according to the first loss function and the second loss function respectively, so as to obtain the target key phrase generation model.

[0022] According to one aspect of the embodiments of this application, a key phrase generation apparatus is provided, comprising:

[0023] The second acquisition module is used to acquire the document to be processed.

[0024] The second processing module is used to input the document to be processed into a pre-trained target key phrase generation model, so that the target key phrase generation model outputs at least one target key phrase corresponding to the document to be processed. The target key phrase generation model is trained by the training method described in the above embodiments.

[0025] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the above embodiments.

[0026] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to perform the method described in the above embodiments.

[0027] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in the above embodiments.

[0028] In some embodiments of this application, the key phrase generation model includes a generator and a selector. The generator uses a Transformer as its backbone, and its decoder has several control codes. The selector is constructed based on a large language model. By acquiring training documents and their corresponding key phrase truth values, and based on optimal transmission theory, the key phrase truth values ​​are assigned to several control codes as supervision signals. The training documents are input into the generator so that the generator outputs several candidate key phrases corresponding to the training documents. These candidate key phrases are then input into the selector so that the selector outputs a decision label corresponding to each candidate key phrase. This decision label indicates whether to retain or discard the corresponding candidate key phrase. Next, a first loss function corresponding to the generator and a second loss function corresponding to the selector are constructed. The generator and selector are then tuned according to the first and second loss functions to obtain the target key phrase.

[0029] Therefore, the technical solution provided in this application embodiment can correctly recall key phrases through the generator and eliminate erroneous candidates through the selector. Thus, the key phrase generation model can balance the accuracy and recall rate of key phrase generation, ensuring the quality of key phrase generation.

[0030] Furthermore, since the number of control codes usually exceeds the number of key phrase truth values, many control codes are undertrained. The technical solution provided in this application is based on optimal transmission theory to determine the allocation scheme between key phrase truth values ​​and control codes, allowing a key phrase truth value to be assigned to multiple control codes, thereby ensuring the rationality of key phrase truth value allocation and the training effect of each control code.

[0031] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0033] Figure 1 A flowchart illustrating a training method for a key phrase generation model according to an embodiment of this application is shown.

[0034] Figure 2 A block diagram of a training apparatus for a key phrase generation model according to an embodiment of this application is shown;

[0035] Figure 3 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0036] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0037] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0038] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0039] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0040] Figure 1 A flowchart illustrating a training method for a key phrase generation model according to an embodiment of this application is shown. It should be noted that this method can be applied to a terminal device or a server, wherein the terminal device may include one or more of a smartphone, tablet computer, laptop, and desktop computer; the server may be a physical server or a cloud server.

[0041] In one embodiment, the key phrase generation model provided in this application includes a generator and a selector. The generator's main objective is to generate a set of candidate key phrases that represent the core concepts of a given document. Specifically, the generator is built based on the ONE2SET paradigm and uses the Transformer model as its backbone. The Transformer model employs an encoder-decoder architecture, where the encoder processes the input sequence and the decoder generates the output sequence. Furthermore, the decoder is assigned multiple control codes, each responsible for generating one candidate key phrase, thereby improving recall.

[0042] The primary function of the selector is to sift through the candidate key phrases generated by the generator to select the most relevant and accurate key phrases. It is built upon a large language model, leveraging its powerful semantic understanding capabilities to perform this task, ensuring that the final set of key phrases is both comprehensive and highly relevant. Specifically, the selector models the key phrase selection process as a sequence labeling task, where each candidate key phrase is labeled either "keep" (T) or "discard" (F). The selector generates these decision labels autoregressively, taking into account previously made choices, reducing semantic redundancy, and improving the accuracy of the selection.

[0043] Please continue to refer to this. Figure 1 The training method for this key phrase generation model includes at least steps S110 to S150, which are described in detail below (the following explanation uses the application of this method to a terminal device as an example, hereinafter referred to as "terminal"):

[0044] In step S110, the training document and its corresponding key phrase truth values ​​are obtained.

[0045] In one embodiment, those skilled in the art can pre-construct a training database for model training. This training database may include several training documents and several key phrase truth values ​​corresponding to each training document. These key phrase truth values ​​can be obtained from manual annotation of the documents or confirmation from authoritative sources, and can accurately represent the core concepts and themes of the corresponding training documents. Therefore, before training, the terminal can obtain the training documents and their corresponding key phrase truth values ​​from the pre-constructed training database for subsequent training.

[0046] In step S120, based on optimal transmission theory, the truth values ​​of several key phrases are assigned to several control codes as monitoring signals.

[0047] In this embodiment, since the number of control codes usually exceeds the number of key phrase truth values, many control codes are prone to insufficient training. To solve this problem, the terminal can adopt optimal transmission theory to transmit key phrase truth values... or φ(y) M Dynamically assigning control codes as supervision signals ensures a reasonable distribution between key phrase truth values ​​and control codes, improving training efficiency and quality. Wherein, y i φ represents the truth value of the i-th key phrase, and there are a total of M-1 non-empty truth values; φ represents the meaningless empty symbol used for padding, i.e., the M-th truth value.

[0048] In one embodiment of this application, based on optimal transmission theory, the truth values ​​of several key phrases are assigned to several control codes as monitoring signals, including:

[0049] Generate several minimum text units for each of the control codes;

[0050] For each of the control codes, determine the matching score between the truth value of each key phrase and the smallest text unit corresponding to the control code;

[0051] Based on optimal transmission theory, the target allocation scheme between the key phrase truth value and the control code is determined by calculating the matching score between the smallest text unit corresponding to each control code and the truth value of each key phrase.

[0052] In this embodiment, K tokens (i.e., at least one smallest text unit) can be generated for each control code as predictions for that control code. The smallest text unit can be a word or a letter, etc. Then, the terminal can use the pairwise matching function C... match (*) Calculate the truth value y for each key phrase. i According to the prediction Matching score μ between ij :

[0053]

[0054]

[0055] Where τ is the normalized hyperparameter. K′=min(|y i |,K), where K is the smallest number of text units to predict. Represents the smallest unit of text In probability distribution The probability value in.

[0056] Once the matching score between the truth value of each key phrase and the prediction of each control code is determined, the allocation between the truth value of the key phrase and the control code can be regarded as an optimal transmission problem based on the matching score, and the optimal allocation with Sinkhorn-Knopp iteration, i.e., the target allocation scheme, can be searched.

[0057] In one embodiment, based on optimal transmission theory, a target allocation scheme between the key phrase truth values ​​and the control codes is determined by calculating the matching score between the smallest text unit corresponding to each control code and the truth value of each key phrase, including:

[0058] The key phrase truth value is used as the supplier and its corresponding supply vector is constructed, and the control code is used as the demand side and its corresponding demand vector is constructed.

[0059] Construct an allocation cost matrix between the key phrase truth values ​​and the control codes, where each element in the allocation cost matrix represents the allocation cost of allocating each key phrase truth value to each control code, and the allocation cost is negatively correlated with the matching score between the two.

[0060] Using optimal transmission theory, the target allocation scheme between the key phrase truth value and the control code is determined based on the supply vector, the demand vector, and the allocation cost matrix.

[0061] In this embodiment, the control code can be regarded as the demand side, and its corresponding demand vector can be constructed. Where N is the number of control codes, d j This represents the number of truth values ​​of the key phrase assigned to the j-th control code.

[0062] Treat the truth values ​​of key phrases as suppliers and construct their corresponding supply vectors. Among them, s i Let y represent the truth value of the i-th key phrase. i The number of control codes that can be assigned.

[0063] Next, an allocation cost matrix is ​​constructed between the truth values ​​of key phrases and control codes. Among them, c ij This indicates that the truth value y of the i-th key phrase is... i The cost allocated to the j-th control code.

[0064] It should be noted that since assigning truth values ​​of multiple key phrases to a single control code simultaneously may cause interference, d can be limited. j The value of is 1. Next, if y... i If it matches more control codes highly, then the y i More control codes should be assigned, therefore, s can be used. i Defined as with The number of positively correlated dynamic quantities:

[0065]

[0066] in, This indicates rounding up, where k is a predefined hyperparameter.

[0067] Furthermore, in order to simulate the truth value y of the key phrase i With prediction The higher the matching score between the two phrases, the higher the truth value of the key phrase y. i If the cost of assigning the j-th control code is lower, then c can be defined. ij for:

[0068]

[0069] With the above vectors and matrices, the optimal allocation of π* can be found according to the following objective function:

[0070]

[0071] Thus, each control code can be assigned the key phrase truth value or φ with the maximum matching score, i.e., the target assignment scheme.

[0072] Please continue to refer to this. Figure 1 In step S130, the training document is input into the generator so that the generator outputs a number of candidate key phrases corresponding to the training document.

[0073] In this embodiment, the terminal can input training documents into the generator so that each control code in the generator generates a candidate key phrase for subsequent screening.

[0074] In step S140, several candidate key phrases are input into the selector so that the selector outputs a decision label corresponding to each candidate key phrase, the decision label being used to indicate whether to retain or discard the corresponding candidate key phrase.

[0075] In this embodiment, after obtaining several key phrases, the terminal can input them into the selector, thereby using the selector to filter out the most relevant and accurate key phrases from the candidate key phrases generated by the generator.

[0076] Specifically, the key phrase selection process can be modeled as a sequence labeling task, where a selector generates a decision label for each candidate key phrase, indicating whether to retain or discard it. In one example, the decision label could include "T" and "F," where "T" indicates that the candidate key phrase should be retained, and "F" indicates that it should be discarded. The selector can generate the decision label autoregressively, thus taking into account previously made choices, reducing semantic redundancy, and improving selection accuracy.

[0077] In one example, the instruction template used when entering a selector could look like this:

[0078] Task Definition: You need to perform a sequence labeling task to select multiple key phrases from numbered candidates based on a given document. Use the label “T” to select a candidate and the label “F” to reject it. For example, the label sequence “TFF” means to select candidate [1] and reject candidates [2] and [3].

[0079] ###enter:

[0080] Document: {document}

[0081] Candidates:

[0082] [1]{candidate1}

[0083] ...

[0084] [n]{candidaten}

[0085] ###response:

[0086] Label sequence: {label_sequence}

[0087] Please continue to refer to this. Figure 1 In step S150, a first loss function corresponding to the generator and a second loss function corresponding to the selector are constructed, and the generator and the selector are tuned according to the first loss function and the second loss function respectively to obtain the target key phrase generation model.

[0088] In this embodiment, a first loss function corresponding to the generator and a second loss function corresponding to the selector can be constructed respectively. The generator can be optimized based on the first loss function and the selector can be optimized based on the second loss function, thereby obtaining the target key phrase generation model. In the future, key phrases can be generated for the specified document based on the target key phrase generation model.

[0089] In one embodiment of this application, a plurality of the control code portions are used to generate existing key phrases, and the remaining portions are used to generate missing key phrases; then, constructing the first loss function corresponding to the generator includes:

[0090] Determine a first target allocation scheme corresponding to the existing key phrases, and a second target allocation scheme corresponding to the missing key phrases;

[0091] Based on the first target allocation scheme and the second target allocation scheme, the cross-entropy loss between the two is determined as the first loss function.

[0092] In this embodiment, several control codes are used to generate existing keyphrases, and the remaining code is used to generate missing keyphrases. Existing keyphrases refer to those phrases that can be directly found or explicitly mentioned in the document, and they directly reflect the core concepts and themes of the document. Missing keyphrases refer to those phrases that are not directly mentioned in a given document, but are essential for understanding the core concepts and themes of the document.

[0093] Therefore, the generator's loss function can consist of two parts: one for existing key phrases and the other for missing key phrases. Specifically, optimal allocation plans can be found for existing key phrases and missing key phrases, respectively. (i.e., the first target allocation scheme) and (i.e., the second objective allocation scheme). Next, the corresponding cross-entropy loss is calculated based on the two allocation schemes to serve as the first loss function of the generator.

[0094] In one example, the cross-entropy loss can be determined using the following formula:

[0095]

[0096] Among them, L p and L a These represent the losses for existing key phrases and the losses for missing key phrases, respectively, where θ represents the model parameters. and These are predictions corresponding to existing key phrases and missing key phrases, respectively.

[0097] In one example, using the calculated first loss function, the generator parameters can be updated via backpropagation, and the model can be optimized using stochastic gradient descent or its variants. This process is repeated until the model's performance on the validation set can no longer be improved or the predetermined number of iterations is reached.

[0098] In one embodiment of this application, the second loss function corresponding to the selector is determined according to the following formula:

[0099]

[0100] Where φ is the parameter of the selector, N T and N F Y represents the number of decision labels "keep" and "discard," respectively, and is determined by N. T One "reservation" and N F The sequence of labels consisting of "discarded" values, where X is the input data excluding the label sequence, i.e., candidate key phrases, p φ (Y t |X,Y<t () is the probability that the selector predicts for the current label given that the previous selections are known.

[0101] In one embodiment of this application, during each round of training, several candidate key phrases are randomly sorted and adjusted before being input into the selector.

[0102] In this embodiment, during training, the input order of candidate key phrases can be randomly sorted and adjusted so that the selector does not depend on a specific input order. This forces the selector to learn how to evaluate the importance and relevance of each candidate key phrase independently of the input order, thereby enhancing the model's generalization ability when dealing with inputs in different orders.

[0103] In one embodiment, during the inference phase of the selector, candidate key phrases can be ranked based on scores generated by the generator, allowing the selector to more effectively process candidates most likely to be selected as the final key phrases. In one example, the score can be calculated based on various factors, such as the semantic relevance of the candidate key phrase to the document topic, its frequency of occurrence in the document, or its position in the document (phrases appearing in the title, summary, or keyword field may be given higher weight). In another example, the score can be the logarithmic probability of the candidate key phrase.

[0104] In one embodiment of this application, a key phrase generation method is also provided, comprising:

[0105] Get the document to be processed;

[0106] The document to be processed is input into a pre-trained target key phrase generation model, so that the target key phrase generation model outputs at least one target key phrase corresponding to the document to be processed. The target key phrase generation model is trained by the training method described in any of the foregoing embodiments.

[0107] Therefore, by using the training method described in any of the foregoing embodiments, a target key phrase generation model is obtained. This target key phrase generation model includes a generator and a selector. The generator is based on a Transformer architecture, and the decoder of the generator has several control codes. The selector is constructed based on a large language model. During training, by acquiring training documents and their corresponding key phrase truth values, and based on optimal transmission theory, the key phrase truth values ​​are assigned to several control codes as supervision signals. The training documents are input into the generator so that the generator outputs several candidate key phrases corresponding to the training documents. Then, the candidate key phrases are input into the selector so that the selector outputs a decision label corresponding to each candidate key phrase. This decision label is used to indicate whether to retain or discard the corresponding candidate key phrase. Next, a first loss function corresponding to the generator and a second loss function corresponding to the selector are constructed, and the generator and selector are tuned according to the first loss function and the second loss function, respectively, to obtain the target key phrases.

[0108] Therefore, the generator can correctly recall key phrases, while the selector can eliminate incorrect candidates. Thus, this key phrase generation model can balance the accuracy and recall of key phrase generation, ensuring the quality of key phrase generation.

[0109] Furthermore, since the number of control codes usually exceeds the number of key phrase truth values, many control codes are undertrained. The technical solution provided in this application is based on optimal transmission theory to determine the allocation scheme between key phrase truth values ​​and control codes, allowing a key phrase truth value to be assigned to multiple control codes, thereby ensuring the rationality of key phrase truth value allocation and the training effect of each control code.

[0110] The following describes an apparatus embodiment of this application, which can be used to perform the methods described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the methods described in the above embodiments of this application.

[0111] Figure 2 A block diagram of a training apparatus for a key phrase generation model according to an embodiment of this application is shown.

[0112] Reference Figure 2 As shown, a training apparatus for a key phrase generation model according to an embodiment of this application is provided. The key phrase generation model includes a generator and a selector, wherein the generator is based on a Transformer backbone, the decoder of the generator has several control codes, and the selector is constructed based on a large language model.

[0113] The device includes:

[0114] The first acquisition module acquires the training documents and the truth values ​​of several corresponding key phrases;

[0115] The allocation module is used to allocate the truth values ​​of several key phrases to several control codes as monitoring signals based on the optimal transmission theory.

[0116] The first input module is used to input the training document into the generator so that the generator outputs a number of candidate key phrases corresponding to the training document;

[0117] The second input module is used to input a number of candidate key phrases into the selector, so that the selector outputs a decision label corresponding to each candidate key phrase, and the decision label is used to indicate whether to retain or discard the corresponding candidate key phrase;

[0118] The first processing module is used to construct a first loss function corresponding to the generator and a second loss function corresponding to the selector, and to fine-tune the generator and the selector according to the first loss function and the second loss function respectively, so as to obtain the target key phrase generation model.

[0119] This application embodiment also provides a key phrase generation apparatus, the apparatus comprising:

[0120] The second acquisition module is used to acquire the document to be processed.

[0121] The second processing module is used to input the document to be processed into a pre-trained target key phrase generation model, so that the target key phrase generation model outputs at least one target key phrase corresponding to the document to be processed. The target key phrase generation model is trained by the training method described in any of the foregoing embodiments.

[0122] Figure 3 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0123] It should be noted that, Figure 3 The computer system of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0124] like Figure 3As shown, the computer system includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 302 or programs loaded from storage portion 308 into Random Access Memory (RAM) 303, such as performing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0125] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0126] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs various functions defined in the system of this application.

[0127] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0129] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0130] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0131] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0132] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0133] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0134] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A training method for a key phrase generation model, characterized in that, The key phrase generation model includes a generator and a selector. Among them, the generator takes Transformer as the backbone, the decoder of the generator has several control codes, and the selector is constructed based on a large language model; The method includes: Obtaining training documents and their corresponding several key phrase ground truths; Based on the optimal transport theory, allocating several of the key phrase ground truths to several of the control codes as supervision signals; Inputting the training documents into the generator so that the generator outputs several candidate key phrases corresponding to the training documents; Inputting several of the candidate key phrases into the selector so that the selector outputs a decision label corresponding to each of the candidate key phrases, and the decision label is used to indicate whether to retain or discard the corresponding candidate key phrase; Constructing a first loss function corresponding to the generator and a second loss function corresponding to the selector, and respectively tuning the generator and the selector according to the first loss function and the second loss function to obtain a target key phrase generation model; Based on the optimal transport theory, allocating several of the key phrase ground truths to several of the control codes as supervision signals, including: Generating several minimum text units for each of the control codes; For each of the control codes, determining the matching scores between each of the key phrase ground truths and the minimum text units corresponding to the control code; Based on the optimal transport theory, calculating according to the matching scores between the minimum text units corresponding to each of the control codes and the key phrase ground truths corresponding to each of the key phrase ground truths to determine the target allocation scheme between the key phrase ground truths and the control codes; Based on the optimal transport theory, calculating according to the matching scores between the minimum text units corresponding to each of the control codes and the key phrase ground truths corresponding to each of the key phrase ground truths to determine the target allocation scheme between the key phrase ground truths and the control codes, including: Regarding the key phrase ground truths as the supply side and constructing its corresponding supply vector, and regarding the control codes as the demand side and constructing its corresponding demand vector; Constructing an allocation cost matrix between the key phrase ground truths and the control codes, and each element in the allocation cost matrix is the allocation cost of allocating each of the key phrase ground truths to each of the control codes, and the allocation cost is negatively correlated with the matching score between the two; Using the optimal transport theory, calculating based on the supply vector, the demand vector, and the allocation cost matrix to determine the target allocation scheme between the key phrase ground truths and the control codes; According to the following formula, determining the second loss function corresponding to the selector: where $\phi$ is the parameter of the selector, $N_T$ and $N_F$ are the numbers of decision labels of "retain" and "discard" respectively, $Y$ is a label sequence composed of $N_T$ "retain" and $N_F$ "discard", $X$ is the input data without the label sequence, that is, the candidate key phrase, and $p_{\phi}(Y_t|X, Y_{<t})$ is the probability that the selector predicts the current label given the previous selections.

2. The method according to claim 1, characterized in that, Several of the control codes are partially used to generate existing key phrases, and the remaining part is used to generate missing key phrases; The first loss function corresponding to the generator is then constructed as follows: Determine a first target allocation scheme corresponding to the existing key phrases, and a second target allocation scheme corresponding to the missing key phrases; Based on the first target allocation scheme and the second target allocation scheme, the cross-entropy loss between the two is determined as the first loss function.

3. The method according to claim 1, characterized in that, In each round of training, several candidate key phrases are randomly sorted and adjusted before being input into the selector.

4. A method for generating key phrases, characterized in that, include: Get the document to be processed; The document to be processed is input into a pre-trained target key phrase generation model, so that the target key phrase generation model outputs at least one target key phrase corresponding to the document to be processed, wherein the target key phrase generation model is trained by the training method as described in any one of claims 1-3.

5. A training device for a key phrase generation model, characterized in that, The key phrase generation model includes a generator and a selector. The generator is based on the Transformer architecture, the decoder of the generator has several control codes, and the selector is constructed based on a large language model. The device includes: The first acquisition module acquires the training documents and the truth values ​​of several corresponding key phrases; The allocation module is used to allocate the truth values ​​of several key phrases to several control codes as monitoring signals based on the optimal transmission theory. The first input module is used to input the training document into the generator so that the generator outputs a number of candidate key phrases corresponding to the training document; The second input module is used to input a number of candidate key phrases into the selector, so that the selector outputs a decision label corresponding to each candidate key phrase, and the decision label is used to indicate whether to retain or discard the corresponding candidate key phrase; The first processing module is used to construct a first loss function corresponding to the generator and a second loss function corresponding to the selector, and to fine-tune the generator and the selector according to the first loss function and the second loss function respectively, so as to obtain the target key phrase generation model. Based on optimal transmission theory, the truth values ​​of several key phrases are assigned to several control codes as monitoring signals, including: Generate several minimum text units for each of the control codes; For each of the control codes, determine the matching score between the truth value of each key phrase and the smallest text unit corresponding to the control code; Based on the optimal transmission theory, the target allocation scheme between the key phrase truth value and the control code is determined by calculating the matching score between the smallest text unit corresponding to each control code and the truth value of each key phrase. Based on optimal transmission theory, a target allocation scheme between the key phrase truth value and the control code is determined by calculating the matching score between the smallest text unit corresponding to each control code and the truth value of each key phrase, including: The key phrase truth value is used as the supplier and its corresponding supply vector is constructed, and the control code is used as the demand side and its corresponding demand vector is constructed. Construct an assignment cost matrix between the critical phrase truth values and the control codes, where each element in the assignment cost matrix is the assignment cost of assigning each critical phrase truth value to each control code, and the assignment cost is negatively correlated with the matching score between the two; Using the optimal transport theory, calculate based on the supply vector, the demand vector, and the assignment cost matrix to determine the target assignment scheme between the critical phrase truth values and the control codes; Determine the second loss function corresponding to the selector according to the following formula: Where ϕ is the parameter of the selector, NT and NF are the numbers of decision labels of "retain" and "discard" respectively, Y is a label sequence consisting of NT "retain" and NF "discard", X is the input data excluding the label sequence, that is, the candidate critical phrase, and pϕ(Yt∣X, Y<t) is the probability that the selector predicts the current label given the previously selected ones.

6. A key phrase generation device, characterized in that, Include: A second acquisition module, configured to acquire a document to be processed; A second processing module, configured to input the document to be processed into a pre-trained target critical phrase generation model, so that the target critical phrase generation model outputs at least one target critical phrase corresponding to the document to be processed, and the target critical phrase generation model is trained by the training method described in any one of claims 1-3.

7. An electronic device, characterized in that, Include: One or more processors; A storage device, configured to store one or more programs, and when the one or more programs are executed by the one or more processors, cause the one or more processors to implement the method described in any one of claims 1 to 4.

Images