Keyword generation method and system based on one2multi seq training paradigm and pre-training model bart

Abstract

The application provides a keyword generation method, system, storage medium and electronic equipment based on One2MultiSeq training paradigm and pre-training model BART, and relates to the technical field of natural language processing. The application proposes One2MultiSeq paradigm, takes into account existing keywords and missing keywords, constructs two keyword sequences with completely opposite keyword sequences, and is used for model training; the influence of keyword sequence on the model can be reduced, the attention of the model to the missing keywords is improved, and therefore the model effect is improved. In addition, in order to solve the problem that the text modeling capability of the existing model is poor, the pre-training model BART is used as a basic framework, and a copy mechanism is introduced for the BART to form a CopyBART, and the model effect is further improved. For multi-modal information, a simple and effective method is used for alignment between modes in the embodiment of the application, and experiments prove that the proposed mode alignment is very effective.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a keyword generation method, system, storage medium, and electronic device based on the One2MultiSeq training paradigm and the pre-trained model BART. Background Technology

[0002] Keyword generation (KG), a fundamental task in Natural Language Processing (NLP), aims to automatically predict key phrases from a given source text. Key phrases are a set of phrases that highlight crucial information in a document and have been widely used in downstream NLP tasks such as recommender systems, opinion mining, and text summarization. Generally, a single text will correspond to multiple keywords.

[0003] Existing keyword generation models include CopyRNN and CatSeq. CopyRNN employs an RNN-based encoder-decoder model structure. To enable the model to discover location-based important information, a copying mechanism is applied to the RNN, forming CopyRNN. CopyRNN is trained using a One2One training paradigm and generates a fixed set of K keywords through beam search. The One2One paradigm states that if a text has M keywords, it will be copied M times during training to ensure that each text contains only one keyword. CatSeq is also an RNN-based model, but its training paradigm is changed to One2Seq. The One2Seq paradigm states that if a text corresponds to M keywords, it will use M-1 delimiters to separate these M keywords. <sep>Connect them to form a sentence containing M keywords.

[0004] However, because the One2One paradigm cannot dynamically determine the number of generated keywords, and because keywords can only be generated independently using bundle search during the generation process, the keywords cannot reference each other, resulting in a significant reduction in keyword diversity. To solve this problem, the One2Seq paradigm pre-defines a keyword order and uses different keywords through... <sep>By concatenating characters, subsequent keywords will reference previously generated keywords during the keyword generation process. This not only allows for the dynamic generation of varying numbers of keywords but also enhances keyword diversity. Although the keywords are initially unordered, experiments demonstrate that placing existing keywords before missing ones yields the best model performance. However, this keyword order can also cause the model to prioritize existing keywords while neglecting missing ones. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a keyword generation method, system, storage medium, and electronic device based on the One2MultiSeq training paradigm and the pre-trained model BART, which solves the technical problem of emphasizing existing keywords while neglecting missing keywords.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A keyword generation method based on the One2MultiSeq training paradigm and the pre-trained model BART, wherein the pre-trained model BART introduces a copy mechanism to form CopyBART, the method comprising:

[0010] Obtain a historical dataset, wherein any historical data in the historical dataset includes source text, as well as the corresponding image and keyword set;

[0011] Based on the comparison between the keyword set and the source text, all keywords are divided into existing keywords and missing keywords; and a first keyword sequence is obtained with existing keywords first and missing keywords second; and a second keyword sequence is obtained with missing keywords first and existing keywords second.

[0012] The image is preprocessed to obtain image attribute text and OCR text, and the source text, image attribute text and OCR text are concatenated sequentially to obtain joint text;

[0013] Using the combined text as input to CopyBART, and combining it with the first keyword sequence or the second keyword sequence, predict the third keyword sequence corresponding to the source text respectively;

[0014] Based on the first, second, and third keyword sequences, a loss function is constructed; the CopyBART is trained based on the loss function until the model converges.

[0015] The data to be analyzed is used as input to the converged model to generate a keyword set for the data to be analyzed.

[0016] Preferably, the processes for obtaining the first keyword sequence and the second keyword sequence are as follows:

[0017] The historical dataset is defined as follows: Where x i ,I i ,y i Let y represent the source text, corresponding image, and keyword set of the i-th historical data, respectively, where D is the total number of data points; and the keyword set y... i All contain M i Keywords Each keyword y (i,j) All include One word,

[0018] y i M in i The keywords are concatenated in a sequence with existing keywords first and missing keywords last, separated by a separator. <sep>sequence;

[0019]

[0020] Copy the sequence and reverse the order of its keywords;

[0021]

[0022] Among them, y i _(1),y i _(2) represent the first and second keyword sequences, respectively.

[0023] Preferably, the process of obtaining the combined text includes:

[0024] The image attribute text is obtained by describing local information in the image using text.

[0025] The OCR text is obtained using optical character recognition (OCR) methods.

[0026] The source text x i Image attribute text and OCR text Perform a connection to obtain the union text X. i ;

[0027]

[0028] Here, "and" represents the separator.

[0029] Preferably, the process of obtaining any of the third keyword sequences includes:

[0030] The joint text X i Input the encoder of CopyBART to obtain the high-dimensional vector h. i ;

[0031] h i =CopyBARTEncoder(X i )

[0032] in, For the joint text X i The length of , where d is the dimension of the encoder's hidden state;

[0033] h i Input the decoder of CopyBART and obtain the hidden state of the decoder at time t.

[0034]

[0035] in, The true target value at time step t-1 is represented by the first keyword sequence y. i _(1) or the second keyword sequence y i (2) Confirm;

[0036] according to Obtain the preliminary probability distribution P for each keyword vocab,t ;

[0037]

[0038] Among them, W vocab ∈R |V|×d Let |V| be a trainable parameter, and let |V| be the vocabulary size of the word list V.

[0039] And according to h i and The cross-attention of the encoder and decoder in the last layer of the decoder is used as the copy distribution, and the overall copy distribution is obtained:

[0040]

[0041]

[0042] in, Indicates the initial replication distribution; Indicates the source text x i Length;

[0043] W s ∈R d×d W h ∈R d×d represent the trainable parameters respectively; T represents the matrix transpose;

[0044] Based on the aforementioned copying mechanism, using the source text x i As a replication distribution, obtain the adjusted replication distribution.

[0045]

[0046] in, For trainable parameters, These are bias parameters;

[0047] According to P vocab,t and Get each word w in the vocabulary list V at time t. k,t The final probability distribution P final,t (w k,t );

[0048]

[0049] Where k = 1, 2, ..., |V| are word indices; p gen,t Indicates weight;

[0050] According to P final,t (w k,t Predict the word at time t;

[0051]

[0052] The argmax function determines the index of the maximum value, which is used to determine the index position of the word to be generated in the next time step, until the third keyword sequence is generated.

[0053] Preferably, the weight p gen,t The acquisition process includes:

[0054] The Pointer Softmax mechanism is used as a switch to determine whether to generate words from vocabulary V or copy words from the source text.

[0055]

[0056] Where sigmoid and tanh are both activation functions.

[0057] Preferably, cross-entropy is used as the loss function:

[0058]

[0059] Where log is the logarithmic function; These represent the third keyword sequence corresponding to the first and second keyword sequences, respectively.

[0060] A keyword generation system based on the One2MultiSeq training paradigm and the pre-trained model BART, wherein the pre-trained model BART introduces a copy mechanism to form CopyBART, the system comprising:

[0061] The acquisition module is used to acquire historical datasets, wherein any historical data in the historical dataset includes source text, as well as corresponding images and keyword sets;

[0062] The segmentation module is used to segment all keywords into existing keywords and missing keywords according to the comparison relationship between the keyword set and the source text; and to obtain a first keyword sequence in the order of existing keywords first and missing keywords second; and to obtain a second keyword sequence in the order of missing keywords first and existing keywords second.

[0063] The joint module is used to preprocess the image, obtain image attribute text and OCR text, and concatenate the source text, image attribute text and OCR text in sequence to obtain the joint text;

[0064] The prediction module is used to take the joint text as input to CopyBART and, in combination with the first keyword sequence or the second keyword sequence, predict the third keyword sequence corresponding to the source text.

[0065] A construction module is used to construct a loss function based on the first, second, and third keyword sequences; and to train the CopyBART model based on the loss function until the model converges.

[0066] The generation module is used to take the data to be analyzed as input to the converged model and generate a keyword set for the data to be analyzed.

[0067] A storage medium storing a computer program for keyword generation based on a One2MultiSeq training paradigm and a pre-trained model BART, wherein the computer program causes a computer to perform the keyword generation method as described above.

[0068] An electronic device, comprising:

[0069] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the keyword generation method as described above. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 A block diagram illustrating a keyword generation method based on the One2MultiSeq training paradigm and the pre-trained model BART, provided in an embodiment of the present invention.

[0072] Figure 2 This is a flowchart illustrating a keyword generation method based on the One2MultiSeq training paradigm and the pre-trained model BART, provided in an embodiment of the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] This application provides a keyword generation method, system, storage medium, and electronic device based on the One2MultiSeq training paradigm and the pre-trained model BART. This solves the technical problem of emphasizing existing keywords while neglecting missing keywords, thereby improving the model's performance.

[0075] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0076] To ensure the model considers both existing and missing keywords, this invention proposes the One2MultiSeq paradigm. This involves copying a text twice and pairing each text with a keyword sentence. These two keyword sentences have completely opposite keyword orders. For example, in the first keyword sentence, existing keywords come first, followed by missing keywords; in the second keyword sentence, the missing keywords come before the existing keywords.

[0077] Furthermore, to address the issue of poor text modeling capabilities in existing models, a pre-trained model, BART, is introduced as the underlying architecture. Additionally, a copying mechanism is introduced for BART, forming CopyBART.

[0078] Because a single post on social media platforms like Twitter may contain not only text but also images, making it multimodal, this invention employs a simple modal alignment method to fully utilize BART's text modeling capabilities: the image is directly converted into text using an image-to-text tool, then appended to the original text, and separated by the "and" character.

[0079] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0080] Example:

[0081] like Figure 1 As shown, this embodiment of the invention provides a keyword generation method based on the One2MultiSeq training paradigm and the pre-trained model BART. The pre-trained model BART introduces a copying mechanism to form CopyBART. The method includes:

[0082] S1. Obtain historical datasets, wherein any historical data in the historical dataset includes source text, as well as corresponding image and keyword sets;

[0083] S2. Based on the comparison relationship between the keyword set and the source text, divide all keywords into existing keywords and missing keywords; obtain the first keyword sequence in the order of existing keywords first and missing keywords last; and obtain the second keyword sequence in the order of missing keywords first and existing keywords last.

[0084] S3. Preprocess the image to obtain image attribute text and OCR text, and concatenate the source text, image attribute text and OCR text in sequence to obtain the joint text;

[0085] S4. Using the combined text as input to CopyBART, and combining it with the first keyword sequence or the second keyword sequence, predict the third keyword sequence corresponding to the source text respectively;

[0086] S5. Construct a loss function based on the first, second, and third keyword sequences; train the CopyBART model based on the loss function until the model converges;

[0087] S6. Use the data to be analyzed as input to the converged model to generate a keyword set for the data to be analyzed.

[0088] This invention proposes a One2MultiSeq paradigm that considers both existing and missing keywords, constructing two keyword sequences with completely opposite keyword orders for model training. This reduces the impact of keyword order on the model, increases the model's attention to missing keywords, and thus improves model performance. Furthermore, to address the poor text modeling capabilities of existing models, a pre-trained model BART is used as the foundation, and a copying mechanism is introduced to BART to form CopyBART, further enhancing model performance.

[0089] Next, we will combine Figure 2 The steps of the above scheme are described in detail:

[0090] First, it should be noted that the embodiments of this invention use the pre-trained model BART as the backbone model. As a standard seq2seq architecture model, BART has been used in many fields, including dialogue generation, question answering systems, and summary generation. The problem description of the keyword generation method provided in these embodiments can be summarized as follows: taking the source text and its corresponding image as input to the improved BART model, CopyBART, to predict its key phrase set.

[0091] In step S1, a historical dataset is obtained, wherein any historical data in the historical dataset includes source text, as well as the corresponding image and keyword set.

[0092] The historical dataset is defined as follows: Where x i ,I i ,y i Let y represent the source text, corresponding image, and keyword set of the i-th historical data, respectively, where D is the total number of data points; and the keyword set y... i All contain M i Keywords Each keyword y (i,j) All include One word,

[0093] In step S2, based on the comparison relationship between the keyword set and the source text, all keywords are divided into existing keywords and missing keywords; and a first keyword sequence is obtained in the order of existing keywords first and missing keywords second; and a second keyword sequence is obtained in the order of missing keywords first and existing keywords second.

[0094] In essence, this step expands upon the One2MultiSeq training paradigm described above, specifically including:

[0095] Based on the comparison between the keyword set and the source text, all keywords are divided into existing keywords and missing keywords. Specifically, if a keyword in the keyword set is contained in the source text, it is classified as an existing keyword; otherwise, it is classified as a missing keyword.

[0096] (1) y i M in i The keywords are concatenated in a sequence with existing keywords first and missing keywords last, separated by a separator. <sep>sequence;

[0097]

[0098] (2) Copy the sequence and reverse the order of its keywords;

[0099]

[0100] Among them, y i _(1),y i _(2) represent the first and second keyword sequences, respectively.

[0101] In step S3, the image is preprocessed to obtain image attribute text and OCR text, and the source text, image attribute text and OCR text are concatenated in sequence to obtain joint text.

[0102] This invention takes into account that Transformer-based pre-trained models have strong text representation capabilities. Therefore, it is necessary to convert image information into text space and perform image-text alignment. In this way, BART can focus on text modeling and leverage its advantages in representing text.

[0103] Accordingly, S3 specifically includes:

[0104] For image attribute text, an important and useful aspect of its ability to convey image information is its attributes. Image attributes include objects within the image, such as people, buildings, landscapes, etc. They are components of the image and can be used to describe local information within the image, representing them as... Image attribute text Append it to the source text and separate them with an "and" character.

[0105] In the context of OCR text: Some images contain text content, which is often used to enrich the image and convey semantic information. To better understand these images, OCR technology is used to recognize the OCR text and represent it as... As supplementary input.

[0106] Specifically The text is also appended to the source text using the "and" character. Note that... This is an optional character; it will be empty if there is no text in the image.

[0107] OCR technology refers to Optical Character Recognition, which uses machine learning algorithms to recognize and convert printed or handwritten text in images or scanned documents into editable and searchable digital text.

[0108] Finally, the source text x i Image attribute text and OCR text Perform a connection to obtain the union text X. i ;

[0109]

[0110] In step S4, the combined text is used as input to CopyBART, and the third keyword sequence corresponding to the source text is predicted by combining the first keyword sequence or the second keyword sequence.

[0111] like Figure 2 As shown, the process of obtaining any of the third keyword sequences specifically includes:

[0112] S41, the combined text X i Input the encoder of CopyBART to obtain the high-dimensional vector h. i ;

[0113] h i =CopyBARTEncoder(X i )

[0114] in, For the joint text X i The length of , where d is the dimension of the encoder's hidden state.

[0115] S42, h i Input the decoder of CopyBART and obtain the hidden state of the decoder at time t.

[0116]

[0117] in, The true target value at time step t-1 is represented by the first keyword sequence y. i _(1) or the second keyword sequence y i _(2) Determined, that is or

[0118]

[0119] The BART model's decoder maximizes the probability. (during training) The data will be input into the model in the same way for training, and all training data will be fine-tuned through the above process.

[0120] S43, according to Obtain the preliminary probability distribution P for each keyword vocab,t ;

[0121]

[0122] Among them, W vocab ∈R |V|×d Let |V| be a trainable parameter, and let |V| be the vocabulary size of the word list V.

[0123] S44, and according to h i and The cross-attention of the encoder and decoder in the last layer of the decoder is used as a copy distribution (the Transformer architecture uses scaled dot products for its self-attention calculation, i.e.) Where Q, K, and V represent Query, Key, and Value respectively, and the overall replication distribution is obtained:

[0124]

[0125]

[0126] in, Indicates the initial replication distribution; Indicates the source text x i Length; W s ∈R d×d W h ∈R d×d ...

[0127] It should be noted that, unlike the standard Transformer architecture, the final layer of the BART decoder is not directly connected to the output softmax function. Instead, it is connected to a linear layer.

[0128] S45. Based on the aforementioned copying mechanism, using the source text x i As a replication distribution, obtain the adjusted replication distribution.

[0129]

[0130] in, For trainable parameters, This is the bias parameter.

[0131] Because the introduced copying mechanism enables the decoder to copy important content from the source text into keywords, this embodiment of the invention uses the copying mechanism for BART to further improve its keyword generation capabilities.

[0132] Considering that the dataset obtained in step S1 is a multimodal dataset, we set it to utilize only the source text x. i As a replication distribution, rather than the joint text X i This is because keywords often exhibit grammatical structures and lexical regularities more consistent with the source text. Therefore, this embodiment of the invention believes that copying only from the source text helps reduce deviations in the keyword generation process. Thus, the above method is used to adjust the copy distribution.

[0133] S46, According to P vocab,t and Get each word w in the vocabulary list V at time t. k,t The final probability distribution P final,t (w k,t );

[0134]

[0135] Where k = 1, 2, ..., |V| are word indices; p gen,t Indicates weight; This means that if a word appears multiple times in a text, its probability will be increased multiple times.

[0136] Specifically, the weight p gen,t The acquisition process includes:

[0137] The Pointer Softmax mechanism is used as a switch to determine whether to generate words from vocabulary V or copy words from the source text.

[0138]

[0139] Where sigmoid and tanh are both activation functions. W st ∈R d×d For trainable parameters (as mentioned above W) s different).

[0140] S47, according to P final,t (w k,t Predict the word at time t;

[0141]

[0142] The argmax function determines the index of the maximum value, which is used to determine the index position of the word to be generated in the next time step, until the third keyword sequence is generated.

[0143] In step S5, a loss function is constructed based on the first, second, and third keyword sequences; the CopyBART is trained based on the loss function until the model converges.

[0144] In this embodiment of the invention, cross-entropy is used as the loss function:

[0145]

[0146] Where log is the logarithmic function; These represent the third keyword sequence corresponding to the first and second keyword sequences, respectively.

[0147] In step S6, the data to be analyzed is used as input to the converged model to generate a keyword set for the data to be analyzed.

[0148] To verify the effectiveness of the method proposed in the embodiments of the present invention, the following example is provided:

[0149] (1) Experimental data

[0150] The tests were conducted on three datasets: the unimodal datasets Twitter and StackExchange, and a multimodal Tweet dataset, as detailed in Table 1.

[0151] Table 1. Statistical Table

[0152]

[0153] (2) Parameter settings

[0154] BART includes two pre-trained general models, namely BART base and BART large These are used as the backbone models, employing BART's default BPE encoding. BPE encoding consists of 50265 tokens, representing numbers, links, and usernames in the Twitter dataset using three special tokens respectively: <number> 、 <url> 、 <mention>In addition, the separator <sep>is also added to the vocabulary. At this time, the size of the vocabulary increases from the original 50265 to 50269. Specific training parameters can be seen in Table 2.

[0155] Table 2 Parameter setting table

[0156]

[0157]

[0158] Wherein, MLP is the abbreviation of Maximum length of predictions.

[0159] (3) Evaluation indicators

[0160] Since the average number of keywords in the multimodal Tweet dataset is 1.33 and that in the unimodal Twitter dataset is 1.13, MacroF1@1 and MacroF1@3 are used as evaluation indicators therefor. The average number of keywords in StackExchange is much higher than that in Twitter, so MacroF1@3 and MacroF1@5 are used as evaluation indicators therefor. In addition, the mean average precision (MAP) of the top five predicted keywords is also used to evaluate all the above datasets.

[0161] For MacroF1@k, given a source text or a source text and its image text, it is assumed that the model predicts a series of different keywords and the target keywords are y. When calculating MacroF1@k (where k=1, 3, 5), the number of keywords is intercepted If k<m, filling is performed with kppad_1, kppad_2, … until the number of keywords reaches k. The definitions of Precision@k, Recall@k and MacroF1@k are:

[0162]

[0163]

[0164]

[0165]

[0166]

[0167] Wherein, N represents the number of test sets.

[0168] (4) Experimental results

[0169] Table 3 Comparison results of single-modal datasets

[0170]

[0171] Note: Bold data indicates the best display.

[0172] Table 4 Comparison Results of Multimodal Datasets

[0173]

[0174]

[0175] As can be seen from Tables 3 and 4, compared to the baseline model, the implementation of this invention achieves a significant performance improvement after using CopyBART as the backbone model. Furthermore, the One2MultiSeq paradigm shows a substantial performance improvement over the One2Seq paradigm under the same conditions. For example, on the StackExchange dataset, based on CopyBART-base, the F1@5 evaluation metric is improved by more than two percentage points compared to One2Seq.

[0176] This invention provides a keyword generation system based on the One2MultiSeq training paradigm and the pre-trained model BART. The pre-trained model BART incorporates a copy mechanism to form CopyBART. The system includes:

[0177] The acquisition module is used to acquire historical datasets, wherein any historical data in the historical dataset includes source text, as well as corresponding images and keyword sets;

[0178] The segmentation module is used to segment all keywords into existing keywords and missing keywords according to the comparison relationship between the keyword set and the source text; and to obtain a first keyword sequence in the order of existing keywords first and missing keywords second; and to obtain a second keyword sequence in the order of missing keywords first and existing keywords second.

[0179] The joint module is used to preprocess the image, obtain image attribute text and OCR text, and concatenate the source text, image attribute text and OCR text in sequence to obtain the joint text;

[0180] The prediction module is used to take the joint text as input to CopyBART and, in combination with the first keyword sequence or the second keyword sequence, predict the third keyword sequence corresponding to the source text.

[0181] A construction module is used to construct a loss function based on the first, second, and third keyword sequences; and to train the CopyBART model based on the loss function until the model converges.

[0182] The generation module is used to take the data to be analyzed as input to the converged model and generate a keyword set for the data to be analyzed.

[0183] This invention provides a storage medium storing a computer program for generating keywords based on the One2MultiSeq training paradigm and the pre-trained model BART, wherein the computer program causes a computer to execute the keyword generation method described above.

[0184] This invention provides an electronic device, comprising:

[0185] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the keyword generation method as described above.

[0186] It is understood that the keyword generation system, storage medium, and electronic device based on the One2MultiSeq training paradigm and the pre-trained model BART provided in the embodiments of the present invention correspond to the keyword generation method based on the One2MultiSeq training paradigm and the pre-trained model BART provided in the embodiments of the present invention. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the keyword generation method, and will not be repeated here.

[0187] In summary, compared with existing technologies, it has the following beneficial effects:

[0188] 1. This invention proposes a One2MultiSeq paradigm that considers both existing and missing keywords, constructing two keyword sequences with completely opposite keyword orders for model training. This reduces the impact of keyword order on the model, increases the model's attention to missing keywords, and thus improves model performance. Furthermore, to address the poor text modeling capabilities of existing models, a pre-trained model BART is used as the foundation, and a copying mechanism is introduced to BART to form CopyBART, further enhancing model performance.

[0189] 2. Since the introduced copying mechanism enables the decoder to copy important content from the source text into the keywords, this embodiment of the invention uses the copying mechanism for BART to further improve its keyword generation capability.

[0190] 3. For multimodal information, the embodiments of the present invention employ a simple and effective method for aligning modes, and experiments have shown that the proposed mode alignment is very effective.

[0191] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0192] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / sep> < / mention> < / url> < / number> < / sep> < / sep> < / sep> < / sep>

Claims

1. A keyword generation method based on the One2MultiSeq training paradigm and the pre-trained model BART, characterized in that, The pre-trained model BART incorporates a copying mechanism to form CopyBART, which includes: Obtain a historical dataset, wherein any historical data in the historical dataset includes source text, as well as the corresponding image and keyword set; Based on the comparison between the keyword set and the source text, all keywords are divided into existing keywords and missing keywords; and a first keyword sequence is obtained with existing keywords first and missing keywords second; and a second keyword sequence is obtained with missing keywords first and existing keywords second. The image is preprocessed to obtain image attribute text and OCR text, and the source text, image attribute text and OCR text are concatenated sequentially to obtain joint text; Using the combined text as input to CopyBART, and combining it with the first keyword sequence or the second keyword sequence, predict the third keyword sequence corresponding to the source text, respectively. Based on the first, second, and third keyword sequences, a loss function is constructed; the CopyBART is trained based on the loss function until the model converges; The data to be analyzed is used as input to the converged model to generate a keyword set for the data to be analyzed.

2. The keyword generation method as described in claim 1, characterized in that, The processes for obtaining the first keyword sequence and the second keyword sequence are as follows: The historical dataset is defined as follows: Where x i I i y i Let y represent the source text, corresponding image, and keyword set of the i-th historical data, respectively, where D is the total number of data points; and the keyword set y... i All contain M i Keywords Each keyword y (i，j) All include One word, y i M in i The keywords are concatenated in a sequence with existing keywords first and missing keywords last, separated by a separator. <sep> sequence;< / sep> Copy the sequence and reverse the order of its keywords; Among them, y i _(1),y i _(2) represent the first and second keyword sequences, respectively.

3. The keyword generation method as described in claim 2, characterized in that, The process of obtaining the combined text includes: The image attribute text is obtained by describing local information in the image using text. The OCR text is obtained using optical character recognition (OCR) methods. The source text x i Image attribute text and OCR text Perform a connection to obtain the union text X. i ; Here, "and" represents the separator.

4. The keyword generation method as described in claim 3, characterized in that, The process of obtaining any of the third keyword sequences includes: The joint text X i Input the encoder of CopyBART to obtain the high-dimensional vector h. i ; i=CopyBARTEncoder(X i ) in, For the joint text X i The length of , where d is the dimension of the encoder's hidden state; h i Input the decoder of CopyBART and obtain the hidden state of the decoder at time t. in, The true target value at time step t-1 is represented by the first keyword sequence y. i _(1) or the second keyword sequence y i (2) Confirm; according to Obtain the preliminary probability distribution P for each keyword vocab，t ; Among them, W vocab ∈R |V|×d Let |V| be a trainable parameter, and let |V| be the vocabulary size of the word list V. And according to h i and The cross-attention of the encoder and decoder in the last layer of the decoder is used as the copy distribution, and the overall copy distribution is obtained: in, Indicates the initial replication distribution; Indicates the source text x i Length; W s ∈R d×d W h ∈R d×d represent the trainable parameters respectively; T represents the matrix transpose; Based on the aforementioned copying mechanism, using the source text x i As a replication distribution, obtain the adjusted replication distribution. in, For trainable parameters, These are bias parameters; According to P vocab，t and Get each word w in the vocabulary list V at time t. k，t The final probability distribution P final，t (w k，t ); Where k = 1, 2, ..., |V| are word indices; p gen，t Indicates weight; According to P final，t (w k，t Predict the word at time t; The argmax function determines the index of the maximum value, which is used to determine the index position of the word to be generated in the next time step, until the third keyword sequence is generated.

5. The keyword generation method as described in claim 4, characterized in that, The weight p gen，t The acquisition process includes: The Pointer Softmax mechanism is used as a switch to determine whether to generate words from vocabulary V or copy words from the source text. Where sigmoid and tanh are both activation functions.

6. The keyword generation method as described in claim 4, characterized in that, Cross-entropy is used as the loss function: Where log is the logarithmic function; These represent the third keyword sequence corresponding to the first and second keyword sequences, respectively.

7. A keyword generation system based on the One2MultiSeq training paradigm and the pre-trained model BART, characterized in that, The pre-trained model BART incorporates a copying mechanism to form CopyBART, which includes: The acquisition module is used to acquire historical datasets, wherein any historical data in the historical dataset includes source text, as well as corresponding images and keyword sets; The segmentation module is used to segment all keywords into existing keywords and missing keywords according to the comparison relationship between the keyword set and the source text; and to obtain a first keyword sequence in the order of existing keywords first and missing keywords second; and to obtain a second keyword sequence in the order of missing keywords first and existing keywords second. The joint module is used to preprocess the image, obtain image attribute text and OCR text, and concatenate the source text, image attribute text and OCR text in sequence to obtain the joint text; The prediction module is used to take the joint text as input to CopyBART and, in combination with the first keyword sequence or the second keyword sequence, predict the third keyword sequence corresponding to the source text. A construction module is used to construct a loss function based on the first, second, and third keyword sequences; and to train the CopyBART model based on the loss function until the model converges. The generation module is used to take the data to be analyzed as input to the converged model and generate a keyword set for the data to be analyzed.

8. A storage medium, characterized in that, It stores a computer program for generating keywords based on the One2MultiSeq training paradigm and the pre-trained model BART, wherein the computer program causes a computer to perform the keyword generation method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the keyword generation method as described in any one of claims 1 to 6.

Images