A long text label recommendation model construction method and a label recommendation method

Abstract

The application provides a long text label recommendation model construction method and a label recommendation method, and relates to the technical field of natural language processing. The model construction method comprises the following steps: obtaining a historical long text, labeling the historical long text to obtain text labels, standard similarity, historical clauses and clause historical weights; obtaining historical key sentences according to the historical weights, expanding the text labels to obtain historical pseudo labels, and obtaining a historical label sequence according to the historical pseudo labels; training an initial prediction model by using the above data to obtain a prediction model; training an initial sorting model by using the above data to obtain a reordering model; and obtaining a label recommendation model according to the prediction model and the reordering model. The application has the beneficial effects that the key sentences representing the meaning of the long text are obtained according to the clause weights, the text labels are expanded to obtain pseudo labels that are convenient for the model to understand, the label recommendation of the long text is realized, and the accuracy of the model is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a method for constructing a tag recommendation model for long texts and a tag recommendation method. Background Art

[0002] With the rapid development of the internet and mobile communication technologies, text-based information can be published by individuals. However, it is inevitable that illegal elements will spread information that does not conform to current mainstream values, such as violence, pornography, and fraud. Therefore, using computer technology to recommend tags for online text, using tags to filter out negative information, and actively disseminating text that conforms to current mainstream values ​​has become an urgent problem to be solved.

[0003] In existing technologies, tag recommendation methods are usually transformed into tag classification problems. However, this requires a large number of tagged datasets for tag classification, and there will still be sample imbalance and special requirements for the relationship between tags in the tagged dataset, which makes it not very practical. Currently, there is little research on tag recommendation methods in the field of values, so there are fewer public datasets available, resulting in poor results in processing long texts in the field of values. In addition, existing technologies have length restrictions on the texts they process. For example, the maximum text length that the current model can accept is 512. However, when recommending tags for text on the Internet, the text is not subject to this length limit, resulting in the inability of existing technologies to recommend tags for long texts. Summary of the Invention

[0004] The problem solved by the present invention is how to achieve accurate long text tag recommendation.

[0005] To solve the above problems, the present invention provides a tag recommendation model construction method and a tag recommendation method for long texts.

[0006] In a first aspect, the present invention provides a method for constructing a tag recommendation model for a long text, comprising:

[0007] Acquire a historical long text, annotate the historical long text, and obtain a plurality of text labels corresponding to the historical long text and a standard similarity between the text labels and the historical long text;

[0008] Perform overlapping segmentation on the long historical text according to a first preset length to obtain a plurality of historical clauses, and mark the historical weight of each of the historical clauses in the long historical text;

[0009] Selecting a first preset number of the historical clauses in descending order of the historical weights for splicing to obtain historical key sentences, deleting the text labels whose standard similarity is less than a preset threshold, performing label expansion on the remaining text labels to obtain historical pseudo-labels, and marking the historical pseudo-similarity of each historical pseudo-label with the historical long text, and sorting the historical pseudo-labels in descending order of the historical pseudo-similarity to obtain a historical label sequence;

[0010] Constructing an initial prediction model, training the initial prediction model with the historical clauses, the historical weights, the historical labels, and the standard similarity to obtain a prediction model, wherein the prediction model is used to output a target weight for each clause to be tested, a plurality of prediction labels, and a predicted similarity of the plurality of prediction labels, wherein a predicted key sentence is obtained according to the target weight, the prediction labels are preprocessed according to the predicted similarity, and the preprocessed prediction labels are expanded to obtain a target pseudo label;

[0011] Constructing an initial re-ranking model, training the initial re-ranking model with the historical key sentence, the historical label sequence, the historical pseudo-label, and the historical pseudo-similarity to obtain a re-ranking model, wherein the re-ranking model is used to obtain a predicted pseudo-similarity based on the predicted key sentence and the target pseudo-label, and output a predicted label sequence sorted in descending order according to the predicted pseudo-similarity;

[0012] A tag recommendation model is obtained according to the prediction model and the re-ranking model.

[0013] In the present invention, the historical long text is subjected to overlapping segmentation processing to obtain multiple historical clauses and historical weights, retaining the connection between each historical clause and obtaining historical key sentences to construct an accurate data set; text tags with a standard similarity less than a preset threshold are deleted, that is, text tags with a low connection with the historical long text are deleted, reducing the amount of data for subsequent label processing. The text tags are expanded to obtain historical pseudo-labels, and a historical label sequence is obtained based on the historical pseudo-labels, effectively expanding the information of the text tags, making the meaning of the text tags clearer and richer, and facilitating the model to understand the meaning of the text tags, thereby increasing the accuracy of label recommendations. The initial prediction model is trained with historical clauses, historical weights, and text labels. While ensuring performance, the accuracy of the prediction model is effectively improved. When the prediction model performs label prediction, it obtains the target weight corresponding to each clause to be tested, and selects the clause to be tested that meets the conditions to obtain the predicted key sentence that can represent the meaning of the long text to be tested, effectively reducing the length of the long text for further label prediction. By training the re-ranking model with historical pseudo-labels and historical label sequences, the re-ranking model can predict more accurate and richer label recommendation combinations using more explicit and rich pseudo-labels. The label recommendation model constructed based on the prediction model and the re-ranking model can effectively predict labels for long texts accurately and richly, obtaining accurate recommended labels without any special requirements for the relationships between labels (such as a certain rank relationship or independence between labels), effectively increasing the versatility of the model.

[0014] Optionally, the method for constructing the initial prediction model includes:

[0015] Get the Bert model, long short-term memory network and fully connected layer;

[0016] The Bert model, the long short-term memory network and the fully connected layer are connected in sequence to obtain the initial prediction model.

[0017] Optionally, the training of the initial prediction model using the historical clauses, the historical weights, the historical labels, and the standard similarity to obtain the prediction model includes:

[0018] Input the history clauses into the Bert model, insert a classification symbol before each history clause, and take the output corresponding to the classification symbol as the embedding vector of the history clause;

[0019] Utilizing the attention mechanism of the Bert model to obtain the prediction weight of each of the history clauses;

[0020] Using the long short-term memory network to fuse the embedding vectors to obtain a history clause vector;

[0021] Obtaining the historical long text vector according to the prediction weight and the historical clause vector;

[0022] The historical long text vector is classified using the fully connected layer to obtain the predicted label and the predicted similarity.

[0023] Optionally, the training of the initial prediction model using the historical clauses, the historical weights, the historical tags, and the standard similarity to obtain the prediction model further comprises:

[0024] According to the predicted label and the text label, the predicted similarity and the standard similarity, the model parameters of the initial prediction model are updated using the Focal loss function to obtain the prediction model.

[0025] Optionally, the step of expanding the remaining text labels to obtain historical pseudo labels includes:

[0026] Splitting the historical key sentence to obtain multiple historical keywords;

[0027] According to each of the text tags, marking the importance of each of the historical keywords;

[0028] According to the text label and the order of importance from high to low, a second preset number of the historical keywords are selected and sorted to obtain the historical pseudo-label, wherein the historical pseudo-label includes the text label and the second preset number of the historical keywords;

[0029] The historical keywords, the text labels and the historical pseudo labels are used to train a preset initial support vector machine to obtain a support vector machine, and the support vector machine is used to output the target pseudo labels.

[0030] Optionally, the historical pseudo-similarity includes positive sample similarity and negative sample similarity, the re-ranking model includes a pseudo-similarity model and a ranking model, and the initial ranking model is trained with the historical key sentences, the historical label sequence, the historical pseudo-label, and the historical pseudo-similarity to obtain the re-ranking model, including:

[0031] Obtaining the first historical pseudo label in the historical label sequence as a positive sample;

[0032] Randomly selecting a third preset number of historical pseudo labels other than the positive samples in the historical label sequence as negative samples;

[0033] Using the pseudo-similarity model, a positive sample similarity is obtained based on the positive sample and the historical key sentence, wherein the positive sample similarity is the closest distance between the positive sample and the historical key sentence in a vector space, and the pseudo-similarity model is trained based on the historical key sentence, the historical pseudo-label, and the similarity of the historical pseudo-label;

[0034] Using the pseudo-similarity model, obtaining a negative sample similarity based on the negative sample and the historical key sentence, wherein the negative sample similarity is less than the positive sample similarity, and the negative sample similarity is the distance between the negative sample and the historical key sentence in a vector space;

[0035] Using a preset initial sorting model, the predicted label sequence is obtained according to the positive sample similarity and the negative sample similarity;

[0036] The model parameters of the initial sorting model are updated according to the predicted label sequence and the historical label sequence to obtain the sorting model.

[0037] Optionally, the acquiring of the historical long text, annotating the historical long text, and obtaining a plurality of text tags corresponding to the historical long text and a standard similarity between the text tags and the historical long text further includes:

[0038] Obtaining the text length of the historical long text and the number of corresponding text tags;

[0039] The historical long text whose text length is less than a second preset length and whose number of text tags is less than a third preset number is deleted.

[0040] The present invention also provides a device for constructing a tag recommendation model for a long text, comprising a memory and a processor;

[0041] The memory is used to store computer programs;

[0042] The processor is configured to implement the above-mentioned method for constructing a tag recommendation model for a long text when executing the computer program.

[0043] The present invention further provides a tag recommendation method for a long text, comprising:

[0044] Obtaining a long text to be tested, and obtaining a plurality of clauses to be tested according to the long text to be tested;

[0045] Inputting the clause to be tested into the prediction model constructed according to the long text label recommendation model construction method as described above, obtaining the target weight, predicted label and similarity of the predicted label for each clause to be tested;

[0046] Obtaining a predicted key sentence according to the target weight, preprocessing the predicted label according to the predicted similarity, and expanding the preprocessed predicted label to obtain a plurality of target pseudo labels;

[0047] Inputting the plurality of target pseudo labels and the predicted key sentences into a re-ranking model constructed according to the long text label recommendation model construction method as described above, to obtain a predicted label sequence;

[0048] A preset target number of target pseudo labels in the predicted label sequence are selected in sequence as recommended labels.

[0049] In the present invention, the long text to be tested is divided into multiple clauses to be tested so that the prediction model can perform label prediction. The prediction model is used to obtain the preliminary target weights, predicted labels and corresponding predicted similarities of the clauses to be tested, and based on the target weights and predicted similarities, the predicted key sentences and target pseudo labels that can represent the meaning of the long text to be tested are obtained. While reducing the length of the long text, the predicted data is preliminarily processed to reduce the processing pressure of the subsequent re-ranking model and increase the processing efficiency of the re-ranking model. The predicted labels are also expanded and described so that the model can better understand the meaning of the predicted labels to increase the accuracy of label recommendations. The target pseudo labels are arranged using the re-ranking model to obtain a predicted label sequence, and recommended labels are obtained based on the predicted labels, effectively increasing the accuracy of long text label recommendations.

[0050] The present invention further provides a tag recommendation device for a long text, comprising a memory and a processor;

[0051] The memory is used to store computer programs;

[0052] The processor is configured to implement the above-mentioned tag recommendation method for a long text when executing the computer program. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of a method for building a tag recommendation model for long texts according to an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of a flow chart of using a prediction model according to an embodiment of the present invention;

[0055] Figure 3 Schematic diagram of the process of the pseudo label expansion method according to an embodiment of the present invention;

[0056] Figure 4 A schematic diagram of a reordering model usage flow according to an embodiment of the present invention;

[0057] Figure 5 Schematic diagram of the flow of a tag recommendation method for long text according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0059] In the description of the embodiments of this application, the term "some embodiments" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same implementation or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in any one or more embodiments or examples.

[0060] like Figure 1 As shown, the embodiment of the present invention provides a method for constructing a tag recommendation model for long texts in the field of values. However, the method is not limited to recommending text tags in the field of values, but can also be used for recommending text tags in multiple fields. The method includes:

[0061] Step S1: Acquire a historical long text, annotate the historical long text, obtain a plurality of text tags corresponding to the historical long text and a standard similarity between the text tags and the historical long text.

[0062] Specifically, in order to ensure the extensiveness of the data, this embodiment obtains historical long texts from major websites such as Sohu News, People's Daily, China Daily, Weibo (official certified account), Twitter, etc. to train the model. The obtained historical long texts are subjected to manual annotation, model annotation, etc., and on the basis of the historical long texts having been annotated, manual inspection is introduced to correct the incorrect historical long text annotations and the omitted unannotated historical long texts, or the annotation model is used to perform multiple rounds of self-training on the historical long texts to increase the accuracy of the historical long text annotations and construct an accurate data set. The data set is divided into a training set, a validation set, and a test set in a ratio of 6:2:2. Among them, the historical long text expresses multiple meanings, that is, each historical long text corresponds to multiple text labels, and according to the connection between the meaning of the historical long text and the text label, the standard similarity between the historical long text and each text label is annotated, and the standard similarity of the text label closest to the meaning expressed by the historical long text is the highest.

[0063] Step S2: performing overlapping segmentation on the historical long text according to a first preset length to obtain a plurality of historical clauses, and marking the historical weight of each of the historical clauses in the historical long text.

[0064] Specifically, the long historical text is first segmented with overlap based on a first preset length. Specifically, the long historical text T is segmented into k historical clauses n1, n2, ..., nk of equal length. The first preset length does not exceed 512 words. For example, the first preset length is 200 words, and the overlap is 30 words. Based on the meaning of each historical clause, the historical weight of each clause in the long historical text is determined and labeled.

[0065] In addition, when the tag recommendation model construction method of the embodiment is applied to tag recommendation of shorter texts, this step can also be appropriately adjusted according to the length of the text, such as adaptively changing the preset length and overlap, or deleting this step.

[0066] Step S3: Select a first preset number of the historical clauses in descending order of the historical weights to perform splicing processing to obtain historical key sentences, delete the text labels whose standard similarity is less than a preset threshold, expand the remaining text labels to obtain historical pseudo-labels, and mark the historical pseudo-similarity of each historical pseudo-label with the historical long text, and sort the historical pseudo-labels in descending order of the historical pseudo-similarity to obtain a historical label sequence.

[0067] Specifically, the above historical clauses are arranged in descending order of historical weight to obtain a historical clause sequence. The first preset number (e.g., 3) of historical clauses with the largest historical weights in the historical clause sequence are sequentially selected to represent the meaning expressed by the entire historical long text. The historical clauses are spliced ​​together to obtain historical key sentences that can represent the meaning of the historical long text. The length of the spliced ​​historical key sentences is less than 512. Historical text labels with a standard similarity less than a preset threshold, such as 0.2, are deleted. However, since most of the text labels obtained during annotation are in the form of four-character idioms, even the longest text label length does not reach 10. Excessively short text labels will prevent the model from fully extracting vector expressions. Therefore, the remaining text labels need to be expanded to increase the accuracy of label recommendations for long texts. After expansion, the model's understanding of the text labels will change, that is, the similarity between the historical pseudo-label and the historical long text will change. Therefore, it is necessary to re-annotate the historical pseudo-similarity of each historical pseudo-label with the historical long text, and use the historical pseudo-similarity to sort the historical pseudo-labels to obtain a historical label sequence.

[0068] It should be noted that in the validation set, in 76% of the samples, after sorting by historical weight, the cumulative historical weight of the first three historical clauses can reach 0.8. This shows that the embodiment of the present invention only needs some key sentences to determine the final recommended tags for long texts. Therefore, the method of using weights to obtain key sentences for tag recommendation is feasible, and reduces the data processing pressure of subsequent models while ensuring accuracy.

[0069] Step S4: construct an initial prediction model, train the initial prediction model with the historical clauses, the historical weights, the historical labels and the standard similarity to obtain a prediction model, wherein the prediction model is used to output a target weight for each clause to be tested, multiple prediction labels and the predicted similarity of the multiple prediction labels, wherein a predicted key sentence is obtained according to the target weight, the prediction label is preprocessed according to the prediction similarity, and the preprocessed prediction label is expanded to obtain a target pseudo label.

[0070] Specifically, an initial prediction model is constructed, such as a Bert model (Bidirectional Encoder Representations from Transformer), a recurrent neural network (RNN), a classifier, etc. The initial prediction model is trained with the above-constructed data set to obtain a trained prediction model, wherein the input of the prediction model is a plurality of clauses to be tested, and the output is the weight, predicted label, and predicted similarity of the predicted label of each clause to be tested obtained by, for example, the attention mechanism in the Bert model. And using the method described in step S3, the predicted key sentence can be obtained according to the target weight, the predicted label can be processed according to the predicted similarity, and the processed predicted label can be expanded to obtain the target pseudo label, which is used as the input of the subsequent re-ranking model when recommending labels for long texts.

[0071] Step S5: construct an initial reordering model, train the initial reordering model with the historical key sentences, the historical label sequence, the historical pseudo-label and the historical pseudo-similarity to obtain a reordering model, and the reordering model is used to obtain the predicted pseudo-similarity based on the predicted key sentences and the target pseudo-label, and output a predicted label sequence sorted in descending order according to the predicted pseudo-similarity.

[0072] Specifically, the reranking model takes as input the predicted key sentence and the target pseudo-label, and outputs a sequence of predicted labels sorted in descending order of predicted pseudo-similarity. Since the expansion of the predicted labels changes the model's understanding of the predicted labels, and therefore the similarity between the predicted labels and the predicted key sentence, it is necessary to first determine the predicted pseudo-similarity of each target pseudo-label based on the predicted key sentence and the target pseudo-label. Then, the target pseudo-labels are ranked according to the target pseudo-labels and predicted pseudo-similarity to obtain a more accurate predicted label sequence. Furthermore, the recommended labels for the long text under test can be obtained based on the predicted label sequence and a preset number of recommended labels. For example, if the predicted label sequence is "selfless, honest, close to the people and love the people... violates the law" and there are three recommended labels, then "selfless," "honest," and "close to the people and love the people" are selected as the recommended labels for the long text under test.

[0073] Step S6: Obtain a tag recommendation model based on the prediction model and the re-ranking model.

[0074] In this embodiment, the historical long text is subjected to overlapping segmentation processing to obtain multiple historical clauses and historical weights, retaining the connections between the historical clauses and obtaining historical key sentences to construct an accurate data set; text tags with a standard similarity less than a preset threshold are deleted, that is, text tags with a low connection to the historical long text are deleted, reducing the amount of data for subsequent label processing. The text tags are expanded to obtain historical pseudo-labels, and historical label sequences are obtained based on the historical pseudo-labels. This effectively expands the information of the text labels, making the meaning of the text labels clearer and richer, making it easier for the model to understand the meaning of the text labels and increasing the accuracy of label recommendations. The initial prediction model is trained with historical clauses, historical weights, and text labels. While ensuring performance, the accuracy of the prediction model is effectively improved. When performing label prediction, the prediction model obtains the target weight corresponding to each clause to be tested, and selects the clauses to be tested that meet the conditions to obtain predicted key sentences that can represent the meaning of the long text to be tested, effectively reducing the length of the long text for further label prediction. By training the re-ranking model with historical pseudo-labels and historical label sequences, the re-ranking model leverages the more explicit and rich historical pseudo-labels to predict more accurate and diverse label recommendation combinations. The label recommendation model, constructed based on the prediction and re-ranking models, effectively predicts and recommends labels for long texts accurately and comprehensively, without requiring specific relationships between labels (e.g., a certain rank relationship or independence between labels), effectively increasing the model's versatility.

[0075] Optionally, the method for constructing the initial prediction model includes:

[0076] Get the Bert model, long short-term memory network and fully connected layer;

[0077] The Bert model, the long short-term memory network and the fully connected layer are connected in sequence to obtain the initial prediction model.

[0078] Specifically, if Figure 2 As shown, the initial prediction model includes a Bert model (Bidirectional Encoder Representations from Transformer), a long short-term memory network (LSTM, Long Short-Term Memory) and a fully connected layer, wherein this embodiment uses the Roberta model, an optimized version of the Bert model, to extract the features of the clauses to increase the accuracy of feature extraction. The Roberta model, LSTM and fully connected layer are connected in sequence to obtain an initial prediction model, that is, the Roberta model is used to extract features and output weights for each clause, the extracted features are input into the LSTM for feature fusion, and the fully connected layer is used for classification, so that the data features are scaled uniformly on the same dimension, which can avoid the loss of data information and accurately predict the labels, the corresponding prediction similarities and the weights of the clauses.

[0079] Optionally, the training of the initial prediction model using the historical clauses, the historical weights, the historical labels, and the standard similarity to obtain the prediction model includes:

[0080] Input the history clauses into the Bert model, insert a classification symbol before each history clause, and take the output corresponding to the classification symbol as the embedding vector of the history clause;

[0081] Utilizing the attention mechanism of the Bert model to obtain the prediction weight of each of the history clauses;

[0082] Using the long short-term memory network to fuse the embedding vectors to obtain a history clause vector;

[0083] Obtaining the historical long text vector according to the prediction weight and the historical clause vector;

[0084] The historical long text vector is classified using the fully connected layer to obtain the predicted label and the predicted similarity.

[0085] Specifically, if Figure 2As shown, TopN represents the top N data. In this embodiment, the Roberta model, an optimized version of the Bert model, is used to extract the features of historical clauses. The input of the Roberta model is the historical clause, and the classification symbol [CLS] is inserted before each historical clause, and the feature vector corresponding to the historical clause and the feature vector corresponding to the classification symbol [CLS] are output, where the output vector corresponding to the classification symbol [CLS] is used as the embedding vector of each historical clause. Compared with other historical clauses already existing in the historical long text, the classification symbol [CLS] has no obvious semantic information and will more fairly integrate the semantic information of each word / word in the historical clause. The attention mechanism in the Roberta model is used to obtain the historical weight α of each historical clause i The embedding vector of each historical clause output by the Roberta model is input into LSTM for fusion to obtain the representation vector of the historical clause (historical clause vector e i ), and the historical long text vector E is obtained by the following formula:

[0086]

[0087] Among them, E represents the historical long text vector, k represents the number of historical clauses, i represents the i-th historical clause, α i represents the historical weight of the i-th historical clause, e i Represents the historical clause vector of the i-th historical clause. To predict the label of the historical long text. At the same time, in the future, the historical weight e i Get the key historical sentences.

[0088] The historical long text vector E is input into the fully connected layer (here the fully connected layer acts as a classifier), and label prediction and phase velocity prediction are performed on the historical long text vector to obtain the predicted label and predicted similarity.

[0089] Optionally, the training of the initial prediction model using the historical clauses, the historical weights, the historical tags, and the standard similarity to obtain the prediction model further comprises:

[0090] According to the predicted label and the text label, the predicted similarity and the standard similarity, the model parameters of the initial prediction model are updated using the Focal loss function to obtain the prediction model.

[0091] Specifically, the Focal loss function is expressed as:

[0092]

[0093] in, Represents a text label, represents the standard similarity, γ represents the callability factor, and γ>0.

[0094] In this embodiment, the Focal loss function increases the weight of difficult-to-distinguish samples in the loss function, so that the Focal loss function tends to favor difficult-to-distinguish samples. Difficult-to-classify samples are usually concentrated in the tail label part of the data set. Because there are few data samples in the tail label part, the model cannot fully learn the tail label data. Therefore, using the Focal loss function helps to improve the accuracy of difficult-to-distinguish samples and effectively alleviate the tail label problem.

[0095] Optionally, the step of expanding the remaining text labels to obtain historical pseudo labels includes:

[0096] Splitting the historical key sentence to obtain multiple historical keywords;

[0097] According to each of the text tags, marking the importance of each of the historical keywords;

[0098] According to the text label and the order of importance from high to low, a second preset number of the historical keywords are selected and sorted to obtain the historical pseudo-label, wherein the historical pseudo-label includes the text label and the second preset number of the historical keywords;

[0099] The historical keywords, the text labels and the historical pseudo labels are used to train a preset initial support vector machine to obtain a support vector machine, and the support vector machine is used to output the target pseudo labels.

[0100] Specifically, since the text labels obtained in step S1 are mostly in the form of four-character idioms, even the longest label length does not reach 10. Too short labels will prevent the model from fully extracting vector expressions, so the labels need to be expanded. The simplest existing way to expand labels is to directly use interpretations, but the interpretations of labels are abstractly summarized by humans. Humans can understand interpretations because they have a lot of prior knowledge, but for the model, it is not necessarily possible to fully understand the interpretations. For example, the label "optimistic" is defined by Baidu Encyclopedia as "referring to a person's positive cognitive orientation towards the people and things around him, which manifests as a personality trait. Cheerful, positive, and confident in the development of things." It is difficult for the model to understand this highly generalized interpretation, and this label interpretation has no connection with the data set itself. Introducing text that is inconsistent with the distribution of the data set itself will also cause the model to be unable to fully train. Therefore, this embodiment proposes a word information expansion method.

[0101] First, the historical key sentence is split into word units to obtain multiple historical keywords. For example, if the historical key sentence is "Our school's graduate tutor insists on cultivating morality, rigorous scholarship, and dedicated education, and has made outstanding contributions to graduate training and education", then the historical keywords are "graduate student", "our school", "tutor", "scholarship", "rigorous"... and for the multiple text tags corresponding to the historical key sentence, the importance of the historical keywords to it is marked respectively. For example, for the text tag "rigorous scholarship", each of the above historical keywords is marked with the importance W, and the following is obtained:

[0102] W 严谨治学，治学 =99%,W 严谨治学，严谨 =90%, W 严谨治学，导师 =95%,W 严谨治学，研究生

[0103] =80%……W 严谨治学，我校 =10%.

[0104] Then, a second preset number such as 4 historical keywords are selected according to importance to obtain pseudo labels, that is, historical pseudo label 1 = rigorous scholarship + scholarship, rigorous, tutor, graduate student.

[0105] Additionally, if Figure 3 As shown in the figure, when subsequently recommending tags for long texts, in order to quickly obtain accurate target pseudo-labels, an initial support vector machine is set up and trained using historical key sentences, historical pseudo-labels, and text labels as a dataset. This results in a support vector machine (SVM). The SVM input is the predicted key sentence and predicted label, and the output is the target pseudo-label corresponding to each predicted label. However, the SVM input has certain limitations, namely, it can only process one predicted label at a time. Therefore, when predicting the target pseudo-label, the relationship between the predicted key sentence and the predicted label must be transformed from <predicted key sentence, predicted label 1, predicted label 2, predicted label 3...> to <predicted key sentence, predicted label 1>, <predicted key sentence, predicted label 2>, <predicted key sentence, predicted label 3>... so that each key sentence is bound to only one predicted label. Furthermore, one SVM can only predict the target pseudo-label for one predicted label. That is, if n predicted labels need to be expanded, n SVMs are used to predict the target pseudo-label. Furthermore, the predicted key sentence must be converted into a key sentence vector for SVM processing.

[0106] In this embodiment, historical pseudo labels are constructed according to the importance of historical keywords, which effectively improves the accuracy of historical pseudo labels.

[0107] Optionally, the historical pseudo-similarity includes positive sample similarity and negative sample similarity, the re-ranking model includes a pseudo-similarity model and a ranking model, and the initial ranking model is trained with the historical key sentences, the historical label sequence, the historical pseudo-label, and the historical pseudo-similarity to obtain the re-ranking model, including:

[0108] Obtaining the first historical pseudo label in the historical label sequence as a positive sample;

[0109] Randomly selecting a third preset number of historical pseudo labels other than the positive samples in the historical label sequence as negative samples;

[0110] Using the pseudo-similarity model, a positive sample similarity is obtained based on the positive sample and the historical key sentence, wherein the positive sample similarity is the closest distance between the positive sample and the historical key sentence in a vector space, and the pseudo-similarity model is trained based on the historical key sentence, the historical pseudo-label, and the similarity of the historical pseudo-label;

[0111] Using the pseudo-similarity model, obtaining a negative sample similarity based on the negative sample and the historical key sentence, wherein the negative sample similarity is less than the positive sample similarity, and the negative sample similarity is the distance between the negative sample and the historical key sentence in a vector space;

[0112] Using a preset initial sorting model, the predicted label sequence is obtained according to the positive sample similarity and the negative sample similarity;

[0113] The model parameters of the initial sorting model are updated according to the predicted label sequence and the historical label sequence to obtain the sorting model

[0114] Specifically, if Figure 4 As shown, the preset initial pseudo-similarity model is first trained to obtain a trained pseudo-similarity model. The historical pseudo-label with the highest standard similarity to the historical long text (i.e., ranked first) in the historical label sequence is taken as a positive sample, and then a third preset number, such as 30 historical pseudo-labels, in the historical label sequence are randomly selected as negative samples. The initial pseudo-similarity model uses the SBERT model (Sentence Bert, SBERT). The initial pseudo-similarity model is trained based on historical key sentences and historical pseudo-similarity to obtain a trained pseudo-similarity model. The input of the pseudo-similarity model is the predicted key sentence and the target pseudo-label, and the output is the predicted pseudo-similarity. Among them, the loss function for training the pseudo-similarity model is Triplet loss, which is specifically expressed as:

[0115] L=max(d(a,p)-d(a,n)+margin,0),

[0116] Among them, a represents the historical key sentence, p represents the positive sample, n represents the negative sample, d(a, p) represents the distance between the positive sample and the historical key sentence in the vector space, d(a, n) represents the distance between the negative sample and the historical key sentence in the vector space, and margin is a constant greater than 0.

[0117] Through this model, the closest distance between the positive sample and the predicted key sentence in the vector space can be obtained. Correspondingly, the distance between the negative sample and the predicted key sentence can also be obtained. The distance between the positive sample and the predicted key sentence is always smaller than the distance between the negative sample and the predicted key sentence, that is, the similarity of the positive sample is always greater than the similarity of the negative sample.

[0118] This embodiment uses contrastive learning to make the positive sample more similar to the key sentence and less similar to other labels (such as negative samples). This results in a closer distance between the key sentence and the positive sample in vector space and a greater distance from other labels, thereby obtaining an accurate predicted pseudo-similarity. The ranking model is trained using the positive and negative sample similarities to obtain a more accurate predicted label sequence.

[0119] Optionally, the acquiring of the historical long text, annotating the historical long text, and obtaining a plurality of text tags corresponding to the historical long text and a standard similarity between the text tags and the historical long text further includes:

[0120] Obtaining the text length of the historical long text and the number of corresponding text tags;

[0121] The historical long text whose text length is less than a second preset length and whose number of text tags is less than a third preset number is deleted.

[0122] Specifically, the lengths of the long historical texts obtained from various websites vary, and the present invention mainly constructs a label recommendation model for long historical texts. Therefore, the obtained long historical texts need to be processed, that is, the long historical texts that are less than a second preset length, such as 200, need to be deleted. Long historical texts represent many meanings, so using long historical texts with more labels for model training can effectively increase the accuracy of model training. Therefore, long historical texts with too few text labels are deleted, that is, long historical texts with less than a third preset number, such as 10, need to be deleted. Finally, 12,759 samples and 164 value labels were obtained, of which 60% of the sample lengths exceeded 500. This is to build an accurate data set, thereby increasing the accuracy of subsequent model training.

[0123] In a specific embodiment, the existing label classification model and the model constructed by the model construction method provided by the present invention were tested using the preset evaluation indicators (micro-average micro@F1 and macro-average macro@F1) and the same value data set. The results are as follows:

[0124] Table 1 Experimental results of different models on the value dataset

[0125]

[0126] Among them, F1 represents the multi-classification label prediction method in the existing technology, precision represents the accuracy rate, Recall represents the recall rate, that is, the number of predicted labels, Bert is the experiment using the Bert model alone, Sentence_Bert is the experiment using the Sentence_Bert model alone, Bert_LSTM_BCE is the experiment using the Bert model and LSTM model, and the loss function uses the BCE loss function for the experiment, Bert_LSTM_FL is the experiment using the Bert model and LSTM model, and the loss function uses the Focal loss function for the experiment, Bert_LSTM_FL_SBert is the experiment using the Bert model, LSTM model and Sentence_Bert model, and the loss function uses the Focal loss function for the experiment, Bert_LSTM_FL_SBert with pseudolabel is the experiment using the Bert model, LSTM model and Sentence_Bert model, and the loss function uses the Focal loss function and introduces pseudo labels. Since some samples in the value dataset have multiple labels, the calculation method is different from the F1 calculation method of multi-classification. Instead, it adopts the calculation method of multi-label classification and takes the top k labels of the model's final prediction results as recommended labels.

[0127] As can be seen from Table 1,

[0128] When the baseline model is the Bert model, the maximum text length of the model is limited to 512 characters, so long input text needs to be truncated. However, this will result in information loss compared to humans. Its micro F1@1 and macro F1@1 reach 0.5189 and 0.4008 respectively. However, its performance in tail labels is unsatisfactory due to insufficient samples.

[0129] After segmenting long texts, the baseline model, using the Bert_LSTM model, surpassed the Bert model in all metrics. Furthermore, experimental results using Bert_LSTM_BCE and Bert_LSTM_FL show that the advantage of using the Focal loss function lies not only in improved overall performance but also in improved processing of tail labels. Compared to the BCE loss function, the Focal loss function has a slightly lower micro F1@1, but improves macro F1@1 by 0.0102 and tail label performance by 0.0892.

[0130] Because the label prediction model needs to identify key sentences, it is inevitably affected by information from non-key clauses, which can cause the actual labels to be ranked lower. Therefore, after completing the model classification, the Sentence Bert model is used to re-rank the candidate label set. Specifically, when the baseline model is Bert_LSTM_FL_SBert, the experimental results exceed those of the Bert_LSTM model alone in terms of micro, macro, and tail labels. Furthermore, the experimental results using pseudo-label descriptions (i.e., Bert_LSTM_FL_SBert with pseudo labels) show improvements of 0.006 in micro F1@1, 0.0094 in macro F1@1, and 0.0026 in tail labels, demonstrating that adding pseudo-descriptions to labels can improve the accuracy of label recommendations.

[0131] The experimental data in Table 1 also show that the experimental results of using the Bert_LSTM model or the Sentence-Bert model alone are not as good as the prediction-then-classification method in all indicators, which also proves the accuracy of the long text tag recommendation method of the present invention.

[0132] An embodiment of the present invention further provides a device for constructing a tag recommendation model for a long text, comprising a memory and a processor;

[0133] The memory is used to store computer programs;

[0134] The processor is configured to implement the above-mentioned method for constructing a tag recommendation model for a long text when executing the computer program.

[0135] The advantages of the apparatus for constructing a tag recommendation model for a long text and the method for constructing a tag recommendation model for a long text in this embodiment are the same as those of the prior art, and are not described in detail here.

[0136] like Figure 5 As shown, an embodiment of the present invention further provides a tag recommendation method for a long text, comprising:

[0137] Step A1: Obtain a long text to be tested, and obtain multiple clauses to be tested based on the long text to be tested.

[0138] Specifically, a long text to be tested for label recommendation is obtained, which may be a paragraph, a long sentence, etc., and the long text to be tested is segmented according to a preset length and overlap to obtain multiple clauses to be tested, where the preset length may be 200 and the overlap may be 30 words.

[0139] Step A2: input the clause to be tested into the prediction model constructed according to the long text label recommendation model construction method as described above, and obtain the target weight, predicted label and similarity of the predicted label of each clause to be tested.

[0140] Step A3: obtaining predicted key sentences according to the target weights, preprocessing the predicted labels according to the predicted similarities, and expanding the preprocessed predicted labels to obtain multiple target pseudo labels.

[0141] Specifically, each clause to be tested is sorted from high to low according to the target weight, and a preset number of clauses to be tested, for example, 3 clauses to be tested, are selected from the sequence of clauses to be tested, and the selected clauses to be tested are spliced ​​to obtain a predicted key sentence (the length of the predicted key sentence is less than 512) that can represent the meaning of the entire text, so as to further perform tag recommendation. Each predicted label is sorted from high to low according to the predicted similarity, and the predicted labels with a similarity less than the preset similarity are deleted, which effectively reduces the subsequent model data processing pressure and increases the model prediction efficiency. In order to more accurately represent the meaning of the predicted label and increase the accuracy of the label recommendation, this embodiment expands the predicted label, for example, using SVM to obtain multiple target pseudo labels (target pseudo label = predicted label + predicted keyword), so that the re-ranking model can rearrange them according to the predicted pseudo similarity between the target pseudo label and the predicted key sentence.

[0142] Step A4: input the plurality of target pseudo labels and the predicted key sentences into the re-ranking model constructed according to the long text label recommendation model construction method as described above to obtain a predicted label sequence.

[0143] Step A5: sequentially select a preset target number of target pseudo-labels in the predicted label sequence as recommended labels.

[0144] Specifically, the predicted label sequence obtained in step A4 is sorted in descending order of the predicted pseudo-similarity between the target pseudo-label and the predicted key sentence. Therefore, the first target pseudo-label in the predicted label sequence has the highest similarity to the predicted key sentence and, therefore, best represents the meaning of the predicted key sentence. Thus, a predetermined number of target pseudo-labels, for example, three, are selected as recommended labels based on the order in which the predicted label sequence is arranged.

[0145] In this embodiment, the long text to be tested is divided into multiple clauses to be tested so that the prediction model can perform label prediction. The prediction model is used to obtain the preliminary target weights, predicted labels and corresponding predicted similarities of the clauses to be tested, and the predicted key sentences and target pseudo labels that can represent the meaning of the long text to be tested are obtained based on the target weights and predicted similarities. While reducing the length of the long text, the predicted data is preliminarily processed to reduce the processing pressure of the subsequent re-ranking model and increase the processing efficiency of the re-ranking model. The predicted labels are also expanded and described so that the model can better understand the meaning of the predicted labels to increase the accuracy of label recommendations. The target pseudo labels are arranged using the re-ranking model to obtain a predicted label sequence, and recommended labels are obtained based on the predicted labels, which effectively increases the accuracy of long text label recommendations.

[0146] The embodiment of the present invention further discloses a tag recommendation device for a long text, comprising a memory and a processor;

[0147] The memory is used to store computer programs;

[0148] The processor is configured to implement the above-mentioned tag recommendation method for a long text when executing the computer program.

[0149] The advantages of the long text tag recommendation device and the long text tag recommendation method of this embodiment are the same as those of the prior art, and will not be described in detail here.

[0150] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A method for constructing a tag recommendation model for long text, characterized in that: include: Acquire a historical long text, annotate the historical long text, and obtain a plurality of text labels corresponding to the historical long text and a standard similarity between the text labels and the historical long text; Perform overlapping segmentation on the long historical text according to a first preset length to obtain a plurality of historical clauses, and mark the historical weight of each of the historical clauses in the long historical text; Selecting a first preset number of the historical clauses in descending order of the historical weights for splicing to obtain historical key sentences, deleting the text labels whose standard similarity is less than a preset threshold, performing label expansion on the remaining text labels to obtain historical pseudo-labels, and marking the historical pseudo-similarity of each historical pseudo-label with the historical long text, and sorting the historical pseudo-labels in descending order of the historical pseudo-similarity to obtain a historical label sequence; Constructing an initial prediction model, training the initial prediction model with the historical clauses, the historical weights, the historical labels, and the standard similarity to obtain a prediction model, wherein the prediction model is used to output a target weight for each clause to be tested, a plurality of prediction labels, and a predicted similarity of the plurality of prediction labels, wherein a predicted key sentence is obtained according to the target weight, the prediction labels are preprocessed according to the predicted similarity, and the preprocessed prediction labels are expanded to obtain a target pseudo label; Constructing an initial re-ranking model, training the initial re-ranking model with the historical key sentence, the historical label sequence, the historical pseudo-label, and the historical pseudo-similarity to obtain a re-ranking model, wherein the re-ranking model is used to obtain a predicted pseudo-similarity based on the predicted key sentence and the target pseudo-label, and output a predicted label sequence sorted in descending order according to the predicted pseudo-similarity; A tag recommendation model is obtained according to the prediction model and the re-ranking model.

2. The method for constructing a tag recommendation model for a long text according to claim 1, wherein: The method for constructing the initial prediction model includes: Get the Bert model, long short-term memory network and fully connected layer; The Bert model, the long short-term memory network and the fully connected layer are connected in sequence to obtain the initial prediction model.

3. The method for constructing a tag recommendation model for a long text according to claim 2, characterized in that: The training of the initial prediction model using the historical clauses, the historical weights, the historical labels, and the standard similarity to obtain the prediction model includes: Input the history clauses into the Bert model, insert a classification symbol before each history clause, and take the output corresponding to the classification symbol as the embedding vector of the history clause; Utilizing the attention mechanism of the Bert model to obtain the prediction weight of each of the history clauses; Using the long short-term memory network to fuse the embedding vectors to obtain a history clause vector; Obtaining the historical long text vector according to the prediction weight and the historical clause vector; The historical long text vector is classified using the fully connected layer to obtain the predicted label and the predicted similarity.

4. The method for constructing a tag recommendation model for a long text according to claim 3, wherein: The step of training the initial prediction model using the historical clauses, the historical weights, the historical tags, and the standard similarity to obtain the prediction model further comprises: According to the predicted label and the text label, the predicted similarity and the standard similarity, the model parameters of the initial prediction model are updated using the Focal loss function to obtain the prediction model.

5. The method for constructing a tag recommendation model for a long text according to claim 1, wherein: The step of expanding the remaining text labels to obtain historical pseudo labels includes: Splitting the historical key sentence to obtain multiple historical keywords; According to each of the text tags, marking the importance of each of the historical keywords; According to the text label and the order of importance from high to low, a second preset number of the historical keywords are selected and sorted to obtain the historical pseudo-label, wherein the historical pseudo-label includes the text label and the second preset number of the historical keywords; The historical keywords, the text labels and the historical pseudo labels are used to train a preset initial support vector machine to obtain a support vector machine, and the support vector machine is used to output the target pseudo labels.

6. The method for constructing a tag recommendation model for a long text according to claim 5, characterized in that: The historical pseudo-similarity includes positive sample similarity and negative sample similarity, the re-ranking model includes a pseudo-similarity model and a ranking model, and the re-ranking model is obtained by training the initial ranking model with the historical key sentences, the historical label sequence, the historical pseudo-label, and the historical pseudo-similarity. Obtaining the first historical pseudo label in the historical label sequence as a positive sample; Randomly selecting a third preset number of historical pseudo labels other than the positive samples in the historical label sequence as negative samples; Using the pseudo-similarity model, a positive sample similarity is obtained based on the positive sample and the historical key sentence, wherein the positive sample similarity is the closest distance between the positive sample and the historical key sentence in a vector space, and the pseudo-similarity model is trained based on the historical key sentence, the historical pseudo-label, and the similarity of the historical pseudo-label; Using the pseudo-similarity model, obtaining a negative sample similarity based on the negative sample and the historical key sentence, wherein the negative sample similarity is less than the positive sample similarity, and the negative sample similarity is the distance between the negative sample and the historical key sentence in a vector space; Using a preset initial sorting model, the predicted label sequence is obtained according to the positive sample similarity and the negative sample similarity; The model parameters of the initial sorting model are updated according to the predicted label sequence and the historical label sequence to obtain the sorting model.

7. The method for constructing a tag recommendation model for a long text according to claim 1, wherein: The acquiring of the historical long text, labeling the historical long text, and obtaining a plurality of text labels corresponding to the historical long text and a standard similarity between the text labels and the historical long text further includes: Obtaining the text length of the historical long text and the number of corresponding text tags; The historical long text whose text length is less than a second preset length and whose number of text tags is less than a third preset number is deleted.

8. A device for constructing a tag recommendation model for long texts, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the method for constructing a tag recommendation model for a long text according to any one of claims 1 to 7 when executing the computer program.

9. A tag recommendation method for long text, characterized in that: include: Obtaining a long text to be tested, and obtaining a plurality of clauses to be tested according to the long text to be tested; Inputting the clause to be tested into the prediction model constructed according to the method for constructing a long text label recommendation model according to claims 1 to 7, and obtaining a target weight, a predicted label, and a similarity of the predicted label for each clause to be tested; Obtaining a predicted key sentence according to the target weight, preprocessing the predicted label according to the predicted similarity, and expanding the preprocessed predicted label to obtain a plurality of target pseudo labels; Inputting the plurality of target pseudo labels and the predicted key sentences into a re-ranking model constructed according to the long text label recommendation model construction method according to claims 1 to 7 to obtain a predicted label sequence; A preset target number of target pseudo labels in the predicted label sequence are selected in sequence as recommended labels.

10. A tag recommendation device for long text, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the tag recommendation method for a long text according to claim 9 when executing the computer program.

Images