Context-based citation recommendation method, device, electronic device and storage medium
Through the context-based citation recommendation method, the citation vector and context vector description network are used to solve the problem of low efficiency in citation recommendation in the prior art, and more accurate citation matching and recommendation are achieved when writing.
Patent Information
- Application Number
- CN202110502375.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-08
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-05-08
AI Technical Summary
Existing citation recommendation methods are inefficient and cannot accurately match the context, making it difficult to find suitable citations when writing.
Using the context-based citation recommendation method, the pre-trained citation recommendation model is used to use the citation vector description network, the context vector description network and the citation matching degree calculation network, and the appropriate citations are automatically recommended from the preset citation library.
It realizes more efficient and accurate recommendations of citations matching the context in a given context, improving the efficiency of citation usage in writing.
Smart Images

Figure CN113191142B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of citation recommendation, and in particular to a context-based citation recommendation method, device, electronic device and storage medium. Background Art
[0002] Quotation is a very common rhetorical device in writing. Quoting famous sayings, proverbs, colloquialisms and other quotations in writing can not only express and emphasize the author's meaning more accurately, but also increase the literary grace and persuasiveness.
[0003] However, the number of famous sayings and proverbs is staggering. For most ordinary people, they can only remember a few quotations from memory, and often cannot think of quotations that are suitable for the current context when writing. Although search engines and some famous quote database websites provide help in finding quotations through quotation searches based on keyword matching or quotation classification based on themes, their role is relatively limited. On the one hand, many quotations are in classical Chinese, and the remaining modern quotations often use rhetorical devices such as metaphors, which leads to the fact that the words used in most quotations are very different from those used in daily writing, so quotation searches based on word matching are invalid. On the other hand, even if the quotations are classified by theme, there are still countless quotations under the same theme, and finding a suitable quotation is still as difficult as finding a needle in a haystack.
[0004] Therefore, how to avoid the existing citation recommendation methods being inefficient and unable to accurately match the description of the context remains a problem that needs to be solved urgently by those skilled in the art. Summary of the invention
[0005] The present invention provides a context-based citation recommendation method, device, electronic device and storage medium to solve the problem that the existing citation recommendation method is inefficient and cannot accurately match the description of the context. By using a pre-trained citation recommendation model to automatically recommend several citations suitable for the current context from a preset citation library based on the content of the context under a given context, the method has high efficiency and recommendation accuracy.
[0006] The present invention provides a context-based citation recommendation method, comprising:
[0007] Determine the context of the citation to be recommended;
[0008] Inputting the context into a citation recommendation model and outputting corresponding recommended citations;
[0009] The citation recommendation model is trained based on sample context and corresponding citation tags, and the network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network and a citation matching degree calculation network connected in sequence.
[0010] According to a context-based citation recommendation method provided by the present invention, the construction of the sample context and the corresponding citation label specifically includes:
[0011] Determine possible recommended citation tags based on a preset citation library;
[0012] For any possible recommended citation tag, if it also exists in the preset corpus, then,
[0013] Determining any of the possible recommended citation tags as a correct citation tag, and determining a context corresponding to the correct citation tag in the preset corpus as a sample context;
[0014] During the training process of the citation recommendation model, for the sample context used in the current round of iterative calculation, a preset number of negative citation labels are randomly selected from other citation sets in the preset citation library that exclude the correct citation labels corresponding to the sample context, and the citation labels corresponding to the sample context used in the current round of iterative calculation are constructed by combining the preset number of negative citation labels and the correct citation labels.
[0015] According to a context-based citation recommendation method provided by the present invention, the citation vector description network is used to combine the input citation with the semantic original information of the citation using a vector representation.
[0016] According to a context-based citation recommendation method provided by the present invention, the network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network, and a citation matching degree calculation network connected in sequence, specifically including:
[0017] During the training process of the citation recommendation model,
[0018] The citation vector description network combines the input citation label with the semantic information of each citation in the citation label and uses vector representation to output the citation vector matrix to the citation matching degree calculation network;
[0019] The context vector description network uses a vector to represent the input corresponding sample context and outputs a context vector to the citation matching degree calculation network;
[0020] The citation matching degree calculation network determines the pseudo matching degree of the correct citation tag in the citation tags based on the citation vector matrix and the context vector;
[0021] The pseudo matching degree is used to construct a loss function during training of the citation recommendation model.
[0022] According to a context-based citation recommendation method provided by the present invention, the citation vector description network combines the input citation label with the semantic information of each citation in the citation label and outputs a citation vector matrix using a vector representation, which specifically includes:
[0023] The citation vector description network calculates any citation q in the input citation label through the following formula k Any word in The vector w′ i :
[0024]
[0025] in, Any of the words The set of semantic primitives, s j yes Zhongyiyuans j The vector representation of w i For any of the words The word vector, α is the preset weight coefficient, s j and w i These are all parameters to be optimized in the network structure during the training of the citation recommendation model;
[0026] The citation vector description network then calculates any citation q by the following formula k The citation vector t k :
[0027]
[0028]
[0029] Among them, PLM q is a citation representation model, [CLS] is a special sentence starter in the citation representation model, and m is a citation q k The number of words, Any citation q output by the citation representation model k Words in The hidden state of Any citation q output by the citation representation model k Words in The hidden state of Any citation q output by the citation representation model k Words in The hidden state of is the hidden state of the special sentence starter [CLS] output by the citation representation model, and is used during the use of the citation representation model The corresponding word vectors w′1,…,w′i ,…,w′ m Participate in the calculation of hidden states;
[0030] The citation vector description network finally calculates the citation vector matrix Q by the following formula * And output:
[0031]
[0032] Among them, k * =1,2,…,n * , n * -1 is the preset training number.
[0033] According to a context-based citation recommendation method provided by the present invention, the context vector description network outputs a context vector using a vector representation of the input corresponding sample context, specifically comprising:
[0034] The context vector describes the learning representation sequence c′ of the input corresponding sample context c through the following formula:
[0035] c′={c l ,[MASK],c r}
[0036] Among them, c l is the character sequence in the context of the corresponding sample, c r is the character sequence of the following text in the corresponding sample context, and [MASK] is a mark character;
[0037] The context vector description network then calculates the context vector u of the corresponding sample context c through the following formula:
[0038]
[0039]
[0040] Among them, PLM c is a context representation model, [CLS] is a special sentence starter in the context representation model, is the hidden state of the tag character [MASK] output by the context representation model, is the hidden state of the special sentence start symbol [CLS] output by the context representation model, using the c pre-stored by the context representation model l and c r The corresponding word vector sequences participate in the calculation of the hidden state;
[0041] Correspondingly, the citation matching degree calculation network determines the pseudo matching degree of the correct citation tag in the citation tags based on the citation vector matrix and the context vector, specifically including:
[0042] The pseudo-matching degree p of the correct citation tag q in the citation tag is calculated by the following formula: * :
[0043]
[0044] in, is the set of negative citations of the citation label except the correct citation label q, u is the context vector of the corresponding sample context c, t is the citation vector of the correct citation label q, and t′ is the citation vector of any negative citation q′ in the set of other negative citations;
[0045] The pseudo-matching degree is used to construct the loss function when training the citation recommendation model, which specifically includes:
[0046] The loss function is constructed by the following formula
[0047]
[0048] Among them, p * is the pseudo matching degree.
[0049] The present invention also provides a context-based citation recommendation device, comprising:
[0050] A determination unit, used for determining the context of the citation to be recommended;
[0051] A recommendation unit, configured to input the context into a citation recommendation model and output a corresponding recommended citation;
[0052] The citation recommendation model is trained based on sample context and corresponding citation tags, and the network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network and a citation matching degree calculation network connected in sequence.
[0053] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described context-based citation recommendation methods are implemented.
[0054] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described context-based citation recommendation methods.
[0055] The context-based citation recommendation method, device, electronic device and storage medium provided by the present invention determine the context of the citation to be recommended; input the context into the citation recommendation model, and output the corresponding recommended citation; wherein the citation recommendation model is obtained by training based on the sample context and the corresponding citation label, and the network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network and a citation matching degree calculation network connected in sequence. The pre-trained citation recommendation model is used to process the context of the citation to be recommended, and the recommended citation matching with the context is found and output, wherein the citation recommendation model is obtained after training based on a large number of sample contexts and corresponding citation labels, and the accuracy of the citation recommendation model is ensured, and the network structure in the citation recommendation model training process includes a citation vector description network, a context vector description network and a citation matching degree calculation network connected in sequence, and the basic algorithm framework for citation recommendation in the citation recommendation model is given. Therefore, the method, device, electronic device and storage medium provided by the present invention realize more efficient and accurate recommendation of citations matching with the context under a given context. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0057] Figure 1 A schematic diagram of a process flow of a context-based citation recommendation method provided by the present invention;
[0058] Figure 2 A schematic diagram of the structure of a context-based citation recommendation device provided by the present invention;
[0059] Figure 3 A schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION
[0060] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0061] The existing question-answering systems generally have a single source of knowledge and cannot consider the various forms of information and knowledge in the current industry. Figure 1 A context-based citation recommendation method of the present invention is described. Figure 1 A flowchart of a context-based citation recommendation method provided by the present invention is shown in FIG. Figure 1 As shown, the method includes:
[0062] Step 110, determining the context of the citation to be recommended.
[0063] Specifically, the context-based citation recommendation method provided by the present invention is performed under a given context, that is, according to the given context, citations matching the context meaning are automatically selected from a preset citation library for recommendation. Therefore, the context content given by the user needs to be determined first.
[0064] Step 120, inputting the context into a citation recommendation model, and outputting corresponding recommended citations;
[0065] The citation recommendation model is trained based on sample context and corresponding citation tags, and the network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network and a citation matching degree calculation network connected in sequence.
[0066] Specifically, after determining the context given by the user, the context is input into the citation recommendation model, and the corresponding recommended citation is output; it should be noted here that the recommended citation can be one or more, which is determined according to the representation rule of the output result in the citation recommendation model. For example, if the representation rule stipulates that only the best recommended citation is output, then the citation recommendation model outputs the only best recommended citation. If the representation rule stipulates that the citation that meets the recommendation condition is output, then the output can be multiple citations that meet the recommendation condition, which is not specifically limited here. Among them, the citation recommendation model is a model pre-trained based on machine learning. The model is trained based on a large number of sample contexts and corresponding citation labels, so that the accuracy of the model can be guaranteed. It should also be noted here that the citation recommendation model also carries a preset citation library. The training of the citation recommendation model is based on the training data set constructed by the preset citation library. When the citation recommendation model is trained and used, its function is also to select accurate citations from the preset citation library for recommendation. Therefore, the preset citation library needs to be carried when the citation recommendation model is used after the training is completed. Finally, the network structure of the model training is further limited. The network structure includes the citation vector description network, the context vector description network and the citation matching degree calculation network connected in sequence. The basic algorithm framework for citation recommendation in the citation recommendation model is given, which includes the following three steps: A. Citation representation learning step: it can learn a vector representation for each citation in the preset citation library; B. Context representation learning step: it can learn a vector representation for a given context; C. Citation matching degree calculation step: it can calculate a matching degree for all citations in the citation library for a given context, and then select the citation with higher matching pairs as the recommended citation suitable for the given context. The above three steps correspond to the citation vector description network, the context vector description network and the citation matching degree calculation network in the network structure during model training. During model training, the parameters to be adjusted in the citation vector description network and the context vector description network are continuously optimized to obtain better vector representations of citations and contexts, so that the citation matching degree calculation network can output more accurate citation matching degrees.
[0067] The method provided by the present invention determines the context of the citation to be recommended; inputs the context into the citation recommendation model, and outputs the corresponding recommended citation; wherein the citation recommendation model is obtained by training based on the sample context and the corresponding citation label, and the network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network, and a citation matching degree calculation network connected in sequence. The pre-trained citation recommendation model is used to process the context of the citation to be recommended, and the recommended citation matching with the context is found and output, wherein the citation recommendation model is obtained after training based on a large number of sample contexts and corresponding citation labels, so as to ensure the accuracy of the citation recommendation model, and the network structure during the citation recommendation model training process includes a citation vector description network, a context vector description network, and a citation matching degree calculation network connected in sequence, and the basic algorithm framework for citation recommendation in the citation recommendation model is given. Therefore, the method provided by the present invention realizes more efficient and accurate recommendation of citations matching with the context under a given context.
[0068] Based on the above embodiment, in the method, the construction of the sample context and the corresponding citation label specifically includes:
[0069] Determine possible recommended citation tags based on a preset citation library;
[0070] For any possible recommended citation tag, if it also exists in the preset corpus, then,
[0071] Determining any of the possible recommended citation tags as a correct citation tag, and determining a context corresponding to the correct citation tag in the preset corpus as a sample context;
[0072] During the training process of the citation recommendation model, for the sample context used in the current round of iterative calculation, a preset number of negative citation labels are randomly selected from other citation sets in the preset citation library that exclude the correct citation labels corresponding to the sample context, and the citation labels corresponding to the sample context used in the current round of iterative calculation are constructed by combining the preset number of negative citation labels and the correct citation labels.
[0073] Specifically, for the training data set used in the training process of the citation recommendation model, there is a specific data source, wherein the specific construction of the sample context and the corresponding citation label that needs to be used in the training process includes the following contents: First, a preset citation library must be determined. The citations in the preset citation library are constructed based on a common attribute among the citations, which can be the same subject or the same field or from the same literature book, without specific limitation. All citations are extracted from the preset citation library, and then each extracted citation is searched in the preset corpus. If the citation q exists in the preset corpus, the context corresponding to the citation q in the preset corpus is extracted as the context sample x, and then the citation is used as the correct citation label q and the screened context sample as the context-citation pair to participate in the training of the citation recommendation model. During the training of the citation recommendation model, when each context-citation pair is used for iterative calculation, it is necessary to temporarily select the negative citation label of the correct citation label in the context-citation pair. The screening method is that when the current correct citation label q is used in the iterative calculation process, a preset number of other citations are randomly selected from the preset citation library except for the correct citation label as the negative example set. The negative example set The final citation label is combined with the correct citation label q to obtain the final citation label. The final citation label matrix and the context sample form a pair of training data. Each pair of training data in the training data set used in the citation recommendation model training process is constructed through the above process, except that the context-citation pair of the "correct answer" can be pre-constructed, and the negative example set of the context-citation pair of the "correct answer" is formed into a pair of training data. It is selected in real time during the training of the citation recommendation model. This is because the same correct citation label may correspond to different sample contexts, and the negative example sets in the multiple context-citation pairs formed by them may not be exactly the same. They need to be randomly screened in real time to ensure the accuracy of model training.
[0074] The preset training number is explained here. The preset training number is set to reduce the amount of calculation in the training process. If all citations in the preset citation library except the correct citation label are used as the negative example set during the training process, since the number of citation items in the preset citation library is very large, the elements in the negative example set constructed in this way will also be very large, resulting in a large amount of data to be calculated in each round of training iteration. Therefore, when constructing the negative example set, a certain number of citations in the preset citation library except the correct citation label are randomly selected as negative examples. The above certain number is the preset training number. The preset training number is usually much smaller than the number of citations in the preset citation library, but in order to balance the accuracy of model training, a larger value will be appropriately taken. Usually, an appropriate value in the middle that balances the amount of calculation and accuracy is taken.
[0075] Based on the above embodiment, in this method, the citation vector description network is used to combine the input citation with the semantic information of the citation and adopt a vector representation.
[0076] Specifically, considering that quotations often have fewer words but rich meanings, general sentence representation learning methods cannot learn better vector representations for short quotations. Therefore, the present invention integrates the semantic meaning knowledge of each word in the quotations into the representation learning of the quotations. Among them, the semantic meaning is the smallest semantic unit in linguistics, and the semantic meaning of a word carries the core semantic information of the word. The citation vector description network first adds the semantic meaning information to each word in the citation to obtain the vector representation of the semantic meaning information of each word, and then learns the vector representation of the entire citation through the citation learning model. During the model learning process, the word vector that comes with the citation learning model is no longer used, but the word vector containing the semantic meaning information calculated previously is used.
[0077] Based on the above embodiment, in this method, the network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network, and a citation matching degree calculation network connected in sequence, specifically including:
[0078] During the training process of the citation recommendation model,
[0079] The citation vector description network combines the input citation label with the semantic information of each citation in the citation label and uses vector representation to output the citation vector matrix to the citation matching degree calculation network;
[0080] The context vector description network uses a vector to represent the input corresponding sample context and outputs a context vector to the citation matching degree calculation network;
[0081] The citation matching degree calculation network determines the pseudo matching degree of the correct citation tag in the citation tags based on the citation vector matrix and the context vector;
[0082] The pseudo matching degree is used to construct a loss function during training of the citation recommendation model.
[0083] Specifically, during the training process of the citation recommendation model, the parameters to be adjusted in the first two networks, the citation vector description network and the context vector description network, which are sequentially connected in the network structure during the training of the citation recommendation model are trained and learned. When the citation recommendation model is trained, the citation vector description network combines the input citation label with the semantic information of each citation in the citation label using vector representation to output the citation vector matrix to the citation matching calculation network, wherein the use of vector representation is divided into two steps: first, each word in the citation is represented by a word vector with added semantic information, and then the citation vector of the entire citation to which the word belongs is learned based on the word vector, and the final citation vector is output to the citation matching calculation network. It should be noted here that since the citation label includes the correct citation label q and the negative example set Therefore, when each citation in the citation label is represented by a citation vector, the citation label with the preset training number + 1 citation is represented by a matrix, the number of rows of the matrix is the preset training number + 1, and the number of columns of the matrix is the vector dimension of any citation in the citation label. The context description network uses vector representation to output the context samples corresponding to the citation labels input in the same batch to the citation matching calculation network, where the vector representation of the context samples is also based on machine learning to learn the vector representation of the context samples as a whole. The context vector description network has its own preset word vector library, and participates in the calculation of the vector representation of the overall context sample by extracting the word vectors of each word in the context sample in the preset word vector library. The citation matching degree calculation network determines the pseudo matching degree of the correct citation label in the citation label based on the citation vector matrix and the context vector. The pseudo matching degree is determined by finding the similarity between each citation vector in the citation vector matrix and the context vector to obtain a matching degree vector (i.e., a similarity vector). Each element in the matching degree vector represents the matching degree between the corresponding citation and the context sample. The matching degree value corresponding to the correct citation label is searched from the matching degree vector as the pseudo matching degree of the correct citation label. It should be noted here that there are many common methods and many similarity calculation functions for calculating the matching degree (i.e., similarity) between two vectors, which are not specifically limited here, but the dimensions of the two vectors for calculating the matching degree need to be the same. Therefore, the dimensions of the citation vector and the context vector need to be set to be consistent during calculation.
[0084] During the use of the citation recommendation model, the context vector description network uses a vector to represent the context to be tested of the input citation to be recommended and outputs the context vector to be tested to the citation matching degree calculation network; the citation matching degree calculation network determines the true matching degree of all citations in the citation vector matrix based on the citation vector matrix and the context vector to be tested, and then selects the citation corresponding to the matching degree of the head area as the recommended citation for the context to be tested based on the true matching degree of all citations, arranged from large to small. It should be noted here that during the use of the citation recommendation model, the citation vector matrix for matching degree calculation is constructed based on all citations in the preset citation library. When the citation recommendation model is trained, if each citation label includes all citation items in the preset citation library, the amount of calculation in the training process will be very large, resulting in too long a training time for the citation recommendation model. When the citation recommendation model is used, the citation matching degree calculation network only performs one calculation for an input context to be tested. Therefore, when the citation recommendation model is used, the matching degree of all citation items in the preset citation library can be calculated to ensure the accuracy of citation recommendation.
[0085] Based on the above embodiment, in this method, the citation vector description network combines the input citation label with the semantic information of each citation in the citation label and outputs the citation vector matrix using vector representation, which specifically includes:
[0086] The citation vector description network calculates any citation q in the input citation label through the following formula k Any word in The vector w′ i :
[0087]
[0088] in, Any of the words The set of semantic primitives, s j yes Zhongyiyuans j The vector representation of w i For any of the words The word vector, α is the preset weight coefficient, s j and w i These are all parameters to be optimized in the network structure during the training of the citation recommendation model;
[0089] The citation vector description network then calculates any citation q by the following formula k The citation vector t k :
[0090]
[0091]
[0092] Among them, PLM q is a citation representation model, [CLS] is a special sentence starter in the citation representation model, and m is a citation q k The number of words, Any citation q output by the citation representation model k Words in The hidden state of Any citation q output by the citation representation model k Words in The hidden state of Any citation q output by the citation representation model k Words in The hidden state of is the hidden state of the special sentence starter [CLS] output by the citation representation model, and is used during the use of the citation representation model The corresponding word vectors w′1,…,w′ i ,…,w′ m Participate in the calculation of hidden states;
[0093] The citation vector description network finally calculates the citation vector matrix Q by the following formula * And output:
[0094]
[0095] Among them, k * =1,2,…,n * , n * -1 is the preset training number.
[0096] Specifically, considering that quotations often have fewer words but rich meanings, general sentence representation learning methods cannot learn good vector representations for short quotations. Therefore, we incorporate the semantic meaning of each word in the quotations into the representation learning of the quotations. The semantic meaning is the smallest semantic unit in linguistics. The semantic meaning of a word carries the core semantic information of the word. We specifically achieve the fusion of semantic meaning by adding the semantic meaning vector to the word vector. Suppose any quotation q in the quotations k Any word in The word vector is w i , after integrating the semantic information, it becomes:
[0097]
[0098] in, Any of the words The set of semantic primitives, s j yes Zhongyiyuans j The vector representation of w i For any of the words The word vector, α is the preset weight coefficient, s j and w i These are all parameters to be optimized in the network structure during the training of the citation recommendation model;
[0099] Then, we use the pre-trained model to get the citation vector representation:
[0100] Here we use a pre-trained model (such as BERT) to learn the vector representation of citations. Therefore, the word vector w in the above formula i The word vectors of each word in the citation in the pre-trained model’s own word vector library are used during the initialization of training, but are continuously optimized and learned as parameters to be adjusted during the subsequent training process. Similarly, the semantic vector s j After random initialization, it is also used as a parameter to be adjusted and continuously optimized and learned during the model training process. Specifically, citation q = w1…w m Passed to the pre-trained model PLM for citation representation learning q After that, we get a series of hidden states corresponding to each word (character):
[0101]
[0102] Among them, [CLS] represents the special sentence start symbol in the pre-trained model, and its corresponding hidden state The vector used to represent citations is
[0103] The vector representation of all citations in the current input citation label constitutes the citation vector matrix Among them, k * =1,2,…,n * , n * -1 is the preset training number.
[0104] Based on the above embodiment, in the method, the context vector description network uses a vector to represent the input corresponding sample context and outputs a context vector, which specifically includes:
[0105] The context vector description network uses a vector to represent the corresponding sample context of the input and outputs the context vector, specifically including:
[0106] The context vector describes the learning representation sequence c′ of the input corresponding sample context c through the following formula:
[0107] c′={c l ,[MASK],c r}
[0108] Among them, c l is the character sequence in the context of the corresponding sample, c r is the character sequence of the following text in the corresponding sample context, and [MASK] is a mark character;
[0109] The context vector description network then calculates the context vector u of the corresponding sample context c through the following formula:
[0110]
[0111]
[0112] Among them, PLM c is the context representation model, [CLS] is the context representation model
[0113] Special sentence starters, is the hidden state of the tag character [MASK] output by the context representation model, is the hidden state of the special sentence start symbol [CLS] output by the context representation model, using the c pre-stored by the context representation model l and c r The corresponding word vector sequences participate in the calculation of the hidden state;
[0114] Correspondingly, the citation matching degree calculation network determines the pseudo matching degree of the correct citation tag in the citation tags based on the citation vector matrix and the context vector, specifically including:
[0115] The pseudo-matching degree p of the correct citation tag q in the citation tag is calculated by the following formula: * :
[0116]
[0117] in, is the set of negative citations of the citation label except the correct citation label q, u is the context vector of the corresponding sample context c, t is the citation vector of the correct citation label q, and t′ is the citation vector of any negative citation q′ in the set of other negative citations;
[0118] The pseudo-matching degree is used to construct the loss function when training the citation recommendation model, which specifically includes:
[0119] The loss function is constructed by the following formula
[0120]
[0121] Among them, p* is the pseudo matching degree.
[0122] Specifically, the context c of a citation consists of the previous c l and c below r The two parts are composed of two parts. Since these two parts are not naturally connected, directly splicing them together for sentence representation learning will affect the effect of representation learning. Therefore, an additional [MASK] character is inserted between the upper and lower texts to indicate that a part of the content (i.e., the quotation) is missing in the middle. The constructed context representation learning sequence is c′={c l ,[MASK],c r}.
[0123] Then, the pre-trained model is used to obtain the context vector representation:
[0124] Here, a pre-trained model is also used to model the constructed context representation learning sequence, and then obtain the vector representation of the context. Specifically, c′ is passed to the pre-trained model PLM for context representation learning c After that, we get a series of hidden states corresponding to each word (character):
[0125]
[0126]
[0127] Here we use the hidden state corresponding to [MASK] To represent the context vector,
[0128] Furthermore, the pseudo matching degree p of the correct citation tag q in the citation tag is calculated by the following formula: * :
[0129]
[0130] in, is the set of negative citations other than the correct citation label q, u is the context vector of the corresponding sample context c, t is the citation vector of the correct citation label q, and t′ is the citation vector of any negative citation q′ in the set of other negative citations; it should be noted that, The citation vector set corresponding to the negative example set and the correct citation vector corresponding to the correct citation label q are combined to construct the citation label matrix, which is the citation vector matrix Q mentioned above. * .
[0131] The pseudo-matching degree is used to construct the loss function when training the citation recommendation model, which specifically includes:
[0132] The loss function is constructed by the following formula
[0133]
[0134] Among them, p * is the pseudo matching degree.
[0135] In the process of using the citation recommendation model, the context vector is calculated by using the context vector description network for the context of the input citation to be recommended, and then the matching degree between the context vector and the citation vector of each citation in the preset citation library is calculated, that is:
[0136] After obtaining the context vector and the citation vector representation of each citation, the matching degree of each citation is obtained by calculating the dot product and sofmax normalization:
[0137] p=softmax(Q T u)
[0138] Among them, p is the normalized vector of all citation matching degrees, and its i-th element is the matching degree of the i-th citation. Then all citations are sorted from high to low according to their matching degrees, and the citations with higher matching degrees are selected as the output results. It should be noted here that the higher matching degree can be directly selecting a recommended citation with the highest matching degree for output, or it can be that under a given matching degree threshold, all citations with matching degrees exceeding the matching degree threshold are output as recommended citations.
[0139] It should be noted here that in the process of using the recommended citation model, the citation vector description network no longer plays any role. Instead, each citation in the preset citation library is equipped with the citation vector learned after the recommended citation model is trained. That is, the preset citation library stores citations and corresponding citation vectors at the same time. In the process of using the recommended citation model, it is only necessary to calculate the vector description of the input context, and then calculate the matching degree of the current real-time calculated context vector with all citation vectors in the preset citation library, and output the recommended citation according to the ranking of the matching degree.
[0140] Based on the above embodiment, in this method, when the citation recommendation model is trained, after the citation vector description network, the context vector description network and the citation matching degree calculation network are trained together, the context vector description network and the citation matching degree calculation network continue to be trained separately until the training of the citation recommendation model is completed.
[0141] Specifically, the context vector description network models a large number of diverse contexts, and more training is needed to obtain better context representation. Therefore, after the citation vector description network and the context vector description network are trained together, the context vector description network is trained separately. In its separate training stage, the citation vector description network is fixed and not trained, that is, the parameters to be adjusted in the citation vector description network have been optimized and will not change. The loss function of context training is the cross entropy loss function based on the normalized citation matching vector p.
[0142] The following is a description of a context-based citation recommendation device provided by the present invention. The context-based citation recommendation device described below and the context-based citation recommendation method described above can be referred to each other.
[0143] Figure 2 A schematic diagram of the structure of the context-based citation recommendation device provided by the present invention, such as Figure 2 As shown, the device includes a determination unit 210 and a recommendation unit 220, wherein:
[0144] The determining unit 210 is used to determine the context of the citation to be recommended;
[0145] The recommendation unit 220 is used to input the context into a citation recommendation model and output a corresponding recommended citation;
[0146] The citation recommendation model is trained based on sample context and corresponding citation tags, and the network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network and a citation matching degree calculation network connected in sequence.
[0147] The device provided by the present invention determines the context of the citation to be recommended; inputs the context into the citation recommendation model, and outputs the corresponding recommended citation; wherein the citation recommendation model is obtained by training based on the sample context and the corresponding citation label, and the network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network, and a citation matching degree calculation network connected in sequence. The pre-trained citation recommendation model is used to process the context of the citation to be recommended, and the recommended citation matching with the context is found and output, wherein the citation recommendation model is obtained by training based on a large number of sample contexts and corresponding citation labels, so as to ensure the accuracy of the citation recommendation model, and the network structure during the citation recommendation model training process includes a citation vector description network, a context vector description network, and a citation matching degree calculation network connected in sequence, and the basic algorithm framework for citation recommendation in the citation recommendation model is given. Therefore, the device provided by the present invention realizes more efficient and accurate recommendation of citations matching with the context under a given context.
[0148] Based on the above embodiment, in the device, the construction of the sample context and the corresponding citation label specifically includes:
[0149] Determine possible recommended citation tags based on a preset citation library;
[0150] For any possible recommended citation tag, if it also exists in the preset corpus, then,
[0151] Determining any of the possible recommended citation tags as a correct citation tag, and determining a context corresponding to the correct citation tag in the preset corpus as a sample context;
[0152] During the training process of the citation recommendation model, for the sample context used in the current round of iterative calculation, a preset number of negative citation labels are randomly selected from other citation sets in the preset citation library that exclude the correct citation labels corresponding to the sample context, and the citation labels corresponding to the sample context used in the current round of iterative calculation are constructed by combining the preset number of negative citation labels and the correct citation labels.
[0153] Based on the above embodiment, in the device, the citation vector description network is used to combine the input citation with the semantic information of the citation and adopt a vector representation.
[0154] Based on the above embodiment, in the device, the network structure of the citation recommendation model training includes a citation vector description network, a context vector description network and a citation matching degree calculation network connected in sequence, specifically including:
[0155] During the training process of the citation recommendation model,
[0156] The citation vector description network combines the input citation label with the semantic information of each citation in the citation label and uses vector representation to output the citation vector matrix to the citation matching degree calculation network;
[0157] The context vector description network uses a vector to represent the input corresponding sample context and outputs a context vector to the citation matching degree calculation network;
[0158] The citation matching degree calculation network determines the pseudo matching degree of the correct citation tag in the citation tags based on the citation vector matrix and the context vector;
[0159] The pseudo matching degree is used to construct a loss function during training of the citation recommendation model.
[0160] Based on the above embodiment, in the device, the citation vector description network combines the input citation label with the semantic information of each citation in the citation label and outputs the citation vector matrix using vector representation, which specifically includes:
[0161] The citation vector description network calculates any citation q in the input citation label through the following formula k Any word in The vector w′ i :
[0162]
[0163] in, Any of the words The set of semantic primitives, s j yes Zhongyiyuans j The vector representation of w i For any of the words The word vector, α is the preset weight coefficient, s j and w i These are all parameters to be optimized in the network structure during the training of the citation recommendation model;
[0164] The citation vector description network then calculates any citation q by the following formula k The citation vector t k :
[0165]
[0166]
[0167] Among them, PLM q is a citation representation model, [CLS] is a special sentence starter in the citation representation model, and m is a citation q k The number of words, Any citation q output by the citation representation model k Words in The hidden state of Any citation q output by the citation representation model k Words in The hidden state of Any citation q output by the citation representation model k Words in The hidden state of is the hidden state of the special sentence starter [CLS] output by the citation representation model, and is used during the use of the citation representation model The corresponding word vectors w′1,…,w′ i ,…,w′ m Participate in the calculation of hidden states;
[0168] The citation vector description network finally calculates the citation vector matrix Q by the following formula* And output:
[0169]
[0170] Among them, k * =1,2,…,n * , n * -1 is the preset training number.
[0171] Based on the above embodiment, in the device, the context vector description network outputs a context vector by using a vector to represent the input corresponding sample context, specifically including:
[0172] The context vector describes the learning representation sequence c′ of the input corresponding sample context c through the following formula:
[0173] c′={c l ,[MASK],c r}
[0174] Among them, c l is the character sequence in the context of the corresponding sample, c r is the character sequence of the following text in the corresponding sample context, and [MASK] is a mark character;
[0175] The context vector description network then calculates the context vector u of the corresponding sample context c through the following formula:
[0176]
[0177]
[0178] Among them, PLM c is the context representation model, [CLS] is the context representation model
[0179] Special sentence starters, is the hidden state of the tag character [MASK] output by the context representation model, is the hidden state of the special sentence start symbol [CLS] output by the context representation model, using the c pre-stored by the context representation model l and c r The corresponding word vector sequences participate in the calculation of the hidden state;
[0180] Correspondingly, the citation matching degree calculation network determines the pseudo matching degree of the correct citation tag in the citation tags based on the citation vector matrix and the context vector, specifically including:
[0181] The pseudo-matching degree p of the correct citation tag q in the citation tag is calculated by the following formula: * :
[0182]
[0183] in, is the set of negative citations of the citation label except the correct citation label q, u is the context vector of the corresponding sample context c, t is the citation vector of the correct citation label q, and t′ is the citation vector of any negative citation q′ in the set of other negative citations;
[0184] The pseudo-matching degree is used to construct the loss function when training the citation recommendation model, which specifically includes:
[0185] The loss function is constructed by the following formula
[0186]
[0187] Among them, p * is the pseudo matching degree.
[0188] Based on the above embodiment, in the device, when the citation recommendation model is trained, after the citation vector description network, the context vector description network and the citation matching degree calculation network are trained together, the context vector description network and the citation matching degree calculation network continue to be trained separately until the training of the citation recommendation model is completed.
[0189] Figure 3 A schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute a context-based citation recommendation method, the method comprising: determining the context of the citation to be recommended; inputting the context into a citation recommendation model, and outputting the corresponding recommended citation; wherein the citation recommendation model is obtained by training based on the sample context and the corresponding citation label, and the network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network and a citation matching degree calculation network connected in sequence.
[0190] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0191] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the context-based citation recommendation method provided by the above-mentioned methods, and the method includes: determining the context of the citation to be recommended; inputting the context into a citation recommendation model, and outputting the corresponding recommended citation; wherein the citation recommendation model is trained based on sample context and corresponding citation labels, and the network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network and a citation matching calculation network connected in sequence.
[0192] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the context-based citation recommendation method provided by the above-mentioned methods, the method comprising: determining the context of the citation to be recommended; inputting the context into a citation recommendation model, and outputting the corresponding recommended citation; wherein the citation recommendation model is trained based on the sample context and the corresponding citation label, and the network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network and a citation matching calculation network connected in sequence.
[0193] The server embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.
[0194] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A context-based citation recommendation method, characterized in that: include: Determine the context of the citation to be recommended; Inputting the context into a citation recommendation model and outputting corresponding recommended citations; The citation recommendation model is trained based on sample context and corresponding citation tags, and the network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network, and a citation matching degree calculation network connected in sequence; The network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network, and a citation matching degree calculation network connected in sequence, specifically including: During the training process of the citation recommendation model, The citation vector description network combines the input citation label with the semantic information of each citation in the citation label and uses vector representation to output the citation vector matrix to the citation matching degree calculation network; The context vector description network uses a vector to represent the input corresponding sample context and outputs a context vector to the citation matching degree calculation network; The citation matching degree calculation network determines the pseudo matching degree of the correct citation tag in the citation tags based on the citation vector matrix and the context vector; The pseudo-matching degree is used to construct a loss function when training the citation recommendation model; The citation vector description network combines the input citation label with the semantic information of each citation in the citation label and outputs the citation vector matrix using vector representation, which specifically includes: The citation vector description network calculates any citation q in the input citation label through the following formula k Any word in The vector w′ i : in, Any of the words The set of semantic primitives, s j yes Zhongyiyuans j The vector representation of w i For any of the words The word vector, α is the preset weight coefficient, s j and w i These are all parameters to be optimized in the network structure during the training of the citation recommendation model; The citation vector description network then calculates any citation q by the following formula k The citation vector t k : Among them, PLM q is a citation representation model, [CLS] is a special sentence starter in the citation representation model, and m is a citation q k The number of words, Any citation q output by the citation representation model k Words in The hidden state of Any citation q output by the citation representation model k Words in The hidden state of Any citation q output by the citation representation model k Words in The hidden state of is the hidden state of the special sentence starter [CLS] output by the citation representation model, and is used during the use of the citation representation model The corresponding word vectors w′1,…,w ′ i ,…,w′ m Participate in the calculation of hidden states; The citation vector description network finally calculates the citation vector matrix Q by the following formula * And output: Among them, k * =1,2,…,n * , n * -1 is the preset number of trainings; The context vector description network uses a vector to represent the corresponding sample context of the input and outputs the context vector, specifically including: The context vector describes the learning representation sequence c of the network corresponding to the input sample context c through the following formula ′ : c ′ ={c l ,[MASK],c r } Among them, c l is the character sequence in the context of the corresponding sample, c r is the character sequence of the following text in the corresponding sample context, and [MASK] is a mark character; The context vector description network then calculates the context vector u of the corresponding sample context c through the following formula: Among them, PLM c is a context representation model, [CLS] is a special sentence starter in the context representation model, is the hidden state of the tag character [MASK] output by the context representation model, The special output of the context representation model The hidden state of the sentence start symbol [CLS] is represented by the c pre-stored by the context representation model. l and c r The corresponding word vector sequences participate in the calculation of the hidden state; Correspondingly, the citation matching degree calculation network determines the pseudo matching degree of the correct citation tag in the citation tags based on the citation vector matrix and the context vector, specifically including: The pseudo-matching degree p of the correct citation tag q in the citation tag is calculated by the following formula: * : in, is the set of negative citations of the citation label except the correct citation label q, u is the context vector of the corresponding sample context c, t is the citation vector of the correct citation label q, and t′ is the citation vector of any negative citation q′ in the set of other negative citations; The pseudo-matching degree is used to construct the loss function when training the citation recommendation model, which specifically includes: The loss function is constructed by the following formula Among them, p * is the pseudo matching degree.
2. The context-based citation recommendation method according to claim 1, characterized in that: The construction of the sample context and the corresponding citation label specifically includes: Determine possible recommended citation tags based on a preset citation library; For any possible recommended citation tag, if it also exists in the preset corpus, then, Determining any of the possible recommended citation tags as a correct citation tag, and determining a context corresponding to the correct citation tag in the preset corpus as a sample context; During the training process of the citation recommendation model, for the sample context used in the current round of iterative calculation, a preset number of negative citation labels are randomly selected from other citation sets in the preset citation library that exclude the correct citation labels corresponding to the sample context, and the citation labels corresponding to the sample context used in the current round of iterative calculation are constructed by combining the preset number of negative citation labels and the correct citation labels.
3. The context-based citation recommendation method according to claim 2, characterized in that: The citation vector description network is used to combine the input citation with the semantic information of the citation and adopt a vector representation.
4. The context-based citation recommendation method according to any one of claims 1 to 3, characterized in that: When the citation recommendation model is trained, after the citation vector description network, the context vector description network and the citation matching degree calculation network are trained together, the context vector description network and the citation matching degree calculation network are trained separately until the training of the citation recommendation model is completed.
5. A context-based citation recommendation device, characterized in that: include: A determination unit, used for determining the context of the citation to be recommended; A recommendation unit, configured to input the context into a citation recommendation model and output a corresponding recommended citation; The citation recommendation model is trained based on sample context and corresponding citation tags, and the network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network, and a citation matching degree calculation network connected in sequence; The network structure of the citation recommendation model during training includes a citation vector description network, a context vector description network, and a citation matching degree calculation network connected in sequence, specifically including: During the training process of the citation recommendation model, The citation vector description network combines the input citation label with the semantic information of each citation in the citation label and uses vector representation to output the citation vector matrix to the citation matching degree calculation network; The context vector description network uses a vector to represent the input corresponding sample context and outputs a context vector to the citation matching degree calculation network; The citation matching degree calculation network determines the pseudo matching degree of the correct citation tag in the citation tags based on the citation vector matrix and the context vector; The pseudo-matching degree is used to construct a loss function when training the citation recommendation model; The citation vector description network combines the input citation label with the semantic information of each citation in the citation label and outputs the citation vector matrix using vector representation, which specifically includes: The citation vector description network calculates any citation q in the input citation label through the following formula k Any word in The vector w′ i : in, Any of the words The set of semantic primitives, s j yes Zhongyiyuans j The vector representation of w i For any of the words The word vector, α is the preset weight coefficient, s j and w i These are all parameters to be optimized in the network structure during the training of the citation recommendation model; The citation vector description network then calculates any citation q by the following formula k The citation vector t k : Among them, PLM q is a citation representation model, [CLS] is a special sentence starter in the citation representation model, and m is a citation q k The number of words, Any citation q output by the citation representation model k Words in The hidden state of Any citation q output by the citation representation model k Words in The hidden state of Any citation q output by the citation representation model k Words in The hidden state of is the hidden state of the special sentence starter [CLS] output by the citation representation model, and is used during the use of the citation representation model The corresponding word vectors w′1,…,w ′ i ,…,w′ m Participate in the calculation of hidden states; The citation vector description network finally calculates the citation vector matrix Q by the following formula * And output: Among them, k * =1,2,…,n * , n * -1 is the preset number of trainings; The context vector description network uses a vector to represent the corresponding sample context of the input and outputs the context vector, specifically including: The context vector describes the learning representation sequence c of the network corresponding to the input sample context c through the following formula ′ : c ′ ={c l ,[MASK],c r } Among them, c l is the character sequence in the context of the corresponding sample, c r is the character sequence of the following text in the corresponding sample context, and [MASK] is a mark character; The context vector description network then calculates the context vector u of the corresponding sample context c through the following formula: Among them, PLM c is a context representation model, [CLS] is a special sentence starter in the context representation model, is the hidden state of the tag character [MASK] output by the context representation model, The special output of the context representation model The hidden state of the sentence start symbol [CLS] is represented by the c pre-stored by the context representation model. l and c r The corresponding word vector sequences participate in the calculation of the hidden state; Correspondingly, the citation matching degree calculation network determines the pseudo matching degree of the correct citation tag in the citation tags based on the citation vector matrix and the context vector, specifically including: The pseudo-matching degree p of the correct citation tag q in the citation tag is calculated by the following formula: * : in, is the set of negative citations of the citation label except the correct citation label q, u is the context vector of the corresponding sample context c, t is the citation vector of the correct citation label q, and t′ is the citation vector of any negative citation q′ in the set of other negative citations; The pseudo-matching degree is used to construct the loss function when training the citation recommendation model, which specifically includes: The loss function is constructed by the following formula Among them, p * is the pseudo matching degree.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the context-based citation recommendation method according to any one of claims 1 to 4 are implemented.
7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the context-based citation recommendation method according to any one of claims 1 to 4 are implemented.
