Word recommendation model training method and device

Abstract

The present invention discloses a word recommendation model training method and device. By establishing a text data set and determining the entity words of each text data, and then determining the first text vector and the first graph vector corresponding to each entity word, the loss function corresponding to the word recommendation model is determined according to the first text vector and the first graph vector, thereby improving the versatility of the word recommendation model in different scenarios and improving the efficiency of word recommendation model training.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a word recommendation model training method and device. Background Art

[0002] Existing word recommendation models include those based on text similarity, those based on dictionaries and contextual information, and those based on knowledge graphs. However, each of these different word recommendation models has its own limitations. For example, methods based on text similarity are complex, those based on dictionaries and contextual information cannot recommend isolated words without context, and those based on knowledge graphs are complex to apply. Furthermore, there are obstacles when combining multiple methods, resulting in poor versatility in word recommendation across different scenarios. Therefore, improving the versatility of word recommendation models across different scenarios is crucial. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a word recommendation model training method and device, by establishing a text data set and determining the entity words of each text data, and then determining the first text vector and first graph vector corresponding to each entity word, and determining the loss function corresponding to the word recommendation model based on the first text vector and the first graph vector, thereby improving the versatility of the word recommendation model in different scenarios and improving the efficiency of word recommendation model training.

[0004] In order to solve the above technical problems, the first aspect of the present invention discloses a word recommendation model training method, comprising:

[0005] Establishing a text data set and determining entity words corresponding to each text data in the text data set;

[0006] Determining, based on the entity words, a first text vector corresponding to each of the entity words; and determining, based on the entity words, a knowledge graph structure;

[0007] Determining a word vector corresponding to each of the entity words in a pre-trained word vector library, and determining a first graph vector corresponding to each of the entity words through a graph structure encoder based on the word vector;

[0008] The graph structure encoder is used to indicate the knowledge graph structure; the pre-trained word vector library is used to indicate the word vector corresponding to the preset text data;

[0009] A loss function of the word recommendation model is determined based on the first text vector and the first graph vector; if the loss function converges, it is determined that the training of the text processing model is completed.

[0010] As an optional implementation manner, determining the entity words corresponding to each text data in the text dataset includes:

[0011] Segmenting each of the text data using a text encoding network to obtain a word sequence corresponding to each of the text data;

[0012] According to the word segmentation sequence, a corresponding digital sequence is obtained through a preset word table;

[0013] Obtaining, according to the digital sequence, a second text vector corresponding to each text data in the text data set;

[0014] Acquiring context information corresponding to each of the text data;

[0015] An entity word corresponding to each of the text data is determined according to the context information and the second text vector.

[0016] As an optional implementation manner, after obtaining the context information corresponding to each of the text data, the method further includes:

[0017] Determining a global optimal sequence of the second text vectors based on the context information;

[0018] replacing the second text vector according to the global optimal sequence, and obtaining a replaced second text vector;

[0019] Determining the entity word corresponding to each of the text data according to the context information and the second text vector includes:

[0020] According to the replaced second text vector, the entity word corresponding to each of the text data is determined.

[0021] As an optional implementation, the knowledge graph structure includes at least one triple relationship; and determining the knowledge graph structure based on the entity word includes:

[0022] Determining the category to which each entity word belongs;

[0023] Determining entity relationships between the entity words according to the categories of the entity words;

[0024] According to the first text vector corresponding to each of the entity words and the entity relationship, a triple relationship corresponding to each of the entity words is determined.

[0025] As an optional implementation, determining, based on the word vector, a first graph vector corresponding to each of the entity words through a graph structure encoder includes:

[0026] Determine a first attention parameter of the node corresponding to each entity word, and a second attention parameter between the node corresponding to each entity word and a neighboring node within a preset number of hops in the knowledge graph structure;

[0027] Calculate the graph attention parameter corresponding to each entity word according to the word vector corresponding to each entity word, the word vector corresponding to the neighbor node, the first attention parameter, and the second attention parameter;

[0028] According to the graph attention parameter, a first graph vector corresponding to each of the entity words is determined through a graph structure encoder.

[0029] As an optional implementation manner, after determining that the text processing model is trained, the method further includes:

[0030] Inputting a text to be recognized and determining whether the length of the text to be recognized is greater than a preset length;

[0031] If it is determined that the length of the text to be recognized is not greater than the preset length, obtaining a third text vector corresponding to the text to be recognized through the text encoding network;

[0032] If it is determined that the length of the text to be recognized is greater than a preset length, obtaining the entity words to be recognized in the text to be recognized through the text encoding network; and obtaining a third text vector corresponding to the text to be recognized based on the entity words to be recognized;

[0033] A first similarity between the third text vector and the first image vector is calculated, and all the entity words whose first similarity is greater than a first preset threshold are output as recommended words for the text to be recognized.

[0034] As an optional implementation manner, after calculating and obtaining the first similarity between the third text vector and the first image vector, the method further includes:

[0035] determining a second image vector corresponding to the third text vector;

[0036] Calculating and obtaining a second similarity between the second graph vector and the first graph vector corresponding to each of the entity words;

[0037] Calculating and obtaining a third similarity between the third text vector and the first text vector corresponding to each entity word;

[0038] Calculating and obtaining a fourth similarity between the second graph vector and the first text vector corresponding to each of the entity words;

[0039] According to the first similarity, and / or the second similarity, and / or the third similarity, and / or the fourth similarity, semantic similarity is obtained, and all the entity words whose semantic similarity is greater than a second preset threshold are output as recommended words for the text to be recognized.

[0040] In a second aspect, the present application provides a word recommendation model training device, the device comprising:

[0041] A data set establishment module, used to establish a text data set and determine the entity words corresponding to each text data in the text data set;

[0042] A text vector determination module, configured to determine, based on the entity words, a first text vector corresponding to each of the entity words;

[0043] A knowledge graph determination module, configured to determine a knowledge graph structure based on the entity words;

[0044] A graph vector determination module is configured to determine a word vector corresponding to each of the entity words in a pre-trained word vector library, and determine a first graph vector corresponding to each of the entity words based on the word vector through a graph structure encoder;

[0045] The graph structure encoder is used to indicate the knowledge graph structure; the pre-trained word vector library is used to indicate the word vector corresponding to the preset text data;

[0046] A loss function determination module is used to determine the loss function of the word recommendation model based on the first text vector and the first graph vector; if the loss function converges, it is determined that the training of the text processing model is completed.

[0047] As an optional implementation manner, the data set establishment module determines a specific manner of an entity word corresponding to each text data in the text data set, including:

[0048] Segmenting each of the text data using a text encoding network to obtain a word sequence corresponding to each of the text data;

[0049] According to the word segmentation sequence, a corresponding digital sequence is obtained through a preset word table;

[0050] Obtaining, according to the digital sequence, a second text vector corresponding to each text data in the text data set;

[0051] Acquiring context information corresponding to each of the text data;

[0052] An entity word corresponding to each of the text data is determined according to the context information and the second text vector.

[0053] As an optional implementation manner, the text vector determination module is further configured to, after the data set establishment module obtains the context information corresponding to each of the text data,

[0054] Determining a global optimal sequence of the second text vectors based on the context information;

[0055] replacing the second text vector according to the global optimal sequence, and obtaining a replaced second text vector;

[0056] The data set establishment module determines a specific method of an entity word corresponding to each of the text data according to the context information and the second text vector, including:

[0057] According to the replaced second text vector, the entity word corresponding to each of the text data is determined.

[0058] As an optional implementation, the knowledge graph structure includes at least one triple relationship; the knowledge graph determination module determines the specific manner of the knowledge graph structure based on the entity word, including:

[0059] Determining the category to which each entity word belongs;

[0060] Determining entity relationships between the entity words according to the categories of the entity words;

[0061] According to the first text vector corresponding to each of the entity words and the entity relationship, a triple relationship corresponding to each of the entity words is determined.

[0062] As an optional implementation, the graph vector determination module determines the first graph vector corresponding to each entity word according to the word vector through a graph structure encoder, including:

[0063] Determine a first attention parameter of the node corresponding to each entity word, and a second attention parameter between the node corresponding to each entity word and a neighboring node within a preset number of hops in the knowledge graph structure;

[0064] Calculate the graph attention parameter corresponding to each entity word according to the word vector corresponding to each entity word, the word vector corresponding to the neighbor node, the first attention parameter, and the second attention parameter;

[0065] According to the graph attention parameter, a first graph vector corresponding to each of the entity words is determined through a graph structure encoder.

[0066] As an optional embodiment, the device further includes a first word recommendation module, which is used to, after the loss function determination module determines that the text processing model training is completed,

[0067] Inputting a text to be recognized and determining whether the length of the text to be recognized is greater than a preset length;

[0068] If it is determined that the length of the text to be recognized is not greater than the preset length, obtaining a third text vector corresponding to the text to be recognized through the text encoding network;

[0069] If it is determined that the length of the text to be recognized is greater than a preset length, obtaining the entity words to be recognized in the text to be recognized through the text encoding network; and obtaining a third text vector corresponding to the text to be recognized based on the entity words to be recognized;

[0070] A first similarity between the third text vector and the first image vector is calculated, and all the entity words whose first similarity is greater than a first preset threshold are output as recommended words for the text to be recognized.

[0071] As an optional embodiment, the device further includes a second word recommendation module, which is configured to, after the first word recommendation module calculates and obtains the first similarity between the third text vector and the first image vector,

[0072] determining a second image vector corresponding to the third text vector;

[0073] Calculating and obtaining a second similarity between the second graph vector and the first graph vector corresponding to each of the entity words;

[0074] Calculating and obtaining a third similarity between the third text vector and the first text vector corresponding to each entity word;

[0075] Calculating and obtaining a fourth similarity between the second graph vector and the first text vector corresponding to each of the entity words;

[0076] According to the first similarity, and / or the second similarity, and / or the third similarity, and / or the fourth similarity, semantic similarity is obtained, and all the entity words whose semantic similarity is greater than a second preset threshold are output as recommended words for the text to be recognized.

[0077] A third aspect of the present invention discloses another word recommendation model training device, comprising:

[0078] a memory storing executable program code;

[0079] a processor coupled to the memory;

[0080] The processor calls the executable program code stored in the memory to execute the word recommendation model training method disclosed in the first aspect of the present invention.

[0081] A fourth aspect of the present invention discloses a computer storage medium, wherein the computer storage medium stores computer instructions. When the computer instructions are called, they are used to execute the word recommendation model training method disclosed in the first aspect of the present invention.

[0082] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: by establishing a text data set and determining the entity words of each text data, and then determining the first text vector and the first graph vector corresponding to each entity word, and determining the loss function corresponding to the word recommendation model based on the first text vector and the first graph vector, the versatility of the word recommendation model in different scenarios is improved, and the efficiency of word recommendation model training is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0084] Figure 1 This is a schematic diagram of an application scenario of a word recommendation model training method disclosed in an embodiment of the present invention;

[0085] Figure 2 This is a flowchart of a word recommendation model training method disclosed in an embodiment of the present invention;

[0086] Figure 3 This is a flowchart of another word recommendation model training method disclosed in an embodiment of the present invention;

[0087] Figure 4 It is a structural diagram of a word recommendation model training device disclosed in an embodiment of the present invention;

[0088] Figure 5 1 is a structural diagram of another word recommendation model training device disclosed in an embodiment of the present invention;

[0089] Figure 6 This is a structural diagram of another word recommendation model training device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0090] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0091] The terms "first," "second," and so on, in the description and claims of the present invention and the accompanying drawings are used to distinguish between different items, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or end comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed therein, or may optionally include other steps or elements inherent to such process, method, product, or end.

[0092] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0093] Existing word recommendation models include those based on text similarity, those based on dictionaries and contextual information, and those based on knowledge graphs. However, each of these different word recommendation models has its own limitations. For example, methods based on text similarity are complex, those based on dictionaries and contextual information cannot recommend isolated words without context, and those based on knowledge graphs are complex to apply. Furthermore, there are obstacles when combining multiple methods, resulting in poor versatility in word recommendation across different scenarios. Therefore, improving the versatility of word recommendation models across different scenarios is crucial.

[0094] Specifically, common word recommendation methods calculate text similarity by first representing the text as a vector and then calculating the cosine similarity between the two vectors. This similarity determines whether the two texts are similar. This similarity score selects the most similar words, but the text of phrases and groups is short, and the vectors calculated using some of the newer large-scale models often contain many redundant features. Furthermore, the recommendation effect is poor for words that are not included in the vocabulary.

[0095] An English synonym recommendation method that integrates dictionary and contextual information. This word recommendation method requires contextual information about the word itself and the text in which it is located, so it cannot recommend isolated words that have been extracted.

[0096] For the word recommendation method based on the industry knowledge graph, the industry search keywords are searched according to the pre-established knowledge graph to obtain the corresponding search results; the search results are displayed on the display interface of the client. This method belongs to the direct query graph database. On the one hand, it requires editing complex query statement pairs. If a word that does not exist in the knowledge graph is input, relevant word recommendations cannot be made.

[0097] See also Figure 1 , Figure 1 This is a schematic diagram of an application scenario of a word recommendation model training method disclosed in an embodiment of the present invention. Figure 1 As shown in the figure, word recommendation can be achieved through the logical relationship between words, word semantics, word attributes and category relationship. Figure 1 The relationship between the indicated words can be used to construct a corresponding knowledge graph structure. In this application, by combining the text vector and the graph vector corresponding to the knowledge graph, a more accurate word recommendation model can be obtained.

[0098] The word recommendation model training method provided in this application implicitly adds the graph structure information of the knowledge graph to the word recommendation, instead of directly querying in the graph database, constructing the relationship between words and building a knowledge graph. In addition, the credibility of the word recommendation is increased through voting in a single and double tower structure.

[0099] The present invention discloses a word recommendation model training method and device. By establishing a text data set and determining the entity words of each text data, and then determining the first text vector and the first graph vector corresponding to each entity word, the loss function corresponding to the word recommendation model is determined according to the first text vector and the first graph vector, thereby improving the versatility of the word recommendation model in different scenarios and improving the efficiency of word recommendation model training.

[0100] Example 1

[0101] See also Figure 2 , Figure 2 This is a flow chart of a word recommendation model training method disclosed in an embodiment of the present invention. Figure 2 As shown, the word recommendation model training method may include the following operations:

[0102] S101, establishing a text data set, and determining entity words corresponding to each text data in the text data set;

[0103] Text datasets can come from public datasets, or they can collect text data from specific fields, such as advertising copy, product data, and promotional materials in the public e-commerce field.

[0104] After completing the acquisition of the text dataset, you can extract the product's brand, category, ingredients, functions, applicable scenarios and population, price, sales volume, sales revenue and other multi-dimensional product parameter information from each text data in the text dataset.

[0105] S102. Determine, based on the entity words, a first text vector corresponding to each of the entity words; and determine, based on the entity words, a knowledge graph structure;

[0106] After completing the entity word extraction, you can build a knowledge graph structure centered on a single product based on the preset category hierarchy or brand hierarchy. You can also build several triples of information through the output results of the entity word extraction model to form a knowledge graph.

[0107] A knowledge graph is a semantic network whose basic architecture consists of several triples [h:r:t], such as [lipstick:brand:ysl], which include a head entity h, a relation r, and a tail entity t. A knowledge graph can be constructed by treating entities as nodes and relations as edges. These triples are stored in a graph database to generate a knowledge graph.

[0108] S103, determining a word vector corresponding to each of the entity words in a pre-trained word vector library, and determining a first graph vector corresponding to each of the entity words based on the word vector through a graph structure encoder;

[0109] The graph structure encoder is used to indicate the knowledge graph structure; the pre-trained word vector library is used to indicate the word vector corresponding to the preset text data;

[0110] The training process of the graph structure encoder can be determined based on the attention values ​​between each entity itself and the nodes within a preset number of hops. For details, please refer to other implementation methods and will not be repeated here.

[0111] S104. Determine a loss function of the word recommendation model based on the first text vector and the first graph vector; if the loss function converges, determine that the training of the text processing model is completed.

[0112] As an optional implementation, the loss function can be a cross-entropy loss function. In the present application, the first text vector corresponding to the entity word in each text data and the first graph vector corresponding to each entity word can be extracted. According to the output results of different principles, the similarity between the first text vector and the first graph vector is calculated to determine the corresponding cross-entropy loss function, and this is used as a benchmark to determine whether the text processing model is trained.

[0113] The cross-entropy function may be a forward cross-entropy function based on the first text vector and the first image vector, or a reverse cross-entropy function, or a combination of bidirectional cross-entropy functions, without limitation herein. It should be noted that this cross-entropy function is used to indicate the difference between the first text vector and the first image vector, and therefore, there is no limitation on its specific form.

[0114] As an optional implementation manner, determining the entity words corresponding to each text data in the text dataset includes:

[0115] Segmenting each of the text data using a text encoding network to obtain a word sequence corresponding to each of the text data;

[0116] The text encoding network can be a neural network model that is a combination of a transformer series model + an rnn series model + a crf layer, which segments the text data into single-word levels to obtain corresponding word sequences.

[0117] According to the word segmentation sequence, a corresponding digital sequence is obtained through a preset word table;

[0118] Obtaining, according to the digital sequence, a second text vector corresponding to each text data in the text data set;

[0119] The text vector is the concatenation of the digital sequences corresponding to each word in the text data. Through the encoder of the multi-layer transformer structure in the model, the output vector after feature extraction, that is, the second text vector, can be output.

[0120] Acquiring context information corresponding to each of the text data;

[0121] For the extraction of contextual information, the contextual information and the global optimal sequence are obtained based on the second text vector obtained above through the RNN layer and CRF layer. Finally, a fully connected layer is connected to classify the vector obtained for each word to determine whether the corresponding text belongs to the aforementioned brand, category or product parameters, or is a non-entity word that does not belong to any category.

[0122] An entity word corresponding to each of the text data is determined according to the context information and the second text vector.

[0123] In a practical application scenario, the brand's starting character label might be B-brand, followed by I-brand until the end. Each category's starting character label is B-cate, followed by I-cate until the end. If an entity word consists of only one character, this character label is B-brand; non-entity character labels are labeled O. The final output label is a predicted label for each character in the text, for example, [O, B-brand, I-brand, B-cate, I-cate]. Based on the classification, the entity words can be deduced backwards.

[0124] According to the output of the entity words and the true label, the loss function can be calculated in the form of a cross-entropy function. The training is iterated until the text encoding network converges. Finally, the entity words corresponding to each text data are determined based on the output of the trained text encoding network.

[0125] The text encoding network is used to obtain a text sequence corresponding to each text data, and further obtains a corresponding digital sequence, thereby obtaining a second text vector corresponding to each text data. Through the second text vector and the context information corresponding to each text data, the entity word corresponding to each text data can be determined, thereby improving the efficiency of word recommendation model training.

[0126] As an optional implementation manner, after obtaining the context information corresponding to each of the text data, the method further includes:

[0127] Determining a global optimal sequence of the second text vectors based on the context information;

[0128] replacing the second text vector according to the global optimal sequence, and obtaining a replaced second text vector;

[0129] After acquiring the context information, the second text vector may be updated in real time, and the second text vector may be replaced according to the obtained global optimal sequence.

[0130] Determining the entity word corresponding to each of the text data according to the context information and the second text vector includes:

[0131] According to the replaced second text vector, the entity word corresponding to each of the text data is determined.

[0132] The global optimal sequence corresponding to the second text vector is determined through context information, and the second text vector is replaced based on this to obtain the replaced second text vector. The corresponding entity word is determined by the replaced second text vector, thereby improving the efficiency and accuracy of word recommendation model training.

[0133] As an optional implementation, determining, based on the word vector, a first graph vector corresponding to each of the entity words through a graph structure encoder includes:

[0134] Determine a first attention parameter of the node corresponding to each entity word, and a second attention parameter between the node corresponding to each entity word and a neighboring node within a preset number of hops in the knowledge graph structure;

[0135] The preset number of hops can usually be 3. There is little relationship between neighboring nodes with a distance greater than 3 hops, while the semantics represented by neighboring nodes with a distance less than or equal to 3 hops and themselves may have a strong correlation. Therefore, the self-attention parameter of each node itself can be determined, and the other-attention parameter between the node and the neighboring nodes within the preset number of hops can be determined, that is, the aforementioned first attention parameter and second attention parameter.

[0136] Calculate the graph attention parameter corresponding to each entity word according to the word vector corresponding to each entity word, the word vector corresponding to the neighbor node, the first attention parameter, and the second attention parameter;

[0137] As an optional implementation, the calculation to obtain the graph attention parameter corresponding to each entity word includes:

[0138] Determine a first weight of each entity word itself, and determine a second weight of each neighbor node corresponding to each entity word;

[0139] Determining a first attention score according to a product of a word vector corresponding to each entity word, the first weight, and / or the first attention parameter;

[0140] Determining a second attention score according to a product of a word vector of each neighbor node corresponding to each entity word, the second weight, and / or the second attention parameter;

[0141] Calculating a graph attention parameter corresponding to each of the entity words according to the sum of the first attention score and the second attention score;

[0142] According to the graph attention parameter, a first graph vector corresponding to each of the entity words is determined through a graph structure encoder.

[0143] By calculating the entity word ontology and its attention parameters with each node within a preset number of hops, the graph attention parameters corresponding to the entity word are obtained, and then the corresponding first graph vector is obtained, which improves the efficiency and accuracy of word recommendation model training.

[0144] For a specific application scenario, the graph structure encoder can be trained using the following principles:

[0145] Using large amounts of pre-trained vector data publicly available online, each character is assigned an N-dimensional vector, with N typically ranging from several hundred to tens of thousands. The head and tail entities of a triple are separated into characters, and character vectors are extracted from the pre-trained vectors. The character vectors are summed and averaged to obtain a word vector, resulting in an initial graph containing k entities, each with a feature vector dimension of 768.

[0146] This application can be trained using the GAT network. The GAT network introduces a self-attention mechanism within the graph convolutional network. Each entity node can assign different attention scores to each neighbor node, identifying more important neighbors. The GAT network implements this by stacking graph attention layers. Each attention layer calculates the attention coefficient for each node-neighbor pair between two nodes. This step can be calculated in parallel and does not require knowledge of the entire knowledge graph structure; it only needs to determine the neighbor nodes of each node.

[0147] The initial representation vectors of all nodes are consistent. If nodes i and j are neighbors, the attention mechanism can be used to calculate the importance of i to j. The feature vectors of nodes i and j are concatenated and the inner product is calculated with a 2*768-dimensional vector a. a is randomly initialized and continuously updated in subsequent backpropagation.

[0148] The feature of node i is equal to the attention coefficient score of itself and all surrounding neighbor nodes multiplied by the representation vector of the corresponding neighbor node, then summed and normalized using softmax, an activation function is added to provide nonlinearity, and then passed to the next layer to repeat the same operation to update each node vector.

[0149] For an attention calculation process, GAT uses multi-head attention to stabilize the process of learning the relationship between nodes. It calculates the hidden state through k independent attention mechanisms, and then connects or averages each attention result to obtain the updated representation vector of node i.

[0150] According to the set number of GAT layers, the updated vector of each node will continue to enter the next layer, and then each layer will calculate k attention calculations for each node to update the node's representation vector again, and so on. By combining the features of the neighboring nodes into the current node, the feature extraction process of the entire GAT is finally completed, the fully connected layer of the GAT is removed, and the multi-layer GAT is used as an encoder. The traditional GAT network directly calculates the neighboring nodes directly associated with node i, while ignoring the nodes that are not directly connected. This application can increase the attention calculation of multi-hop nodes. Since the connection between physical nodes greater than three hops is already relatively weak, for the calculation of the attention mechanism, the attention feature of node i is equal to the attention coefficient score of itself and the neighboring nodes within 3 hops multiplied by the representation vector of the corresponding neighboring node, and then summed and normalized using softmax, adding an activation function to provide nonlinearity, and then repeating multiple layers of calculations to update each node vector to complete the training of the graph structure encoder.

[0151] In a specific training process, 30% of entities are randomly selected from the graph and replaced with the following: 80% of the time, the mask is replaced, 10% of the time, any other entity is replaced, and 10% of the time, the entity remains unchanged. The model is then trained to predict the value of the replaced node by learning from neighboring nodes within one to three hops of the replaced node. The vector output by the model at the masked location is compared with the actual node vector. The network parameters are adjusted using gradient descent using a cross-entropy loss function until convergence, completing network training.

[0152] The present embodiment provides a method for training a word recommendation model, the method comprising: establishing a text data set and determining entity words corresponding to each text data in the text data set; determining a first text vector corresponding to each entity word based on the entity words; and determining a knowledge graph structure based on the entity words; determining a word vector corresponding to each entity word in a pre-trained word vector library, and determining a first graph vector corresponding to each entity word through a graph structure encoder based on the word vector; wherein the graph structure encoder is used to indicate the knowledge graph structure; the pre-trained word vector library is used to indicate the word vector corresponding to the preset text data; determining a loss function of the word recommendation model based on the first text vector and the first graph vector; if the loss function converges, determining that the training of the text processing model is completed. By establishing a text data set and determining the entity words of each text data, and then determining the first text vector and the first graph vector corresponding to each entity word, and determining the loss function corresponding to the word recommendation model based on the first text vector and the first graph vector, the versatility of the word recommendation model in different scenarios is improved, and the efficiency of word recommendation model training is improved.

[0153] Example 2

[0154] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of another word recommendation model training method disclosed in an embodiment of the present invention. Figure 3 As shown, based on any other implementation manner, the knowledge graph structure includes at least one triple relationship, and the method includes:

[0155] S201, establishing a text data set, and determining entity words corresponding to each text data in the text data set;

[0156] S202: Determine, based on the entity words, a first text vector corresponding to each of the entity words;

[0157] S203, determining the category to which each entity word belongs;

[0158] S204, determining the entity relationship between the entity words according to the categories of the entity words;

[0159] S205 : Determine a triple relationship corresponding to each entity word according to the first text vector corresponding to each entity word and the entity relationship.

[0160] As described in Example 1, categories may include brands, categories or product parameters, etc. After determining the category to which the entity word belongs, the entity relationship between the entity words can be determined accordingly, and then based on the entity relationship and the first text vector of each entity word, at least one triple relationship can be determined. Based on each triple relationship, the corresponding knowledge graph structure can be obtained or directly generated.

[0161] S206. Determine a word vector corresponding to each entity word in a pre-trained word vector library, and determine a first graph vector corresponding to each entity word through a graph structure encoder based on the word vector;

[0162] The graph structure encoder is used to indicate the knowledge graph structure; the pre-trained word vector library is used to indicate the word vector corresponding to the preset text data;

[0163] S207. Determine a loss function of the word recommendation model based on the first text vector and the first graph vector; if the loss function converges, determine that the training of the text processing model is completed.

[0164] It should be noted that, for the relevant descriptions of S201, S202, S206 and S207, reference may be made to the relevant descriptions of S101-S104 in the first embodiment, which will not be repeated here.

[0165] As an optional implementation manner, after determining that the text processing model is trained, the method further includes:

[0166] Inputting a text to be recognized and determining whether the length of the text to be recognized is greater than a preset length;

[0167] If it is determined that the length of the text to be recognized is not greater than the preset length, obtaining a third text vector corresponding to the text to be recognized through the text encoding network;

[0168] If it is determined that the length of the text to be recognized is greater than a preset length, obtaining the entity words to be recognized in the text to be recognized through the text encoding network; and obtaining a third text vector corresponding to the text to be recognized based on the entity words to be recognized;

[0169] According to the length of the input text, it can be determined whether the input is a word or a sentence. If the input is a word, the corresponding third text vector is directly obtained. If the input is a sentence, word segmentation is required to obtain the entity words to be identified in the text, and then the corresponding third text vector is obtained. For example, in an actual application scenario, the user inputs any text into the entity recognition network. When the text length is less than 5 characters, the user input is considered to be a word. The input text is directly input into the text encoder to obtain the word vector A, and then input into the graph encoder to obtain the graph vector B. If the text length is greater than 5 characters, the user input is considered to be text, and entity recognition is required. The result after the crf layer is output, the entity words in the text and the corresponding word vector A are extracted, and the entity words are input into the graph encoder to obtain the graph vector B.

[0170] A first similarity between the third text vector and the first image vector is calculated, and all the entity words whose first similarity is greater than a first preset threshold are output as recommended words for the text to be recognized.

[0171] For the aforementioned application scenario, the cosine similarity between the input word vector A and the graph vectors of all entities can be calculated to return the top K words A with the greatest similarity. k can be freely set to 100 or 500, etc. In this way, the recommended words will comprehensively consider words that are semantically similar to the input word and have an indirect or direct relationship with the input word in the graph.

[0172] By inputting the text to be recognized, the corresponding third text vector is obtained. By calculating the similarity between the third text vector and the first graph vector, several entity words are obtained as corresponding recommended words, which improves the efficiency and accuracy of word recommendation and the versatility and practicality of the word recommendation model in different scenarios.

[0173] As an optional implementation manner, after calculating and obtaining the first similarity between the third text vector and the first image vector, the method further includes:

[0174] determining a second image vector corresponding to the third text vector;

[0175] Calculating and obtaining a second similarity between the second graph vector and the first graph vector corresponding to each of the entity words;

[0176] Calculating and obtaining a third similarity between the third text vector and the first text vector corresponding to each entity word;

[0177] Calculating and obtaining a fourth similarity between the second graph vector and the first text vector corresponding to each of the entity words;

[0178] According to the first similarity, and / or the second similarity, and / or the third similarity, and / or the fourth similarity, semantic similarity is obtained, and all the entity words whose semantic similarity is greater than a second preset threshold are output as recommended words for the text to be recognized.

[0179] The first to fourth similarities are used to represent the four similarities between the text vectors and graph vectors corresponding to the entity words in the input text and all the text vectors and graph vectors in the entity library. Based on the combination of one or more of these four similarities, the final semantic similarity can be determined. When applied, all the entity words with a semantic similarity greater than the second preset threshold can be output as recommended words for the text to be recognized.

[0180] By obtaining a second graph vector corresponding to a third text vector and calculating a second similarity, a third similarity, and a fourth similarity, semantic similarity is obtained based on the first similarity, and / or the second similarity, and / or the third similarity, and / or the fourth similarity, and a number of recommended words corresponding to the text to be recognized are obtained through the semantic similarity, thereby improving the efficiency and accuracy of word recommendation, and improving the versatility and practicality of the word recommendation model in different scenarios.

[0181] The word recommendation model training method provided in this embodiment uses regular expression matching to determine the data type, data encoding and data annotation of the metadata reflected in the metadata process document, and then determines the metadata quality and generates a metadata quality document, thereby improving the standardization level of word recommendation model training and thus improving the efficiency and accuracy of word recommendation model training.

[0182] Example 3

[0183] The embodiment of the present invention also provides a word recommendation model training device to implement the above method, please refer to Figure 4 , Figure 4 This is a structural diagram of a word recommendation model training device disclosed in an embodiment of the present invention. Figure 4 As shown, based on any other embodiment, the device includes:

[0184] A data set establishment module 31 is used to establish a text data set and determine entity words corresponding to each text data in the text data set;

[0185] A text vector determination module 32 is configured to determine, based on the entity words, a first text vector corresponding to each entity word;

[0186] A knowledge graph determination module 33 is used to determine a knowledge graph structure based on the entity words;

[0187] A graph vector determination module 34 is configured to determine a word vector corresponding to each entity word in a pre-trained word vector library, and determine a first graph vector corresponding to each entity word based on the word vector through a graph structure encoder;

[0188] The graph structure encoder is used to indicate the knowledge graph structure; the pre-trained word vector library is used to indicate the word vector corresponding to the preset text data;

[0189] The loss function determination module 35 is used to determine the loss function of the word recommendation model based on the first text vector and the first graph vector; if the loss function converges, it is determined that the training of the word recommendation model is completed.

[0190] By establishing a text dataset and determining the entity words of each text data, and then determining the first text vector and first graph vector corresponding to each entity word, the loss function corresponding to the word recommendation model is determined based on the first text vector and the first graph vector, which improves the versatility of the word recommendation model in different scenarios and the efficiency of word recommendation model training.

[0191] As an optional implementation, the data set establishment module 31 determines a specific method of an entity word corresponding to each text data in the text data set, including:

[0192] Segmenting each of the text data using a text encoding network to obtain a word sequence corresponding to each of the text data;

[0193] According to the word segmentation sequence, a corresponding digital sequence is obtained through a preset word table;

[0194] Obtaining, according to the digital sequence, a second text vector corresponding to each text data in the text data set;

[0195] Acquiring context information corresponding to each of the text data;

[0196] An entity word corresponding to each of the text data is determined according to the context information and the second text vector.

[0197] The text encoding network is used to obtain a text sequence corresponding to each text data, and further obtains a corresponding digital sequence, thereby obtaining a second text vector corresponding to each text data. Through the second text vector and the context information corresponding to each text data, the entity word corresponding to each text data can be determined, thereby improving the efficiency of word recommendation model training.

[0198] As an optional implementation, the text vector determination module 32 is further configured to, after the data set establishment module obtains the context information corresponding to each of the text data,

[0199] Determining a global optimal sequence of the second text vectors based on the context information;

[0200] replacing the second text vector according to the global optimal sequence, and obtaining a replaced second text vector;

[0201] The data set establishment module 31 determines the specific manner of the entity words corresponding to each of the text data according to the context information and the second text vector, including:

[0202] According to the replaced second text vector, the entity word corresponding to each of the text data is determined.

[0203] The global optimal sequence corresponding to the second text vector is determined through context information, and the second text vector is replaced based on this to obtain the replaced second text vector. The corresponding entity word is determined by the replaced second text vector, thereby improving the efficiency and accuracy of word recommendation model training.

[0204] As an optional implementation, the knowledge graph structure includes at least one triple relationship; the knowledge graph determination module 33 determines the specific manner of the knowledge graph structure according to the entity word, including:

[0205] Determining the category to which each entity word belongs;

[0206] Determining entity relationships between the entity words according to the categories of the entity words;

[0207] According to the first text vector corresponding to each of the entity words and the entity relationship, a triple relationship corresponding to each of the entity words is determined.

[0208] The entity relationships between entity words are determined by the categories to which each entity word belongs. Based on the first text vector corresponding to each entity word and the determined entity relationship, several triple relationships can be determined to form a knowledge graph, which improves the efficiency and accuracy of word recommendation model training.

[0209] As an optional implementation, the graph vector determination module 34 determines the first graph vector corresponding to each entity word according to the word vector through the graph structure encoder, including:

[0210] Determine a first attention parameter of the node corresponding to each entity word, and a second attention parameter between the node corresponding to each entity word and a neighboring node within a preset number of hops in the knowledge graph structure;

[0211] Calculate the graph attention parameter corresponding to each entity word according to the word vector corresponding to each entity word, the word vector corresponding to the neighbor node, the first attention parameter, and the second attention parameter;

[0212] According to the graph attention parameter, a first graph vector corresponding to each of the entity words is determined through a graph structure encoder.

[0213] By calculating the entity word ontology and its attention parameters with each node within a preset number of hops, the graph attention parameters corresponding to the entity word are obtained, and then the corresponding first graph vector is obtained, which improves the efficiency and accuracy of word recommendation model training.

[0214] See also Figure 5 , Figure 5 This is a structural diagram of another word recommendation model training device disclosed in an embodiment of the present invention. Figure 5 As shown, as an optional embodiment, the device further includes a first word recommendation module 36, which is used to, after the loss function determination module 35 determines that the word recommendation model training is completed,

[0215] Inputting a text to be recognized and determining whether the length of the text to be recognized is greater than a preset length;

[0216] If it is determined that the length of the text to be recognized is not greater than the preset length, obtaining a third text vector corresponding to the text to be recognized through the text encoding network;

[0217] If it is determined that the length of the text to be recognized is greater than a preset length, obtaining the entity words to be recognized in the text to be recognized through the text encoding network; and obtaining a third text vector corresponding to the text to be recognized based on the entity words to be recognized;

[0218] A first similarity between the third text vector and the first image vector is calculated, and all the entity words whose first similarity is greater than a first preset threshold are output as recommended words for the text to be recognized.

[0219] By inputting the text to be recognized, the corresponding third text vector is obtained. By calculating the similarity between the third text vector and the first graph vector, several entity words are obtained as corresponding recommended words, which improves the efficiency and accuracy of word recommendation and the versatility and practicality of the word recommendation model in different scenarios.

[0220] like Figure 5 As shown, as an optional embodiment, the device further includes a second word recommendation module 37, which is configured to, after the first word recommendation module 36 calculates and obtains the first similarity between the third text vector and the first image vector,

[0221] determining a second image vector corresponding to the third text vector;

[0222] Calculating and obtaining a second similarity between the second graph vector and the first graph vector corresponding to each of the entity words;

[0223] Calculating and obtaining a third similarity between the third text vector and the first text vector corresponding to each entity word;

[0224] Calculating and obtaining a fourth similarity between the second graph vector and the first text vector corresponding to each of the entity words;

[0225] According to the first similarity, and / or the second similarity, and / or the third similarity, and / or the fourth similarity, semantic similarity is obtained, and all the entity words whose semantic similarity is greater than a second preset threshold are output as recommended words for the text to be recognized.

[0226] By obtaining a second graph vector corresponding to a third text vector and calculating a second similarity, a third similarity, and a fourth similarity, semantic similarity is obtained based on the first similarity, and / or the second similarity, and / or the third similarity, and / or the fourth similarity, and a number of recommended words corresponding to the text to be recognized are obtained through the semantic similarity, thereby improving the efficiency and accuracy of word recommendation, and improving the versatility and practicality of the word recommendation model in different scenarios.

[0227] Example 4

[0228] See also Figure 6 , Figure 6 This is a structural diagram of another word recommendation model training device disclosed in an embodiment of the present invention. Figure 6 As shown, the word recommendation model training device may include:

[0229] The device includes a processor 291 and a memory 292 storing executable program code. It may also include a communication interface 293 and a bus 294. The processor 291, memory 292, and communication interface 293 can communicate with each other via bus 294. Communication interface 293 can be used for information transmission. The processor 291 is coupled to the memory 292 and can invoke the logic instructions (executable program code) in the memory 292 to execute the word recommendation model training method described in any of the above embodiments.

[0230] In addition, the logic instructions in the memory 292 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0231] Memory 292, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of the present application. Processor 291 executes the software programs, instructions, and modules stored in memory 292 to perform functional applications and data processing, thereby implementing the methods in the above-mentioned method embodiments.

[0232] Memory 292 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the use of the terminal device. Memory 292 may also include high-speed random access memory and non-volatile memory.

[0233] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called, they are used to implement the method described in any one of the embodiments.

[0234] An embodiment of the present invention also discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps in the word recommendation model training method described in any embodiment.

[0235] The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Those skilled in the art can understand and implement the present invention without inventive effort.

[0236] Through the detailed description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by means of hardware. Based on this understanding, the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0237] Finally, it should be noted that the word recommendation model training method and device disclosed in the embodiments of the present invention only disclose preferred embodiments of the present invention, which are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A word recommendation model training method, characterized in that: The method comprises: Establishing a text data set and determining entity words corresponding to each text data in the text data set; Determining, based on the entity words, a first text vector corresponding to each of the entity words; and determining, based on the entity words, a knowledge graph structure; Determining a word vector corresponding to each of the entity words in a pre-trained word vector library; Determine a first attention parameter of the node corresponding to each entity word, and a second attention parameter between the node corresponding to each entity word and a neighboring node within a preset number of hops in the knowledge graph structure; Calculate the graph attention parameter corresponding to each entity word according to the word vector corresponding to each entity word, the word vector corresponding to the neighbor node, the first attention parameter, and the second attention parameter; Determining, by a graph structure encoder according to the graph attention parameter, a first graph vector corresponding to each of the entity words; The graph structure encoder is used to indicate the knowledge graph structure; the pre-trained word vector library is used to indicate the word vector corresponding to the preset text data; A loss function of the word recommendation model is determined based on the first text vector and the first graph vector; if the loss function converges, it is determined that the training of the word recommendation model is completed.

2. The method according to claim 1, characterized in that The determining of the entity words corresponding to each text data in the text data set includes: Segmenting each of the text data using a text encoding network to obtain a word sequence corresponding to each of the text data; According to the word segmentation sequence, a corresponding digital sequence is obtained through a preset word table; Obtaining, according to the digital sequence, a second text vector corresponding to each text data in the text data set; Acquiring context information corresponding to each of the text data; An entity word corresponding to each of the text data is determined according to the context information and the second text vector.

3. The method according to claim 2, characterized in that After obtaining the context information corresponding to each of the text data, the method further includes: Determining a global optimal sequence of the second text vectors based on the context information; replacing the second text vector according to the global optimal sequence, and obtaining a replaced second text vector; Determining the entity word corresponding to each of the text data according to the context information and the second text vector includes: According to the replaced second text vector, the entity word corresponding to each of the text data is determined.

4. The method according to claim 1, wherein The knowledge graph structure includes at least one triple relationship; and determining the knowledge graph structure based on the entity word includes: Determining the category to which each entity word belongs; Determining entity relationships between the entity words according to the categories of the entity words; According to the first text vector corresponding to each of the entity words and the entity relationship, a triple relationship corresponding to each of the entity words is determined.

5. The method according to any one of claims 1 to 4, characterized in that After determining that the word recommendation model is trained, the method further includes: Inputting a text to be recognized and determining whether the length of the text to be recognized is greater than a preset length; If it is determined that the length of the text to be recognized is not greater than the preset length, obtaining a third text vector corresponding to the text to be recognized through the text encoding network; If it is determined that the length of the text to be recognized is greater than a preset length, obtaining the entity words to be recognized in the text to be recognized through the text encoding network; and obtaining a third text vector corresponding to the text to be recognized based on the entity words to be recognized; A first similarity between the third text vector and the first image vector is calculated, and all the entity words whose first similarity is greater than a first preset threshold are output as recommended words for the text to be recognized.

6. The method according to claim 5, characterized in that After calculating and obtaining the first similarity between the third text vector and the first image vector, the method further includes: determining a second image vector corresponding to the third text vector; Calculating and obtaining a second similarity between the second graph vector and the first graph vector corresponding to each of the entity words; Calculating and obtaining a third similarity between the third text vector and the first text vector corresponding to each entity word; Calculating and obtaining a fourth similarity between the second graph vector and the first text vector corresponding to each of the entity words; According to the first similarity, and / or the second similarity, and / or the third similarity, and / or the fourth similarity, semantic similarity is obtained, and all the entity words whose semantic similarity is greater than a second preset threshold are output as recommended words for the text to be recognized.

7. A word recommendation model training device, characterized in that: The device comprises: A data set establishment module, used to establish a text data set and determine the entity words corresponding to each text data in the text data set; A text vector determination module, configured to determine, based on the entity words, a first text vector corresponding to each of the entity words; A knowledge graph determination module, configured to determine a knowledge graph structure based on the entity words; A graph vector determination module is configured to determine a word vector corresponding to each entity word in a pre-trained word vector library; determine a first attention parameter of the node corresponding to each entity word, and a second attention parameter between the node corresponding to each entity word and a neighboring node with no more than a preset number of hops in the knowledge graph structure; calculate a graph attention parameter corresponding to each entity word based on the word vector corresponding to each entity word, the word vector corresponding to the neighboring node, the first attention parameter, and the second attention parameter; and determine a first graph vector corresponding to each entity word through a graph structure encoder based on the graph attention parameter; The graph structure encoder is used to indicate the knowledge graph structure; the pre-trained word vector library is used to indicate the word vector corresponding to the preset text data; A loss function determination module is used to determine the loss function of the word recommendation model based on the first text vector and the first graph vector; if the loss function converges, it is determined that the training of the word recommendation model is completed.

8. A word recommendation model training device, characterized in that: The device comprises: a memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the word recommendation model training method according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that The computer storage medium stores computer instructions, which, when called, are used to execute the word recommendation model training method according to any one of claims 1 to 6.

Images