Similar word search method, device, equipment and medium for text words
By calculating the representative characteristics and corpus expansion of core words in the target text, combined with dependency analysis, the problem of difficulty in identifying similar words in texts in small corpus fields is solved, and accurate search of low-frequency words and representativeness of similar words are achieved. promote.
Patent Information
- Application Number
- CN202011560879.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-25
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-12-25
AI Technical Summary
The prior art is difficult to accurately identify similar words of words from small corpus domain texts, especially those in domain texts with low frequency in texts, because they depend on the frequency of words appearing in texts and cannot be effectively identified. Similar words for low frequency words.
By calculating the representative characteristics of different vectors of each core word in the target text, obtain the user's search preset word vector, determine the core node words and candidate word vectors of the target text, use the similarity threshold to determine the similar word set, and combine it with corpus expansion. and dependency analysis, eliminate invalid words, calculate transfer probability and inverse text frequency index, and generate candidate word vectors.
The ability to accurately search for similar words in texts from small corpus fields simplifies workload, improves the accuracy of identifying similar words with low frequency in expert systems, and improves the representativeness of similar words.
Smart Images

Figure CN112559693B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of natural language processing technology, and in particular to a method, device, equipment and medium for searching similar words in text. Background Art
[0002] With the development of digital technology, more and more data is generated within enterprises, forming a large amount of text. These texts are stored in the enterprise's document management system and expert system, and contain rich knowledge. When using document management systems and expert systems, users usually need to search and understand the knowledge in the literature based on similar words of text words. Since these texts generally involve enterprise-specific management models, business processes, and operating procedures, and the number of texts related to a single type of knowledge is not large, it is impossible to use common natural language processing methods that require a large amount of text data for training to extract similar words of text words. Therefore, it becomes extremely important to search for similar words in the text of the expert system's small corpus.
[0003] At present, the search for similar words to text words mainly utilizes the similarity / correlation between the word vectors of each word, and combines the frequency of occurrence of each word in the text to calculate the key indicator value of each word, so as to determine the similar words of the words in the text.
[0004] The drawback of the above scheme is that it mainly relies on the frequency of occurrence of words in the text to determine text similar words, and it is difficult to identify similar words in the field text where the frequency of occurrence of each word in the text is low (such as expert system). Summary of the invention
[0005] The embodiments of the present application provide a method, apparatus, device and medium for searching similar words of text words, which can accurately search for similar words of text words from a small corpus domain text by calculating representative features of different vectors of core words in the target text.
[0006] In a first aspect, an embodiment of the present invention provides a method for searching similar words in a text, comprising:
[0007] Obtaining a user's preset search word vector; wherein the preset search word vector includes a first preset word vector and a second preset word vector;
[0008] Determine a core node word of a target text; and determine a candidate word vector of the core node word; wherein the candidate word vector includes a first vector and a second vector;
[0009] According to the similarity between the preset search word vector and the candidate word vector of the core node word, a similar word set of the preset search word is determined.
[0010] Optionally, before determining the core node words of the target text, the method further includes:
[0011] Obtain the original text of the core node word to be determined;
[0012] The original text is expanded with corpus to obtain a target text; wherein the corpus expansion is to perform a text translation operation on the original text.
[0013] Optionally, determine the core node words of the target text, including:
[0014] Obtaining each text sentence of the target text; and determining the dependency relationship between each word in the text sentence;
[0015] Constructing an initial node word chain of the target text according to the dependency relationship;
[0016] According to the initial node word chain and the preset stop word dictionary, the core node words of the target text are determined.
[0017] Optionally, determining the core node words of the target text according to the initial node word chain and a preset stop word dictionary includes:
[0018] If it is detected that there is a second node word in the initial node word chain, which has dependency relationships with the first node word and the third node word respectively, and the second node word is stored in a preset stop word dictionary, then the second node word is removed;
[0019] A dependency relationship between the first node word and the third node word is established to obtain an updated node word chain, and the first node word and the third node word are used as core node words of the target text.
[0020] Optionally, determining the candidate word vector of the core node word includes:
[0021] Obtaining the weight value of each edge in the update node word chain; and calculating the distance between the core node word and its adjacent core node words in the update node word chain, and the inverse text frequency index of the core node word;
[0022] Determining the transition probability between the core node word and its adjacent core node words according to the edge weight value, the distance, the transition parameter and the inverse text frequency index;
[0023] According to the transition probability, a candidate word vector of the core node word is determined.
[0024] Optionally, determining the candidate word vector of the core node word according to the transition probability includes:
[0025] The transition parameter in the transition probability is set to a first set of values to obtain a first set of node search sequences; the first set of node search sequences is trained using a word vector training model to obtain a first vector of the core node word;
[0026] The transfer parameter in the transfer probability is set to a second set of values to obtain a second set of node search sequences; the second set of node search sequences is trained using a word vector training model to obtain a second vector of the core node word.
[0027] Optionally, determining a similar word set of the preset search word according to the similarity between the preset search word vector and the candidate word vector of the core node word includes:
[0028] Calculating the similarity between the first vector and the first preset word vector; taking the core node words whose similarity is greater than or equal to the first similarity threshold as the first node word set of the search preset words;
[0029] Calculate the similarity between the second vector and the second preset word vector; and use the core node words whose similarity is greater than a second similarity threshold as the second node word set of the search preset words.
[0030] In a second aspect, an embodiment of the present invention provides a similar word search device for text words, comprising:
[0031] A vector acquisition module, used to acquire a user's search preset word vector; wherein the search preset word vector includes a first preset word vector and a second preset word vector;
[0032] A vector determination module is used to determine the core node words of the target text; and determine the candidate word vectors of the core node words; wherein the candidate word vectors include a first vector and a second vector;
[0033] The similar word set determination module is used to determine the similar word set of the search preset word according to the similarity between the search preset word vector and the candidate word vector of the core node word.
[0034] Optionally, also include:
[0035] An original text determination module is used to obtain the original text of the core node word to be determined;
[0036] The target text determination module is used to perform corpus expansion on the original text to obtain the target text; wherein the corpus expansion is to perform a text translation operation on the original text.
[0037] Optionally, the vector determination module includes a dependency determination unit, an initial node word chain construction unit, and a node word determination unit;
[0038] A dependency relationship determination unit, used to obtain each text sentence of the target text; and determine the dependency relationship between each word in the text sentence;
[0039] An initial node word chain construction unit, used to construct an initial node word chain of the target text according to the dependency relationship;
[0040] The node word determination unit is used to determine the core node words of the target text according to the initial node word chain and a preset stop word dictionary.
[0041] Optionally, a node word determination unit is specifically used to:
[0042] If it is detected that there is a second node word in the initial node word chain, which has dependency relationships with the first node word and the third node word respectively, and the second node word is stored in a preset stop word dictionary, then the second node word is removed;
[0043] A dependency relationship between the first node word and the third node word is established to obtain an updated node word chain, and the first node word and the third node word are used as core node words of the target text.
[0044] Optionally, the vector determination module includes a calculation unit, a transition probability determination unit and a vector determination unit;
[0045] A calculation unit, used to obtain the weight value of each edge in the update node word chain; and calculate the distance between the core node word and its adjacent core node words in the update node word chain, and the inverse text frequency index of the core node word;
[0046] A transition probability determination unit, used to determine the transition probability between the core node word and its adjacent core node words according to the edge weight value, the distance, the transition parameter and the inverse text frequency index;
[0047] A vector determination unit is used to determine the candidate word vector of the core node word according to the transition probability.
[0048] Optionally, a vector determination unit is specifically used to:
[0049] The transition parameter in the transition probability is set to a first set of values to obtain a first set of node search sequences; the first set of node search sequences is trained using a word vector training model to obtain a first vector of the core node word;
[0050] The transfer parameter in the transfer probability is set to a second set of values to obtain a second set of node search sequences; the second set of node search sequences is trained using a word vector training model to obtain a second vector of the core node word.
[0051] Optionally, a similar word set determination module is specifically used to:
[0052] Calculating the similarity between the first vector and the first preset word vector; taking the core node words whose similarity is greater than or equal to the first similarity threshold as the first node word set of the search preset words;
[0053] Calculate the similarity between the second vector and the second preset word vector; and use the core node words whose similarity is greater than a second similarity threshold as the second node word set of the search preset words.
[0054] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:
[0055] one or more processors;
[0056] a storage device for storing one or more programs,
[0057] When the one or more programs are executed by the one or more processors, the one or more processors implement any one of the similarity word searching methods for text terms in the embodiments of the present invention.
[0058] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the similarity search methods for text terms described in the embodiments of the present invention.
[0059] The embodiment of the present invention obtains a user's search preset word vector; wherein the search preset word vector includes a first preset word vector and a second preset word vector; determines the core node word of the target text; and determines the candidate word vector of the core node word; wherein the candidate word vector includes a first vector and a second vector; and determines a similar word set of the search preset word according to the similarity between the search preset word vector and the candidate word vector of the core node word. The embodiment of the present invention can accurately search for similar words of text words from a small corpus domain text by calculating the representative features of different vectors of each core word in the target text. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 1 is a flow chart of a similar word search method for text words in the first embodiment of the present invention;
[0061] Figure 2 is a flow chart of a similar word search method for text words in Embodiment 2 of the present invention;
[0062] Figure 3 is an undirected weighted schematic diagram of the initial node word chain in the second embodiment of the present invention;
[0063] Figure 4 Schematic diagram of an undirected weighted update node word chain in the second embodiment of the present invention;
[0064] Figure 5 Schematic diagram of the movement of the core node words in the second embodiment of the present invention;
[0065] Figure 6 is a schematic diagram of the structure of a similar word search device for text words in Embodiment 3 of the present invention;
[0066] Figure 7 It is a schematic diagram of the structure of the electronic device in the fourth embodiment. DETAILED DESCRIPTION
[0067] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0068] Embodiment 1
[0069] Figure 1 It is a flowchart of the text word similarity search method in the first embodiment of the present invention. This embodiment can be applied to the case of determining similar words of text words in a small corpus field. The method of this embodiment can be executed by a text word similarity search device, which can be implemented in hardware and / or software and can be configured in an electronic device. The text word similarity search method described in any embodiment of the present application can be implemented. Figure 1 As shown, the method specifically includes the following:
[0070] S110. Obtain the user's preset search word vector; wherein the preset search word vector includes a first preset word vector and a second preset word vector.
[0071] In this embodiment, the search preset word vector may be a word vector corresponding to one or two similar words in the target text that are considered to be more important and marked by the user in advance; in traditional document management systems and expert systems, users are required to read the text and mark multiple similar words based on understanding and analysis, which takes a lot of time. However, in this embodiment, only one or two similar words that are considered to be the most important are provided, which simplifies the workload while introducing expert knowledge; for expert systems, the meanings represented by nouns and verbs are more critical, so users can provide a noun and a verb as the search preset word vector.
[0072] Wherein, obtaining the user's preset search word vector may include: obtaining the user's preset search word; and determining the preset search word vector according to the preset search word. Specifically, the preset search word is selected by the user from the target text in advance based on experience. Specifically, the first preset word vector may be a homogeneity vector, and the second preset word vector may be a structural vector.
[0073] S120, determining the core node words of the target text; and determining the candidate word vectors of the core node words; wherein the candidate word vectors include a first vector and a second vector.
[0074] In this embodiment, the target text is a text in a small corpus scenario in which a single word in the text appears less frequently; the small corpus scenario may be an expert system containing specific expert knowledge; in an expert system, the text may involve more concept definitions or noun explanations, and the subject word of a sentence is often the most important, but in the field of the expert system, the subject word appears less frequently in the full text and may appear only once or twice.
[0075] The core node words are key words with dependency relationships in the target text, which can express the content of the target text intuitively and clearly. First, the individual words of the target text are screened to obtain the core node words, and then the similar words of the words of the target text are determined among the core node words, which can effectively simplify the implementation complexity of the similar words in the target text.
[0076] In the prior art, a large amount of text data is usually required as training samples to obtain relatively accurate results. For example, algorithms such as PageRank are used to extract similar words in the text. However, this method tends to obtain words with higher frequency of occurrence and is not applicable to the domain text of the expert system.
[0077] Therefore, this embodiment uses the candidate word vectors of the core node words to effectively solve the problem that similar text words with low frequency of occurrence are difficult to identify in the expert system. Among them, the first vector reflects the sequential co-occurrence relationship of the core node words in the context of the target text, and the second vector reflects the structural similarity relationship of the core node words in the context of the target text; specifically, the first vector in this embodiment can be a homogeneity vector, and the second vector can be a structural vector. This embodiment can use the Node2vec improved method to perform random walks to obtain a random walk node sequence, and then use the BERT method to train separately to obtain the candidate word vectors of each core node word in the target text.
[0078] S130: Determine a similar word set of the search preset word according to the similarity between the search preset word vector and the candidate word vector of the core node word.
[0079] In this embodiment, the similarity between the preset search word vector and the first vector and the second vector can be used to effectively determine the similar word set of the preset search word. This embodiment introduces a few key preset text similar words to determine the similar word set of the preset search word, which can greatly simplify the workload.
[0080] The embodiment of the present invention obtains a user's search preset word vector; wherein the search preset word vector includes a first preset word vector and a second preset word vector; determines the core node word of the target text; and determines the candidate word vector of the core node word; wherein the candidate word vector includes a first vector and a second vector; and determines a similar word set of the search preset word according to the similarity between the search preset word vector and the candidate word vector of the core node word. The embodiment of the present invention can accurately search for similar words of text words from a small corpus domain text by calculating the representative features of different vectors of each core word in the target text.
[0081] Embodiment 2
[0082] Figure 2 1 is a flow chart of a similar word search method for text words in the second embodiment of the present invention. This embodiment is further expanded and optimized on the basis of the above embodiment, and can be combined with any optional solution in the above technical solution. Figure 2 As shown, the method includes:
[0083] S210. Obtain the user's preset search word vector; wherein the preset search word vector includes a first preset word vector and a second preset word vector.
[0084] S220, obtaining each text sentence of the target text; and determining the dependency relationship between each word in the text sentence.
[0085] In this embodiment, the target text is sentence-processed to obtain the text sentences of the target text; and each text sentence is syntactically analyzed using dependency syntax to obtain the dependency relationship between the words in the text sentence; for example, the dependency relationship can be a subject-predicate relationship, a verb-object relationship, an indirect object relationship, a preposed object, a paraphrase, a attributive-predicate relationship, an adverbial-predicate structure, a verb-complement structure, a parallel relationship, and a preposition-object relationship, etc.
[0086] Specifically, dependency syntax explains the syntactic structure of a language unit by analyzing the dependency relationship between components within the unit. It argues that the core verb in a sentence is the central word that governs other components, but it itself is not governed by any other component. All governed components are subordinate to the ruler in some way. It is used to analyze the syntactic units contained in a sentence and the relationship between these syntactic units, eliminate invalid words, and obtain directed relationships between words at the syntactic level.
[0087] In this embodiment, optionally, before determining the core node words of the target text, the method of this embodiment further includes:
[0088] Obtain the original text of the core node word to be determined;
[0089] The original text is expanded to obtain the target text; wherein the corpus expansion is to perform text translation operations on the original text.
[0090] The original text is the expert knowledge text selected in the text field of the expert system; the corpus expansion is to enable the original text to express as much content as possible, and the expanded content is similar to the text content of the original text.
[0091] Specifically, back-translation technology can be used to expand the corpus of the original text; for example, the original text is translated from Chinese into English using an online translation tool, and then the English output is translated into Chinese using an online translation tool, and the back-translated Chinese output is appended to the original text to obtain the target text. Typical online translation tools include Baidu Translate, Youdao Translate, Sogou Translate, Tencent Translate, and Google Translate.
[0092] It should be noted that in the back-translation process, different online translation tools can be used interchangeably for Chinese-to-English and English-to-Chinese translation.
[0093] In this embodiment, by expanding the corpus of the original text to obtain the target text, the text content of the similar words to be extracted can be increased, so that the determined similar words are more accurate.
[0094] S230, constructing an initial node word chain of the target text according to the dependency relationship; and determining the core node words of the target text according to the initial node word chain and a preset stop word dictionary.
[0095] In this embodiment, the node word chain is a word chain composed of individual words in the target text, which can clearly and intuitively express the dependency relationship between the words; wherein the preset stop word dictionary stores the individual words that need to be deleted in the text.
[0096] In this embodiment, optionally, the core node words of the target text are determined according to the initial node word chain and the preset stop word dictionary, including:
[0097] If it is detected that there is a second node word in the initial node word chain that has a dependency relationship with the first node word and the third node word respectively, and the second node word is stored in the preset stop word dictionary, then the second node word is removed;
[0098] A dependency relationship between the first node word and the third node word is established to obtain an updated node word chain, and the first node word and the third node word are used as core node words of the target text.
[0099] In this embodiment, the connection relationship between the words can be directly seen through the initial node word chain; the initial node word chain can be represented by an undirected weighted graph, Figure 3 is an undirected weighted schematic diagram of the initial node word chain; for example, a single sentence target text in a resource management theory document, "Industry rules can adjust the degree of competition and the degree of uncertainty caused by the intensity of competition", is taken as an example, and dependency syntactic analysis is performed on it. The analysis results are shown in Figure 3 shown.
[0100] At the same time, when constructing an undirected weighted graph, stop words are specially processed. When stop words are removed, the relationship between core words formed with stop words as the medium is extracted. For example, if there are dependency relationships A->B and B->C, if B is a stop word and A and C are non-stop words, then the relationships A->B and B->C are removed and a new relationship A->C is added; the updated node word chain obtained is shown in Figure 4 , Figure 4 Schematic diagram of the undirected weighted update of the node word chain.
[0101] From the above dependency syntax analysis results, it can be seen that the core predicate of the text to be processed "industry rules can adjust the degree of competition and the degree of uncertainty caused by the intensity of competition" is "adjustment", the subject is "rules", and the object of "adjustment" is "degree"; when constructing an undirected weighted graph, since the undirected edges "adjustment"-"degree" and "competition"-"degree" appear twice, their edge weights are 2, and the weights of the remaining edges are 1.
[0102] Before the target text "Industry rules can adjust the degree of competition and the degree of uncertainty caused by the intensity of competition" is specially treated with stop words, there are undirected edges "competition"-"intense" and "intense"-"degree", among which "intense" is removed as a stop word, and then the relationship between the core words formed with the stop word "intense" as the center is extracted as "competition"-"degree".
[0103] In this embodiment, the initial node word chain is updated by using a preset stop word dictionary to obtain the core node words of the target text, which can effectively eliminate useless words in the target text and simplify the target text.
[0104] S240, obtaining the weight value of each edge in the update node word chain; and calculating the distance between the core node word and the adjacent core node word of the core node word in the update node word chain, and the inverse text frequency index of the core node word.
[0105] In this embodiment, the Node2vec method is used to perform a walk search on each core node word in the target text, and the distance between the current core node word and the next adjacent core node word is calculated; for example, assuming that a random walk starts from node T and now reaches node V, see Figure 5 , Figure 5 The figure is a schematic diagram of the movement of core node words; the circle is the core node word in the target text, and the straight line indicates that there is a dependency relationship between two core node words.
[0106] The inverse text frequency index of the core node word can be obtained by the following formula (1).
[0107]
[0108] Where |D| is the total number of files in all file sets, |{J:X∈d J}| represents the number of documents containing word node X.
[0109] S250, determining the transition probability between the core node word and its adjacent core node words according to the edge weight value, distance, transition parameter and inverse text frequency index.
[0110] In this embodiment, the transition parameter is a bias parameter of the transition probability, and the characteristic of the transition probability expression can be controlled.
[0111] The transition probability between a core node word and its adjacent core node words can be obtained according to the following formula (2).
[0112]
[0113] d T,X is the shortest distance between nodes T and X, and assuming that the Markov condition is met, d T,X The value of can only be one of {0, 1, 2}. V,X represents the edge weight between node V and node X in an undirected weighted graph, I X Represents the inverse text frequency index of node X.
[0114] When T,X = 0, it means returning from node V to node T. At this time, the transition probability is
[0115] When T,X =1, then node X is the direct neighbor of node T, and the transition probability is W V,X I X ;
[0116] When T,X =2, then node X is the neighbor of node T's neighbor, and the transition probability is
[0117] S260. Determine the candidate word vector of the core node word according to the transition probability.
[0118] In this embodiment, the difference vectors of each core node word in the target text can be obtained by adjusting the transfer parameter in the transfer probability; wherein, the parameter p is the return parameter, and the smaller p is, the greater the possibility of the random walk returning to the node T; the parameter q is called the in-out parameter, which controls the relationship between the breadth-first traversal and the depth-first traversal; if q>1, Node2vec tends to perform breadth-first traversal, and pays more attention to reflecting the structural similarity relationship of the core node words in the context of the target text; if q<1, Node2vec tends to perform depth-first traversal, and pays more attention to reflecting the sequential co-occurrence relationship of the core node words in the context of the target text. This embodiment can more accurately determine the different word vectors corresponding to each core node word in the target text through the transfer probability between the core node word and its adjacent core node word.
[0119] In this embodiment, optionally, determining the candidate word vector of the core node word according to the transition probability includes:
[0120] The transition parameter in the transition probability is set to a first set of values to obtain a first set of node search sequences; the first set of node search sequences is trained using a word vector training model to obtain a first vector of a core node word;
[0121] The transfer parameter in the transfer probability is set to a second set of values to obtain a second set of node search sequences; the second set of node search sequences is trained using a word vector training model to obtain a second vector of the core node word.
[0122] In this embodiment, the first set of values includes the values of two transfer parameters. Similarly, the second set of values includes the values of the above two transfer parameters. The node search sequence is a different search order for each core node word in the target text.
[0123] Specifically, we can set the transfer parameters {p1, q1}; where p1>>1 means that node T is not returned as much as possible; q1<1 makes Node2vec tend to perform depth-first traversal, and we get the first set of node search sequences. We can set the transfer parameters {p2, q2}; where p2>>1 means that node T is not returned as much as possible; q2>1 makes Node2vec tend to perform breadth-first traversal, and we get the second set of node search sequences.
[0124] For the first set of node search sequences and the second set of node search sequences, the BERT method is used to train the first vector (e.g., homogeneity vector) and the second vector (e.g., structural vector) of all core node words in the target text. Specifically, in the Node2vec training process, the initial vector of each word in the target text can be a word vector trained by a large amount of public Internet text data. Typical training methods include Word2vec, FastText or BERT.
[0125] Exemplarily, this embodiment takes all documents in the expert knowledge base as the file set, and the above-mentioned resource management theory document as the target text document; calculates the inverse text frequency index of the core node words therein, and obtains the node transition probability in combination with the edge weight; generates two sets of node search sequences according to the two sets of transition probability parameters; and finally obtains the first vector and second vector of each core node word in the resource management theory document, and some of the results are shown in the following table.
[0126]
[0127]
[0128] S270: Determine a similar word set of the search preset word according to the similarity between the search preset word vector and the candidate word vector of the core node word.
[0129] On the basis of the above embodiment, optionally, according to the similarity between the preset search word vector and the candidate word vector of the core node word, a similar word set of the preset search word is determined, including:
[0130] Calculate the similarity between the first vector and the first preset word vector; use the core node words whose similarity is greater than or equal to the first similarity threshold as the first node word set for searching the preset word;
[0131] Calculate the similarity between the second vector and the second preset word vector; and use the core node words whose similarity is greater than the second similarity threshold as the second node word set for searching the preset words.
[0132] In this embodiment, according to the above example, the search preset word vectors may be "resources" and "management". The cosine distance is used to calculate the vector similarity between the first vector and the first preset word vector, and the vector similarity between the second vector and the second preset word vector, as shown in the following formula (3).
[0133]
[0134] in, is the first vector or the second vector; is the first preset word vector or the second preset word vector.
[0135] It should be noted that, in this embodiment, the first vector may be a homogeneity vector, and the second vector may be a structural vector.
[0136] This embodiment combines the search with preset word vectors, which can greatly improve the accuracy of similar word determination for small corpus scenarios, and at the same time make the extracted similar words more representative.
[0137] Embodiment 3
[0138] Figure 6 1 is a schematic diagram of the structure of the text word similarity search device in the third embodiment of the present invention. This embodiment can be applied to the case of determining the text word similarity in a small corpus field. The device is configured in an electronic device and can implement the text word similarity search method described in any embodiment of the present application. The device specifically includes the following:
[0139] The vector acquisition module 610 is used to acquire the user's search preset word vector; wherein the search preset word vector includes a first preset word vector and a second preset word vector;
[0140] The vector determination module 620 is used to determine the core node words of the target text; and determine the candidate word vectors of the core node words; wherein the candidate word vectors include a first vector and a second vector;
[0141] The similar word set determination module 630 is used to determine a similar word set of the preset search word according to the similarity between the preset search word vector and the candidate word vector of the core node word.
[0142] Based on the above embodiment, optionally, the device of this embodiment further includes:
[0143] An original text determination module is used to obtain the original text of the core node word to be determined;
[0144] The target text determination module is used to perform corpus expansion on the original text to obtain the target text; wherein the corpus expansion is to perform a text translation operation on the original text.
[0145] Based on the above embodiment, optionally, the vector determination module 620 includes a dependency determination unit, an initial node word chain construction unit and a node word determination unit;
[0146] A dependency relationship determination unit, used to obtain each text sentence of the target text; and determine the dependency relationship between each word in the text sentence;
[0147] An initial node word chain construction unit, used to construct an initial node word chain of the target text according to the dependency relationship;
[0148] The node word determination unit is used to determine the core node words of the target text according to the initial node word chain and a preset stop word dictionary.
[0149] Based on the above embodiment, optionally, the node word determination unit is specifically used to:
[0150] If it is detected that there is a second node word in the initial node word chain, which has dependency relationships with the first node word and the third node word respectively, and the second node word is stored in a preset stop word dictionary, then the second node word is removed;
[0151] A dependency relationship between the first node word and the third node word is established to obtain an updated node word chain, and the first node word and the third node word are used as core node words of the target text.
[0152] On the basis of the above embodiment, optionally, the vector determination module 620 includes a calculation unit, a transition probability determination unit and a vector determination unit;
[0153] A calculation unit, used to obtain the weight value of each edge in the update node word chain; and calculate the distance between the core node word and its adjacent core node words in the update node word chain, and the inverse text frequency index of the core node word;
[0154] A transition probability determination unit, used to determine the transition probability between the core node word and its adjacent core node words according to the edge weight value, the distance, the transition parameter and the inverse text frequency index;
[0155] A vector determination unit is used to determine the candidate word vector of the core node word according to the transition probability.
[0156] Based on the above embodiment, optionally, the vector determination unit is specifically configured to:
[0157] The transition parameter in the transition probability is set to a first set of values to obtain a first set of node search sequences; the first set of node search sequences is trained using a word vector training model to obtain a first vector of the core node word;
[0158] The transfer parameter in the transfer probability is set to a second set of values to obtain a second set of node search sequences; the second set of node search sequences is trained using a word vector training model to obtain a second vector of the core node word.
[0159] Based on the above embodiment, optionally, the similar word set determination module 630 is specifically used to:
[0160] Calculating the similarity between the first vector and the first preset word vector; taking the core node words whose similarity is greater than or equal to the first similarity threshold as the first node word set of the search preset words;
[0161] Calculate the similarity between the second vector and the second preset word vector; and use the core node words whose similarity is greater than a second similarity threshold as the second node word set of the search preset words.
[0162] By using the similar word search device for text words of the third embodiment of the present invention, similar words of text words can be accurately searched from the small corpus domain text by calculating the representative features of different vectors of each core word in the target text.
[0163] The similar word search device for text words provided in the embodiment of the present invention can execute the similar word search method for text words provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0164] Embodiment 4
[0165] Figure 7 is a schematic diagram of the structure of an electronic device in Embodiment 4 of the present invention. Figure 7 As shown, the electronic device includes a processor 710, a memory 720, an input device 730, and an output device 740; the number of the processor 710 in the electronic device can be one or more. Figure 7 A processor 710 is taken as an example; the processor 710, the memory 720, the input device 730 and the output device 740 in the electronic device can be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.
[0166] The memory 720, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the text word similarity search method in the embodiment of the present invention. The processor 710 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 720, that is, the text word similarity search method provided in the embodiment of the present invention is implemented.
[0167] The memory 720 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 720 may further include a memory remotely arranged relative to the processor 710, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0168] The input device 730 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device, and may include a keyboard, a mouse, etc. The output device 740 may include a display device such as a display screen.
[0169] Embodiment 5
[0170] This embodiment provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to implement the similar word search method for text terms provided in the embodiment of the present invention.
[0171] Of course, the computer executable instructions of a storage medium containing computer executable instructions provided by an embodiment of the present invention are not limited to the method operations described above, and can also execute related operations in the similar word search method for text words provided by any embodiment of the present invention.
[0172] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.
[0173] It is worth noting that in the embodiment of the above-mentioned search device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
[0174] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A similar word search method for text words, characterized in that: The method comprises: Obtaining a user's preset search word vector; wherein the preset search word is selected by the user from the target text; the preset search word vector includes a first preset word vector and a second preset word vector; Determining the core node words of a target text includes: obtaining each text sentence of the target text; and determining the dependency relationship between each word in the text sentence; constructing an initial node word chain of the target text according to the dependency relationship; determining the core node words of the target text according to the initial node word chain and a preset stop word dictionary; wherein the target text is a text in a small corpus scenario and a single word in the text has a low frequency of occurrence; the small corpus scenario is an expert system containing specific expert knowledge; Determine a candidate word vector of the core node word; wherein the candidate word vector includes a first vector and a second vector; According to the similarity between the preset search word vector and the candidate word vector of the core node word, a similar word set of the preset search word is determined.
2. The method according to claim 1, characterized in that Before determining the core node words of the target text, the method further includes: Obtain the original text of the core node word to be determined; The original text is expanded with corpus to obtain a target text; wherein the corpus expansion is to perform a text translation operation on the original text.
3. The method according to claim 1, characterized in that Determine the core node words of the target text according to the initial node word chain and the preset stop word dictionary, including: If it is detected that there is a second node word in the initial node word chain, which has dependency relationships with the first node word and the third node word respectively, and the second node word is stored in a preset stop word dictionary, then the second node word is removed; A dependency relationship between the first node word and the third node word is established to obtain an updated node word chain, and the first node word and the third node word are used as core node words of the target text.
4. The method according to claim 3, characterized in that Determining the candidate word vector of the core node word includes: Obtaining the weight value of each edge in the update node word chain; and calculating the distance between the core node word and its adjacent core node words in the update node word chain, and the inverse text frequency index of the core node word; Determining the transition probability between the core node word and its adjacent core node words according to the edge weight value, the distance, the transition parameter and the inverse text frequency index; According to the transition probability, a candidate word vector of the core node word is determined.
5. The method according to claim 4, characterized in that Determining the candidate word vector of the core node word according to the transition probability includes: The transition parameter in the transition probability is set to a first set of values to obtain a first set of node search sequences; the first set of node search sequences is trained using a word vector training model to obtain a first vector of the core node word; The transfer parameter in the transfer probability is set to a second set of values to obtain a second set of node search sequences; the second set of node search sequences is trained using a word vector training model to obtain a second vector of the core node word.
6. The method according to claim 1, characterized in that Determining a similar word set of the preset search word according to the similarity between the preset search word vector and the candidate word vector of the core node word includes: Calculating the similarity between the first vector and the first preset word vector; taking the core node words whose similarity is greater than or equal to the first similarity threshold as the first node word set of the search preset words; Calculate the similarity between the second vector and the second preset word vector; and use the core node words whose similarity is greater than a second similarity threshold as the second node word set of the search preset words.
7. A similar word search device for text words, characterized in that: The device comprises: A vector acquisition and determination module is used to obtain a user's search preset word vector; wherein the search preset word is selected by the user from the target text; the search preset word vector includes a first preset word vector and a second preset word vector; A vector determination module is used to determine the core node words of the target text; and determine the candidate word vectors of the core node words; wherein the target text is a text in a small corpus scenario and the frequency of occurrence of a single word in the text is low; the small corpus scenario is an expert system containing specific expert knowledge; the candidate word vectors include a first vector and a second vector; The vector determination module includes a dependency determination unit, an initial node word chain construction unit and a node word determination unit; the dependency determination unit is used to obtain each text sentence of the target text; and determine the dependency relationship between each word in the text sentence; The initial node word chain construction unit is used to construct the initial node word chain of the target text according to the dependency relationship; The node word determination unit is used to determine the core node words of the target text according to the initial node word chain and a preset stop word dictionary; The similar word set determination module is used to determine the similar word set of the search preset word according to the similarity between the search preset word vector and the candidate word vector of the core node word.
8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the similar word search method for text terms as described in any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the similar word search method for text words as described in any one of claims 1 to 6 is implemented.
