A method and device for analyzing text relations, and a method for constructing a text relation network
Patent Information
- Application Number
- CN201911197739.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-11-29
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2039-11-29
AI Technical Summary
[0004]有鉴于此,本发明提供了一种文本关系的分析方法及装置、文本关系网络 的构建方法,用以解决或者至少部分解决现有技术中的方法存在的结果不够准 确的技术问题
[0036] The text relationship analysis method provided by the present invention first preprocesses each text contained in a text collection containing m independent texts; then, each word of the corpus is used as a concept subject heading, and all the texts obtained after preprocessing are traversed. Sentences and vocabulary, words that co-occur in the same sentence with concept topic words are included in the vocabulary set corresponding to the concept topic words, and then each vocabulary set is screened for vocabulary elements to construct concepts, and finally based on different texts in the text collection Does it contain the same concept? determine the relationship between texts.
Smart Images

Figure CN112883705B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to a method and device for analyzing text relations, and a method for constructing a text relation network. Background Art
[0002] Currently in the field of natural language processing, the quantitative evaluation of the relationship between texts is mainly based on the similarity of vocabulary, word frequency and word organization between texts. However, analyzing text relationships based solely on vocabulary similarity often leads to incorrect evaluation of the relationship between texts with different words but the same theme. At the same time, this defect also makes it difficult to measure the relationship between complex contents of multiple texts.
[0003] It can be seen from this that the methods in the prior art have the technical problem that the analysis results are not accurate enough. Summary of the invention
[0004] In view of this, the present invention provides a method and device for analyzing text relations, and a method for constructing a text relation network, so as to solve or at least partially solve the technical problem that the results of the methods in the prior art are not accurate enough.
[0005] In order to solve the above technical problems, the present invention provides a method for analyzing text relations in a first aspect, comprising:
[0006] Preprocess each text contained in a text set containing m independent texts to obtain all sentences and words of the corpus consisting of the m independent texts, where m is a positive integer;
[0007] Take each word in the corpus as a concept keyword, traverse all sentences and words, and add the words that co-appear with the concept keyword in the same sentence into the vocabulary set corresponding to the concept keyword, where the vocabulary set includes the concept keyword and vocabulary elements;
[0008] Screen the vocabulary elements of each vocabulary set and construct concepts;
[0009] Determine the relationship between texts based on whether different texts in the text collection contain the same concept.
[0010] In one implementation, preprocessing is performed on each text included in a text set including m independent texts, including:
[0011] Each text contained in a text collection containing m independent texts is segmented into sentences, words, and stop words are removed.
[0012] In one implementation, each vocabulary set is screened for vocabulary elements to construct concepts, including:
[0013] Count each vocabulary element x in the vocabulary set j and concept keyword x i The number of co-occurring texts z, where z≤m;
[0014] It is determined whether the text quantity z is greater than or equal to a first threshold value. If so, the vocabulary element is taken as a valid vocabulary of the vocabulary set and retained in the vocabulary set. Otherwise, the vocabulary element is removed from the vocabulary set.
[0015] In one implementation, determining the relationship between texts based on whether different texts in a text set contain the same concept includes:
[0016] According to the appearance of the concept subject words or vocabulary elements in the concept in the text, determine whether the text contains the concept;
[0017] The concepts contained in every two texts in the text collection are compared. If the two texts contain the same concept, the two texts are associated.
[0018] In one implementation, judging whether a text contains a concept based on the appearance of a concept keyword or a vocabulary element in the concept in the text includes:
[0019] When a concept X I When the total number of vocabulary elements is 0, if the concept keyword x i Appears in the text w i In the above example, we can determine the text w i Contains concept X I ;
[0020] When a concept X I When the total number of vocabulary elements is greater than 0 and less than the second threshold, if there is a vocabulary element that appears in the text w i If the text w i Contains concept X I ;
[0021] When a concept X I When the total number of vocabulary elements is greater than or equal to the second threshold, then according to X I The number of necessary words in the text w i Does it include concept X? I , where the necessary vocabulary is based on the vocabulary elements x in the vocabulary set j and concept keyword x i The number of co-occurring texts and lexical elements x j and concept keyword x i The relationship between them is determined.
[0022] In one embodiment, according to X I The number of necessary words in the text w i Does it include concept X? I ,include:
[0023] When a concept X I The number of necessary words in the text w is greater than or equal to the third threshold, if a preset proportion of necessary words appears in the text w i If the text w i Contains concept X I ;
[0024] When a concept X I The number of necessary words in the text w is less than the third threshold, if there is a preset proportion of vocabulary elements appearing in the text w i In the vocabulary of i Contains concept X I .
[0025] Based on the same inventive concept, the second aspect of the present invention provides a text relationship analysis device, comprising:
[0026] A preprocessing module preprocesses each text contained in a text set containing m independent texts to obtain all sentences and words of the corpus consisting of the m independent texts, where m is a positive integer;
[0027] The vocabulary set construction module is used to take each word in the corpus as a concept keyword, traverse all sentences and words, and include the words that co-appear with the concept keyword in the same sentence into the vocabulary set corresponding to the concept keyword, wherein the vocabulary set includes the concept keyword and vocabulary elements;
[0028] The concept building module is used to screen the vocabulary elements of each vocabulary set and build concepts;
[0029] The text relationship determination module is used to determine the relationship between texts based on whether different texts in the text collection contain the same concept.
[0030] Based on the same inventive concept, the third aspect of the present invention provides a method for constructing a text relationship network, which is implemented based on the text relationship analysis method described in the first aspect, and the construction method includes:
[0031] Construct a text relationship network based on the relationship between texts.
[0032] In one implementation, constructing a text relationship network based on the relationship between texts includes:
[0033] A weighted acyclic and undirected network is constructed as a text relationship network, with texts as nodes, the same concepts in texts as edges, and the number of the same concepts in two texts as the weight of the edge.
[0034] Based on the same inventive concept, the fourth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed, the method described in the first aspect is implemented. , including:
[0035] The above one or more technical solutions in the embodiments of the present application have at least one or more of the following technical effects:
[0036] The text relationship analysis method provided by the present invention first pre-processes each text contained in a text set containing m independent texts; then, each word in the corpus is used as a concept keyword, all sentences and words obtained after the pre-processing are traversed, and the words that appear together with the concept keyword in the same sentence are included in the vocabulary set corresponding to the concept keyword, and then the vocabulary elements of each vocabulary set are screened to construct concepts, and finally the relationship between the texts is determined according to whether different texts in the text set contain the same concept.
[0037] Due to the method provided by the present invention, a vocabulary set centered on a single word (conceptual keyword) is constructed through text co-occurrence rules, and vocabulary elements are screened for each vocabulary set to construct concepts. Then, according to whether different texts in the text set contain the same concept, the relationship between texts is determined. By providing a new method for constructing concepts, calculating the concepts contained in the texts, and then using the concepts to calculate the relationship between texts, the relationship between different texts can be analyzed more comprehensively, the accuracy of the analysis is improved, and the technical problem that the analysis results of the methods in the prior art are not accurate enough is solved.
[0038] Furthermore, based on the analysis of the relationships between different texts, the present invention also provides a method for constructing a text relationship network, which can display the relationships between different texts through the network, which is conducive to understanding the content hierarchy between multiple texts and expanding the application scenarios based on the relationship between texts. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0040] Figure 1A schematic diagram of a flow chart of a text relationship analysis method of the present invention;
[0041] Figure 2 A schematic diagram of a text relationship network constructed by the present invention in one embodiment;
[0042] Figure 3 A schematic diagram of a text relationship network constructed by the present invention in another embodiment;
[0043] Figure 4 is a structural block diagram of a text relationship analysis device in an embodiment of the present invention;
[0044] Figure 5 The present invention is a block diagram of a computer-readable storage medium. DETAILED DESCRIPTION
[0045] The purpose of the present invention is to provide a method and device for analyzing text relations and a method for constructing a text relation network, so as to improve the comprehensiveness and accuracy of the analysis, in order to solve the technical problem that the analysis results of the methods in the prior art are not accurate enough.
[0046] In order to achieve the above object, the main concepts of the present invention are as follows:
[0047] Through the text co-occurrence rules, a vocabulary set centered on a single word is constructed, and concepts are constructed. Then, the relationship between texts is determined based on whether different texts in the text set contain the same concept. The set of words that often co-occur with a certain word can be used to represent the concept pointed to by the word, and the relationship between different texts can be determined based on whether different texts contain the same concept. Based on this, a method for constructing a text relationship network is also provided, and texts are connected based on whether the same concept exists and a network composed of texts is constructed.
[0048] By constructing concepts in the form of vocabulary sets and then using concepts to calculate the relationships between texts, we can more comprehensively analyze the relationships between different texts and improve the accuracy of the analysis. In addition, by constructing a text relationship network, we can understand the content hierarchy between multiple texts and expand the application scenarios based on the relationship between texts.
[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] Embodiment 1
[0051] This embodiment provides a method for analyzing text relations. Figure 1 , the method comprising:
[0052] Step S1: pre-process each text contained in a text set containing m independent texts to obtain all sentences and words of a corpus consisting of the m independent texts, where m is a positive integer.
[0053] Specifically, m independent texts are the texts to be analyzed, and the number of m can be determined according to actual conditions. Preprocessing includes sentence segmentation, word segmentation, and stop word removal.
[0054] Step S2: Take each word in the corpus as a concept keyword, traverse all sentences and words, and add the words that co-appear in the same sentence with the concept keyword into the vocabulary set corresponding to the concept keyword, where the vocabulary set includes the concept keyword and vocabulary elements.
[0055] Specifically, step S1 can obtain all the words in the corpus, and this step takes each word as a benchmark to find words that appear in the same sentence as the word, and constructs a vocabulary set with the words that appear together with the word. If the words appear together with the concept subject word, it means that the two words are related. Among them, the vocabulary set includes two types of words, one is the concept subject word, and the other is the vocabulary element.
[0056] Step S3: Screen vocabulary elements for each vocabulary set and construct concepts.
[0057] Specifically, in order to improve the accuracy of concepts, this step further screens the vocabulary elements, for example, by judging whether the vocabulary elements often appear together with concept keywords in a text to determine whether to retain the vocabulary elements, wherein whether the vocabulary elements often appear can be determined according to a set threshold.
[0058] Step S4: Determine the relationship between the texts according to whether different texts in the text set contain the same concept.
[0059] Specifically, after constructing the concepts of the text corpus, the relationship between the texts is determined based on whether the texts contain the same concepts. It is necessary to determine whether a text contains a certain concept. In the specific implementation process, it can be determined based on whether the concept (conceptual subject word or lexical element) appears in the text and the number of times it appears in the text.
[0060] In one embodiment, step S1 includes:
[0061] Each text contained in a text collection containing m independent texts is segmented into sentences, words, and stop words are removed.
[0062] For example, for a text data set W containing m independent texts, W = {w1, w2…w m}, and remove stop words to obtain all sentences and vocabulary x1, x2...x in the corpus. n .
[0063] In one embodiment, step S3 includes:
[0064] Step S3.1: Count each vocabulary element x in the vocabulary set j and concept keyword x i The number of co-occurring texts z, where z≤m;
[0065] Step S3.2: Determine whether the text quantity z is greater than or equal to a first threshold value. If so, the vocabulary element is taken as a valid vocabulary of the vocabulary set and retained in the vocabulary set. Otherwise, the vocabulary element is removed from the vocabulary set.
[0066] Specifically, the first threshold can be set according to actual conditions, for example, 3, 5, 6, etc. Through vocabulary element screening, words that often appear in the same text as concept keywords can be selected as valid words, thereby improving the accuracy of concepts.
[0067] In one embodiment, step S4 includes:
[0068] Step S4.1: judging whether the text contains the concept according to the appearance of the concept subject words or vocabulary elements in the concept in the text;
[0069] Step S4.2: Compare the concepts contained in every two texts in the text collection. If two texts contain the same concept, the two texts are associated.
[0070] Specifically, the occurrence of conceptual keywords or lexical elements in the concept in the text includes whether they appear and the number of occurrences.
[0071] In one embodiment, step S4.1 comprises:
[0072] Step S4.1.1: When a concept X I When the total number of vocabulary elements is 0, if the concept keyword x i Appears in the text w i If the text w i Contains concept X I ;
[0073] Step S4.1.2: When a concept X I When the total number of vocabulary elements is greater than 0 and less than the second threshold, if there is a vocabulary element that appears in the text w i If the text w i Contains concept X I ;
[0074] Step S4.1.3: When a concept X I When the total number of vocabulary elements is greater than or equal to the second threshold, then according to X I The number of necessary words in the text w i Does it include concept X? I , where the necessary vocabulary is based on the vocabulary elements x in the vocabulary set j and concept keyword x i The number of co-occurring texts and lexical elements x j and concept keyword x i The relationship between them is determined.
[0075] Specifically, if the number of lexical elements of a concept is 0, it is determined based on whether the concept subject word appears in the text (step S4.1.1), where whether the concept subject word appears in the text can be determined based on whether the concept subject word appears in the text w i The words contained in the text w i The included vocabulary can be obtained by sentence segmentation, word segmentation and removal of stop words. j and concept keyword x i The relationship between can be determined based on the distance between the corresponding word vectors.
[0076] If the number of vocabulary elements is greater than 0, the judgment is made according to steps S4.1.2 and S4.1.3, and the second threshold can be determined according to the actual situation, for example, 3, 4, 5, etc.
[0077] Among them, the necessary words are further processed from the valid words retained in the vocabulary set. The relationship between necessary words and valid words is explained below. Necessary words must exist as valid words first, but valid words are not necessarily necessary words. The necessary words acquisition steps can be:
[0078] (1) Statistical concept keyword x i The number of occurrences of text y (y≤m). Calculate the vocabulary element x j and concept keyword x i The co-occurrence score scoreP j =z / y.
[0079] (2) Calculate the vocabulary element xj The word vector v j and concept keyword x i The word vector v i The Euclidean distance D j =v i -v j , then the Euclidean distance D j Normalize and get the vocabulary element in the vocabulary set X I The distance score scoreD j .
[0080] Among them, the word vector can be obtained by training the initial word vector through word embedding. The word vector is obtained as follows:
[0081] First, a text data set W = {w1, w2...w m}, and then we can segment each text into sentences and words, remove stop words, and get all the words x1, x2...x in the corpus. n .
[0082] Then, word embedding is performed through the word2vec method to obtain word vectors of n words, and the dimension of the word vector is 200 dimensions.
[0083] Lexical element x j The word vector v j and concept keyword x i The word vector can be matched from the initial word vector obtained through training.
[0084] The distance D between two words j The closer they are, the closer their relationship is, and the distance score is scoreD j The higher the distance D between two words j The farther they are, the farther their relationship is, and the distance score is scoreD j The lower the distance score, the higher the scoreD j The calculation method is as follows:
[0085]
[0086] (3) For each vocabulary element x in the set j , put it into the rectangular coordinate system composed of two dimensions scoreD and scoreP, the vocabulary element x j The coordinates of scoreD j , scoreP j ) to the origin is recorded as the comprehensive score of the vocabulary element
[0087] (4) If the vocabulary element x j The comprehensive score S j > 0.6, then the vocabulary element is regarded as the vocabulary set X I Necessary vocabulary.
[0088] In one embodiment, step S4.1.3 comprises:
[0089] Step S4.1.3.1: When a concept X I The number of necessary words in the text w is greater than or equal to the third threshold, if a preset proportion of necessary words appears in the text w i If the text w i Contains concept X I ;
[0090] Step S4.1.3.2: When a concept X I The number of necessary words in the text w is less than the third threshold, if there is a preset proportion of vocabulary elements appearing in the text w i In the vocabulary of i Contains concept X I .
[0091] Specifically, the third threshold value can be determined according to actual conditions, such as 5, 6, 8, etc., and the preset ratio can be determined according to actual conditions, such as 60%, 65%, 70%, etc.
[0092] Taking the third threshold of 5 and the preset ratio of 60% as an example, if the number of necessary words in a concept is greater than or equal to 5, the ratio of the necessary words is used to measure whether the text contains a certain concept.
[0093] If the number of necessary words in a concept is less than 5, it means that the number of necessary words is not enough to determine whether the text contains this concept. Therefore, it is necessary to calculate based on all the lexical elements. If more than 60% of the lexical elements of a concept appear in the text, it is determined that the text contains this concept.
[0094] In order to more clearly illustrate the specific implementation process of the method of the present invention, the following is introduced through specific examples:
[0095] Embodiment 2
[0096] Based on the same inventive concept, this embodiment provides a text relationship analysis device, see Figure 4 ,include:
[0097] A preprocessing module 201 preprocesses each text contained in a text set containing m independent texts to obtain all sentences and words of a corpus consisting of the m independent texts, where m is a positive integer;
[0098] The vocabulary set building module 202 is used to take each word of the corpus as a concept keyword, traverse all sentences and words, and add the words that co-appear with the concept keyword in the same sentence into the vocabulary set corresponding to the concept keyword, wherein the vocabulary set includes the concept keyword and vocabulary elements;
[0099] A concept construction module 203 is used to screen vocabulary elements for each vocabulary set and construct concepts;
[0100] The text relationship determination module 204 is used to determine the relationship between texts according to whether different texts in the text set contain the same concept.
[0101] In one embodiment, the preprocessing module 201 is specifically used for:
[0102] Each text contained in a text collection containing m independent texts is segmented into sentences, words, and stop words are removed.
[0103] In one implementation, the concept building module 203 is specifically used to:
[0104] Count each vocabulary element x in the vocabulary set j and concept keyword x i The number of co-occurring texts z, where z≤m;
[0105] It is determined whether the text quantity z is greater than or equal to a first threshold value. If so, the vocabulary element is taken as a valid vocabulary of the vocabulary set and retained in the vocabulary set. Otherwise, the vocabulary element is removed from the vocabulary set.
[0106] In one implementation, the text relationship determination module 204 is specifically configured to:
[0107] According to the appearance of the concept subject words or vocabulary elements in the concept in the text, determine whether the text contains the concept;
[0108] The concepts contained in every two texts in the text collection are compared. If the two texts contain the same concept, the two texts are associated.
[0109] In one implementation, the text relationship determination module 204 is further configured to:
[0110] When a concept X I When the total number of vocabulary elements is 0, if the concept keyword x i Appears in the text wi In the above example, we can determine the text w i Contains concept X I ;
[0111] When a concept X I When the total number of vocabulary elements is greater than 0 and less than the second threshold, if there is a vocabulary element that appears in the text w i If the text w i Contains concept X I ;
[0112] When a concept X I When the total number of vocabulary elements is greater than or equal to the second threshold, then according to X I The number of necessary words in the text w i Does it include concept X? I , where the necessary vocabulary is based on the vocabulary elements x in the vocabulary set j and concept keyword x i The number of co-occurring texts and lexical elements x j and concept keyword x i The relationship between them is determined.
[0113] In one implementation, the text relationship determination module 204 is further configured to:
[0114] When a concept X I The number of necessary words in the text w is greater than or equal to the third threshold, if a preset proportion of necessary words appears in the text w i If the text w i Contains concept X I ;
[0115] When a concept X I The number of necessary words in the text w is less than the third threshold, if there is a preset proportion of vocabulary elements appearing in the text w i In the vocabulary of i Contains concept X I .
[0116] Since the device introduced in the second embodiment of the present invention is a device used to implement the text relationship analysis method in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the device, so it is not repeated here. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.
[0117] Embodiment 3
[0118] Based on the same inventive concept, the present application also provides a method for constructing a text relationship network, which is implemented based on the text relationship analysis method described in Example 1. The construction method includes:
[0119] Construct a text relationship network based on the relationship between texts.
[0120] Among them, according to the relationship between texts, a text relationship network is constructed, including:
[0121] A weighted acyclic and undirected network is constructed as a text relationship network, with texts as nodes, the same concepts in texts as edges, and the number of the same concepts in two texts as the weight of the edge.
[0122] Specifically, for a text set W containing n independent texts, n}, calculate the relationship between texts. Compare the texts in the set W two by two and compare the concepts contained in the texts. If two texts have the same concept, they are linked (there is an edge between the two nodes), and the number of the same concepts is the weight of the link, where the weight indicates the strength of the relationship between the two texts. Then, a weighted acyclic undirected network between the texts is constructed based on the links and weights between the texts.
[0123] Several steps of constructing a text relationship network using the method provided by the present invention are described below through specific examples.
[0124] 1. Example of building a concept library. All concepts constitute the concept library.
[0125] The concept keyword is "rootstock", which appears in 51 articles in the corpus. All the lexical elements in the table below are words that have co-appeared in the same sentence with the word "rootstock" (excerpt)
[0126]
[0127]
[0128] The distance can be calculated by computing the vocabulary element x j The word vector and concept keyword x i The distance score is obtained by normalizing the distance so that it is between 0 and 1. The closer the distance, the higher the distance score, and the farther the distance, the lower the distance score.
[0129] 2. Based on the existing concept library, determine which concepts the text contains
[0130] Obtained from the matching of concepts in the existing literature "Relevant Research Based on Problems" (excerpt)
[0131]
[0132] 3. Text network construction and drawing.
[0133] In a text network, nodes represent texts, edges represent the relationship between texts, and the weight of the edge represents the strength of the relationship between texts.
[0134] in, Figure 2 This is an example of a text relationship network diagram containing 42 texts (number of nodes: 42, number of edges: 47).
[0135] The network of inter-textual relations drawn for m texts is as follows Figure 3 As shown, the denser the edges, the stronger the association between the corresponding texts, and the sparser the edges, the weaker the association.
[0136] The method provided by the present invention constructs a vocabulary set centered on a single vocabulary (conceptual subject word) through text co-occurrence rules, and screens vocabulary elements for each vocabulary set to construct concepts, and then determines the relationship between texts based on whether different texts in the text set contain the same concept. By providing a new method for constructing concepts, calculating the concepts contained in the texts, and then using the concepts to calculate the relationship between the texts, the relationship between different texts can be analyzed more comprehensively, thereby improving the accuracy of the analysis.
[0137] Furthermore, based on the analysis of the relationships between different texts, the present invention also provides a method for constructing a text relationship network, which can display the relationships between different texts through the network, which is conducive to understanding the content hierarchy between multiple texts and expanding the application scenarios based on the relationship between texts.
[0138] Embodiment 4
[0139] See also Figure 5 Based on the same inventive concept, the present application also provides a computer-readable storage medium 300 on which a computer program 311 is stored. When the program is executed, the method described in the first embodiment is implemented.
[0140] Since the computer-readable storage medium introduced in the third embodiment of the present invention is a computer device used to implement the text relationship analysis method in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the computer-readable storage medium, so it is not repeated here. All computer-readable storage media used in the method in the first embodiment of the present invention belong to the scope of protection of the present invention.
[0141] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0142] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0143] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0144] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A method for analyzing textual relations, characterized in that: include: Preprocess each text contained in a text set containing m independent texts to obtain all sentences and words of the corpus consisting of the m independent texts, where m is a positive integer; Take each word in the corpus as a concept keyword, traverse all sentences and words, and add the words that co-appear with the concept keyword in the same sentence into the vocabulary set corresponding to the concept keyword, where the vocabulary set includes the concept keyword and vocabulary elements; Screen the vocabulary elements of each vocabulary set and construct concepts; Determine the relationship between texts based on whether different texts in the text collection contain the same concept; Among them, each vocabulary set is screened for vocabulary elements and concepts are constructed, including: Count each word element in the vocabulary collection Concept Keywords Number of co-occurring texts ,in, ; Determine the number of texts Is it greater than or equal to a first threshold? If so, the vocabulary element is taken as a valid vocabulary of the vocabulary set and retained in the vocabulary set; otherwise, the vocabulary element is removed from the vocabulary set.
2. The method according to claim 1, characterized in that Preprocess each text contained in a text collection containing m independent texts, including: Each text contained in a text collection containing m independent texts is segmented into sentences, words, and stop words are removed.
3. The method according to claim 1, characterized in that Determine the relationship between texts based on whether different texts in the text collection contain the same concept, including: Determine whether the text contains the concept based on the appearance of the concept subject words or vocabulary elements in the concept in the text; The concepts contained in every two texts in the text collection are compared. If the two texts contain the same concept, the two texts are associated.
4. The method according to claim 3, characterized in that Determine whether the text contains the concept based on the appearance of the concept subject words or vocabulary elements in the concept, including: When a concept When the total number of vocabulary elements is 0, if the concept keyword Appears in text , then determine the text Contains concepts ; When a concept When the total number of vocabulary elements is greater than 0 and less than the second threshold, if there is 1 vocabulary element appearing in the text , then determine the text Contains concepts ; When a concept When the total number of vocabulary elements is greater than or equal to the second threshold, The necessary number of words in the text Does it contain concepts? , where the necessary words are based on the vocabulary elements in the vocabulary set Concept Keywords Number of co-occurring texts and lexical elements Concept Keywords The relationship between them is determined.
5. The method according to claim 4, characterized in that according to The necessary number of words in the text Does it contain concepts? ,include: When a concept The number of necessary words in the text is greater than or equal to the third threshold, if a preset proportion of necessary words appears in the text , then determine the text Contains concepts ; When a concept The number of necessary words in the text is less than the third threshold, if there is a preset proportion of vocabulary elements appearing in the text , then determine the text Contains concepts .
6. A text relationship analysis device, characterized in that: include: A preprocessing module preprocesses each text contained in a text set containing m independent texts to obtain all sentences and words of the corpus consisting of the m independent texts, where m is a positive integer; The vocabulary set construction module is used to take each word in the corpus as a concept keyword, traverse all sentences and words, and include the words that co-appear with the concept keyword in the same sentence into the vocabulary set corresponding to the concept keyword, wherein the vocabulary set includes the concept keyword and vocabulary elements; The concept building module is used to screen the vocabulary elements of each vocabulary set and build concepts; A text relationship determination module is used to determine the relationship between texts according to whether different texts in the text collection contain the same concept; The concept building module is specifically used for: Count each word element in the vocabulary collection Concept Keywords Number of co-occurring texts ,in, ; Determine the number of texts Is it greater than or equal to a first threshold? If so, the vocabulary element is taken as a valid vocabulary of the vocabulary set and retained in the vocabulary set; otherwise, the vocabulary element is removed from the vocabulary set.
7. A method for constructing a text relationship network, characterized in that: Based on the text relationship analysis method described in any one of claims 1 to 5, the construction method includes: Construct a text relationship network based on the relationship between texts.
8. The method according to claim 7, characterized in that According to the relationship between texts, a text relationship network is constructed, including: A weighted acyclic and undirected network is constructed as a text relationship network, with texts as nodes, the same concepts in texts as edges, and the number of the same concepts in two texts as the weight of the edge.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 6 is implemented.
