A standard intelligent association method based on text mining

Abstract

The present invention relates to the technical field of text processing, and in particular to a standard intelligent association method based on text mining, comprising: pre-processing each text file to obtain corresponding vocabulary sets; obtaining corresponding keyword sets based on each vocabulary set; pre-screening the text file to be mined based on the keyword set to obtain a number of coarse-grained text files to form a coarse-grained file set; performing pointing processing on each keyword of each keyword set in the coarse-grained file set to form each keyword group set; calculating the similarity between the several keyword sets corresponding to the coarse-grained file set based on each keyword group set to obtain a fine-grained file set; determining the association logic map of each text file in the fine-grained file set according to the attributes of the keyword set corresponding to each text file in the fine-grained file set. The present invention determines similar files by a two-step method of coarse-grained screening and fine-grained screening, thereby improving the accuracy of similarity calculation.

Description

Technical Field

[0001] The present invention relates to the technical field of text processing, and in particular to a standard intelligent association method based on text mining. Background Art

[0002] Thanks to the continuous advancement of computer technology and the rapid development of machine learning algorithms, text mining has developed rapidly in the similarity judgment and relevance judgment of large-scale document collections. Through similarity and relevance judgment, the time and energy of manual judgment and identification are reduced, and the mining efficiency of text mining is improved.

[0003] Chinese Patent Publication No.: CN113688954A. A method, system, device and storage medium for calculating text similarity are disclosed, the method comprising: preprocessing the text to be compared and the reference text respectively; counting the frequency of each word in the text to be compared and the reference text respectively, and converting the preprocessed text to be compared and the reference text into feature vectors according to the frequency; using a vector space model algorithm to calculate the similarity value of the feature vector of the text to be compared and the feature vector of the reference text, and judging whether the similarity value reaches a threshold; and in response to the similarity value reaching the threshold, highlighting the similar part.

[0004] It can be seen that although the above technical solution realizes the calculation of similarity between any two texts and improves the accuracy of text similarity calculation, the following problems still exist: the text similarity is calculated by calculating cosine similarity, ignoring the word order information inside the text, which will cause two groups of phrases with different semantics in different texts, such as "the mouse is chasing the cat" and "the cat is chasing the mouse", to be judged as similar, thereby affecting the accuracy of the calculated text similarity, and failing to determine the association logic between different files. Summary of the invention

[0005] To this end, the present invention provides a standard intelligent association method based on text mining to overcome the problems of low text similarity calculation accuracy and low association judgment accuracy in the prior art.

[0006] To achieve the above object, on the one hand, the present invention provides a standard intelligent association method based on text mining, comprising:

[0007] Step S1, preprocessing each text file to obtain each vocabulary set of the corresponding text file, wherein the words in the vocabulary set are sorted in order according to the text content;

[0008] Step S2, extracting corresponding keywords based on each of the vocabulary sets to obtain corresponding keyword sets;

[0009] Step S3, preliminarily screening the text files to be mined based on the keyword set of each text file mined to obtain a number of coarse-grained text files to form a coarse-grained file set;

[0010] Step S4, performing pointing processing on each keyword of each keyword set in the coarse-grained document set to form each keyword group set;

[0011] Step S5, calculating the similarities between the plurality of keyword sets corresponding to the coarse-grained document set based on each of the keyword group sets, to obtain a fine-grained document set;

[0012] Step S6, determining the logical relationship between any two text files in the fine-grained file set to construct an association logic map of the text files in the fine-grained file set.

[0013] Further, in the step S1, the preprocessing includes:

[0014] Step S11, denoising a number of files, including removing punctuation marks and stop words;

[0015] Step S12, performing text segmentation on the denoised files to obtain a number of segmentation sets;

[0016] Step S13, extracting stems from the plurality of word segments to obtain a plurality of vocabulary sets.

[0017] Furthermore, in step S2, corresponding keywords are extracted based on each vocabulary set to obtain corresponding keyword sets, including:

[0018] Step S21, counting the number of occurrences of each word in the plurality of vocabulary sets;

[0019] Step S22, the words whose occurrence times are greater than or equal to a preset number are determined as keywords;

[0020] Step S23, integrating the keywords to obtain a keyword set.

[0021] Furthermore, in step S3, the text files to be mined are preliminarily screened based on the keyword set of each text file mined to obtain a number of coarse-grained text files, forming a coarse-grained file set, including:

[0022] Step S31, obtaining the union of two keyword sets corresponding to any two files among the plurality of files;

[0023] Step S32, creating a vocabulary based on the union, and setting an index for each keyword in the vocabulary;

[0024] Step S33, creating corresponding vector A and vector B based on the two keyword sets and the index;

[0025] Step S34, calculating the cosine similarity between the vector A and the vector B;

[0026] Step S35, judging whether the two files corresponding to the two keyword sets are similar based on the cosine similarity and the set coarse-grained association degree, includes:

[0027] If the cosine similarity is greater than or equal to the coarse-grained association, the two files corresponding to the two keyword sets are similar, and the two files are placed in the same coarse-grained file set;

[0028] If the cosine similarity is less than the coarse-grained correlation, the two files corresponding to the two keyword sets are not similar, and any one of the two files is used as a reference file, and steps S31 to S35 are continued to be performed, and the cosine similarity between the remaining files in the plurality of files and the reference file is continuously judged until the number of judgments reaches a preset number of judgments;

[0029] The preset number of determinations is the number of files that are not included in the coarse-grained file set minus 1.

[0030] Furthermore, in step S4, it includes:

[0031] Dividing any of the plurality of files into a plurality of groups of sentences based on punctuation marks;

[0032] Based on the keyword set corresponding to any one of the files, the keyword replacement is performed on the groups of sentences to obtain corresponding keyword groups.

[0033] Furthermore, the plurality of keyword groups also includes: removing repeated keyword groups from the plurality of keyword groups respectively to form a keyword group set.

[0034] Further, each keyword group is sequentially placed into the corresponding keyword group set based on the order of each keyword group in the corresponding file, including:

[0035] If the keyword group does not exist in the keyword group set, adding the keyword group to the corresponding keyword group set;

[0036] If the elements and the order of the elements in the keyword group are different from the elements and the order of the elements in the keyword group already added in the corresponding keyword group set, then add it to the keyword group set;

[0037] If the elements and the order of the elements in the keyword group are the same as the elements in the keyword group already added in the corresponding keyword group set and the order of the elements is the same, then the keyword group will not be added to the corresponding keyword group set.

[0038] Furthermore, in step S5, it includes:

[0039] respectively determining the longest common keyword group between any two keyword groups in the keyword group sets;

[0040] Calculate phrase similarity based on the longest common keyword group and the two keyword groups corresponding to the longest common keyword group;

[0041] The comprehensive similarity of two keyword group sets is calculated based on the phrase similarity, wherein:

[0042] If the comprehensive similarity is greater than or equal to the fine-grained relevance, the two files corresponding to the two keyword groups are determined to be similar, and the two files corresponding to the two keyword groups are placed in the same fine-grained file set.

[0043] Furthermore, a logical relationship judgment is performed on any two files in a single fine-grained file set, including:

[0044] Determine two text files in the single fine-grained file set for logical relationship judgment;

[0045] Determine the union of the keyword sets corresponding to the two text files;

[0046] Respectively determine the positions of the keywords in the union in any two text files;

[0047] A logical relationship between the two text files is determined based on the positions.

[0048] Furthermore, in step S1, the preprocessing further includes: performing hypernym replacement on the plurality of word segmentation sets.

[0049] Compared with the prior art, the beneficial effect of the present invention is that, by preprocessing several files and extracting keywords respectively, a keyword set specific to each file is formed. Based on several keyword sets, the present invention improves the accuracy of file screening, obtains several coarse-grained files with certain similarities by preliminary screening of several files, reduces the interference of irrelevant files, and further calculates the similarity between the coarse-grained files, thereby obtaining a fine-grained file set with higher similarity, thereby determining the associated logical map of each text file in the fine-grained file set. The present invention improves the accuracy of similarity calculation between different files, and ensures the fine-grained and high-quality text mining.

[0050] Furthermore, the present invention preprocesses several files to remove noise and redundant information in the text, converts the original text data into a form that can be better understood and processed by the computer, saves computer resources, and further improves the accuracy of similarity calculation between different files, ensuring the fine-grained and high-quality text mining.

[0051] Furthermore, the present invention determines high-frequency keywords by counting the number of occurrences of each word in a number of vocabulary sets, and integrates the high-frequency keywords to obtain a keyword set. The present invention improves the accuracy of keyword selection, and also greatly shortens the time for keyword set construction, improves work efficiency, and is particularly suitable for processing large amounts of text data. It further improves the accuracy of similarity calculations between different files, ensuring the fine-grained and high-quality text mining.

[0052] Furthermore, the present invention realizes the judgment of the similarity between different files through the steps of keyword set union extraction, file vectorization, cosine similarity calculation, etc., and classifies similar files into the same coarse-grained file set, thereby laying the foundation for subsequent fine-grained similarity judgment, further improving the accuracy of similarity calculation between different files, and ensuring the fine-grained and high-quality text mining.

[0053] Furthermore, the present invention firstly effectively groups the text content of any file based on punctuation marks to form several groups of sentences, which provides convenience for subsequent information processing. Secondly, based on the previously extracted keyword set, several groups of sentences are replaced with corresponding keyword groups, which retains the core information of the file and greatly improves the speed of information retrieval and processing. Furthermore, by removing the same repeated groups in several keyword groups, a keyword group set is obtained, which effectively avoids the redundancy of text information, further improves the accuracy of similarity calculation between different files, and ensures the fine-grained and high-quality text mining.

[0054] Furthermore, the present invention sets a keyword group set, and adds the keyword group to the keyword group set in an orderly manner based on the keywords and keyword order in the keyword group, effectively avoiding the duplication of keyword groups, greatly improving data processing efficiency and accuracy, reducing the interference of repeated information, further improving the accuracy of similarity calculation between different files, and ensuring the fine-grained and high-quality text mining.

[0055] Furthermore, the present invention determines the relevance between two text files by respectively determining the longest common keyword group between each keyword group in any two keyword group sets, and then calculates the phrase similarity based on the determined longest common keyword group and the two keyword groups corresponding to the longest common keyword group, which not only improves the scientificity and rationality of the calculation process, but also makes the subsequent comprehensive similarity calculation results closer to the similarity of the actual content. Furthermore, the present invention calculates the comprehensive similarity based on several phrase similarities and corresponding weights, realizes the classification of text files at a fine-grained level, and further ensures the fine-grained and high-quality text mining.

[0056] Furthermore, the present invention effectively improves the accuracy and efficiency of judging the logical relationship between files by determining two text files in a single fine-grained file set to be judged, and determining the union of the keyword sets corresponding to the two files. This avoids misjudgment of logical relationships due to inaccurate keyword selection. Then, based on the positions of the keywords in the union in any two text files, the logical relationship between the two text files is determined to ensure the objectivity and scientificity of the logical relationship judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of the standard intelligent association method based on text mining of the present invention;

[0058] Figure 2 A flow chart of determining a coarse-grained file set based on a plurality of text files according to the present invention;

[0059] Figure 3 A flow chart of determining a fine-grained file set based on a coarse-grained file set according to the present invention;

[0060] Figure 4 This is a flow chart of the present invention for judging the logical relationship between any two files in a single fine-grained file set. DETAILED DESCRIPTION

[0061] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0062] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0063] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0064] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0065] See also Figure 1 As shown, it is a flow chart of a standard intelligent association method based on text mining of the present invention. The present invention provides a standard intelligent association method based on text mining, comprising:

[0066] Step S1, preprocessing each text file to obtain each vocabulary set of the corresponding text file, wherein the words in the vocabulary set are sorted in order according to the text content;

[0067] Step S2, extracting corresponding keywords based on each of the vocabulary sets to obtain corresponding keyword sets;

[0068] Step S3, preliminarily screening the text files to be mined based on the keyword set of each text file mined to obtain a number of coarse-grained text files to form a coarse-grained file set;

[0069] Step S4, performing pointing processing on each keyword of each keyword set in the coarse-grained document set to form each keyword group set;

[0070] Step S5, calculating the similarities between the plurality of keyword sets corresponding to the coarse-grained document set based on each of the keyword group sets, to obtain a fine-grained document set;

[0071] Step S6, determining the logical relationship between any two text files in the fine-grained file set to construct an association logic map of the text files in the fine-grained file set.

[0072] It can be understood that the coarse-grained file set is files with roughly similar texts, and the fine-grained file set is files with further similar texts. The present invention obtains a fine-grained file set through two-step screening of coarse-grained and fine-grained files, and constructs an associated logical map of each text file therein based on the fine-grained file set.

[0073] The present invention forms a keyword set specific to each file by preprocessing several files and extracting keywords respectively. Based on several keyword sets, the present invention improves the accuracy of file screening, obtains several coarse-grained files with certain similarities through preliminary screening of several files, reduces interference from irrelevant files, and further calculates the similarity between coarse-grained files, thereby obtaining a fine-grained file set with higher similarity, thereby determining the associated logical map of each text file in the fine-grained file set. The present invention improves the accuracy of similarity calculation between different files and ensures the fine-grained and high-quality text mining.

[0074] Specifically, in step S1, the preprocessing includes:

[0075] Step S11, denoising a number of files, including removing punctuation marks and stop words;

[0076] Step S12, performing text segmentation on the denoised files to obtain a number of segmentation sets;

[0077] Step S13, extracting stems from the plurality of word segments to obtain a plurality of vocabulary sets.

[0078] It is understandable that in text mining, removing punctuation marks, stop words, replacing words, and segmenting text in a file is a very important preprocessing step, which facilitates subsequent computer similarity calculation and comparison of several files.

[0079] For example, for a specific text: Xiao Ming got up and prepared to make zongzi with his mother. The table was already full of zongzi materials. His mother showed Xiao Ming the basic steps of making zongzi. His mother skillfully folded the zong leaves into a funnel shape, then scooped a spoonful of glutinous rice into it, then put in a piece of stuffing, and covered it with a layer of glutinous rice.

[0080] After preprocessing, the corresponding vocabulary set is obtained: Xiao Ming gets up and prepares the table for making zongzi, materials for making zongzi, mother, Xiao Ming shows the basic steps of making zongzi, mother is proficient in making zongzi leaves, funnel-shaped glutinous rice filling, glutinous rice.

[0081] The present invention removes noise and redundant information in the text by preprocessing several files, converts the original text data into a form that can be better understood and processed by computers, saves computer resources, and further improves the accuracy of similarity calculation between different files, ensuring the fine-grained and high-quality text mining.

[0082] Specifically, in step S2, corresponding keywords are extracted based on each vocabulary set to obtain corresponding keyword sets, including:

[0083] Step S21, counting the number of occurrences of each word in the plurality of vocabulary sets;

[0084] Step S22, the words whose occurrence times are greater than or equal to a preset number are determined as keywords;

[0085] Step S23, integrating the keywords to obtain a keyword set.

[0086] It is understandable that when a word appears in the plurality of word sets more than or equal to the preset number of times, it indicates that the word is important in the corresponding document and may represent the theme or core content of the corresponding document. In implementation, the preset number of times is 5 to 20 times, preferably, the preset number of times is 10 times, and the preferred value of the preset number of times needs to be determined according to actual conditions, which will not be elaborated here.

[0087] The present invention determines high-frequency keywords by counting the number of occurrences of each word in a number of vocabulary sets, and integrates the high-frequency keywords to obtain a keyword set. The present invention improves the accuracy of keyword selection, and also greatly shortens the time for keyword set construction, improves work efficiency, and is particularly suitable for processing large amounts of text data. It further improves the accuracy of similarity calculations between different files, ensuring the fine-grained and high-quality text mining.

[0088] See also Figure 2 , which is a flow chart of determining a coarse-grained file set based on a plurality of text files of the present invention. Specifically, in the step S3, the text files to be mined are preliminarily screened based on the keyword set of each text file mined to obtain a plurality of coarse-grained text files, forming a coarse-grained file set, including:

[0089] Step S31, obtaining the union of two keyword sets corresponding to any two files among the plurality of files;

[0090] Step S32, creating a vocabulary based on the union, and setting an index for each keyword in the vocabulary;

[0091] Step S33, creating corresponding vector A and vector B based on the two keyword sets and the index;

[0092] Step S34, calculating the cosine similarity between the vector A and the vector B;

[0093] Step S35, judging whether the two files corresponding to the two keyword sets are similar based on the cosine similarity and the set coarse-grained association degree, includes:

[0094] If the cosine similarity is greater than or equal to the coarse-grained association, the two files corresponding to the two keyword sets are similar, and the two files are placed in the same coarse-grained file set;

[0095] If the cosine similarity is less than the coarse-grained correlation, the two files corresponding to the two keyword sets are not similar, and any one of the two files is used as a reference file, and steps S31 to S35 are continued to be performed, and the cosine similarity between the remaining files in the plurality of files and the reference file is continuously judged until the number of judgments reaches a preset number of judgments;

[0096] The preset number of determinations is the number of files that are not included in the coarse-grained file set minus 1.

[0097] It is understandable that the computer randomly selects two keyword sets corresponding to any two files from a number of files, and finds the union of the two keyword sets to create a vocabulary. Each keyword in the vocabulary is assigned a unique index, which will be used to create vector A and vector B. The length of vector A and vector B is the number of common keywords, and the content is the corresponding occurrence position of each keyword in each keyword set in the vocabulary. If the keyword appears in the vocabulary, a non-zero value (such as 1) will be set at the occurrence position, otherwise it will be set to zero, so that each file is represented as a vector related to the vocabulary. The cosine similarity is calculated based on vector A and vector B. The closer the cosine similarity is to 1, the more similar the two files are.

[0098] In implementation, assuming that the two keyword sets are (grapes, kiwi, apple, lychee, pear) and (apple, grapes, kiwi, lychee, banana, orange), the union is (apple, grapes, kiwi, lychee, banana, orange, pear), so as to create a vocabulary, the keywords in the vocabulary are the keywords in the union, vector A is (1, 1, 1, 1, 1, 0), and vector B is (1, 1, 1, 1, 0, 0, 1). The formula for calculating cosine similarity is as follows:

[0099] (1)

[0100] in, is the dot product of vector A and vector B, is the magnitude of vector A, is the magnitude of vector B.

[0101] The cosine similarity between the above vector A (1, 1, 1, 1, 1, 0) and vector B (1, 1, 1, 1, 0, 0, 1) is approximately 0.8165.

[0102] It is understandable that, if the similarity of any two selected files is greater than or equal to the coarse-grained correlation, the two files are included in a coarse-grained file set; if the similarity of any two selected files is less than the coarse-grained correlation, any one of the two files is used as a reference file, and then the similarity is compared with the remaining files in the plurality of files, and the preset number of judgments (number of comparisons) is the number of files in the plurality of files that are not included in the coarse-grained file set minus 1. In implementation, the coarse-grained correlation ranges from 0.3 to 0.6, preferably, the coarse-grained value is 0.6, and the range and preferred value of the coarse-grained value need to be determined according to actual conditions, which will not be repeated here.

[0103] The present invention realizes the judgment of the similarity between different files through the steps of keyword set union extraction, file vectorization, cosine similarity calculation and other steps, and classifies similar files into the same coarse-grained file set, thereby laying a foundation for subsequent fine-grained similarity judgment, further improving the accuracy of similarity calculation between different files, and ensuring the fine-grained and high-quality text mining.

[0104] Specifically, in step S4,

[0105] Dividing any of the plurality of files into a plurality of groups of sentences based on punctuation marks;

[0106] Based on the keyword set corresponding to any one of the files, the keyword replacement is performed on the groups of sentences to obtain corresponding keyword groups.

[0107] Specifically, the plurality of keyword groups further includes: removing repeated keyword groups from the plurality of keyword groups respectively to form a keyword group set.

[0108] It is understandable that any file is grouped based on punctuation marks into several groups of sentences, and based on the previously extracted keyword sets, the several groups of sentences are replaced with corresponding several groups of keyword groups, so that the file is represented as a set of several groups of keyword groups. At the same time, the repeated groups with the same content in several keyword groups are removed to obtain a keyword group set, thereby avoiding text redundancy and facilitating subsequent similarity calculation.

[0109] In a specific embodiment, any document may be divided into groups of sentences based on commas (or spaces equivalent to commas).

[0110] In a specific embodiment, any document may be divided into a number of groups of sentences based on periods or semicolons.

[0111] In another specific embodiment, if there are no punctuation marks in the sentences, any document may be divided into a number of groups of sentences based on the text semantic context in any document.

[0112] The present invention firstly effectively groups the text content of any file based on punctuation marks to form several groups of sentences, which provides convenience for subsequent information processing. Secondly, based on the previously extracted keyword set, several groups of sentences are replaced with corresponding keyword groups, which retains the core information of the file and greatly improves the speed of information retrieval and processing. Furthermore, by removing the same repeated groups in several keyword groups, a keyword group set is obtained, which effectively avoids the redundancy of text information, further improves the accuracy of similarity calculation between different files, and ensures the fine-grained and high-quality text mining.

[0113] Specifically, placing each keyword group into the corresponding keyword group set in sequence based on the order of each keyword group in the corresponding file includes:

[0114] If the keyword group does not exist in the keyword group set, adding the keyword group to the corresponding keyword group set;

[0115] If the elements and the order of the elements in the keyword group are different from the elements and the order of the elements in the keyword group already added in the corresponding keyword group set, then add it to the keyword group set;

[0116] If the elements and the order of the elements in the keyword group are the same as the elements in the keyword group already added in the corresponding keyword group set and the order of the elements is the same, then the keyword group will not be added to the corresponding keyword group set.

[0117] It is understandable that the elements are keywords in the keyword group, and the element order is the keyword order in the keyword group. The keyword group set is initially an empty set, and several keyword groups are sequentially put into the keyword group set based on the order of several keyword groups: if the keyword group does not exist in the keyword group set, the keyword group is added to the keyword group set, and if a keyword group to be added to the keyword group set has the same elements and element order as the keyword group added to the keyword group set last time, the keyword group is no longer added to the keyword group set, otherwise, it is added to avoid duplication of keyword groups in the keyword group set.

[0118] In a specific embodiment, for example, the keyword groups are [I like oranges], [I like bananas and oranges], and [I like oranges], and the keyword group set finally formed is ([I like oranges], [I like bananas and oranges]).

[0119] The present invention sets a keyword group set, and adds the keyword group to the keyword group set in an orderly manner based on the keywords and keyword order in the keyword group, effectively avoiding the duplication of keyword groups, greatly improving data processing efficiency and accuracy, reducing the interference of repeated information, further improving the accuracy of similarity calculation between different files, and ensuring the fine-grained and high-quality text mining.

[0120] See also Figure 3 , which is a flow chart of determining a fine-grained file set based on a coarse-grained file set according to the present invention. Specifically, in step S5, it includes:

[0121] Step S51, respectively determining the longest common keyword group between any two keyword groups in the keyword group sets;

[0122] Step S52, calculating phrase similarity based on the longest common keyword group and the two keyword groups corresponding to the longest common keyword group;

[0123] Step S53, calculating the comprehensive similarity of the two keyword group sets based on the phrase similarity, wherein:

[0124] If the comprehensive similarity is greater than or equal to the fine-grained relevance, the two files corresponding to the two keyword groups are determined to be similar, and the two files corresponding to the two keyword groups are placed in the same fine-grained file set.

[0125] The phrase similarity is calculated based on the longest common keyword group and the two keyword groups corresponding to the longest common keyword group. The phrase similarity calculation formula is as follows:

[0126] (2)

[0127] in, for A keyword group in the collection The number of keywords in the longest common keyword group among all keyword groups in the set, for The number of keywords in the keyword group corresponding to the longest common keyword in the set, for The number of keywords in the keyword group that corresponds to the longest common keyword in the set.

[0128] It can be understood that a common keyword group is a set of identical keywords of any two keyword groups in any two keyword group sets, and the longest common keyword group is a set of keywords that has the most identical keywords between a keyword group in a keyword group set and all keyword groups in another keyword group set.

[0129] For example, any two keyword groups are =([I like cars], [Cars are cheap], [New energy vehicles]), = ([Like cars], [New energy cars]), keyword group set [I like cars] and keyword group The longest common keyword groups of [like cars] and [new energy vehicles] are [like cars] and [cars] respectively. The longest common keyword group with the most identical keywords is [like cars]. is 2, is 3, is 2, is 2 / 3, and the other two similarities are 1 / 3 and 0 respectively. The benchmark set.

[0130] In implementation, the comprehensive similarity is calculated based on the phrase similarity, the retained items of each phrase similarity are determined according to the phrase similarity and a preset separation threshold, and the comprehensive similarity is determined according to the median or average value of each retained item. Preferably, the value of the separation threshold is set between 0.3 and 0.6 to ensure that the retained items of the phrase similarity have a characterization meaning. The redundant keyword groups in the text can be eliminated by the separation threshold to determine the similarity of the core semantics of the two keyword groups.

[0131] For example, if the phrase similarities are 0.35, 0.89, 0.78, 0.5, 0.75, 0.62, and 0.2, respectively, the phrase similarities are sorted from large to small, becoming 0.89, 0.78, 0.75, 0.62, 0.5, 0.35, and 0.2, and the separation threshold is set to 0.5, and the phrase similarities 0.89, 0.78, 0.75, 0.62, and 0.5 greater than or equal to the separation threshold 0.5 are retained, and the median or average is calculated based on the retained five phrase similarities, and the median or average is used to represent the comprehensive similarity. The preferred value of the separation threshold needs to be selected according to the actual situation, which will not be repeated here.

[0132] It is understandable that if the comprehensive similarity is greater than or equal to the fine-grained correlation, the two files corresponding to the two keyword group sets are similar, and the two files are placed in the same fine-grained file set, and then the similarity judgment is performed to form the final fine-grained file sets. In implementation, the value range of the comprehensive similarity is 0.8 to 0.95, and preferably the comprehensive similarity value is 0.9. The value range and preferred value of the comprehensive similarity need to be determined based on actual conditions, which will not be repeated here.

[0133] The present invention determines the relevance between two text files by respectively determining the longest common keyword group between each keyword group in any two keyword group sets, and then calculates the phrase similarity based on the determined longest common keyword group and the two keyword groups corresponding to the longest common keyword group, which not only improves the scientificity and rationality of the calculation process, but also makes the subsequent comprehensive similarity calculation results closer to the similarity of the actual content. Furthermore, the present invention calculates the comprehensive similarity based on a number of phrase similarities and corresponding weights, realizes the classification of text files at a fine-grained level, and further ensures the fine-grained and high-quality text mining.

[0134] See also Figure 4 As shown, it is a flow chart of the present invention for judging the logical relationship between any two files in a single fine-grained file set. Specifically, judging the logical relationship between any two files in a single fine-grained file set includes:

[0135] Step S61, determining two text files in a single fine-grained file set for logical relationship judgment;

[0136] Step S62, determining the union of the keyword sets corresponding to the two text files;

[0137] Step S63, respectively determining the positions of the keywords in the union in any two text files;

[0138] Step S64: determining the logical relationship between the two text files based on the positions.

[0139] It is understandable that by selecting two text files from the fine-grained file set, the union of the keyword sets corresponding to the two text files is found, and for each keyword in the union, its position (such as paragraph, sentence, etc.) in the two text files is determined respectively. In implementation, a text matching algorithm or NLP technology can be used to locate the position of the keyword in the text. The specific position search method can be selected according to the actual situation and will not be described in detail here.

[0140] It can be understood that the logical relationship between the two text files can be determined based on the positions of the keywords in the two text files, and the logical relationship may include: a parallel relationship, a reference relationship, a time sequence relationship, and the like.

[0141] In a specific embodiment, if the keywords in the union of the keyword sets appear throughout text file A, and the keywords in the union of the keyword sets only appear in the middle of text file B, then the logical relationship between the text files can be explained as text file B referring to text file A.

[0142] The present invention effectively improves the accuracy and efficiency of judging the logical relationship between files by determining two text files in a single fine-grained file set to be judged, and determining the union of the keyword sets corresponding to the two files. This avoids misjudgment of the logical relationship caused by inaccurate keyword selection. Then, based on the positions of the keywords in the union in any two text files, the logical relationship between the two text files is determined to ensure the objectivity and scientificity of the logical relationship judgment.

[0143] Specifically, in step S1, the preprocessing further includes: performing hypernym replacement on the plurality of word segments.

[0144] It can be understood that by replacing the hypernyms of the segmented word set, it is convenient to better identify the category or theme of the text file in the text file classification or theme identification task, and using hypernyms to replace the words in the segmented word set can better capture the semantic relationship in the text file and improve the accuracy and effectiveness of keyword extraction.

[0145] Preferably, the preprocessing further includes, when extracting keywords from the plurality of word segments, retaining text information such as "not", "not" and similar words, so as to further improve the accuracy of similarity calculation between different text files.

[0146] In order to further illustrate the present invention, the standard intelligent association method based on text mining provided by the present invention is described in detail below, but it should not be understood as limiting the protection scope of the present invention.

[0147] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A standard intelligent association method based on text mining, characterized in that: include: Step S1, preprocessing each text file to obtain each vocabulary set of the corresponding text file, wherein the words in the vocabulary set are sorted in order according to the text content; Step S2, extracting corresponding keywords based on each of the vocabulary sets to obtain corresponding keyword sets; Step S3, preliminarily screening the text files to be mined based on the keyword set of each text file mined to obtain a number of coarse-grained text files to form a coarse-grained file set; Step S4, performing pointing processing on each keyword of each keyword set in the coarse-grained document set to form each keyword group set; Step S5, calculating the similarities between the plurality of keyword sets corresponding to the coarse-grained document set based on each of the keyword group sets, to obtain a fine-grained document set; Step S6, determining the associated logic map of each text file in the fine-grained file set according to the attributes of the keyword set corresponding to each text file in the fine-grained file set; Among them, in step S5, respectively determining the longest common keyword group between any two keyword groups in the keyword group sets; Calculate phrase similarity based on the longest common keyword group and the two keyword groups corresponding to the longest common keyword group; The comprehensive similarity of two keyword group sets is calculated based on the phrase similarity, wherein: Calculating the comprehensive similarity based on the phrase similarity, determining the retained items of each phrase similarity according to the phrase similarity and a preset separation threshold, and determining the comprehensive similarity according to the median or average of each retained item; If the comprehensive similarity is greater than or equal to the fine-grained relevance, the two files corresponding to the two keyword groups are determined to be similar, and the two files corresponding to the two keyword groups are placed into the same fine-grained file set; In step S6, a logical relationship is determined between any two files in a single fine-grained file set, including: Determine two text files in a single fine-grained file set for logical relationship judgment; Determine the union of the keyword sets corresponding to the two text files; Respectively determine the positions of the keywords in the union in any two text files; A logical relationship between the two text files is determined based on the positions.

2. The standard intelligent association method based on text mining according to claim 1 is characterized in that: In step S1, the preprocessing includes: Step S11, denoising a number of files, including removing punctuation marks and stop words; Step S12, performing text segmentation on the denoised files to obtain a number of segmentation sets; Step S13, extracting stems from the plurality of word segments to obtain a plurality of vocabulary sets.

3. The standard intelligent association method based on text mining according to claim 1 is characterized in that: In the step S2, corresponding keywords are extracted based on each of the vocabulary sets to obtain corresponding keyword sets, including: Step S21, counting the number of occurrences of each word in the plurality of vocabulary sets; Step S22, the words whose occurrence times are greater than or equal to a preset number are determined as keywords; Step S23, integrating the keywords to obtain a keyword set.

4. The standard intelligent association method based on text mining according to claim 1 is characterized in that: In step S3, the text files to be mined are preliminarily screened based on the keyword set of each text file obtained through text mining to obtain a number of coarse-grained text files, forming a coarse-grained file set, including: Step S31, obtaining the union of two keyword sets corresponding to any two files among the plurality of files; Step S32, creating a vocabulary based on the union, and setting an index for each keyword in the vocabulary; Step S33, creating corresponding vector A and vector B based on the two keyword sets and the index; Step S34, calculating the cosine similarity between the vector A and the vector B; Step S35, judging whether the two files corresponding to the two keyword sets are similar based on the cosine similarity and the set coarse-grained association degree, includes: If the cosine similarity is greater than or equal to the coarse-grained association, the two files corresponding to the two keyword sets are similar, and the two files are placed in the same coarse-grained file set; If the cosine similarity is less than the coarse-grained correlation, the two files corresponding to the two keyword sets are not similar, and any one of the two files is used as a reference file, and steps S31 to S35 are continued to be performed, and the cosine similarity between the remaining files in the plurality of files and the reference file is continuously judged until the number of judgments reaches a preset number of judgments; The preset number of determinations is the number of files that are not included in the coarse-grained file set minus 1.

5. The standard intelligent association method based on text mining according to claim 1 is characterized in that: In step S4, it includes: Dividing any of the plurality of files into a plurality of groups of sentences based on punctuation marks; Based on the keyword set corresponding to any one of the files, the keyword replacement is performed on the groups of sentences to obtain corresponding keyword groups.

6. The standard intelligent association method based on text mining according to claim 5 is characterized in that: The plurality of keyword groups further comprises: removing repeated keyword groups from the plurality of keyword groups respectively to form a keyword group set.

7. The standard intelligent association method based on text mining according to claim 6 is characterized in that: Putting each keyword group into the corresponding keyword group set in sequence based on the order of each keyword group in the corresponding file includes: If the keyword group does not exist in the keyword group set, adding the keyword group to the corresponding keyword group set; If the elements and the order of the elements in the keyword group are different from the elements and the order of the elements in the keyword group already added in the corresponding keyword group set, then add it to the keyword group set; If the elements and the order of the elements in the keyword group are the same as the elements in the keyword group already added in the corresponding keyword group set and the order of the elements is the same, then the keyword group will not be added to the corresponding keyword group set.

8. The standard intelligent association method based on text mining according to claim 1 is characterized in that: In step S1, the preprocessing further includes: performing hypernym replacement on the plurality of word segments.

Images