Keyword extraction method, system, device and storage medium

Abstract

The application discloses a keyword extraction method, system, device and storage medium. The method comprises the following steps: obtaining a sensitive word group list, removing stop words and splicing words in each sensitive word group, and obtaining preprocessed text; based on the preprocessed text, a word vector is constructed, and a mapping relationship between a word group in the preprocessed text and the splicing words is established; an initial cluster center vector is selected in the word vector, a Hamming distance between a remaining word vector and the initial cluster center vector is calculated, the word vector is divided into clusters according to the Hamming distance, and a preliminarily divided keyword cluster is obtained; based on the word vector and the preliminarily divided keyword cluster, an adaptive clustering algorithm is adopted to obtain a similar word vector cluster; and the similar word vector cluster is spliced with the splicing words according to the mapping relationship, so that a keyword list is generated; and it is ensured that the obtained keywords can accurately reflect the characteristics of the text content.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and in particular to a keyword extraction method, system, device and storage medium. BACKGROUND

[0002] With the continuous improvement of information retrieval technology, the natural language processing algorithm can filter out irrelevant information according to the text provided by the user, and extract key information to provide more accurate retrieval results.

[0003] For features that can be represented by numerical vectors, clustering algorithms can be applied to clustering to obtain clustering results. Clustering is often disturbed by a lot of redundant information, which has a great impact on the clustering prediction model, resulting in inaccurate results. For example, text semantic ambiguity can affect the accuracy of the algorithm, and the diversity of natural language, such as different cultural backgrounds and regional language differences, can also affect the algorithm effect. In related technologies, the text is usually filtered by stop words before clustering processing, such as the patent application document with publication number CN111985228A, which filters the stop words of the text to be processed, and then uses a clustering algorithm for clustering processing. However, the stop word filtering method has the following defects: important information is easily lost, the accuracy of the search results is affected, it is relatively difficult to process complex text, and it is difficult to process multiple languages.

[0004] In the patent application document with publication number CN114969330A, an lstm model is used to extract effective information from long text data, and an encoder and a decoder of a transformer layer are used to further extract effective features to obtain effective text vectors, and then the text vectors are clustered. However, the use of the lstm model has high actual deployment complexity, large calculation amount, time-consuming calculation, and parallel processing disadvantages.

[0005] In addition, in the traditional keyword simplification algorithm based on clustering, the selection of the initial clustering center has a great influence on the algorithm result. If the initial clustering center is not selected properly, the algorithm may converge to a local optimal solution, rather than a global optimal solution, and the determination of the number of clusters is sensitive. If the number of clusters is not selected properly, the quality of the simplification result may be poor. SUMMARY

[0006] The technical problem to be solved by the present application is how to improve the accuracy of keyword extraction.

[0007] The present application solves the above technical problems by the following technical means:

[0008] In a first aspect, the present application provides a keyword extraction method, which comprises:

[0009] obtaining a sensitive word group list and removing stop words and concatenation words in each sensitive word group to obtain preprocessed text, wherein the concatenation word is a word that affects the uniqueness of the meaning of the sensitive word group;

[0010] Based on the preprocessed text, a word vector is constructed, and a mapping relationship between the word group in the preprocessed text and the concatenation word is established;

[0011] Selecting an initial cluster center vector in the word vector, calculating the Hamming distance between the remaining word vector and the initial cluster center vector, and performing cluster division on the word vector according to the Hamming distance to obtain a preliminary divided keyword cluster;

[0012] Based on the word vector and the preliminary divided keyword cluster, an adaptive clustering algorithm is used to obtain a similar word vector cluster;

[0013] According to the mapping relationship, the similar word vector cluster is concatenated with the concatenation word to generate a keyword list.

[0014] Further, the obtaining of the sensitive word group list and the removal of the stop words and the concatenation words in each sensitive word group to obtain the preprocessed text comprises

[0015] Each of the sensitive word groups is subjected to stop word filtering processing, and the removed stop words are added to a stop word list, wherein the stop words include general vocabulary;

[0016] The concatenation words in each of the sensitive word groups are extracted and added to a concatenation word list in an association array, wherein the concatenation word is a word with a frequency higher than a set number of times and affecting the uniqueness of the meaning of the sensitive word group;

[0017] The text after removing the stop words and the concatenation words is used as the preprocessed text.

[0018] Further, the construction of the word vector based on the preprocessed text and the establishment of the mapping relationship between the word group in the preprocessed text and the concatenation word comprises:

[0019] The preprocessed text is subjected to word segmentation processing to obtain a plurality of word groups;

[0020] A bag-of-words model is constructed based on the plurality of word groups, and the plurality of word groups are converted into a word frequency-based word vector, which is expressed by the following formula:

[0021] v(x)=[f1(x),f2(x),f3(x),......,f n (x)]

[0022] In the formula, v(x) represents a binary word vector; f i(x) represents converting each input word group into a word vector representation feature, i = 1, 2, 3, …, n;

[0023] A mapping relationship between a word group in the preprocessed text and the spliced word is established.

[0024] Further, the initial cluster center vector is selected in the word vector, the Hamming distance between the remaining word vector and the initial cluster center vector is calculated, and the word vector is clustered according to the Hamming distance to obtain a preliminary divided keyword cluster, including:

[0025] An initial cluster center vector is randomly selected in the word vector, and the Hamming distance between each remaining word vector and the initial cluster center vector is calculated, and the calculation formula is:

[0026]

[0027] In the formula, w represents a weighted Hamming distance; w i w represents the weight of the i-th word vector representation feature, N represents the number of all word entries, n i f represents the number of documents containing the i-th word; f i (x) and f i (y) respectively represent the feature values of texts x and y at the i-th position, R represents a constant and R>1, k represents the length of the binary word vector of the texts x and y, and n represents the length of the current word entry text input;

[0028] The word vectors with a Hamming distance greater than a set threshold value are clustered into a cluster.

[0029] Further, based on the word vector and the preliminary divided keyword cluster, an adaptive clustering algorithm is used to obtain a similar word vector cluster, including:

[0030] The distance between each word vector and all preliminary divided keyword clusters is calculated, and the word vector is classified into the nearest keyword cluster;

[0031] For each keyword cluster, the cluster center is recalculated, and the number of clusters is adjusted according to the change of the cluster center;

[0032] When the maximum number of iterations is reached or the cluster center is unchanged, the similar word vector cluster is output.

[0033] Further, after obtaining the similar word vector cluster, the method further includes:

[0034] The similar word vector cluster is screened, and the keywords in the M radius hit in the same cluster are output, and M is a constant and M>1;

[0035] Accordingly, the similar word vector cluster is spliced with the splicing word according to the mapping relationship to generate a keyword list, specifically as follows:

[0036] The keyword is spliced with the splicing word according to the mapping relationship to generate a keyword list.

[0037] Further, the similar word vector cluster is spliced with the splicing word according to the mapping relationship to generate a keyword list, including:

[0038] An association array is established, the splicing word is stored in a splicing word list in the association array, and an index value is associated with the mapping relationship;

[0039] The corresponding splicing word in the association array is found by indexing, and the similar word vector cluster is spliced with the splicing word to generate a keyword list.

[0040] In a second aspect, the present application further provides a keyword extraction system, which comprises:

[0041] A preprocessing module is configured to obtain a sensitive word group list and remove stop words and splicing words in each sensitive word group to obtain preprocessed text, wherein the splicing word is a word that affects the uniqueness of the meaning of the sensitive word group;

[0042] A word vector construction module is configured to construct a word vector based on the preprocessed text, and establish a mapping relationship between a word group in the preprocessed text and the splicing word;

[0043] A preliminary clustering module is configured to select an initial cluster center vector in the word vector, calculate a Hamming distance between a remaining word vector and the initial cluster center vector, divide the word vector into clusters according to the Hamming distance, and obtain a preliminarily divided keyword cluster;

[0044] An adaptive clustering module is configured to obtain a similar word vector cluster by using an adaptive clustering algorithm based on the word vector and the preliminarily divided keyword cluster;

[0045] A splicing module is configured to splice the similar word vector cluster with the splicing word according to the mapping relationship to generate a keyword list.

[0046] In a third aspect, the present application further provides a keyword extraction device, which comprises a memory and a processor; wherein the processor runs a program corresponding to executable program code stored in the memory by reading the executable program code, so as to implement the keyword extraction method as described above.

[0047] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the keyword extraction method.

[0048] The present application has the advantages of:

[0049] (1) The present application firstly pre-processes a batch of sensitive word group list, removes stop words and concatenation words, and obtains a pre-processed pure text, which does not contain redundant information, so as to reduce resource waste caused by repeated search entries and facilitate subsequent calculation; then, the pre-processed text is segmented, and each word is converted into a feature vector, and then the clustering algorithm is used to cluster the feature vectors, and the mapping relationship between the segmented word groups in the pre-processed text and the concatenation words is established, finally, the clustering results and the concatenation word list are combined as the text output, so as to improve the accuracy of data processing and avoid information loss caused by algorithm processing; and the weighted Hamming distance is used for text processing, which increases the distance between different texts and reduces the distance between similar texts, and is used for accurate text clustering, so as to ensure that the obtained keywords can accurately reflect the characteristics of the text content.

[0050] (2) The present application can adjust the concatenation word list and the stop word list according to the specific application scene to obtain more suitable results.

[0051] (3) The present application uses the adaptive clustering algorithm to dynamically adjust the clustering parameters according to the characteristics of the data, so as to adapt to the distribution characteristics of the data, and to ensure that the obtained keywords can accurately reflect the characteristics of the text content.

[0052] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0053] Fig. 1 is a flowchart of a keyword extraction method according to an embodiment of the present application;

[0054] Fig. 2 is a flowchart of a keyword extraction method according to an embodiment of the present application;

[0055] Fig. 3 is a structural diagram of a keyword extraction system according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0057] As shown in FIG. 1, the first embodiment of the present application proposes a keyword extraction method, which comprises the following steps: Figs. 1-2

[0058] S10, obtaining a sensitive word group list and removing stop words and concatenation words in each sensitive word group to obtain preprocessed text, wherein the concatenation word is a word affecting the uniqueness of the meaning of the sensitive word group;

[0059] It should be noted that the initial text of the input end in the present embodiment is composed of a plurality of sensitive word groups rather than a complete sentence, so that only single words can be reduced but not deleted in the keyword extraction process, and the removal of stop words and concatenation words in each sensitive word group can reduce resource waste caused by repeated word entries and facilitate subsequent calculation.

[0060] S20, based on the preprocessed text, constructing a word vector and establishing a mapping relationship between the word groups in the preprocessed text and the concatenation words;

[0061] It should be noted that the concatenation words can be used as elements of an associated array, and the array subscripts can be marked as index values by establishing an associated array.

[0062] It should be understood that the concatenation words and stop words are self-defined and can be maintained, and the specific concatenation words and stop words can be set by those skilled in the art according to actual application requirements.

[0063] S30, selecting an initial cluster center vector in the word vector, calculating the Hamming distance between the remaining word vector and the initial cluster center vector, and performing cluster division on the word vector according to the Hamming distance to obtain a preliminary divided keyword cluster;

[0064] S40, based on the word vector and the preliminary divided keyword cluster, using an adaptive clustering algorithm to obtain a similar word vector cluster;

[0065] S50, according to the mapping relationship, concatenating the similar word vector cluster with the concatenation word to generate a keyword list.

[0066] ​The embodiment carries out word segmentation on the preprocessed text, converts each word into a feature vector, and then uses a clustering algorithm to cluster the feature vectors, and establishes a mapping relationship between the segmented word groups and the spliced words in the preprocessed text. Finally, the clustering results and the spliced word list are combined as the text output, so as to improve the accuracy of data processing and avoid information loss caused by algorithm processing; and the weighted Hamming distance is used for text processing, which increases the distance between different texts and reduces the distance between similar texts, and is used for accurate text clustering, so as to ensure that the obtained keywords can accurately reflect the characteristics of the text content.

[0067] In an embodiment, the step S10 of obtaining a sensitive word group list and removing stop words and spliced words in each sensitive word group to obtain a preprocessed text specifically includes:

[0068] S11, performing stop word filtering processing on each of the sensitive word groups, and adding the removed stop words to a stop word list, the stop words including general words;

[0069] It should be noted that the input sensitive word groups generally have similar or repeated meanings, and some general words have little effect on subsequent business and interfere with actual business execution, such as “group”, “company” and the like, which are not characteristic and unique general words. The embodiment removes these general words from the sensitive word groups and adds them to the stop word list.

[0070] S12, extracting the spliced words in each of the sensitive word groups and adding them to a spliced word list in an association array, the spliced words being words with a frequency higher than a set number and affecting the uniqueness of the meaning of the sensitive word groups;

[0071] It should be noted that some words in the sensitive word groups have a high repetition frequency, but once removed, they will affect the uniqueness of the meaning of the original text, such as “.idss-cn”, “@”, “.com”, “security” and the like, which can be used as spliced words and added to the spliced word list.

[0072] S13, the text after removing the stop words and the spliced words is used as the preprocessed text.

[0073] The embodiment improves the accuracy of the word vector used for clustering processing by preprocessing the sensitive word groups to be processed, thereby improving the accuracy of keyword extraction, and reducing resource waste caused by repeated search words and improving keyword extraction efficiency.

[0074] It should be understood that those skilled in the art can also adjust the spliced word list and the stop word list according to specific application scenarios to obtain more suitable results.

[0075] In an embodiment, the step S20 of constructing a word vector based on the preprocessed text and establishing a mapping relationship between the word groups in the preprocessed text and the concatenated words comprises the following steps:

[0076] S21, performing word segmentation processing on the preprocessed text to obtain a plurality of word groups;

[0077] S22, constructing a bag-of-words model based on the plurality of word groups to convert the plurality of word groups into a word frequency-based word vector, which is expressed by a formula:

[0078] v(x) = [f1(x), f2(x), f3(x),..., fn(x)] n

[0079] In the formula, v(x) represents a binary word vector; f i (x) represents converting each input word group into a word vector representation feature, i = 1, 2, 3,..., n;

[0080] S23, establishing a mapping relationship between the word groups in the preprocessed text and the concatenated words.

[0081] For the preprocessed text, the embodiment first performs word segmentation processing to divide it into word groups, and can establish a mapping relationship between the word groups and the concatenated words. Then, a bag-of-words model is constructed to convert the word groups into a word frequency-based word vector for representing text features, which is converted into a form that can be processed by a computer for subsequent machine learning tasks.

[0082] In an embodiment, the step S30 of selecting an initial cluster center vector in the word vector, calculating the Hamming distance between the remaining word vectors and the initial cluster center vector, and performing cluster division on the word vectors according to the Hamming distance to obtain a preliminary divided keyword cluster comprises the following steps:

[0083] S31, randomly selecting an initial cluster center vector in the word vector and calculating the Hamming distance between each remaining word vector and the initial cluster center vector, which is calculated by a formula:

[0084]

[0085] In the formula, w i represents the weight of the i-th word vector representation feature, N represents the number of all word entries, and n i represents the number of documents containing the i-th word; f i (x) and f i ​(y) respectively represent the feature values of the texts x and y at the i-th position; R represents a constant and R > 1, used to increase the Hamming distance between different texts; k represents the length of the binary word vector of the texts x and y, and n represents the length of the current word text input;

[0086] S32, the word vectors with a Hamming distance greater than a set threshold value M are clustered into a cluster.

[0087] It should be noted that, in this embodiment, an initial cluster center is randomly selected first, and after obtaining the word vector, the similarity between subsequent text samples is calculated using a weighted Hamming distance, and the feature f i (x) is calculated, and the clustering basis is obtained. M is set as the cluster distance radius (M can be defined by the user), and if there are two or more text entries containing multiple same initial characters, then the text entries are recorded as f i (x) {.*}, where {.*} represents a wildcard, representing one or more of all characters. Then, the text entries [f i (x) {.*}] can be clustered into a cluster.

[0088] In this embodiment, the weighted Hamming distance is used for text processing, which increases the distance between different texts and reduces the distance between similar texts, and is used for accurate text clustering.

[0089] In an embodiment, the step S40 of obtaining similar word vector clusters based on the word vector and the initially divided keyword clusters using an adaptive clustering algorithm specifically includes the following steps:

[0090] S41, the distance between each word vector and all initially divided keyword clusters is calculated, and the word vector is assigned to the nearest keyword cluster;

[0091] S42, for each keyword cluster, the cluster center is recalculated, and the number of clusters is adjusted according to the change of the cluster center;

[0092] S43, when the maximum number of iterations is reached or the cluster center is unchanged, the similar word vector clusters are output.

[0093] It should be noted that the adaptive clustering algorithm used in this embodiment can dynamically adjust the clustering parameters according to the characteristics of the data, and does not require the number of clusters in the final output to be a fixed value. For example, according to different input data and different businesses, the number and size of the clusters are dynamically adjusted to adapt to the distribution characteristics of the data.

[0094] In an embodiment, after obtaining the similar word vector clusters, the method further includes the following steps:

[0095] Screening the similar word vector clusters outputs keywords in the same cluster that hit m times, where m is a constant and m>1.

[0096] Accordingly, the similar word vector clusters are spliced with the splicing words according to the mapping relationship to generate a keyword list, specifically as follows:

[0097] The keywords are spliced with the splicing words according to the mapping relationship to generate a keyword list.

[0098] It should be noted that the present embodiment screens the clusters obtained through preliminary clustering in combination with actual business and outputs keywords that hit multiple times in the same cluster. For example, if the keywords in the cluster include “## cloud network security” and “## power security”, “##{.*} security” is outputted.

[0099] It is worth noting that isolated clusters in the clustering process are not considered as noise or outliers and are not excluded from the clustering results. The isolated clusters are retained and outputted. In actual business, structured text is not required to be inputted, and it is not required that the inputted text is related to the company name, location, and main business each time.

[0100] For example, the inputted sensitive word group is [“## group limited company”, “## holding”, “## wisdom technology”, “## technology”, “@upperhills.com”, “@##jituan.com”, “##keji.com”, “@syland.com”],

[0101] The stop words are removed and the splicing words are removed. The stop words are “group” and “limited company”, and the splicing words are “@” and “.com”. The preprocessed text is clustered and processed to output:

[0102] ##j{.*}

[0103] @upperhills.com

[0104] @##j{.*}.com

[0105] @syland.com

[0106] It can be seen that @syland.com is an isolated cluster output and is not considered as noise or outliers.

[0107] In an embodiment, the step S50 of splicing the similar word vector clusters with the splicing words according to the mapping relationship to generate a keyword list specifically includes the following steps:

[0108] S51, an association array is established, the splicing word is stored in a splicing word list in the association array, and an index value is associated with the mapping;

[0109] S52, a corresponding splicing word in the association array is found through the index, and the similar word vector cluster is spliced with the splicing word to generate a keyword list.

[0110] For example, the input sensitive word group is ["Beijing##information", "Shanghai##information", "##data security", "##network security"], the stop word list is ["Beijing", "Shanghai"], and the splicing word list is ["security"], and the output obtained after the above processing should be ["##information", "##{.*}security"], and {.*} represents a wildcard, representing one or more characters.

[0111] It should be noted that the embodiment establishes the mapping relationship between the text word and the splicing word list to improve the accuracy of data processing and avoid information loss caused by algorithm processing.

[0112] As shown in Fig. 3 The second embodiment of the present application provides a keyword extraction system, which comprises:

[0113] A preprocessing module 10 is configured to obtain a sensitive word group list and remove stop words and splicing words in each sensitive word group to obtain preprocessed text, wherein the splicing word is a word that affects the uniqueness of the meaning of the sensitive word group.

[0114] A word vector construction module 20 is configured to construct a word vector based on the preprocessed text and establish a mapping relationship between a word group in the preprocessed text and the splicing word.

[0115] A preliminary clustering module 30 is configured to select an initial cluster center vector in the word vector, calculate a Hamming distance between the remaining word vector and the initial cluster center vector, perform cluster division on the word vector according to the Hamming distance, and obtain a preliminarily divided keyword cluster.

[0116] An adaptive clustering module 40 is configured to obtain a similar word vector cluster by using an adaptive clustering algorithm based on the word vector and the preliminarily divided keyword cluster.

[0117] A splicing module 50 is configured to splice the similar word vector cluster with the splicing word according to the mapping relationship to generate a keyword list.

[0118] The embodiment preprocesses a batch of sensitive word group lists, removes stop words and concatenated words, obtains a preprocessed pure text, the pure text does not contain redundant information, reduces resource waste caused by repeated search entries, and is beneficial to subsequent calculation; then the preprocessed text is segmented, each word is converted into a feature vector, and then a clustering algorithm is used to cluster the feature vectors, and a mapping relationship between the segmented word groups in the preprocessed text and the concatenated words is established, finally the clustering results and the concatenated word list are combined as the text output, so as to improve the accuracy of data processing and avoid information loss caused by algorithm processing; and a weighted Hamming distance is used for text processing, which increases the distance between different texts and reduces the distance between similar texts, for accurate text clustering, so as to ensure that the obtained keywords can accurately reflect the characteristics of the text content.

[0119] In an embodiment, the preprocessing module 10 comprises:

[0120] A stop word filtering unit is configured to perform stop word filtering processing on each sensitive word group, and add the removed stop words to a stop word list, wherein the stop words include general vocabulary;

[0121] A concatenated word extraction unit is configured to extract concatenated words in each sensitive word group and add the concatenated words to a concatenated word list in an association array, wherein the concatenated words are words with a frequency higher than a set number of times and affecting the uniqueness of the meaning of the sensitive word group;

[0122] A preprocessed text determination unit is configured to determine the text after removing the stop words and the concatenated words as a preprocessed text.

[0123] In an embodiment, the word vector construction module 20 comprises:

[0124] A segmentation unit is configured to perform segmentation processing on the preprocessed text to obtain a plurality of word groups;

[0125] A conversion unit is configured to construct a bag-of-words model based on the plurality of word groups, and convert the plurality of word groups into a word frequency-based word vector, which is expressed by a formula as follows:

[0126] v(x) = [f1(x), f2(x), f3(x),..., fn(x)] n (x)

[0127] In the formula, v(x) represents a binary word vector; f i (x) represents converting each input word group into a word vector representation feature, i = 1, 2, 3,..., n;

[0128] A mapping relationship establishment unit is configured to establish a mapping relationship between the word groups in the preprocessed text and the concatenated words.

[0129] In an embodiment, the preliminary clustering module 30 specifically comprises:

[0130] The first distance calculation unit is configured to randomly select an initial cluster center vector in the word vector, and calculate the Hamming distance between each remaining word vector and the initial cluster center vector, with the formula being:

[0131]

[0132] In the formula, w represents the weighted Hamming distance; w i represents the weight of the feature represented by the i-th word vector, N represents the number of all word entries, n i represents the number of documents containing the i-th word; f i (x) and f i (y) represent the feature values of the texts x and y at the i-th position, respectively; R represents a constant and R>1, k represents the length of the binary word vector of the texts x and y, and n represents the length of the current word entry text input;

[0133] The preliminary clustering unit is configured to cluster the word vectors with a Hamming distance greater than a set threshold into a cluster.

[0134] In an embodiment, the adaptive clustering module 40 specifically comprises:

[0135] The second distance calculation unit is configured to calculate the distance between each word vector and all the key word clusters preliminarily divided, and to assign the word vector to the key word cluster with the closest distance;

[0136] The adaptive clustering unit is configured to, for each key word cluster, recalculate the cluster center thereof, and adjust the number of clusters according to the change of the cluster center;

[0137] The output unit is configured to output the similar word vector clusters when a maximum number of iterations is reached or the cluster center is unchanged.

[0138] In an embodiment, the system further comprises:

[0139] The screening module is configured to screen the similar word vector clusters, and output the key words in the M radius hits in the same cluster, M being a constant and M>1;

[0140] Correspondingly, the splicing module 50 is specifically configured to: splice the key words and the splicing words according to the mapping relationship, to generate a key word list.

[0141] In an embodiment, the splicing module 50 is specifically configured to:

[0142] establishing an association array, storing the concatenated words in a concatenated word list in the association array, and associating an index value with the mapping;

[0143] finding the corresponding concatenated word in the association array through the index, and concatenating the similar word vector cluster with the concatenated word to generate a keyword list.

[0144] It should be noted that other embodiments of the keyword extraction system or the implementation method of the present application can refer to the above method embodiments, which will not be repeated here.

[0145] In addition, the third embodiment of the present application further provides a keyword extraction device, which comprises a memory and a processor; wherein the processor runs a program corresponding to the executable program code stored in the memory by reading the executable program code, so as to realize the keyword extraction method as described in the first embodiment.

[0146] In addition, the fourth embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the keyword extraction method as described in the first embodiment.

[0147] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logical functions, which can be embodied in any computer readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instruction execution systems, apparatus or devices. For the purpose of this specification, "computer readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices. More specific examples (non-exhaustive list) of computer readable medium include the following: electrical connections having one or more wires (electronic devices), portable computer disk boxes (magnetic devices), random access memories (RAM), read only memories (ROM), erasable programmable read only memories (EPROM or flash memories), fiber optic devices, and portable compact disc read only memories (CDROM). In addition, the computer readable medium can even be paper or other suitable medium on which the program can be printed, because the program can be obtained electronically, for example, by optical scanning of the paper or other medium, followed by editing, interpreting or processing as necessary, and then stored in the computer memory, if necessary.

[0148] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the description of the specification, the illustrative description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0149] In addition, the terms "first", "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0150] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and cannot be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. A keyword extraction method characterized by, The method comprises: obtaining a sensitive word group list and removing stop words and splicing words in each sensitive word group to obtain preprocessed text, wherein the splicing word is a word that affects the uniqueness of the meaning of the sensitive word group; based on the preprocessed text, constructing a word vector, and establishing a mapping relationship between the word group in the preprocessed text and the splicing word; selecting an initial cluster center vector in the word vector, calculating the Hamming distance between the remaining word vectors and the initial cluster center vector, and performing cluster division on the word vector according to the Hamming distance to obtain a preliminarily divided keyword cluster; based on the word vector and the preliminarily divided keyword cluster, an adaptive clustering algorithm is used to obtain a similar word vector cluster, including calculating the distance between each word vector and all preliminarily divided keyword clusters, and assigning the word vector to the nearest keyword cluster; for each keyword cluster, the cluster center is recalculated, and the number of clusters is adjusted according to the change of the cluster center; when the maximum number of iterations or the cluster center is constant, the similar word vector cluster is output; according to the mapping relationship, the similar word vector cluster and the splicing word are spliced to generate a keyword list.

2. The keyword extraction method of claim 1, wherein, The method comprises: performing stop word filtering processing on each sensitive word group, and adding the removed stop words to a stop word list, wherein the stop words include general vocabulary; extracting the splicing words in each sensitive word group and adding them to the splicing word list in the association array, wherein the splicing word is a word with a frequency higher than a set number and affecting the uniqueness of the meaning of the sensitive word group; the text after removing the stop words and the splicing words is used as the preprocessed text.

3. The keyword extraction method of claim 1, wherein, The method comprises: performing word segmentation processing on the preprocessed text to obtain a plurality of word groups; constructing a bag-of-words model based on the plurality of word groups to convert the plurality of word groups into a word frequency-based word vector, and the formula is: where v(x) represents a binary word vector; f i (x) represents converting each input word group into a word vector representation feature, i = 1, 2, 3,..., n; establishing a mapping relationship between the word group in the preprocessed text and the splicing word.

4. The keyword extraction method of claim 1, wherein, The method comprises: randomly selecting an initial cluster center vector in the word vector, and calculating the Hamming distance between each remaining word vector and the initial cluster center vector, and the formula is: wherein, denotes the weighted Hamming distance; denotes the i-th word vector representation feature weight, N denotes the number of all word entries, denotes the number of documents containing the i-th word; denotes the feature values of the texts x and y at the i-th position, respectively; R denotes a constant and R > 1, k denotes the length of the binary word vector of the texts x and y, and n denotes the current word entry text length of the input. clustering the word vectors with a Hamming distance greater than a set threshold into a cluster.

5. The keyword extraction method of claim 1, wherein, After obtaining the similar word vector cluster, the method further comprises: screening the similar word vector cluster, and outputting the keywords in the M radius in the same cluster, wherein M is a constant and M>1; correspondingly, the method comprises: splicing the keywords and the splicing words according to the mapping relationship to generate a keyword list.

6. The keyword extracting method of claim 1, wherein, The similar word vector cluster is spliced with the splicing word according to the mapping relationship to generate a keyword list, including: An association array is established, the splicing word is stored in a splicing word list in the association array, and an index value is associated with the mapping relationship; The corresponding splicing word in the association array is found by indexing, and the similar word vector cluster is spliced with the splicing word to generate a keyword list.

7. A keyword extraction system characterized by, The system comprises: A preprocessing module is configured to obtain a sensitive word group list and remove stop words and splicing words in each sensitive word group to obtain preprocessed text, wherein the splicing word is a word that affects the uniqueness of the meaning of the sensitive word group; A word vector construction module is configured to construct a word vector based on the preprocessed text and establish a mapping relationship between a word group in the preprocessed text and the splicing word; A preliminary clustering module is configured to select an initial cluster center vector in the word vector, calculate a Hamming distance between a remaining word vector and the initial cluster center vector, perform cluster division on the word vector according to the Hamming distance, and obtain a preliminarily divided keyword cluster; An adaptive clustering module is configured to obtain a similar word vector cluster by using an adaptive clustering algorithm based on the word vector and the preliminarily divided keyword cluster; A splicing module is configured to splice the similar word vector cluster with the splicing word according to the mapping relationship to generate a keyword list; The adaptive clustering module specifically comprises: A second distance calculation unit is configured to calculate a distance between each word vector and all preliminarily divided keyword clusters, and to classify the word vector into a keyword cluster with the closest distance; An adaptive clustering unit is configured to recalculate a cluster center for each keyword cluster, and to adjust the number of clusters according to the change of the cluster center; An output unit is configured to output the similar word vector cluster when a maximum iteration number is reached or the cluster center is unchanged.

8. A keyword extraction device characterized by comprising: The device comprises a memory and a processor; the processor runs a program corresponding to executable program code stored in the memory by reading the executable program code, to implement the method of any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-6.

Images